valitydev/riak_test
14
0
Fork 0
mirror of https://github.com/valitydev/riak_test.git synced 2024-11-06 08:35:22 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Merge pull request #57 from basho/sdc-happy-scripts

Browse Source 
Make the rtdev scripts friendlier.
This commit is contained in:
Joe DeVivo 2012-09-19 14:55:25 -07:00
commit c56fe12bae
5 changed files with 235 additions and 49 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

118
README.md
View File

@ -1,40 +1,132 @@
# Riak Test
Welcome to the exciting world of `riak_test`.
## What is Riak Test?
`riak_test` is a system for testing Riak clusters. Tests are written
in Erlang, and can interact with the cluster using distributed Erlang.
### How does it work?
`riak_test` runs tests in a sandbox, typically `/tmp/rt`. The sanbox
uses `git` to reset back to a clean state after tests are run. The
contents of `/tmp/rt` might look something like this:
```
$ ls /tmp/rt
current riak-1.0.3 riak-1.1.4 riak-1.2.0
```
Inside each of these directories is a `dev` folder, typically
created with your normal `make [stage]devrel`. So how does
this sandbox get populated to begin with?
You'll create another directory that will contain full builds
of different version of Riak for your platform. Typically this directory
has been `~/test-releases` but it can be called anything and be anywhere
that you'd like. The `dev/` directory from each of these
releases will be copied into the sandbox (`/tmp/rt`).
There are helper scripts in `bin/` which will
help you get both `~/test-releases` and `/tmp/rt` all set up. A full
tutorial for using them exists further down in this README.
There is one folder in `/tmp/rt` that does not come from
`~/test-releases`: `current`. The `current` folder can refer
to any version of Riak, but is typically used for something
like the `master` branch, a feature branch, or a release candidate.
The `/tmp/rt/current` dev release gets populated from a devrel of Riak
that can come from anywhere, but is usually your 'normal' git checkout
of Riak. The `bin/rtdev-current.sh` can be run from within that folder
to copy `dev/` into `/tmp/rt/current`.
Once you have everything set up (again, instructions for this are below),
you'll want to run and write tests. This repository also holds code for
an Erlang application called `riak_test`. The actual tests exist in
the `test/` directory.
## Bootstraping Your Test Environment
Welcome to the exciting world of riak_test. Running tests against a development version of Riak is just one of the things that you can do with riak_test. You can also test things involving upgrading from previous versions of Riak. Together, we'll get your test environment up and running. Scripts to help in this process are located in the `bin` directory of this project.
Running tests against a
development version of Riak is just one of the things that you can do
with riak_test. You can also test things involving upgrading from
previous versions of Riak. Together, we'll get your test environment
up and running. Scripts to help in this process are located in the
`bin` directory of this project.
### rtdev-all.sh
This script is for the lazy. It performs all of the setup steps described
in the other scripts, including installing the current "master" branch from
Github into "current". The releases will be built in your current working
directory, so create an empty one in a place you'd like to store these
builds for posterity, so that you don't have to rebuild them if your
installation path (`/tmp/rt` by the way this script installs it) gets into
a bad state, or deleted as tmpfs things tend to get during reboot.
If you do want to restore your `/tmp/rt` folder to factory condition, see
`rtdev-setup-releases.sh` and if you want to change the current riak under
test, see `rtdev-current.sh`.
### rtdev-build-releases.sh
The first one that we want to look at is `rtdev-build-releases.sh`. If left unchanged, this script is going to do the following:
1. Download the source for the past three major Riak versions (e.g. 1.0.3, 1.1.4, and 1.2.0)
1. Build the proper version of Erlang that release was built with, using kerl (which it will also download)
The first one that we want to look at is `rtdev-build-releases.sh`. If
left unchanged, this script is going to do the following:
1. Download the source for the past three major Riak versions (e.g.
1.0.3, 1.1.4, and 1.2.0)
1. Build the proper version of Erlang that release was built with,
using kerl (which it will also download)
1. Build those releases of Riak.
You'll want to run this script from an empty directory. Also, you might be thinking that you already have all the required versions of erlang. Great! You can crack open the script and set the paths to your installation
You'll want to run this script from an empty directory. Also, you
might be thinking that you already have all the required versions of
erlang. Great! You can crack open the script and set the paths to your
installation:
```bash
R14B04=${R14B04:-$HOME/erlang-R14B04}
```
**Kerlveat**: If you want kerl to build erlangs with serious 64-bit macintosh action, you'll need a `~/.kerlrc` file that looks like this:
**Kerlveat**: If you want kerl to build erlangs with serious 64-bit
macintosh action, you'll need a `~/.kerlrc` file that looks like this:
```
KERL_CONFIGURE_OPTIONS="--disable-hipe --enable-smp-support --enable-threads --enable-kernel-poll --enable-darwin-64bit"
```
The script will check that all these paths exist. If even one of them is missing, it will prompt you to install kerl, even if you already have kerl. If you say no, the script quits. If you say yes, or all of your erlang paths check out, then go get a cup of coffee, you'll be building for a little while.
The script will check that all these paths exist. If even one of them
is missing, it will prompt you to install kerl, even if you already
have kerl. If you say no, the script quits. If you say yes, or all of
your erlang paths check out, then go get a cup of coffee, you'll be
building for a little while.
**Warning**: If you are running OS X 10.7+ and trying to build Riak 1.0.3, then the erlang_js dependency won't compile for you, but it fails silently. Fortunately, the precomipled OS X build includes this dependency in it's working form. Just run `rtdev-lion-fix.sh` after `rtdev-build-releases.sh` to patch it. **Please run this patch before proceeding on to the next script**
**Warning**: If you are running OS X 10.7+ and trying to build Riak
1.0.3, then the erlang_js dependency won't compile for you, but it
fails silently. Fortunately, the precomipled OS X build includes this
dependency in it's working form. Just run `rtdev-lion-fix.sh` after
`rtdev-build-releases.sh` to patch it. **Please run this patch before
proceeding on to the next script**
### rtdev-setup-releases.sh
The `rtdev-setup-releases.sh` will get the releases you just built into a local git repository. Currently, running this script from the same directory that you just built all of your releases into. Currently this script initializes this repository into `/tmp/rt` but it's probably worth making that configurable in the near term.
The `rtdev-setup-releases.sh` will get the releases you just built
into a local git repository. Currently, running this script from the
same directory that you just built all of your releases into.
Currently this script initializes this repository into `/tmp/rt` but
it's probably worth making that configurable in the near term.
### rtdev-current.sh
`rtdev-current.sh` is where it gets interesting. You need to run that from the Riak source folder you're wanting to test as the current version of Riak. Also, make sure that you've already run `make devrel` or `make stagedevrel` before you run `rtdev-current.sh`.
`rtdev-current.sh` is where it gets interesting. You need to run that
from the Riak source folder you're wanting to test as the current
version of Riak. Also, make sure that you've already run `make devrel`
or `make stagedevrel` before you run `rtdev-current.sh`.
### Config file.
Now that you've got your releases all ready and gitified, you'll need to tell riak_test about them. The method of choice is to create a `~/.riak_test.config` that looks something like this:
Now that you've got your releases all ready and gitified, you'll need
to tell riak_test about them. The method of choice is to create a
`~/.riak_test.config` that looks something like this:
```erlang
{rtdev_mixed, [
@ -52,6 +144,8 @@ Now that you've got your releases all ready and gitified, you'll need to tell ri
```
### Running riak_test for the first time
Run a test! `./riak_test -c rtdev_mixed -t verify_build_cluster`
Did that work? Great, try something harder: `./riak_test -c rtdev_mixed -t upgrade`
Did that work? Great, try something harder: `./riak_test -c
rtdev_mixed -t upgrade`

62
bin/rtdev-all.sh Executable file
View File

@ -0,0 +1,62 @@
#!/bin/bash
ORIGDIR=`pwd`
pushd `dirname $0` > /dev/null
SCRIPT_DIR=`pwd`
popd > /dev/null
CURRENT_OTP=${CURRENT_OTP:-$HOME/erlang-R15B01}
if [ -n "$DEBUG_RTDEV" ]; then
echo "= Configuration ================================================="
echo "Build dir: $ORIGDIR"
echo "rtdev-* scripts: $SCRIPT_DIR"
echo "Erlang: $CURRENT_OTP"
echo
fi
echo "================== riak_test Omnibus Installer =================="
echo
echo "This is an omnibus script that builds all the necessary versions "
echo "of Erlang and Riak (including the latest from Github) for running"
echo "riak_test and installs them into /tmp/rt. "
echo
echo -n "Are you sure you want to continue? [Y|n] "
read continue
if [[ $continue == n || $continue == N ]]; then
echo
echo "Aborting install!"
exit 1
fi
echo
echo "= Building Riak Releases ========================================"
echo
source $SCRIPT_DIR/rtdev-build-releases.sh
if [[ `uname -s` =~ ^Darwin ]]; then
if [[ `sw_vers|grep ProductVersion|awk '{print $2}'` > "10.7" ]]; then
echo
echo "= Patching OSX > 10.7 ======================================"
echo
source $SCRIPT_DIR/rtdev-lion-fix.sh
fi
fi
echo "= Installing Riak Releases ======================================"
echo
source $SCRIPT_DIR/rtdev-setup-releases.sh
echo
echo "= Building and Installing Riak from Git ========================="
echo
cd $ORIGDIR
build "current" $CURRENT_OTP "" "git://github.com/basho/riak.git"
echo
cd current
source $SCRIPT_DIR/rtdev-current.sh
cd $ORIGDIR
echo
echo "= Build complete! ==============================================="

72
bin/rtdev-build-releases.sh
View File

@ -1,7 +1,7 @@
#!/bin/bash
# You need to use this script once to build a set of devrels for prior
# releases of Riak (for mixed version / upgrade testing). You should
# releases of Riak (for mixed version / upgrade testing). You should
# create a directory and then run this script from within that directory.
# I have ~/test-releases that I created once, and then re-use for testing.
#
@ -23,12 +23,19 @@ R15B01=${R15B01:-$HOME/erlang-R15B01}
checkbuild()
{
ERLROOT=$1
if [ ! -d $ERLROOT ]; then
echo -n "$ERLROOT cannot be found, install kerl? [y|N]: "
echo -n " - $ERLROOT cannot be found, install with kerl? [Y|n]: "
read ans
if [[ $ans == n || $ans == N ]]; then
echo
echo " [ERROR] Can't build $ERLROOT without kerl, aborting!"
exit 1
else
if [ ! -x kerl ]; then
echo " - Fetching kerl."
curl -O https://raw.github.com/spawngrid/kerl/master/kerl > /dev/null 2>&1; chmod a+x kerl
fi
fi
fi
}
@ -38,51 +45,56 @@ kerl()
RELEASE=$1
BUILDNAME=$2
if [ ! -x kerl ]; then
curl -O https://raw.github.com/spawngrid/kerl/master/kerl; chmod a+x kerl
fi
./kerl build $RELEASE $BUILDNAME
./kerl install $BUILDNAME $HOME/$BUILDNAME
echo " - Building Erlang $RELEASE (this could take a while)"
./kerl build $RELEASE $BUILDNAME > /dev/null 2>&1
echo " - Installing $RELEASE into $HOME/$BUILDNAME"
./kerl install $BUILDNAME $HOME/$BUILDNAME > /dev/null 2>&1
}
build()
{
SRCDIR=$1
ERLROOT=$2
DOWNLOAD=$3
GITURL=$4
echo "Building $SRCDIR:"
checkbuild $ERLROOT
if [ ! -d $ERLROOT ]; then
BUILDNAME=`basename $ERLROOT`
RELEASE=`echo $BUILDNAME | awk -F- '{ print $2 }'`
kerl $RELEASE $BUILDNAME
fi
echo
echo "Building $SRCDIR"
if [ -n "$DOWNLOAD" ]; then
echo " - Fetching $DOWNLOAD"
wget -q -c $DOWNLOAD
TARBALL=`basename $DOWNLOAD`
echo " - Expanding $TARBALL"
tar xzf $TARBALL > /dev/null 2>&1
fi
if [ -n "$GITURL" ]; then
echo " - Cloning $GITURL"
git clone $GITURL $SRCDIR > /dev/null 2>&1
fi
echo " - Building devrel in $SRCDIR (this could take a while)"
cd $SRCDIR
RUN="env PATH=$ERLROOT/bin:$ERLROOT/lib/erlang/bin:$PATH \
C_INCLUDE_PATH=$ERLROOT/usr/include \
LD_LIBRARY_PATH=$ERLROOT/usr/lib"
echo $RUN
$RUN make && $RUN make devrel
$RUN make all devrel > /dev/null 2>&1
cd ..
echo " - $SRCDIR built."
}
checkbuild $R14B03
checkbuild $R14B04
checkbuild $R15B01
# Download Riak release source
wget -c http://s3.amazonaws.com/downloads.basho.com/riak/1.0/1.0.3/riak-1.0.3.tar.gz
wget -c http://s3.amazonaws.com/downloads.basho.com/riak/1.1/1.1.4/riak-1.1.4.tar.gz
wget -c http://s3.amazonaws.com/downloads.basho.com/riak/1.2/1.2.0/riak-1.2.0.tar.gz
tar -xzf riak-1.0.3.tar.gz
build "riak-1.0.3" $R14B03
tar -xzf riak-1.1.4.tar.gz
build "riak-1.1.4" $R14B04
tar -xvf riak-1.2.0.tar.gz
build "riak-1.2.0" $R15B01
build "riak-1.0.3" $R14B03 http://s3.amazonaws.com/downloads.basho.com/riak/1.0/1.0.3/riak-1.0.3.tar.gz
echo
build "riak-1.1.4" $R14B04 http://s3.amazonaws.com/downloads.basho.com/riak/1.1/1.1.4/riak-1.1.4.tar.gz
echo
build "riak-1.2.0" $R15B01 http://s3.amazonaws.com/downloads.basho.com/riak/1.2/1.2.0/riak-1.2.0.tar.gz
echo

24
bin/rtdev-current.sh
View File

@ -1,13 +1,27 @@
#!/bin/bash
echo "Making $(pwd) the current release:"
cwd=$(pwd)
echo -n " - Determining version: "
if [ -f $cwd/dependency_manifest.git ]; then
VERSION=`cat $cwd/dependency_manifest.git | awk '/^-/ { print $NF }'`
else
VERSION="$(git describe --tags)-$(git branch | awk '/\*/ {print $2}')"
fi
echo $VERSION
cd /tmp/rt
git reset HEAD --hard
git clean -fd
echo " - Resetting existing /tmp/rt"
git reset HEAD --hard > /dev/null 2>&1
git clean -fd > /dev/null 2>&1
echo " - Removing and recreating /tmp/rt/current"
rm -rf /tmp/rt/current
mkdir /tmp/rt/current
cd $cwd
echo " - Copying devrel to /tmp/rt/current"
cp -a dev /tmp/rt/current
echo " - Writing /tmp/rt/current/VERSION"
echo -n $VERSION > /tmp/rt/current/VERSION
cd /tmp/rt
echo " - Reinitializing git state"
git add .
git commit -a -m "riak_test init" --amend
git commit -a -m "riak_test init" --amend > /dev/null 2>&1

8
bin/rtdev-setup-releases.sh
View File

@ -5,14 +5,18 @@
# that contains devrels for prior Riak releases. Easy way to create this
# is to use the rtdev-build-releases.sh script
echo "Setting up releases from $(pwd):"
echo " - Creating /tmp/rt"
rm -rf /tmp/rt
mkdir /tmp/rt
for rel in */dev; do
vsn=$(dirname "$rel")
echo " - Initializing /tmp/rt/$vsn"
mkdir "/tmp/rt/$vsn"
cp -a "$rel" "/tmp/rt/$vsn"
done
cd /tmp/rt
git init
echo " - Creating the git repository"
git init > /dev/null 2>&1
git add .
git commit -a -m "riak_test init"
git commit -a -m "riak_test init" > /dev/null 2>&1