diff --git a/README.md b/README.md index 61e0010f..f8e766ce 100644 --- a/README.md +++ b/README.md @@ -1,40 +1,132 @@ # Riak Test +Welcome to the exciting world of `riak_test`. + +## What is Riak Test? + +`riak_test` is a system for testing Riak clusters. Tests are written +in Erlang, and can interact with the cluster using distributed Erlang. + +### How does it work? + +`riak_test` runs tests in a sandbox, typically `/tmp/rt`. The sanbox +uses `git` to reset back to a clean state after tests are run. The +contents of `/tmp/rt` might look something like this: + +``` +$ ls /tmp/rt +current riak-1.0.3 riak-1.1.4 riak-1.2.0 +``` + +Inside each of these directories is a `dev` folder, typically +created with your normal `make [stage]devrel`. So how does +this sandbox get populated to begin with? + +You'll create another directory that will contain full builds +of different version of Riak for your platform. Typically this directory +has been `~/test-releases` but it can be called anything and be anywhere +that you'd like. The `dev/` directory from each of these +releases will be copied into the sandbox (`/tmp/rt`). +There are helper scripts in `bin/` which will +help you get both `~/test-releases` and `/tmp/rt` all set up. A full +tutorial for using them exists further down in this README. + +There is one folder in `/tmp/rt` that does not come from +`~/test-releases`: `current`. The `current` folder can refer +to any version of Riak, but is typically used for something +like the `master` branch, a feature branch, or a release candidate. +The `/tmp/rt/current` dev release gets populated from a devrel of Riak +that can come from anywhere, but is usually your 'normal' git checkout +of Riak. The `bin/rtdev-current.sh` can be run from within that folder +to copy `dev/` into `/tmp/rt/current`. + +Once you have everything set up (again, instructions for this are below), +you'll want to run and write tests. This repository also holds code for +an Erlang application called `riak_test`. The actual tests exist in +the `test/` directory. ## Bootstraping Your Test Environment -Welcome to the exciting world of riak_test. Running tests against a development version of Riak is just one of the things that you can do with riak_test. You can also test things involving upgrading from previous versions of Riak. Together, we'll get your test environment up and running. Scripts to help in this process are located in the `bin` directory of this project. +Running tests against a +development version of Riak is just one of the things that you can do +with riak_test. You can also test things involving upgrading from +previous versions of Riak. Together, we'll get your test environment +up and running. Scripts to help in this process are located in the +`bin` directory of this project. + +### rtdev-all.sh + +This script is for the lazy. It performs all of the setup steps described +in the other scripts, including installing the current "master" branch from +Github into "current". The releases will be built in your current working +directory, so create an empty one in a place you'd like to store these +builds for posterity, so that you don't have to rebuild them if your +installation path (`/tmp/rt` by the way this script installs it) gets into +a bad state, or deleted as tmpfs things tend to get during reboot. + +If you do want to restore your `/tmp/rt` folder to factory condition, see +`rtdev-setup-releases.sh` and if you want to change the current riak under +test, see `rtdev-current.sh`. ### rtdev-build-releases.sh -The first one that we want to look at is `rtdev-build-releases.sh`. If left unchanged, this script is going to do the following: -1. Download the source for the past three major Riak versions (e.g. 1.0.3, 1.1.4, and 1.2.0) -1. Build the proper version of Erlang that release was built with, using kerl (which it will also download) +The first one that we want to look at is `rtdev-build-releases.sh`. If +left unchanged, this script is going to do the following: + +1. Download the source for the past three major Riak versions (e.g. + 1.0.3, 1.1.4, and 1.2.0) +1. Build the proper version of Erlang that release was built with, + using kerl (which it will also download) 1. Build those releases of Riak. -You'll want to run this script from an empty directory. Also, you might be thinking that you already have all the required versions of erlang. Great! You can crack open the script and set the paths to your installation +You'll want to run this script from an empty directory. Also, you +might be thinking that you already have all the required versions of +erlang. Great! You can crack open the script and set the paths to your +installation: ```bash R14B04=${R14B04:-$HOME/erlang-R14B04} ``` -**Kerlveat**: If you want kerl to build erlangs with serious 64-bit macintosh action, you'll need a `~/.kerlrc` file that looks like this: +**Kerlveat**: If you want kerl to build erlangs with serious 64-bit +macintosh action, you'll need a `~/.kerlrc` file that looks like this: ``` KERL_CONFIGURE_OPTIONS="--disable-hipe --enable-smp-support --enable-threads --enable-kernel-poll --enable-darwin-64bit" ``` -The script will check that all these paths exist. If even one of them is missing, it will prompt you to install kerl, even if you already have kerl. If you say no, the script quits. If you say yes, or all of your erlang paths check out, then go get a cup of coffee, you'll be building for a little while. +The script will check that all these paths exist. If even one of them +is missing, it will prompt you to install kerl, even if you already +have kerl. If you say no, the script quits. If you say yes, or all of +your erlang paths check out, then go get a cup of coffee, you'll be +building for a little while. -**Warning**: If you are running OS X 10.7+ and trying to build Riak 1.0.3, then the erlang_js dependency won't compile for you, but it fails silently. Fortunately, the precomipled OS X build includes this dependency in it's working form. Just run `rtdev-lion-fix.sh` after `rtdev-build-releases.sh` to patch it. **Please run this patch before proceeding on to the next script** +**Warning**: If you are running OS X 10.7+ and trying to build Riak +1.0.3, then the erlang_js dependency won't compile for you, but it +fails silently. Fortunately, the precomipled OS X build includes this +dependency in it's working form. Just run `rtdev-lion-fix.sh` after +`rtdev-build-releases.sh` to patch it. **Please run this patch before +proceeding on to the next script** ### rtdev-setup-releases.sh -The `rtdev-setup-releases.sh` will get the releases you just built into a local git repository. Currently, running this script from the same directory that you just built all of your releases into. Currently this script initializes this repository into `/tmp/rt` but it's probably worth making that configurable in the near term. + +The `rtdev-setup-releases.sh` will get the releases you just built +into a local git repository. Currently, running this script from the +same directory that you just built all of your releases into. +Currently this script initializes this repository into `/tmp/rt` but +it's probably worth making that configurable in the near term. ### rtdev-current.sh -`rtdev-current.sh` is where it gets interesting. You need to run that from the Riak source folder you're wanting to test as the current version of Riak. Also, make sure that you've already run `make devrel` or `make stagedevrel` before you run `rtdev-current.sh`. + +`rtdev-current.sh` is where it gets interesting. You need to run that +from the Riak source folder you're wanting to test as the current +version of Riak. Also, make sure that you've already run `make devrel` +or `make stagedevrel` before you run `rtdev-current.sh`. ### Config file. -Now that you've got your releases all ready and gitified, you'll need to tell riak_test about them. The method of choice is to create a `~/.riak_test.config` that looks something like this: + +Now that you've got your releases all ready and gitified, you'll need +to tell riak_test about them. The method of choice is to create a +`~/.riak_test.config` that looks something like this: ```erlang {rtdev_mixed, [ @@ -52,6 +144,8 @@ Now that you've got your releases all ready and gitified, you'll need to tell ri ``` ### Running riak_test for the first time + Run a test! `./riak_test -c rtdev_mixed -t verify_build_cluster` -Did that work? Great, try something harder: `./riak_test -c rtdev_mixed -t upgrade` +Did that work? Great, try something harder: `./riak_test -c +rtdev_mixed -t upgrade` diff --git a/bin/rtdev-all.sh b/bin/rtdev-all.sh new file mode 100755 index 00000000..2b9f7690 --- /dev/null +++ b/bin/rtdev-all.sh @@ -0,0 +1,62 @@ +#!/bin/bash + +ORIGDIR=`pwd` +pushd `dirname $0` > /dev/null +SCRIPT_DIR=`pwd` +popd > /dev/null +CURRENT_OTP=${CURRENT_OTP:-$HOME/erlang-R15B01} + +if [ -n "$DEBUG_RTDEV" ]; then + echo "= Configuration =================================================" + echo "Build dir: $ORIGDIR" + echo "rtdev-* scripts: $SCRIPT_DIR" + echo "Erlang: $CURRENT_OTP" + echo +fi + +echo "================== riak_test Omnibus Installer ==================" +echo +echo "This is an omnibus script that builds all the necessary versions " +echo "of Erlang and Riak (including the latest from Github) for running" +echo "riak_test and installs them into /tmp/rt. " +echo +echo -n "Are you sure you want to continue? [Y|n] " +read continue +if [[ $continue == n || $continue == N ]]; then + echo + echo "Aborting install!" + exit 1 +fi + +echo +echo "= Building Riak Releases ========================================" +echo + +source $SCRIPT_DIR/rtdev-build-releases.sh + +if [[ `uname -s` =~ ^Darwin ]]; then + if [[ `sw_vers|grep ProductVersion|awk '{print $2}'` > "10.7" ]]; then + echo + echo "= Patching OSX > 10.7 ======================================" + echo + source $SCRIPT_DIR/rtdev-lion-fix.sh + fi +fi + +echo "= Installing Riak Releases ======================================" +echo +source $SCRIPT_DIR/rtdev-setup-releases.sh + +echo +echo "= Building and Installing Riak from Git =========================" +echo + +cd $ORIGDIR +build "current" $CURRENT_OTP "" "git://github.com/basho/riak.git" +echo +cd current +source $SCRIPT_DIR/rtdev-current.sh + +cd $ORIGDIR +echo +echo "= Build complete! ===============================================" diff --git a/bin/rtdev-build-releases.sh b/bin/rtdev-build-releases.sh index 847d80bb..060128bf 100755 --- a/bin/rtdev-build-releases.sh +++ b/bin/rtdev-build-releases.sh @@ -1,7 +1,7 @@ #!/bin/bash # You need to use this script once to build a set of devrels for prior -# releases of Riak (for mixed version / upgrade testing). You should +# releases of Riak (for mixed version / upgrade testing). You should # create a directory and then run this script from within that directory. # I have ~/test-releases that I created once, and then re-use for testing. # @@ -23,12 +23,19 @@ R15B01=${R15B01:-$HOME/erlang-R15B01} checkbuild() { ERLROOT=$1 - + if [ ! -d $ERLROOT ]; then - echo -n "$ERLROOT cannot be found, install kerl? [y|N]: " + echo -n " - $ERLROOT cannot be found, install with kerl? [Y|n]: " read ans if [[ $ans == n || $ans == N ]]; then + echo + echo " [ERROR] Can't build $ERLROOT without kerl, aborting!" exit 1 + else + if [ ! -x kerl ]; then + echo " - Fetching kerl." + curl -O https://raw.github.com/spawngrid/kerl/master/kerl > /dev/null 2>&1; chmod a+x kerl + fi fi fi } @@ -38,51 +45,56 @@ kerl() RELEASE=$1 BUILDNAME=$2 - if [ ! -x kerl ]; then - curl -O https://raw.github.com/spawngrid/kerl/master/kerl; chmod a+x kerl - fi - - ./kerl build $RELEASE $BUILDNAME - ./kerl install $BUILDNAME $HOME/$BUILDNAME + echo " - Building Erlang $RELEASE (this could take a while)" + ./kerl build $RELEASE $BUILDNAME > /dev/null 2>&1 + echo " - Installing $RELEASE into $HOME/$BUILDNAME" + ./kerl install $BUILDNAME $HOME/$BUILDNAME > /dev/null 2>&1 } build() { SRCDIR=$1 ERLROOT=$2 + DOWNLOAD=$3 + GITURL=$4 + echo "Building $SRCDIR:" + + checkbuild $ERLROOT if [ ! -d $ERLROOT ]; then BUILDNAME=`basename $ERLROOT` RELEASE=`echo $BUILDNAME | awk -F- '{ print $2 }'` kerl $RELEASE $BUILDNAME fi - echo - echo "Building $SRCDIR" + if [ -n "$DOWNLOAD" ]; then + echo " - Fetching $DOWNLOAD" + wget -q -c $DOWNLOAD + + TARBALL=`basename $DOWNLOAD` + echo " - Expanding $TARBALL" + tar xzf $TARBALL > /dev/null 2>&1 + fi + + if [ -n "$GITURL" ]; then + echo " - Cloning $GITURL" + git clone $GITURL $SRCDIR > /dev/null 2>&1 + fi + + echo " - Building devrel in $SRCDIR (this could take a while)" cd $SRCDIR RUN="env PATH=$ERLROOT/bin:$ERLROOT/lib/erlang/bin:$PATH \ C_INCLUDE_PATH=$ERLROOT/usr/include \ LD_LIBRARY_PATH=$ERLROOT/usr/lib" - echo $RUN - $RUN make && $RUN make devrel + $RUN make all devrel > /dev/null 2>&1 cd .. + echo " - $SRCDIR built." } -checkbuild $R14B03 -checkbuild $R14B04 -checkbuild $R15B01 - -# Download Riak release source -wget -c http://s3.amazonaws.com/downloads.basho.com/riak/1.0/1.0.3/riak-1.0.3.tar.gz -wget -c http://s3.amazonaws.com/downloads.basho.com/riak/1.1/1.1.4/riak-1.1.4.tar.gz -wget -c http://s3.amazonaws.com/downloads.basho.com/riak/1.2/1.2.0/riak-1.2.0.tar.gz - -tar -xzf riak-1.0.3.tar.gz -build "riak-1.0.3" $R14B03 - -tar -xzf riak-1.1.4.tar.gz -build "riak-1.1.4" $R14B04 - -tar -xvf riak-1.2.0.tar.gz -build "riak-1.2.0" $R15B01 +build "riak-1.0.3" $R14B03 http://s3.amazonaws.com/downloads.basho.com/riak/1.0/1.0.3/riak-1.0.3.tar.gz +echo +build "riak-1.1.4" $R14B04 http://s3.amazonaws.com/downloads.basho.com/riak/1.1/1.1.4/riak-1.1.4.tar.gz +echo +build "riak-1.2.0" $R15B01 http://s3.amazonaws.com/downloads.basho.com/riak/1.2/1.2.0/riak-1.2.0.tar.gz +echo diff --git a/bin/rtdev-current.sh b/bin/rtdev-current.sh index c338de0b..d0b04082 100755 --- a/bin/rtdev-current.sh +++ b/bin/rtdev-current.sh @@ -1,13 +1,27 @@ +#!/bin/bash + +echo "Making $(pwd) the current release:" cwd=$(pwd) +echo -n " - Determining version: " +if [ -f $cwd/dependency_manifest.git ]; then + VERSION=`cat $cwd/dependency_manifest.git | awk '/^-/ { print $NF }'` +else + VERSION="$(git describe --tags)-$(git branch | awk '/\*/ {print $2}')" +fi +echo $VERSION cd /tmp/rt -git reset HEAD --hard -git clean -fd +echo " - Resetting existing /tmp/rt" +git reset HEAD --hard > /dev/null 2>&1 +git clean -fd > /dev/null 2>&1 +echo " - Removing and recreating /tmp/rt/current" rm -rf /tmp/rt/current mkdir /tmp/rt/current cd $cwd +echo " - Copying devrel to /tmp/rt/current" cp -a dev /tmp/rt/current +echo " - Writing /tmp/rt/current/VERSION" +echo -n $VERSION > /tmp/rt/current/VERSION cd /tmp/rt +echo " - Reinitializing git state" git add . -git commit -a -m "riak_test init" --amend - - +git commit -a -m "riak_test init" --amend > /dev/null 2>&1 diff --git a/bin/rtdev-setup-releases.sh b/bin/rtdev-setup-releases.sh index e92a71ab..a42f06d3 100755 --- a/bin/rtdev-setup-releases.sh +++ b/bin/rtdev-setup-releases.sh @@ -5,14 +5,18 @@ # that contains devrels for prior Riak releases. Easy way to create this # is to use the rtdev-build-releases.sh script +echo "Setting up releases from $(pwd):" +echo " - Creating /tmp/rt" rm -rf /tmp/rt mkdir /tmp/rt for rel in */dev; do vsn=$(dirname "$rel") + echo " - Initializing /tmp/rt/$vsn" mkdir "/tmp/rt/$vsn" cp -a "$rel" "/tmp/rt/$vsn" done cd /tmp/rt -git init +echo " - Creating the git repository" +git init > /dev/null 2>&1 git add . -git commit -a -m "riak_test init" +git commit -a -m "riak_test init" > /dev/null 2>&1