16:22:30.141 [info] Deployed nodes: ['riak@192.168.60.10','riak@192.168.60.11','riak@192.168.60.12']
16:22:30.141 [info] Ensure each nodes 100% of it's own ring
16:22:30.143 [info] Join nodes together
16:22:30.153 [debug] [join] 'riak@192.168.60.11' to ('riak@192.168.60.10'): ok
16:22:30.164 [debug] [join] 'riak@192.168.60.12' to ('riak@192.168.60.10'): ok
16:22:30.164 [info] Wait until all nodes are ready and there are no pending changes
16:22:49.433 [info] Ensure each node owns a portion of the ring
16:22:49.436 [info] verify_build_cluster: PASS
'''
As shown, the tests look similar to standard eunit tests and are
straightforward Erlang. The {@link rt} module provides the basic
commands to control Riak nodes, and abstracts everything away from
the underlying harness: devrel or basho_expect. The `rt' module
also provides a set of useful existing built-in operations. It is
expected that the `rt' module will be extended over time with more
reusable functions.
Writing additional checks and preconditions is the same as with an
eunit test. For example, `rt:owners_according_to(Node)' is just a
basic RPC call:
```
owners_according_to(Node) ->
{ok, Ring} = rpc:call(Node, riak_core_ring_manager, get_raw_ring, []),
Owners = [Owner || {_Idx, Owner} <- riak_core_ring:all_owners(Ring)],
lists:usort(Owners).
'''
And the `rt:wait_until_nodes_ready', which leverages the built-in `wait_until'
primitive:
```
wait_until_nodes_ready(Nodes) ->
[?assertEqual(ok, wait_until(Node, fun is_ready/1)) || Node <- Nodes],
ok.
is_ready(Node) ->
case rpc:call(Node, riak_core_ring_manager, get_raw_ring, []) of
{ok, Ring} ->
lists:member(Node, riak_core_ring:ready_members(Ring));
_ ->
false
end.
'''
Much like basho_bench, riak_test uses Erlang term based config files. The
references `rtdev.config' file above:
```
%% Deps should include the path to compiled Riak libs
{rt_deps, ["/Users/jtuple/basho/riak/deps"]}.
%% Maximum time in milliseconds for wait_until to wait
{rt_max_wait_time, 180000}.
%% Delay between each retry in wait_until
{rt_retry_delay, 500}.
%% Use the devrel harness
{rt_harness, rtdev}.
%% Path to generated devrel of Riak that is git-versioned
{rtdev_path, "/tmp/rt"}.
'''
And the `rtbe.config' file:
```
%% Deps should include Riak libs and basho_expect
{rt_deps, ["/Users/jtuple/basho/riak/deps",
"/Users/jtuple/basho/basho_expect"]}.
{rt_max_wait_time, 180000}.
{rt_retry_delay, 500}.
%% Use the rtbe harness (in basho_expect source tree)
{rt_harness, rtbe}.
%% Host/Port to connect to basho_expect server.
{rtbe_host, "127.0.0.1"}.
{rtbe_port, 9000}.
'''
And to conclude, a slightly longer test that uses most of the built-in
primitives including starting and stopping Riak nodes:
```
verify_claimant() ->
Nodes = build_cluster(3),
[Node1, Node2, _Node3] = Nodes,
%% Ensure all nodes believe node1 is the claimant
lager:info("Ensure all nodes believe ~p is the claimant", [Node1]),
[?assertEqual(Node1, claimant_according_to(Node)) || Node <- Nodes],
%% Stop node1
lager:info("Stop ~p", [Node1]),
stop(Node1),
?assertEqual(ok, wait_until_unpingable(Node1)),
%% Ensure all nodes still believe node1 is the claimant
lager:info("Ensure all nodes still believe ~p is the claimant", [Node1]),
Remaining = Nodes -- [Node1],
[?assertEqual(Node1, claimant_according_to(Node)) || Node <- Remaining],
%% Mark node1 as down and wait for ring convergence
lager:info("Mark ~p as down", [Node1]),
down(Node2, Node1),
?assertEqual(ok, wait_until_ring_converged(Remaining)),
[?assertEqual(down, status_of_according_to(Node1, Node)) || Node <- Remaining],
%% Ensure all nodes now believe node2 to be the claimant
lager:info("Ensure all nodes now believe ~p is the claimant", [Node2]),
[?assertEqual(Node2, claimant_according_to(Node)) || Node <- Remaining],
%% Restart node1 and wait for ring convergence
lager:info("Restart ~p and wait for ring convergence", [Node1]),
start(Node1),
?assertEqual(ok, wait_until_nodes_ready([Node1])),
?assertEqual(ok, wait_until_ring_converged(Nodes)),
%% Ensure node has rejoined and is no longer down
lager:info("Ensure ~p has rejoined and is no longer down", [Node1]),
[?assertEqual(valid, status_of_according_to(Node1, Node)) || Node <- Nodes],
%% Ensure all nodes still believe node2 is the claimant
lager:info("Ensure all nodes still believe ~p is the claimant", [Node2]),
[?assertEqual(Node2, claimant_according_to(Node)) || Node <- Nodes],
ok.
'''
Installation and Usage
Checkout riak_test from Github and build: `http://github.com/basho/riak_test'
riak_test comes with two pre-written config files: `rtdev.config' for
devrel mode and `rtbe.config' for basho_expect mode. The devrel
operating mode comes built into riak_test, and can be used right away.
To use the basho_expect mode, you must checkout a version of basho_expect that
includes the basho_expect_thrift server mode, and make sure to run `make' from
within the `basho_expect/basho_expect_thrift' directory.
For both modes, ensure the `rt_deps' setting in the config file is accurate,
and adjust other settings as appropriate. The comments should be self explanatory.
To run riak_test in devrel mode, you must first generate a git-versioned devrel
directory and ensure the `rtdev_path' setting in `rtdev.config' is accurate.
To generate the versioned devrel, modify the following as necessary:
```
riak$ make devrel
riak$ mkdir /tmp/rt
riak$ cp -r dev /tmp/rt/dev
riak$ cd /tmp/rt/dev
/tmp/rt/dev$ git init
/tmp/rt/dev$ git add .
/tmp/rt/dev$ git commit -m "initial"
'''
You can then launch tests from within the riak_test directory:
```
./riak_test rtdev.config verify_build_cluster
'''
riak_test currently assumes tests are written such that there is one
test per module and that the module contains a same named zero arity
function that implements the test. For example, the above calls
`verify_build_cluster:verify_build_cluster()'.
To run run in basho_expect mode, you must first launch the basho_expect
server and then run riak_test:
```
basho_expect$ python run_be_thrift.py &
riak_test$ ./riak_test rtbe.config verify_build_cluster ubuntu10-64-1 ubuntu10-64-2 ubuntu10-64-3
'''
To make this easier, there is an included script in basho_expect: `Run-riak-test' that operates
more like traditional basho_expect tests. Just make sure to modify it to point to the right
riak_test directory. The above simplifies to the following:
```
basho_expect$ ./Run-riak-test verify_build_cluster ubuntu10-64-1 ubuntu10-64-2 ubuntu10-64-3
'''
TODO: integrate with general basho_expect systest harness and ensure things work with
Guttersnipe.