valitydev/riak_test
14
0
Fork 0
mirror of https://github.com/valitydev/riak_test.git synced 2024-11-06 16:45:29 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
a1bd419973
riak_test/tests/repl_aae_fullsync_util.erl

81 lines
3.2 KiB
Erlang
Raw Normal View History

Refactor common AAE test code and add custom N test. * Common code in the cluster setup and data setup is factored out into a utility module. * AAE fullsync test and benchmark get skinnier and call the common setup code. * Document parameters in the setup for tests. * Add a new AAE fullsync test that changes the N value (source is 2, sink is 3).
2013-05-30 21:02:10 +00:00
%% @doc
%% This module implements a riak_test to exercise the Active Anti-Entropy Fullsync replication.
%% It sets up two clusters and starts a single fullsync worker for a single AAE tree.
-module(repl_aae_fullsync_util).
-export([make_clusters/3,
prepare_cluster_data/5]).
-include_lib("eunit/include/eunit.hrl").
Ensure riak_repl service is up across the board Trying to use the repl features before newly started nodes have riak_repl completely initialized leads to all sorts of nasty crashes and noise. Frequently it makes fullsync stuck forever, which makes a lot of the tests fail. This also tweaks the AAE fullsync tests to remove assumptions about failure stats when AAE transient errors occur. The behavior in the handling of those errors has changed recently with the introduction of soft exits.
2014-12-18 21:07:00 +00:00
-import(rt, [deploy_nodes/3,
Refactor common AAE test code and add custom N test. * Common code in the cluster setup and data setup is factored out into a utility module. * AAE fullsync test and benchmark get skinnier and call the common setup code. * Document parameters in the setup for tests. * Add a new AAE fullsync test that changes the N value (source is 2, sink is 3).
2013-05-30 21:02:10 +00:00
join/2,
log_to_nodes/2,
Ensure riak_repl service is up across the board Trying to use the repl features before newly started nodes have riak_repl completely initialized leads to all sorts of nasty crashes and noise. Frequently it makes fullsync stuck forever, which makes a lot of the tests fail. This also tweaks the AAE fullsync tests to remove assumptions about failure stats when AAE transient errors occur. The behavior in the handling of those errors has changed recently with the introduction of soft exits.
2014-12-18 21:07:00 +00:00
log_to_nodes/3]).
Refactor common AAE test code and add custom N test. * Common code in the cluster setup and data setup is factored out into a utility module. * AAE fullsync test and benchmark get skinnier and call the common setup code. * Document parameters in the setup for tests. * Add a new AAE fullsync test that changes the N value (source is 2, sink is 3).
2013-05-30 21:02:10 +00:00
make_clusters(NumNodesWanted, ClusterSize, Conf) ->
NumNodes = rt_config:get(num_nodes, NumNodesWanted),
ClusterASize = rt_config:get(cluster_a_size, ClusterSize),
lager:info("Deploy ~p nodes", [NumNodes]),
Ensure riak_repl service is up across the board Trying to use the repl features before newly started nodes have riak_repl completely initialized leads to all sorts of nasty crashes and noise. Frequently it makes fullsync stuck forever, which makes a lot of the tests fail. This also tweaks the AAE fullsync tests to remove assumptions about failure stats when AAE transient errors occur. The behavior in the handling of those errors has changed recently with the introduction of soft exits.
2014-12-18 21:07:00 +00:00
Nodes = deploy_nodes(NumNodes, Conf, [riak_kv, riak_repl]),
Refactor common AAE test code and add custom N test. * Common code in the cluster setup and data setup is factored out into a utility module. * AAE fullsync test and benchmark get skinnier and call the common setup code. * Document parameters in the setup for tests. * Add a new AAE fullsync test that changes the N value (source is 2, sink is 3).
2013-05-30 21:02:10 +00:00
{ANodes, BNodes} = lists:split(ClusterASize, Nodes),
lager:info("ANodes: ~p", [ANodes]),
lager:info("BNodes: ~p", [BNodes]),
lager:info("Build cluster A"),
repl_util:make_cluster(ANodes),
lager:info("Build cluster B"),
repl_util:make_cluster(BNodes),
{ANodes, BNodes}.
Add an exhaustive test for AAE based replication. Ensure that AAE replication is tested using all possible failure cases when dealing with the riak_kv_index_hashtrees and failed connections. First, use intercepts on riak_kv_vnode and riak_kv_index_hashtree to ensure that we simulate errors on a per node basis, starting with the source cluster and moving to the sink. Simulate ownership transfers, locked and incomplete hashtrees. Verify partitions generate the correct error count, after using a bounded set of retries, and finally remove all intercepts and verify that the fullsync completes and all keys have been migrated between the two clusters.
2013-12-18 04:21:29 +00:00
prepare_cluster_data(TestBucket, NumKeysAOnly, _NumKeysBoth, [AFirst|_] = ANodes, [BFirst|_] = BNodes) ->
Refactor common AAE test code and add custom N test. * Common code in the cluster setup and data setup is factored out into a utility module. * AAE fullsync test and benchmark get skinnier and call the common setup code. * Document parameters in the setup for tests. * Add a new AAE fullsync test that changes the N value (source is 2, sink is 3).
2013-05-30 21:02:10 +00:00
AllNodes = ANodes ++ BNodes,
log_to_nodes(AllNodes, "Starting AAE Fullsync test"),
%% clusters are not connected, connect them
repl_util:name_cluster(AFirst, "A"),
repl_util:name_cluster(BFirst, "B"),
%% we'll need to wait for cluster names before continuing
rt:wait_until_ring_converged(ANodes),
rt:wait_until_ring_converged(BNodes),
lager:info("waiting for leader to converge on cluster A"),
?assertEqual(ok, repl_util:wait_until_leader_converge(ANodes)),
lager:info("waiting for leader to converge on cluster B"),
?assertEqual(ok, repl_util:wait_until_leader_converge(BNodes)),
%% get the leader for the first cluster
LeaderA = rpc:call(AFirst, riak_core_cluster_mgr, get_leader, []),
{ok, {_IP, Port}} = rpc:call(BFirst, application, get_env,
[riak_core, cluster_mgr]),
lager:info("connect cluster A:~p to B on port ~p", [LeaderA, Port]),
repl_util:connect_cluster(LeaderA, "127.0.0.1", Port),
?assertEqual(ok, repl_util:wait_for_connection(LeaderA, "B")),
%% make sure we are connected
lager:info("Wait for cluster connection A:~p -> B:~p:~p", [LeaderA, BFirst, Port]),
?assertEqual(ok, repl_util:wait_for_connection(LeaderA, "B")),
%%---------------------------------------------------
%% TEST: write data, NOT replicated by RT or fullsync
%% keys: 1..NumKeysAOnly
%%---------------------------------------------------
lager:info("Writing ~p keys to A(~p)", [NumKeysAOnly, AFirst]),
Update calls to rt:systest_read to handle identical siblings Update the calls to rt:systest_read in repl_util and repl_aae_fullsync_util to treat identical siblings resulting from the use of DVV as a single value. These changes are specifically to address failures seen in the repl_aae_fullsync_custom_n and replication_object_reformat tests, but should be generally useful for replication tests using the utility modules that and that have allow_mult set to true.
2014-06-18 17:32:00 +00:00
?assertEqual([], repl_util:do_write(AFirst, 1, NumKeysAOnly, TestBucket, 2)),
Refactor common AAE test code and add custom N test. * Common code in the cluster setup and data setup is factored out into a utility module. * AAE fullsync test and benchmark get skinnier and call the common setup code. * Document parameters in the setup for tests. * Add a new AAE fullsync test that changes the N value (source is 2, sink is 3).
2013-05-30 21:02:10 +00:00
%% check that the keys we wrote initially aren't replicated yet, because
%% we've disabled fullsync_on_connect
lager:info("Check keys written before repl was connected are not present"),
Update calls to rt:systest_read to handle identical siblings Update the calls to rt:systest_read in repl_util and repl_aae_fullsync_util to treat identical siblings resulting from the use of DVV as a single value. These changes are specifically to address failures seen in the repl_aae_fullsync_custom_n and replication_object_reformat tests, but should be generally useful for replication tests using the utility modules that and that have allow_mult set to true.
2014-06-18 17:32:00 +00:00
Res2 = rt:systest_read(BFirst, 1, NumKeysAOnly, TestBucket, 1, <<>>, true),
Refactor common AAE test code and add custom N test. * Common code in the cluster setup and data setup is factored out into a utility module. * AAE fullsync test and benchmark get skinnier and call the common setup code. * Document parameters in the setup for tests. * Add a new AAE fullsync test that changes the N value (source is 2, sink is 3).
2013-05-30 21:02:10 +00:00
?assertEqual(NumKeysAOnly, length(Res2)),
%% wait for the AAE trees to be built so that we don't get a not_built error
Replacing repl_util AAE wait with rt version
2014-05-12 18:55:59 +00:00
rt:wait_until_aae_trees_built(ANodes),
rt:wait_until_aae_trees_built(BNodes),
Refactor common AAE test code and add custom N test. * Common code in the cluster setup and data setup is factored out into a utility module. * AAE fullsync test and benchmark get skinnier and call the common setup code. * Document parameters in the setup for tests. * Add a new AAE fullsync test that changes the N value (source is 2, sink is 3).
2013-05-30 21:02:10 +00:00
ok.
Reference in New Issue Copy Permalink