Now ensemble peers are prevented from starting up until the riak_kv
service is up to avoid nasty races that could even lead to node crashes
as the ensembles frantically query for data that isn't ready.
AAE is disabled. (defect https://github.com/basho/riak_kv/issues/959)
- Adds additional console output to reset-current-env to explain
configuration and steps being executed
- Adds the -n option to the reset-current-env script to specify the
number of nodes to build. By default, 5 will be created.
Prior to this commit, the various riak_ensemble related tests would
manually enable the consensus system on one-and-only-one node in a
given cluster in order to work around issue basho/riak_core#571.
This commit changes the tests to work properly after the above issue
has been fixed.
In addition to removing the call to riak_ensemble_manager:enable()
that is now handled automatically by Riak, this commit also removes
a few wait_until_stable/2 checks against 1-node clusters. These
checks no longer apply, since Riak is now designed to only enable
the consensus system after the cluster contains at least 3 nodes.
Once riak_ensemble_manager:enable() is called, we need to call
riak_core_ring_manager:force_update() so that the members will be
created and added to the ensembles trying to get a quorum. During ticks
in core, new members are created only if the ring has changed. There is
a race that can sometimes prevent the members from starting and thus the
quorum from being achieved indefinitely. This small change to the test
infrastructure resolves this issue, but it still requires fixing in
riak_core and/or riak_kv.
Add ensemble_basic, ensemble_basic2, and ensemble_basic3 tests.
These tests test that Riak correctly generates proper consensus
groups, these groups reach quorum, handle leader failures, etc.
ensemble_basic3 tests basic consistent K/V API as well as behavior
during simple network partitions.
