Part of the condition checking done in the replication_object_reformat
test is to validate the results of a fullsync using
repl_util:validate_completed_fullsync/6. The way in which the the
function is called from the test expects fullsync to complete with 0
error_exit or retry_exit conditions occurring. This requires that sink
cluster be in a steady state with all partitions available. The test
failed to wait for such conditions to occur and instead relied on
performing a node downgrade asynchronously and waiting for up to 60
seconds for a completion message before continuing with the test. The
test was continually failing after a node was downgraded to `previous`
due to partitions being reported as `down` on that node. To resolve
the issue the node downgrade process is now done in the primary test
process instead of in a separate spawned process. After the version
downgrade is complete, the test now waits for the riak_repl and the
riak_kv services, calls rt:wait_until_nodes_ready/1, calls
rt:wait_until_no_pending_changes/1, and finally waits for the
riak_repl2_fs_node_reserver named process to be registered on the
downgraded node. This process is responsible for handling partition
reservation requests and is key to determining the the new node is
able to handle a fullsync without partition errors.
Update the calls to rt:systest_read in repl_util and
repl_aae_fullsync_util to treat identical siblings resulting from the
use of DVV as a single value. These changes are specifically to
address failures seen in the repl_aae_fullsync_custom_n and
replication_object_reformat tests, but should be generally useful for
replication tests using the utility modules that and that have
allow_mult set to true.
Fix problem with cacertdir specification in replication_ssl test. The
code used load cert files in v2 replication expects the path specific
by the cacertdir key to only be a directory. With v3 replication the
code used is flexible enough to allow a directory or a file. Also
correct a typo in the certfile path for the SSLConfig1 configuration.
* Do not attempt to cancel fullsync if the initial attempt to start
and wait for completion fails. It has not been observed that the
problem is fullsync starting and not completing in time, but rather
the issue is that the initial call to start fullsync does not take
effect. Therefore the cancellation is unnecessary.
* Replace the call to repl_util:wait_for_connection/2 in the node
upgrade process with a call to
replication:wait_until_connection/1. This function is geared towards
v2 replication and should speed up test execution.
Replace use of a 40 second sleep in the test_supervision test case
with a wait condition to better handle variances in the time it takes
to progress through 10 retry attempts.
AAE is disabled. (defect https://github.com/basho/riak_kv/issues/959)
- Adds additional console output to reset-current-env to explain
configuration and steps being executed
- Adds the -n option to the reset-current-env script to specify the
number of nodes to build. By default, 5 will be created.
As of commit 3044839456 tests that
return something other than the prescribed success atom 'pass' to
indicate success result in test failure. Change the
replication_upgrade and replication2_upgrade tests that return the
result of the a call to lists:foreach/2 to instead return 'pass' to
indicate success.
Prior to this commit, the various riak_ensemble related tests would
manually enable the consensus system on one-and-only-one node in a
given cluster in order to work around issue basho/riak_core#571.
This commit changes the tests to work properly after the above issue
has been fixed.
In addition to removing the call to riak_ensemble_manager:enable()
that is now handled automatically by Riak, this commit also removes
a few wait_until_stable/2 checks against 1-node clusters. These
checks no longer apply, since Riak is now designed to only enable
the consensus system after the cluster contains at least 3 nodes.
Changed intercept to explicitly return `{error, econnrefused}`. Moved
helper functions to `repl_util` and added a new helper to distinguish
between disconnects on `cluster_by_name` and `cluster_by_address`
connections.
Added asserts to all wait_for functions.
ensemble_remove_node2 uses an intercept to prevent a riak_ensemble
related transition that is necessary for nodes to completely exit and
shutdown after removal. In fact, testing for this scenario is the
entire point of this test, since it is testing logic that was added to
solve basho/riak_core#572 and that logic prevents nodes from exiting
until that transition occurs.
However, even without this new logic, there is an unrelated
riak_ensemble related bug that can trigger a race condition that also
prevents nodes from shutting down.
The good news is that other changes made as part of the solution to
solve basho/riak_core#572 also fix this unrelated bug. Therefore this
commit extends ensemble_remove_node2 to remove the intercept at the
end of the test and verify that the removed nodes do actually end up
exiting as expected. Thus, the test now tests for both the negative
and positive scenarios and serves as a test against future regressions
that stall node removal/shutdown.
Adding a test to verify a bucket type is visible from a number of nodes
since the active status is given as long as the claimant sees it. But
requests to other nodes can end up hitting the dreaded {error, no_type}.
Also added a general utility that can be used for bucket type checks and
for general verification of bucket properties across nodes.
an undefined bucket type is specified. (defect #875)
- Adds a description of the reset-current-env.sh script and its
usage to README.md
- Corrects a spelling mistake in an information message emitted by
the reset-current-env.sh script
While r16b02-basho5 did not need the cacertfile path put in, r16b03 did.
The test still passes r16b02-basho5 with the added cacertfile line. Since
there is no harm in putting it in, better for forwards compatibility than
not.
