- Update make_certs to support wildcard certificates by passing an additional Path parameter.
- add replication2_ssl test to ensure using wildcard certificates works after PR to riak_core_ssl_util.
even in the rare and pathological case where the cluster is partitioned before all 3 nodes
have received the update. riakc_flag:disable(F) requires context, which isn’t there in the
new map that would be created on the side of the partition with no data.
This test's a little confused in the sources as-is since it prints
like it's based on the number of requests, even though the actual
comparison is done against a function of THRESHOLD. I've reverted
to the comparison used currently, since it looks to me like this
test should really expect to have ~NUM_REQUESTS processes, and a
vnode queue pretty close to THRESHOLD. I'd appreciate review here
though, particularly if anyone recalls the original intent of
these comparison numbers.
Previously we'd used a sort of fuzzy 'metric' where we expect the
number of successful requests/fsms to be less than some fudge
factor over the overload threshold. This tends to kick up spurious
failures on the test board without offering much more in the way
of assurances about overload's functionality.
This change instead bases test success on the number of requests
only, not the threshold — if some amount of work was shed at all
we consider that a passing test.
In the future we should revisit this and change the request
accounting machinery to just explicitly track denials instead of
fsm processes / vnode queue depth.
In the case that no advanced.config file exists (everycase!) rt
would not add any advanced config settings to the conf.
This PR teaches rtdev to create an advanced.comfig file if none exists
so that tests may set advanced config.
In this case we set ring_size and also the `crdt_mixed_versions` app env
Still have not completed upgrade and feature-flag switch.
I changed the versions from atoms to "2.0.2" and "2.0.4", we can
bikeshed that with build/test czars on Monday.
Added some useful logging statements to the plain-upgrade test.
Removed unnecessary clean_cluster and systest_read calls.
This test is getting Biig, and there is still a lot to add
(see comments at the end of the test.)
Maybe we should break it out into a few tests, and there are some
questions still.
* Checks the list of stats keys returned from the HTTP endpoint
is complete -- delineating between riak and riak_ee. The test will
fail if the list returned from the HTTP endpoint does not exactly match
the expected list. This behavior acts as a forcing function to ensure
that the expected list is properly maintained as stats are added and
removed.
* Modifies reset-current-env to properly clean dependencies when a
full clean is requested and remove the current directory in the
target test instance.
* Adds logging to verify_riak_stats to explain the addition steps
being performed
* Adds rt:product/1 to determine whether a node is running riak,
riak_ee, or riak_cs
* Adds tools.mk support and eunit scaffolding to rebar.config
* Modifies reset-current-env.sh to remove the current directory in
the target test instance
Because list-keys and list-buckets use coverage, we might hit latent
replicas depending on the coverage plan. This gives each call some extra
tries to complete successfully.
It was previously possible for the 'minority' network partition to
become the majority network partition by a naive network partitioning
strategy. Previously, when a preference list of 5 keyspace partitions
was created on only four distinct nodes, it became possible for a 2 node
'minority' network partition group to actually have a majority of
keyspace partitions because 2 keyspace partitions were assigned to 1
node in the 'minority' group. This was fixed so that the 'majority'
group now always has a majority of keyspace partitions by preventing
nodes with greater than 1 keyspace partition from becoming part of the
'minority' group.
Now green when run in sequence:
Test Results:
pb_cipher_suites-bitcask: pass
pb_security-bitcask : pass
---------------------------------------------
0 Tests Failed
2 Tests Passed
That's 100.0% for those keeping score
This changes the test assertion so that it retries fetching the value
from the second cluster until it is the expected value, at which point
the test will either pass if the sibling count is reasonable or fail if
it is too damn high.
Fetch the sink object on each iteration of the wait_until, just in case
that the entire set of siblings didn't make it across the repl link.
This also gives read-repair a chance to happen, in case the version the
sink wrote didn't make it to all replicas.
Trying to use the repl features before newly started nodes have
riak_repl completely initialized leads to all sorts of nasty crashes and
noise. Frequently it makes fullsync stuck forever, which makes a lot of
the tests fail.
This also tweaks the AAE fullsync tests to remove assumptions about
failure stats when AAE transient errors occur. The behavior in the
handling of those errors has changed recently with the introduction of
soft exits.
Reduces the probability of a race condition between the calculation of spiral/histogram metrics and SNMP stat cache refresh by reducing the SNMP poll interval to 1 second during test execution
Don't wait for convergence of the ring, because bucket properties are no
longer stored in the ring; instead, wait until the property changes,
which means the gossip has stabilized.
Add an rt:admin/3 function that accepts a list of options as the third
parameter. Currently the only valid option is return_exit_code. The
rtdev, rtssh, and rt_cs_dev harnesses have been updated to support
this option. If return_to_exit is specified the return from a
?HARNESS:admin call is a pair with the exit code as the first member
and the command output as the second member. Finally the
basic_command_line test has been changed to use return_for_exit to
verify the changes.
Due to the refactor for the cluster manager/connection manager system to
use otp behaviors, the raw message method of getting stats has been ousted.
Instead, it uses a call. To allow the riak_test to be able to check older
clusters as well as the method, the function was extended to try new and
then the old.
Add testing of the handoff heartbeat change from the following pull
request: https://github.com/basho/riak_core/pull/560. Add an intercept
module for the riak_core_handoff_sender module to introduce artificial
delay on item visitation during a handoff fold. This delay along with
the changes to the verify_handoff test induces test failure when run
without the heartbeat change. The handoff_receive_timeout is exceeded,
handoff stalls, and the test eventually fails due to timeout. The test
succeeds when run with the heartbeat change.
Ensemble_ring_changes tests writing a value, expanding the cluster, then
updating and reading that value after ring expansion has completed. It
also creates a bucket using a bucket type with a different n_val from
the default bucket type. The latter tests basho/riak_kv#1008 and it's
corresponding riak_core PR.
Use riak_test_runner:metadata/0 to get the configured backend instead of
defaulting to bitcask. Additionally we use rt:clean_data_dir/2 to safely
remove backend directories.
This is the first iteration of creating byzantine dataloss tests that
show both recoverable and unrecoverable, but detectable errors. This tests the
following scenarios.
* Lose one partition worth of data, but no synctrees and recover.
* Lose all but one partition of ensemble data, but no synctrees and
recover.
* Lose minority of synctrees. Only the peers with the missing
synctrees are restarted. System remains available.
* Loss of majority of synctrees. Majority peers are restarted. System
recovers when they all come back online.
* Loss of majority of synctrees with one node partitioned. All peers
restarted except partitioned one. System does not recover with that
node partitioned. When the partition is healed the system recovers.
* Loss of all data and synctree except on one peer recovers.
* Backing up and restoring old data but not synctrees results in
detected errors. Restoring newer data fixes this.
* Delete all data on all nodes, but not synctrees. This is detected and
an error returned to the user.
Change the ACL test case in the replication_ssl and replication2_ssl
tests to use certificates generated within the tests instead of
relying on certificates created outside the test that are prone to
expire and cause spurious test failure.
Also change the replication_ssl and replication2_ssl tests to avoid a
cycle of standing up the test clusters and then immediately restarting
them before any tests cases execute. This should make the test
execution slightly faster for both test modules.
This commit also changes the tests to be a bit more robust in checking
for cluster state when restarting nodes and removes an unnecessary
five second sleep call in the replication_ssl test.
Change replication_ssl to use the wait_for_site_ips function from the
replication module introduced in
297090ded6 instead of the defunct
verify_site_ips function.
Avoid a race condition in the replication test module when checking
for site IP addresses in the replication status output. The test
waits for a connection on the leader, but it only queries the
replication status to check for the expected site IP addresses a
single time. Change the test to wait and re-check the status output to
give greater assurance that if the expected site IP addresses are not
present it is due to legitimate failure and not a race condition in
checking the replication status. This change affects the replication
and replication_upgrade tests as well as any other tests that call the
replication:replication function.
