* Wait for services after upgrading a node
* Remove hardcoded sleep call
* Reduce time between upgrades from 300 to 120 seconds
* Refactor worker functions to accept a report pid parameter instead
of assuming a loaded_upgrade named process.
* Remove the mapreduce load. The results of the mapreduce queries
during node shutdown are not predictable enough at this time to rely
on for this test.
loaded upgrade has a tendency to fail on listkeys time out, which should
not make the fail test. Made the PB client have the auto_reconnect flag
set and added functions to retry reconnecting (as it doesn't happen
instantaneously) and retry the list keys a few times with exponentially
bigger time outs.
Use a synchronous command to populate the search buckets, rather than
an asynchronous command. Otherwise, there is a data race between data
being rewritten and search queries being performed.
Change the search verify script to have the proper solr path, otherwise
the script always fails because it queries the wrong index.
Modify the search tester to allow search failures while the cluster has
mixed versions. There is currently a known bug in Riak that causes search
queries to potentially fail against a mixed cluster. The test now only
requires the tester to succeed after all nodes have been upgraded.
Add loaded_upgrade test that spawns k/v and map/reduce testers that run
concurrently as cluster nodes are upgraded. Each tester is spawned before
upgrading a node, and then verified after the upgrade. If the test fails,
the test is then re-run until it passes (or timeout) in order to distinguish
between transient failure and permanent data loss failures.
Current k/v and map/reduce testers use basho_bench.
