First, convert all of the node operations that write/delete to use the
Riak PB connection opened at the start of the test. When deleting,
ensure we wait for the object to be fully deleted.
Here's where it gets more interesting:
In the original allow_mult=false behaviour, original objects are
replaced with modified objects, and then when performing the restore
operation, the originals are abandoned for the modified objects because
they have a greater timestamp.
In the new allow_mult=true behaviour, original objects are replaced with
modified objects, and then the restore operations restores the original
object as a *sibling*, because the objects have divergent vector clocks.
Adapt the test for this.
Using the new function to detect when tombstones are really, really gone
so the test is more reliable.
Also, a bit of refactoring of the delete verification and wait_until
functions to make it cleaner.
This is a port of systest-backup-and-restore from basho_expect with some
changes and additions. Biggest change was not using search-cmd to index
data for search, as it does not seem to update docs, just indexes. Then
added restore on a clean cluster. Basically it does this:
* Writes data, including a search enabled bucket
* Verifies written data, including searches
* Runs backup
* Deletes, modifies some data
* Verifies mods and deletes took
* Runs restore
* Verifies deleted data is back, mods stay (last write wins)
* Stops cluster and wipes out all data dirs
* Verifies clean cluster
* Runs restore
* Verifies data was restored, including searches
Also, adding some more noise to some riak test setup functions. This was
useful when initially debugging the test before it worked the first
time.
