* `xmlrunner` proved to be a way better choice than `junitxml` and it's even parsable by Bamboo. The output of `junitxml` is not.
* The output directory can be tweaked with the environment variable `SALT_XML_TEST_REPORTS`.
* Raised max open files by 1024 since running in a vagrant machine 2048 was not enough, though, I think travis-ci defaults to a higher value.
* Wait a bit longer on the events testing.
* Separated the initial minions connected and minions synced into two different events and functions. Also corrected the `if` expected value when a timeout occurs.
While running the tests within a VirtualBox(vagrant) machine, if the salt source is mounted using shared folders, any tests involving hard links WILL fail. Any tests involving symlinks, can be made to work by setting some properties on the VirtualBox shared folder. By moving the tests `tmp` directory to `tempfile.gettempdir()` we avoid this false test case errors.
* All logging messages are now logged on a log file. If verbose is enabled, the log messages are also logged to the console.
* The tests report now also let's us know if there were no glitches.
* Some tests might be considered destructive, such as adding or removing users from the system. This change allows those tests to be skipped unless stated otherwise.
```python
from saltunittest import destructiveTest
class ATestCase(TestCase):
@destructiveTest
def test_some_destructive_code(self):
"""Some destructive code."""
```
On the console we get something like:
```
test_destructive (unit.config_test.ConfigTestCase) ... skipped 'Destructive tests are disabled'
```
Previously we have to go through the scroll history to see which tests failed, which were skipped, etc. Now, besides test failures, errors and skips being reported like they always were, we also provide a nice overall report and the end.
* Even when the tests logging handler stream is directed to `/dev/null` we still need a low logging level so that our custom logging handler, used to see if certain log messages are emitted or not, catches messages.
--no-clean option speeds up repeating test execution by skipping the cleanup
process before and after testing.
Typical use case command could look like
./runtests.py -vv --no-clean --name integration.modules.ssh
In addition to traditional file system path test discovery it is
useful to run a test by stating it's dotted python path. For example,
this will only run the cmdmod integration tests from the
module suite::
$ runtests.py -n tests.integration.modules.cmdmod
The option parsing logic has changed without affecting the test
invocation commandline interface.
So, it should be possible to invoke tests like this:
python tests/runtests.py -u #only run unit tests
python tests/runtests.py -m #only run module tests
python tests/runtests.py #run all tests
python tests/runtests.py -v #run all tests with verbose output
python tests/runtests.py -vu #run unit tests with verbose output
Integration tests requiring a salt master and daemon were seperated
from pure unit tests. For now both are run with runtest.py. In
the future it could take arguments for which type of tests to run.
After a great deal of thought, I feel that the only way to reliably run
a test suite it to start a minion and master, and condition the data to
reflect the literal returns (take out the transport overhead).
This commit is the initial push towards this method of unit testing,
these test will be able to acctually test every aspect of salt.
This allows for tests to be written without too much change from the
normal unittest workflow. Now they should use the 'saltunittest'
namespace and we will need to import anything we need from the original
'unittest' or 'unittest2' namespace.
