mirror of
https://github.com/valitydev/salt.git
synced 2024-11-07 08:58:59 +00:00
249 lines
8.5 KiB
ReStructuredText
249 lines
8.5 KiB
ReStructuredText
=================
|
|
Running The Tests
|
|
=================
|
|
|
|
There are requirements, in addition to Salt's requirements, which
|
|
need to be installed in order to run the test suite. Install one of
|
|
the lines below, depending on the relevant Python version:
|
|
|
|
.. code-block:: bash
|
|
|
|
pip install -r dev_requirements_python26.txt
|
|
pip install -r dev_requirements_python27.txt
|
|
|
|
.. note::
|
|
|
|
In Salt 0.17, testing libraries were migrated into their own repo. To install them:
|
|
|
|
.. code-block:: bash
|
|
|
|
pip install git+https://github.com/saltstack/salt-testing.git#egg=SaltTesting
|
|
|
|
|
|
Failure to install SaltTesting will result in import errors similar to the following:
|
|
|
|
.. code-block:: bash
|
|
|
|
ImportError: No module named salttesting
|
|
|
|
Once all require requirements are set, use ``tests/runtests.py`` to
|
|
run all of the tests included in Salt's test suite. For more information,
|
|
see ``--help``.
|
|
|
|
An alternative way of invoking the test suite is available in ``setup.py``:
|
|
|
|
.. code-block:: bash
|
|
|
|
./setup.py test
|
|
|
|
Instead of running the entire test suite, there are several ways to run only
|
|
specific groups of tests or individual tests:
|
|
|
|
* Run unit tests only: ``./tests/runtests.py --unit-tests``
|
|
* Run unit and integration tests for states: ``./tests/runtests.py --state``
|
|
* Run integration tests for an individual module: ``./tests/runtests.py -n integration.modules.virt -vv``
|
|
* Run unit tests for an individual module: ``./tests/runtests.py -n unit.modules.virt_test -vv``
|
|
|
|
|
|
Running Destructive Integration Tests
|
|
=====================================
|
|
|
|
Salt is used to change the settings and behavior of systems. In order to
|
|
effectively test Salt's functionality, some integration tests are written to
|
|
make actual changes to the underlying system. These tests are referred to as
|
|
"destructive tests". Some examples of destructive tests are changes may be
|
|
testing the addition of a user or installing packages. By default,
|
|
destructive tests are disabled and will be skipped.
|
|
|
|
Generally, destructive tests should clean up after themselves by attempting to
|
|
restore the system to its original state. For instance, if a new user is created
|
|
during a test, the user should be deleted after the related test(s) have
|
|
completed. However, no guarantees are made that test clean-up will complete
|
|
successfully. Therefore, running destructive tests should be done with caution.
|
|
|
|
.. note::
|
|
|
|
Running destructive tests will change the underlying system. Use caution when running destructive tests.
|
|
|
|
To run tests marked as destructive, set the ``--run-destructive`` flag:
|
|
|
|
.. code-block:: bash
|
|
|
|
./tests/runtests.py --run-destructive
|
|
|
|
|
|
Running Cloud Provider Tests
|
|
============================
|
|
|
|
Salt's testing suite also includes integration tests to assess the successful
|
|
creation and deletion of cloud instances using :ref:`Salt-Cloud<salt-cloud>` for
|
|
providers supported by Salt-Cloud.
|
|
|
|
The cloud provider tests are off by default and run on sample configuration files
|
|
provided in ``tests/integration/files/conf/cloud.providers.d/``. In order to run
|
|
the cloud provider tests, valid credentials, which differ per provider, must be
|
|
supplied. Each credential item that must be supplied is indicated by an empty
|
|
string value and should be edited by the user before running the tests. For
|
|
example, Digital Ocean requires a client key and an api key to operate. Therefore,
|
|
the default cloud provider configuration file for Digital Ocean looks like this:
|
|
|
|
.. code-block:: yaml
|
|
|
|
digitalocean-config:
|
|
provider: digital_ocean
|
|
client_key: ''
|
|
api_key: ''
|
|
location: New York 1
|
|
|
|
As indicated by the empty string values, the ``client_key`` and the ``api_key``
|
|
must be provided:
|
|
|
|
.. code-block:: yaml
|
|
|
|
digitalocean-config:
|
|
provider: digital_ocean
|
|
client_key: wFGEwgregeqw3435gDger
|
|
api_key: GDE43t43REGTrkilg43934t34qT43t4dgegerGEgg
|
|
location: New York 1
|
|
|
|
.. note::
|
|
|
|
When providing credential information in cloud provider configuration files,
|
|
do not include the single quotes.
|
|
|
|
Once all of the valid credentials for the cloud provider have been supplied, the
|
|
cloud provider tests can be run by setting the ``--cloud-provider-tests`` flag:
|
|
|
|
.. code-block:: bash
|
|
|
|
./tests/runtests.py --cloud-provider-tests
|
|
|
|
|
|
Running The Tests In A Docker Container
|
|
=======================================
|
|
|
|
The test suite can be executed under a `docker`_ container using the
|
|
``--docked`` option flag. The `docker`_ container must be properly configured
|
|
on the system invoking the tests and the container must have access to the
|
|
internet.
|
|
|
|
Here's a simple usage example:
|
|
|
|
.. code-block:: bash
|
|
|
|
tests/runtests.py --docked=ubuntu-12.04 -v
|
|
|
|
The full `docker`_ container repository can also be provided:
|
|
|
|
.. code-block:: bash
|
|
|
|
tests/runtests.py --docked=salttest/ubuntu-12.04 -v
|
|
|
|
|
|
The SaltStack team is creating some containers which will have the necessary
|
|
dependencies pre-installed. Running the test suite on a container allows
|
|
destructive tests to run without making changes to the main system. It also
|
|
enables the test suite to run under a different distribution than the one
|
|
the main system is currently using.
|
|
|
|
The current list of test suite images is on Salt's `docker repository`_.
|
|
|
|
Custom `docker`_ containers can be provided by submitting a pull request
|
|
against Salt's `docker Salt test containers`_ repository.
|
|
|
|
.. _`docker`: https://www.docker.io/
|
|
.. _`docker repository`: https://index.docker.io/u/salttest/
|
|
.. _`docker Salt test containers`: https://github.com/saltstack/docker-containers
|
|
|
|
|
|
===================
|
|
Automated Test Runs
|
|
===================
|
|
|
|
SaltStack maintains a Jenkins server to allow for the execution of tests
|
|
across supported platforms. The tests executed from Salt's Jenkins server
|
|
create fresh virtual machines for each test run, then execute destructive
|
|
tests on the new, clean virtual machine.
|
|
|
|
When a pull request is submitted to Salt's repository on GitHub, Jenkins
|
|
runs Salt's test suite on a couple of virtual machines to gauge the pull
|
|
request's viability to merge into Salt's develop branch. If these initial
|
|
tests pass, the pull request can then merged into Salt's develop branch
|
|
by one of Salt's core developers, pending their discretion. If the initial
|
|
tests fail, core developers may request changes to the pull request. If the
|
|
failure is unrelated to the changes in question, core developers may merge
|
|
the pull request despite the initial failure.
|
|
|
|
Once the pull request is merged into Salt's develop branch, a new set of
|
|
Jenkins virtual machines will begin executing the test suite. The develop
|
|
branch tests have many more virtual machines to provide more comprehensive
|
|
results.
|
|
|
|
There are a few other groups of virtual machines that Jenkins tests against,
|
|
including past and current release branches. For a full list of currently
|
|
running test environments, go to http://jenkins.saltstack.com.
|
|
|
|
|
|
Using Salt-Cloud on Jenkins
|
|
===========================
|
|
|
|
For testing Salt on Jenkins, SaltStack uses :ref:`Salt-Cloud<salt-cloud>` to
|
|
spin up virtual machines. The script using Salt-Cloud to accomplish this is
|
|
open source and can be found here: :blob:`tests/jenkins.py`
|
|
|
|
|
|
=============
|
|
Writing Tests
|
|
=============
|
|
|
|
Salt uses a test platform to verify functionality of components in a simple
|
|
way. Two testing systems exist to enable testing salt functions in somewhat
|
|
real environments. The two subsystems available are integration tests and
|
|
unit tests.
|
|
|
|
Salt uses the python standard library unittest2 system for testing.
|
|
|
|
Naming Conventions
|
|
==================
|
|
|
|
Any function in either integration test files or unit test files that is
|
|
doing the actual testing, such as functions containing assertions, must
|
|
start with ``test_``:
|
|
|
|
.. code-block:: python
|
|
|
|
def test_user_present(self):
|
|
|
|
When functions in test files are not prepended with ``test_``,
|
|
the function acts as a normal, helper function and is not run as a test
|
|
by the test suite.
|
|
|
|
Integration Tests
|
|
=================
|
|
|
|
The integration tests start up a number of salt daemons to test functionality
|
|
in a live environment. These daemons include 2 salt masters, 1 syndic, and 2
|
|
minions. This allows the syndic interface to be tested and master/minion
|
|
communication to be verified. All of the integration tests are executed as
|
|
live salt commands sent through the started daemons.
|
|
|
|
Integration tests are particularly good at testing modules, states and shell
|
|
commands.
|
|
|
|
* :doc:`Writing integration tests <integration>`
|
|
|
|
Unit Tests
|
|
==========
|
|
|
|
Direct unit tests are also available. These tests are good for testing internal
|
|
functions.
|
|
|
|
* :doc:`Writing unit tests <unit>`
|
|
|
|
|
|
.. toctree::
|
|
:hidden:
|
|
|
|
integration
|
|
unit
|