Fixes#38683
When two states combine the `require`, `failhard`, and `order` options,
the `order` option should be ignored because `require` is present. Then
the `failhard` option should make the state run fail.
The check for "failhard" in the `check_failhard` function in the state
compiler was too broad. We want "failhard" to be true AND tag to be in
the `running` dict. Without the parens around the first OR statement,
we were bypassing the AND requirement because failhard was found.
Since the second state's tag was not found in the running dict, the state
run stacktraces on a KeyError.
This commit changes how we handle Docker authentication. We no longer
try to auth when pushing/pulling images. This was initially done based
on a misunderstanding of how authentication was handled in docker-py.
docker-py will automagically use the base64-encoded username/password
combination stored in config.json if present, so we don't need to auth
at all for push or pull, docker-py will handle all of that. We just need
to make sure that we get the auth info into the config.json. To do this
I have added a login() function to the execution module, which uses the
Docker CLI to authenticate (with output_loglevel set to "quiet" to
suppress any logging of the credentials). The CLI is used in this
instance because docker-py provides no interface by which to persist a
login in the config.json; it can read from the file, but is not designed
to *write* to it. Rather than trying to write to this file ourselves,
and potentially breaking it when (not *if*) Docker decides to change the
internal format of the JSON data structure, using the CLI is a way of
future-proofing the authentication logic.
Context caching of the docker-py client instance has also been removed.
Context caching was used based on the same incorrect understanding of
how docker-py handles authentication, and sought to avoid repeated login
attempts. As that is no longer a concern, there is no need to cache the
client instance in the context dunder because we don't really gain a
performance benefit from it.
The _image_wrapper() function has been removed, as it no longer serves
any purpose, and the image-specific code in it that was still needed
has been absorbed into the _client_wrapper() helper. The functions which
used the _image_wrapper() helper (like push() and pull()) now use
_client_wrapper().
Additionally, the decorators used to enforce a minimum Docker version
(or Docker API version) have been removed. These are not necessary since
docker-py handles raising exceptions when a given feature is not
supported by the effective API version. The _client_wrapper() helper
function now checks for miscellaneous docker exceptions by catching the
docker.errors.DockerException class, which is the base class for the
custom exceptions raised by docker-py, and by doing so catches exceptions
raised due to API version incompatibility (and more).
The list of functions in the top-level docstring has been removed as it
is very out-of-date, and is somewhat superfluous anyway given that we
have for some time now had a list of the functions on the right side of
the page in the documentation.
Other changes:
- Fixed a bug I introduced when I overhauled the
docker_container.running state, in which a container with no changes
to be made, which was not running, would be started by the state even
when test=True was passed.
- Custom registry image names are now properly identified. The colon in
the hostname of the custom registry was previously causing incorrect
identification of the image name and tag, when no explicit tag was
being passed (e.g. localhost:5000/myimage).
- When configuring credentials, the creds for the Docker Hub can be
configured under a registry named ``hub``. This keeps the user from
having to figure out the registry URL and configure it in their Pillar
data, and thus makes using this module easier.
- Removes the email address from the documentation for credential
configuration. This was probably initially added because docker-py
accepts it as an argument, but it is entirely ignored for purposes of
authentication (even by docker-py) and is thus unnecessary. Relic of
an earlier time in Docker's history, perhaps?
- Fixed RST references to functions which weren't caught when this
module was renamed to dockermod.py
- Allow filter arguments in docker.networks to be passed as a
comma-separated list as well as a Python list, in keeping with general
usage elsewhere in Salt.
* Add Ingest pipeline methods to Elasticsearch execution module
* Refactor Elasticsearch execution module and properly handle exceptions
* Throw CommandExecutionError in methods applicable for different Elasticsearch versions
* Refactor Elasticsearch states to reflect execution module changes
* Add state for managing Elasticsearch pipelines
* Fix few typos in Elasticsearch module, return None when deleted document doesn't exist
* Implements stats and health methods for Elasticsearch
* Add Elasticsearch methods to open/close index, manage search templates and repositories
* Merge existing Elasticsearch states into single one, add Search Template handling
* Add index alias state for Elasticsearch, fix documentation
* Catch all global exceptions in Elasticsearch states, unit test all of them
* Implement few Unit tests for Elasticsearch execution module, merge fixes into deprecated elastic states
* Implement additional unit tests for Elasticsearch execution module
* Finalize Elasticsearch module documentation
This test occassionally fails on the develop branch and I cannot
reproduce it. @cachedout recommended to add the flaky decorator
to the test until we can circle back around and look at this more
closely.
The existing fix did not work if the profile name itself had a dash `-`.
For example - `virtual-guest`. This commit fixes that by using `split('- ')`
rather than `split('-')`. This commit also provides two simple tests for the
`list_()` function to emulate behaviour of both old and new tuned-adm versions
Fixes#39692
When we hit that attribute error, we need to set the docker version
variable to a tuple starting with 0 so we can use it in other test
comparisons. If docker is installed, but at a lower version, we will
see this attribute error.
Fixes the test failure on the 2016.11 branch on CentOS 6 VMs.
This test module was updated for both the 2016.11 and develop
branches in different ways. This commits removes some of the
changes that were merged-forward from the 2016.11 branch and
restores the develop branch tests functionality.
Also remove unused and unnecessary behavior from this function when the
first path component is of single length. An empty first parameter to
this function will normalze to "." and result in an incorrectly-joined
result.
The tests happened to work because py2 does not default to absolute
imports, contrary to py3.
Additionally, with py3 absolute imports we must remove imported libs
from `sys.modules`(not salt loader modules)
Now that we are not caching these in the context dunder, we should pass
them into this function so that we don't repeat gathering the systemd
services inside _get_sysv_services().
Also, this fixes an incorrect change to the mocking that was apparently
made at some point to the get_all unit test.
With the merging of #39996, salt/modules/docker.py now imports
salt.utils.docker. However, our loader appends the module dir (in
this case salt/modules/) to sys.path temporarily for the length of the
loading process. So, as the docker execution module tries to import
salt.utils.docker, when salt.utils.docker attempts to do an "import
docker", and docker-py is *not* installed, this results in
salt/modules/docker.py (the docker execution module) being loaded in its
place, which results in tracebacks in the minion log.
Renaming the docker execution module keeps this import shadowing from
occurring. Note that we don't need to do this for the placeholder
salt/states/docker.py as it does not import salt.utils.docker.
Extend pillar include support so that nesting is possible
for the key directive.
Given the following pillar files:
foo1.sls:
include:
- foo:
key: two:levels
foo2.sls:
foon: blah
After this commit, it'll result in the following pillar data:
two:
levels:
foon: blah
Currently, it results in the following data which is far less useful
for usage, and in ability to be addressed:
two:levels:
foon: blah
Alternatives should say it is setting the symlink to `path` instead of
`current` which it is already set to
ssh_auth.absent should say it is going to remove the key, unless the status is
set to `add`, then it is already absent
Fixes#40322Fixes#40321
The tests pass fine when run independently, but when the full unit test suite
runs, the dockerng state tests are setting global values. This change makes the
module tests avoid those problem by relying on its own mocks.
The rabbitmq module was using `__context__` to store paths of
rabbitmq executables. `__context__` may be cleared but the module
still could remain in use, in which case it would fail to work
correctly. Move the paths of the rabbitmq executables to their own
global variables so that they are not affected by the lifespan of
`__context__`.
Signed-off-by: Sergey Kizunov <sergey.kizunov@ni.com>
The -G/--grains/--grains-tests option was being ignored because a suite
was not defined for it. This adds the option, and also adds a dunder
init to satisfy the test parser.
This does the following:
- Splits states for container/volume/image/network management into four
separate state modules.
- Preserves backward compatibility by making ``docker.image_present``
invoke ``docker_image.present``, etc.
- Changes how Salt detects that a container needs to be replaced.
Instead of comparing each passed argument to the named container's
configuration, it creates a temporary container, and compares that
container to the named container. If the two differ, then the older
container is removed, and the new one is renamed and started, becoming
the named container.
- Removes the unit tests for container management and replaces them with
integration tests.
- Adds unit tests for the new salt.utils.docker
The tornado web aplication that was set up in the archive tests, and then
duplicated in the remote file integration tests, starts the web server,
but never stops it. This creates a stacktrace that hangs the other test
file that attempts to start the web server.
The Application class has a `listen()` function, but not a `stop()` function.
The change uses the `HTTPServer` class to set up the listening server, but
also has the necessary `stop()` function. (The `listen()` function from the
`Application` class just calls out to the `HTTPServer`'s `listen()` function,
so this works nicely here.)
We can then call the `stop()` function in the `tearDownClass` class method.
I also removed some duplicate STATE_DIR definitions.
Put the skipIf and the destructiveTest decorators on the class instead
of each function. Also added a user.absent to the tearDown to reduce
code duplication.
Python 3 didn't like some of the stuff that we were doing with
``subprocess.check_call()`` in these tests, so to fix this I have redone
that stuff with calls to functions in the git execution module. In order
to avoid problems with running tests with no global gitconfig, I needed
to add an argument called ``git_opts`` to most of the funcs in the git
execution module (well I didn't *need* to do it to most of the funcs, it
just seemed like we shouldn't only be supporting this argument in a
single function).
This new ``git_opts`` argument is specifically for passing arguments to
the git command itself (not the subcommand). For example, ``git -c
user.name="Foo Bar" commit .....`` is different than running ``git
commit -c user.name="Foo Bar" .....``, because the ``commit`` subcommand
for git also accepts ``-c``.
The tearDown appears to only be removing the grain if it matches a
specific value. This may be leading to the grain value not being blank
at the time the next test is run.
Instead of only deleting the grain if it matches a specific value,
instead delete all items from that grain to ensure that it is empty for
the next test.
http://bugs.python.org/issue1322
python 3.7 is deprecating the platform.{linux_distribution,dist}
functions. They are being moved to the `distro` module on pypi. This
adds support for using the distro module if platform does not have the
needed functions.
