AWS recently added support for tagging CloudFront distributions,
which allows us to start managing them via Salt
as we can insert a Salt-controlled identifier as a `Name` tag.
(CloudFront distributions get unique IDs generated by AWS,
which we can't predict and thus use to manage them idempotently.)
Moves the original function to `salt.utils.files.py`, adds a deprecation
warning to the original function, and updates all of the istextfile
references in salt code.
Having a jid that is occasionally not unique (eg two processes creating
jobs at about the same time) causes problems in my usage due to
job tracking assuming unique jids. Add a config option called `unique_jid`
that will enforce unique jids, but will change the jid format by
appending an underscore and the process id.
Signed-off-by: Sergey Kizunov <sergey.kizunov@ni.com>
Some states are complicated and multiple subparts,
or maybe cross-call into __states__ if they manage subresources.
In these cases, they will have multiple comments.
Make this more ergonomic by supporting a list of strings as the
value for ret['comment'] in state returns and documenting this.
By joining comments on newlines, it is possible to combine
single-line and multi-line comments cleanly, as opposed to e.g. commas.
The driving impetus for this is some of the boto modules.
An update to the boto_sqs module is included as an example.
Add a check that outgoing state return data has the right shape,
and add a testcase as well.
Fix the NPM state tests and the saltmod runner & wheel state functions
to comply with the prescribed format.
Through merge forwards, some of the old-style paths have made their
way into develop. This PR corrects some of these instances on the more
popular functions.
On some platforms, for reason which I do not yet grok, images pulled
from the Hub are prefixed with "docker.io/". This causes the
docker_image states to fail unless the user manually adds "docker.io/"
before the image name.
This commit adds a new function called "docker.resolve_tag" which
disambiguates this variance and allows images to be specified without
the "docker.io/" prefix.
Resolves#42935.
Re-factoring a little following the previous commit, so that the same
container connection logic can be used in both places, rather than it
being repeated.
Previously any changes to network states wouldn't result in any change
to the network when the state was applied. This was less of an issue
the only attributes you could set for a network were its name and
driver, but with the recent support for driver options and now IPAM
config being added, it's useful for the state to be able to modify the
network if it needs to.
The Docker API doesn't provide any facility to modify an existing
network, so the only option is to recreate it with the new options,
which means disconnecting and reconnecting all containers.
Fixes#43047
moto versions >= 1.0.0 have changed the way the mocked connections through
boto are handled with the @mock_ec2 decorator. They use the boto3 connection
method. However, since we are still using boto in many places, we need to use
the new @mock_ec2_deprecated decorator instead to handle the boto connection
functions for the unit tests.
Versions of moto < 1.0.0 are not Python 3 compatible, so salt-jenkins should
be installing newer versions of moto for those tests. Unfortunately, we cannot
install an older version of moto for Python2 that use the original @mock_ec2 call
and also import the @mock_ec2_deprecated function for newer versions of moto
simultaneously as the @mock_ec2_deprecated function doesn't exist in older
versions of moto.
This moves the following functions from salt.utils to salt.utils.versions:
- warn_until
- kwargs_warn_until
- compare_versions (as salt.utils.versions.compare)
- version_cmp
moto versions >= 1.0.0 have changed the way the mocked connections through
boto are handled with the @mock_ec2 decorator. They use the boto3 connection
method. However, since we are still using boto in many places, we need to use
the new @mock_ec2_deprecated decorator instead to handle the boto connection
functions for the unit tests.
Versions of moto < 1.0.0 are not Python 3 compatible, so salt-jenkins should
be installing newer versions of moto for those tests. Unfortunately, we cannot
install an older version of moto for Python2 that use the original @mock_ec2 call
and also import the @mock_ec2_deprecated function for newer versions of moto
simultaneously as the @mock_ec2_deprecated function doesn't exist in older
versions of moto.
This PR is part of what will be an ongoing effort to use explicit
unicode strings in Salt. Because Python 3 does not suport Python 2's raw
unicode string syntax (i.e. `ur'\d+'`), we must use
`salt.utils.locales.sdecode()` to ensure that the raw string is unicode.
However, because of how `salt/utils/__init__.py` has evolved into the
hulking monstrosity it is today, this means importing a large module in
places where it is not needed, which could negatively impact
performance. For this reason, this PR also breaks out some of the
functions from `salt/utils/__init__.py` into new/existing modules under
`salt/utils/`. The long term goal will be that the modules within this
directory do not depend on importing `salt.utils`.
A summary of the changes in this PR is as follows:
* Moves the following functions from `salt.utils` to new locations
(including a deprecation warning if invoked from `salt.utils`):
`to_bytes`, `to_str`, `to_unicode`, `str_to_num`, `is_quoted`,
`dequote`, `is_hex`, `is_bin_str`, `rand_string`,
`contains_whitespace`, `clean_kwargs`, `invalid_kwargs`, `which`,
`which_bin`, `path_join`, `shlex_split`, `rand_str`, `is_windows`,
`is_proxy`, `is_linux`, `is_darwin`, `is_sunos`, `is_smartos`,
`is_smartos_globalzone`, `is_smartos_zone`, `is_freebsd`, `is_netbsd`,
`is_openbsd`, `is_aix`
* Moves the functions already deprecated by @rallytime to the bottom of
`salt/utils/__init__.py` for better organization, so we can keep the
deprecated ones separate from the ones yet to be deprecated as we
continue to break up `salt.utils`
* Updates `salt/*.py` and all files under `salt/client/` to use explicit
unicode string literals.
* Gets rid of implicit imports of `salt.utils` (e.g. `from salt.utils
import foo` becomes `import salt.utils.foo as foo`).
* Renames the `test.rand_str` function to `test.random_hash` to more
accurately reflect what it does
* Modifies `salt.utils.stringutils.random()` (née `salt.utils.rand_string()`)
such that it returns a string matching the passed size. Previously
this function would get `size` bytes from `os.urandom()`,
base64-encode it, and return the result, which would in most cases not
be equal to the passed size.
Headline changes:
- Move to boto3
- Use result-style error handling
- Optimize number of API calls:
- Set attributes at creation time
- Set all attributes in one API call
- Show diff when attribute changes detected
These test the scenarios where another network with a similar name
already exists, verifying that absent doesn't attempt to remove a
network which isn't specified, and present still attempts to create the
specified network despite a similarly named network already being present.
This started failing following commit 515c612, which relied on the
'Name' key being present in the return value of docker.networks -
as the mock didn't have this set the test started failing.
Separates out the comment from the rest of the state return to make this
test more stable on Python 3. This state run in PY3 adds the minion
names to the comment in a random order, so we can't rely on asserting
against this order in the test.
This PR updates the test so that we are still checking for all of the
elements in the comment, but not relying on the order the minions are
added the comment list.
These test the scenarios where another network with a similar name
already exists, verifying that network_absent doesn't attempt to remove
a network which isn't specified, and network_present still attempts to
create the specified network despite a similarly named network already
being present.
This started failing following commit 515c612, which relied on the
'Name' key being present in the return value of dockerng.networks -
as the mock didn't have this set the test started failing.
This ensures we do not introduce a regression which breaks
the usage of the `batch` option when using states in orchestration
runners.
Related to #40635 and #39169
Currently, the RVM state has no way to pass environment variables (ie, RUBY_CONFIGURE_OPTS)
or compilation flags (-C, --patch) to the Ruby installer. This patch enables this
functionality, permitting installation of customized Ruby installs. For example, the following
should now work:
ruby-2.3.0:
rvm.installed:
- env:
- RUBY_CONFIGURE_OPTS: --with-jemalloc
- opts:
- '--patch /path/to/my.patch'
- '-C --enable-shared,--with-readline-dir=$HOME/.rvm/usr'
* Add Ingest pipeline methods to Elasticsearch execution module
* Refactor Elasticsearch execution module and properly handle exceptions
* Throw CommandExecutionError in methods applicable for different Elasticsearch versions
* Refactor Elasticsearch states to reflect execution module changes
* Add state for managing Elasticsearch pipelines
* Fix few typos in Elasticsearch module, return None when deleted document doesn't exist
* Implements stats and health methods for Elasticsearch
* Add Elasticsearch methods to open/close index, manage search templates and repositories
* Merge existing Elasticsearch states into single one, add Search Template handling
* Add index alias state for Elasticsearch, fix documentation
* Catch all global exceptions in Elasticsearch states, unit test all of them
* Implement few Unit tests for Elasticsearch execution module, merge fixes into deprecated elastic states
* Implement additional unit tests for Elasticsearch execution module
* Finalize Elasticsearch module documentation
With the merging of #39996, salt/modules/docker.py now imports
salt.utils.docker. However, our loader appends the module dir (in
this case salt/modules/) to sys.path temporarily for the length of the
loading process. So, as the docker execution module tries to import
salt.utils.docker, when salt.utils.docker attempts to do an "import
docker", and docker-py is *not* installed, this results in
salt/modules/docker.py (the docker execution module) being loaded in its
place, which results in tracebacks in the minion log.
Renaming the docker execution module keeps this import shadowing from
occurring. Note that we don't need to do this for the placeholder
salt/states/docker.py as it does not import salt.utils.docker.
Alternatives should say it is setting the symlink to `path` instead of
`current` which it is already set to
ssh_auth.absent should say it is going to remove the key, unless the status is
set to `add`, then it is already absent
Fixes#40322Fixes#40321
This does the following:
- Splits states for container/volume/image/network management into four
separate state modules.
- Preserves backward compatibility by making ``docker.image_present``
invoke ``docker_image.present``, etc.
- Changes how Salt detects that a container needs to be replaced.
Instead of comparing each passed argument to the named container's
configuration, it creates a temporary container, and compares that
container to the named container. If the two differ, then the older
container is removed, and the new one is renamed and started, becoming
the named container.
- Removes the unit tests for container management and replaces them with
integration tests.
- Adds unit tests for the new salt.utils.docker
The previous way this test was written was over-mocking some of the
state returns as well as tested and asserted against incorrect
behavior as shown in issue #39255. (The test asserted that a return
of '' from the call to _checkblk() was correct, which was hiding a
bug as noted in #39255 and fixed in #39272.)
Stabilizes these tests by avoiding errors when cheese, crackers
order changes in the comment and gives errors like:
- 'comment': 'Kernel modules crackers, cheese are already present',
? --------
+ 'comment': 'Kernel modules cheese, crackers are already present',
? ++++++++
* openvswitch: add a basic unit test for openvswitch
* openvswitch: enclose comments_* variables into a map
* openvswitch: only prepare error messages if type parameter is present
* openvswitch: add test case: create a GRE port
* openvswitch: remove trailing whitespace
* openvswitch: add missing "comments" variable
* openvswitch: pep8 compliance
* openvswitch: mock dig.check_ip
* openvswitch: rename "type" argument into "tunnel_type"
* openvswitch: reference bridge_exists before assignment
* openvswitch: reference port_list before assignment
* openvswitch: add upgrade note about "type" option renaming
