AWS recently added support for tagging CloudFront distributions,
which allows us to start managing them via Salt
as we can insert a Salt-controlled identifier as a `Name` tag.
(CloudFront distributions get unique IDs generated by AWS,
which we can't predict and thus use to manage them idempotently.)
Moves the original function to `salt.utils.files.py`, adds a deprecation
warning to the original function, and updates all of the istextfile
references in salt code.
Having a jid that is occasionally not unique (eg two processes creating
jobs at about the same time) causes problems in my usage due to
job tracking assuming unique jids. Add a config option called `unique_jid`
that will enforce unique jids, but will change the jid format by
appending an underscore and the process id.
Signed-off-by: Sergey Kizunov <sergey.kizunov@ni.com>
Some states are complicated and multiple subparts,
or maybe cross-call into __states__ if they manage subresources.
In these cases, they will have multiple comments.
Make this more ergonomic by supporting a list of strings as the
value for ret['comment'] in state returns and documenting this.
By joining comments on newlines, it is possible to combine
single-line and multi-line comments cleanly, as opposed to e.g. commas.
The driving impetus for this is some of the boto modules.
An update to the boto_sqs module is included as an example.
Add a check that outgoing state return data has the right shape,
and add a testcase as well.
Fix the NPM state tests and the saltmod runner & wheel state functions
to comply with the prescribed format.
Through merge forwards, some of the old-style paths have made their
way into develop. This PR corrects some of these instances on the more
popular functions.
On some platforms, for reason which I do not yet grok, images pulled
from the Hub are prefixed with "docker.io/". This causes the
docker_image states to fail unless the user manually adds "docker.io/"
before the image name.
This commit adds a new function called "docker.resolve_tag" which
disambiguates this variance and allows images to be specified without
the "docker.io/" prefix.
Resolves#42935.
Re-factoring a little following the previous commit, so that the same
container connection logic can be used in both places, rather than it
being repeated.
Previously any changes to network states wouldn't result in any change
to the network when the state was applied. This was less of an issue
the only attributes you could set for a network were its name and
driver, but with the recent support for driver options and now IPAM
config being added, it's useful for the state to be able to modify the
network if it needs to.
The Docker API doesn't provide any facility to modify an existing
network, so the only option is to recreate it with the new options,
which means disconnecting and reconnecting all containers.
Fixes#43047
moto versions >= 1.0.0 have changed the way the mocked connections through
boto are handled with the @mock_ec2 decorator. They use the boto3 connection
method. However, since we are still using boto in many places, we need to use
the new @mock_ec2_deprecated decorator instead to handle the boto connection
functions for the unit tests.
Versions of moto < 1.0.0 are not Python 3 compatible, so salt-jenkins should
be installing newer versions of moto for those tests. Unfortunately, we cannot
install an older version of moto for Python2 that use the original @mock_ec2 call
and also import the @mock_ec2_deprecated function for newer versions of moto
simultaneously as the @mock_ec2_deprecated function doesn't exist in older
versions of moto.
This moves the following functions from salt.utils to salt.utils.versions:
- warn_until
- kwargs_warn_until
- compare_versions (as salt.utils.versions.compare)
- version_cmp
moto versions >= 1.0.0 have changed the way the mocked connections through
boto are handled with the @mock_ec2 decorator. They use the boto3 connection
method. However, since we are still using boto in many places, we need to use
the new @mock_ec2_deprecated decorator instead to handle the boto connection
functions for the unit tests.
Versions of moto < 1.0.0 are not Python 3 compatible, so salt-jenkins should
be installing newer versions of moto for those tests. Unfortunately, we cannot
install an older version of moto for Python2 that use the original @mock_ec2 call
and also import the @mock_ec2_deprecated function for newer versions of moto
simultaneously as the @mock_ec2_deprecated function doesn't exist in older
versions of moto.
This PR is part of what will be an ongoing effort to use explicit
unicode strings in Salt. Because Python 3 does not suport Python 2's raw
unicode string syntax (i.e. `ur'\d+'`), we must use
`salt.utils.locales.sdecode()` to ensure that the raw string is unicode.
However, because of how `salt/utils/__init__.py` has evolved into the
hulking monstrosity it is today, this means importing a large module in
places where it is not needed, which could negatively impact
performance. For this reason, this PR also breaks out some of the
functions from `salt/utils/__init__.py` into new/existing modules under
`salt/utils/`. The long term goal will be that the modules within this
directory do not depend on importing `salt.utils`.
A summary of the changes in this PR is as follows:
* Moves the following functions from `salt.utils` to new locations
(including a deprecation warning if invoked from `salt.utils`):
`to_bytes`, `to_str`, `to_unicode`, `str_to_num`, `is_quoted`,
`dequote`, `is_hex`, `is_bin_str`, `rand_string`,
`contains_whitespace`, `clean_kwargs`, `invalid_kwargs`, `which`,
`which_bin`, `path_join`, `shlex_split`, `rand_str`, `is_windows`,
`is_proxy`, `is_linux`, `is_darwin`, `is_sunos`, `is_smartos`,
`is_smartos_globalzone`, `is_smartos_zone`, `is_freebsd`, `is_netbsd`,
`is_openbsd`, `is_aix`
* Moves the functions already deprecated by @rallytime to the bottom of
`salt/utils/__init__.py` for better organization, so we can keep the
deprecated ones separate from the ones yet to be deprecated as we
continue to break up `salt.utils`
* Updates `salt/*.py` and all files under `salt/client/` to use explicit
unicode string literals.
* Gets rid of implicit imports of `salt.utils` (e.g. `from salt.utils
import foo` becomes `import salt.utils.foo as foo`).
* Renames the `test.rand_str` function to `test.random_hash` to more
accurately reflect what it does
* Modifies `salt.utils.stringutils.random()` (née `salt.utils.rand_string()`)
such that it returns a string matching the passed size. Previously
this function would get `size` bytes from `os.urandom()`,
base64-encode it, and return the result, which would in most cases not
be equal to the passed size.
Headline changes:
- Move to boto3
- Use result-style error handling
- Optimize number of API calls:
- Set attributes at creation time
- Set all attributes in one API call
- Show diff when attribute changes detected
These test the scenarios where another network with a similar name
already exists, verifying that absent doesn't attempt to remove a
network which isn't specified, and present still attempts to create the
specified network despite a similarly named network already being present.
This started failing following commit 515c612, which relied on the
'Name' key being present in the return value of docker.networks -
as the mock didn't have this set the test started failing.
Separates out the comment from the rest of the state return to make this
test more stable on Python 3. This state run in PY3 adds the minion
names to the comment in a random order, so we can't rely on asserting
against this order in the test.
This PR updates the test so that we are still checking for all of the
elements in the comment, but not relying on the order the minions are
added the comment list.
These test the scenarios where another network with a similar name
already exists, verifying that network_absent doesn't attempt to remove
a network which isn't specified, and network_present still attempts to
create the specified network despite a similarly named network already
being present.
This started failing following commit 515c612, which relied on the
'Name' key being present in the return value of dockerng.networks -
as the mock didn't have this set the test started failing.
This ensures we do not introduce a regression which breaks
the usage of the `batch` option when using states in orchestration
runners.
Related to #40635 and #39169
