When calling an incorrectly formatted wheel or runner function, we should
be raising a SaltInvocationError with a hint to check function syntax rather
that raising an Eauth authentication error.
This PR does several things:
- Adds a dictionary error return when the function syntax passed through to
`utils.minions.CkMinions.spec_check` does not match the expected `module.function`
syntax
- Handles the return of this new dictionary error (instead of previous `False` return)
wherever the spec_check function is called. This is handled up the stack in
`master.py` and `masterapi.py`.
- Reworks the runner and wheel functions in `master.py` and `masterapi.py` to
help make those functions more DRY (see `salt.auth.check_authentication` function).
- Adds tests for all of these changes (written before the runner and wheel functions
were moved to use the new salt.auth.check_authentication function) to help prevent
regressions.
- Fixes a couple of places where unit tests exposed potential stacktraces.
- Adjusts one previous unit test concerning the dictionary error change from spec_check
Some states are complicated and multiple subparts,
or maybe cross-call into __states__ if they manage subresources.
In these cases, they will have multiple comments.
Make this more ergonomic by supporting a list of strings as the
value for ret['comment'] in state returns and documenting this.
By joining comments on newlines, it is possible to combine
single-line and multi-line comments cleanly, as opposed to e.g. commas.
The driving impetus for this is some of the boto modules.
An update to the boto_sqs module is included as an example.
Add a check that outgoing state return data has the right shape,
and add a testcase as well.
Fix the NPM state tests and the saltmod runner & wheel state functions
to comply with the prescribed format.
Moves the following functions from `salt.utils` to `salt.utils.state`:
- gen_state_tag
- search_onfail_requisites
- check_onfail_requisites
- check_state_result
Original functions have a deprecation warning added and all references
in Salt code were updated to use the new function path, including moving
some unit tests.
Uses os.sep instead of unix-style paths in the test
Uses salt.utils.path.islink() to detect symlinks instead of
os.path.islink(). os.path.islink() does not correctly detect symlinks in
Windows
Put grp and pwd imports inside a try/except block
Moves the `get_colors` and `get_color_theme` out of `salt.utils.py`
and into a new file named `salt.utils.color.py`.
This PR moves the original functions, adds a deprecation warning to
the old function paths, and updates any references to the functions
in Salt code to the new location. This includes moving a test as well.
Through merge forwards, some of the old-style paths have made their
way into develop. This PR corrects some of these instances on the more
popular functions.
This makes the following changes:
1. Renames the valid_providers param in GitBase to git_providers,
allowing for a dictionary mapping provider names to their associated
classes. This allows for alternate providers to be used with a
GitBase subclass.
2. Renames the get_provider function to verify_provider to reduce
confusion with git_providers.
3. Uses super() to run a parent class' dunder init instead of invoking
the parent class directly.
Removal of these deprecation warnings has no behavioral effect, as
support for "env" instead of "saltenv" in these cases has already
been removed. Only the warning text will no longer display to users.
On some platforms, for reason which I do not yet grok, images pulled
from the Hub are prefixed with "docker.io/". This causes the
docker_image states to fail unless the user manually adds "docker.io/"
before the image name.
This commit adds a new function called "docker.resolve_tag" which
disambiguates this variance and allows images to be specified without
the "docker.io/" prefix.
Resolves#42935.
These tests were testing things present in the legacy git pillar syntax
and are no longer needed. The new git pillar syntax has many more integration
tests to cover various cases.
This fixes an issue where inspecting the container returns an image ID
instead of an image name, resulting in a spurious report of a changed
image. By resolving the image down to its ID for both the existing and
new containers, we ensure we're comparing ID to ID.
This test wasn't really written with Windows in mind. Uses PATHEXT that
actually resembles a Windows environment. The test value has the correct
path seperator for Windows.
Re-factoring a little following the previous commit, so that the same
container connection logic can be used in both places, rather than it
being repeated.
Previously any changes to network states wouldn't result in any change
to the network when the state was applied. This was less of an issue
the only attributes you could set for a network were its name and
driver, but with the recent support for driver options and now IPAM
config being added, it's useful for the state to be able to modify the
network if it needs to.
The Docker API doesn't provide any facility to modify an existing
network, so the only option is to recreate it with the new options,
which means disconnecting and reconnecting all containers.
Fixes#43047
moto versions >= 1.0.0 have changed the way the mocked connections through
boto are handled with the @mock_ec2 decorator. They use the boto3 connection
method. However, since we are still using boto in many places, we need to use
the new @mock_ec2_deprecated decorator instead to handle the boto connection
functions for the unit tests.
Versions of moto < 1.0.0 are not Python 3 compatible, so salt-jenkins should
be installing newer versions of moto for those tests. Unfortunately, we cannot
install an older version of moto for Python2 that use the original @mock_ec2 call
and also import the @mock_ec2_deprecated function for newer versions of moto
simultaneously as the @mock_ec2_deprecated function doesn't exist in older
versions of moto.
This moves the following functions from salt.utils to salt.utils.versions:
- warn_until
- kwargs_warn_until
- compare_versions (as salt.utils.versions.compare)
- version_cmp
PyYAML works with and without bytestrings on PY3, and Python 3 can read
all of the test data as regular strings, so bytestrings are unnecessary
here.
This also adds use of textwrap.dedent to make the YAML easier to read,
so it can be indented away from the far left side of the line.
moto versions >= 1.0.0 have changed the way the mocked connections through
boto are handled with the @mock_ec2 decorator. They use the boto3 connection
method. However, since we are still using boto in many places, we need to use
the new @mock_ec2_deprecated decorator instead to handle the boto connection
functions for the unit tests.
Versions of moto < 1.0.0 are not Python 3 compatible, so salt-jenkins should
be installing newer versions of moto for those tests. Unfortunately, we cannot
install an older version of moto for Python2 that use the original @mock_ec2 call
and also import the @mock_ec2_deprecated function for newer versions of moto
simultaneously as the @mock_ec2_deprecated function doesn't exist in older
versions of moto.
This PR is part of what will be an ongoing effort to use explicit
unicode strings in Salt. Because Python 3 does not suport Python 2's raw
unicode string syntax (i.e. `ur'\d+'`), we must use
`salt.utils.locales.sdecode()` to ensure that the raw string is unicode.
However, because of how `salt/utils/__init__.py` has evolved into the
hulking monstrosity it is today, this means importing a large module in
places where it is not needed, which could negatively impact
performance. For this reason, this PR also breaks out some of the
functions from `salt/utils/__init__.py` into new/existing modules under
`salt/utils/`. The long term goal will be that the modules within this
directory do not depend on importing `salt.utils`.
A summary of the changes in this PR is as follows:
* Moves the following functions from `salt.utils` to new locations
(including a deprecation warning if invoked from `salt.utils`):
`to_bytes`, `to_str`, `to_unicode`, `str_to_num`, `is_quoted`,
`dequote`, `is_hex`, `is_bin_str`, `rand_string`,
`contains_whitespace`, `clean_kwargs`, `invalid_kwargs`, `which`,
`which_bin`, `path_join`, `shlex_split`, `rand_str`, `is_windows`,
`is_proxy`, `is_linux`, `is_darwin`, `is_sunos`, `is_smartos`,
`is_smartos_globalzone`, `is_smartos_zone`, `is_freebsd`, `is_netbsd`,
`is_openbsd`, `is_aix`
* Moves the functions already deprecated by @rallytime to the bottom of
`salt/utils/__init__.py` for better organization, so we can keep the
deprecated ones separate from the ones yet to be deprecated as we
continue to break up `salt.utils`
* Updates `salt/*.py` and all files under `salt/client/` to use explicit
unicode string literals.
* Gets rid of implicit imports of `salt.utils` (e.g. `from salt.utils
import foo` becomes `import salt.utils.foo as foo`).
* Renames the `test.rand_str` function to `test.random_hash` to more
accurately reflect what it does
* Modifies `salt.utils.stringutils.random()` (née `salt.utils.rand_string()`)
such that it returns a string matching the passed size. Previously
this function would get `size` bytes from `os.urandom()`,
base64-encode it, and return the result, which would in most cases not
be equal to the passed size.
