moto versions >= 1.0.0 have changed the way the mocked connections through
boto are handled with the @mock_ec2 decorator. They use the boto3 connection
method. However, since we are still using boto in many places, we need to use
the new @mock_ec2_deprecated decorator instead to handle the boto connection
functions for the unit tests.
Versions of moto < 1.0.0 are not Python 3 compatible, so salt-jenkins should
be installing newer versions of moto for those tests. Unfortunately, we cannot
install an older version of moto for Python2 that use the original @mock_ec2 call
and also import the @mock_ec2_deprecated function for newer versions of moto
simultaneously as the @mock_ec2_deprecated function doesn't exist in older
versions of moto.
This moves the following functions from salt.utils to salt.utils.versions:
- warn_until
- kwargs_warn_until
- compare_versions (as salt.utils.versions.compare)
- version_cmp
PyYAML works with and without bytestrings on PY3, and Python 3 can read
all of the test data as regular strings, so bytestrings are unnecessary
here.
This also adds use of textwrap.dedent to make the YAML easier to read,
so it can be indented away from the far left side of the line.
moto versions >= 1.0.0 have changed the way the mocked connections through
boto are handled with the @mock_ec2 decorator. They use the boto3 connection
method. However, since we are still using boto in many places, we need to use
the new @mock_ec2_deprecated decorator instead to handle the boto connection
functions for the unit tests.
Versions of moto < 1.0.0 are not Python 3 compatible, so salt-jenkins should
be installing newer versions of moto for those tests. Unfortunately, we cannot
install an older version of moto for Python2 that use the original @mock_ec2 call
and also import the @mock_ec2_deprecated function for newer versions of moto
simultaneously as the @mock_ec2_deprecated function doesn't exist in older
versions of moto.
This PR is part of what will be an ongoing effort to use explicit
unicode strings in Salt. Because Python 3 does not suport Python 2's raw
unicode string syntax (i.e. `ur'\d+'`), we must use
`salt.utils.locales.sdecode()` to ensure that the raw string is unicode.
However, because of how `salt/utils/__init__.py` has evolved into the
hulking monstrosity it is today, this means importing a large module in
places where it is not needed, which could negatively impact
performance. For this reason, this PR also breaks out some of the
functions from `salt/utils/__init__.py` into new/existing modules under
`salt/utils/`. The long term goal will be that the modules within this
directory do not depend on importing `salt.utils`.
A summary of the changes in this PR is as follows:
* Moves the following functions from `salt.utils` to new locations
(including a deprecation warning if invoked from `salt.utils`):
`to_bytes`, `to_str`, `to_unicode`, `str_to_num`, `is_quoted`,
`dequote`, `is_hex`, `is_bin_str`, `rand_string`,
`contains_whitespace`, `clean_kwargs`, `invalid_kwargs`, `which`,
`which_bin`, `path_join`, `shlex_split`, `rand_str`, `is_windows`,
`is_proxy`, `is_linux`, `is_darwin`, `is_sunos`, `is_smartos`,
`is_smartos_globalzone`, `is_smartos_zone`, `is_freebsd`, `is_netbsd`,
`is_openbsd`, `is_aix`
* Moves the functions already deprecated by @rallytime to the bottom of
`salt/utils/__init__.py` for better organization, so we can keep the
deprecated ones separate from the ones yet to be deprecated as we
continue to break up `salt.utils`
* Updates `salt/*.py` and all files under `salt/client/` to use explicit
unicode string literals.
* Gets rid of implicit imports of `salt.utils` (e.g. `from salt.utils
import foo` becomes `import salt.utils.foo as foo`).
* Renames the `test.rand_str` function to `test.random_hash` to more
accurately reflect what it does
* Modifies `salt.utils.stringutils.random()` (née `salt.utils.rand_string()`)
such that it returns a string matching the passed size. Previously
this function would get `size` bytes from `os.urandom()`,
base64-encode it, and return the result, which would in most cases not
be equal to the passed size.
Without this, we stacktrace because it does not appear that setting
'gather_facts' to False prevents the library from assuming the presence
of facts. I believe this to be an upstream bug with jnpr.
Because they have listed this as being a deprecated option in the future
this may re-break in the future.
Headline changes:
- Move to boto3
- Use result-style error handling
- Optimize number of API calls:
- Set attributes at creation time
- Set all attributes in one API call
- Show diff when attribute changes detected
