This PR is part of what will be an ongoing effort to use explicit
unicode strings in Salt. Because Python 3 does not suport Python 2's raw
unicode string syntax (i.e. `ur'\d+'`), we must use
`salt.utils.locales.sdecode()` to ensure that the raw string is unicode.
However, because of how `salt/utils/__init__.py` has evolved into the
hulking monstrosity it is today, this means importing a large module in
places where it is not needed, which could negatively impact
performance. For this reason, this PR also breaks out some of the
functions from `salt/utils/__init__.py` into new/existing modules under
`salt/utils/`. The long term goal will be that the modules within this
directory do not depend on importing `salt.utils`.
A summary of the changes in this PR is as follows:
* Moves the following functions from `salt.utils` to new locations
(including a deprecation warning if invoked from `salt.utils`):
`to_bytes`, `to_str`, `to_unicode`, `str_to_num`, `is_quoted`,
`dequote`, `is_hex`, `is_bin_str`, `rand_string`,
`contains_whitespace`, `clean_kwargs`, `invalid_kwargs`, `which`,
`which_bin`, `path_join`, `shlex_split`, `rand_str`, `is_windows`,
`is_proxy`, `is_linux`, `is_darwin`, `is_sunos`, `is_smartos`,
`is_smartos_globalzone`, `is_smartos_zone`, `is_freebsd`, `is_netbsd`,
`is_openbsd`, `is_aix`
* Moves the functions already deprecated by @rallytime to the bottom of
`salt/utils/__init__.py` for better organization, so we can keep the
deprecated ones separate from the ones yet to be deprecated as we
continue to break up `salt.utils`
* Updates `salt/*.py` and all files under `salt/client/` to use explicit
unicode string literals.
* Gets rid of implicit imports of `salt.utils` (e.g. `from salt.utils
import foo` becomes `import salt.utils.foo as foo`).
* Renames the `test.rand_str` function to `test.random_hash` to more
accurately reflect what it does
* Modifies `salt.utils.stringutils.random()` (née `salt.utils.rand_string()`)
such that it returns a string matching the passed size. Previously
this function would get `size` bytes from `os.urandom()`,
base64-encode it, and return the result, which would in most cases not
be equal to the passed size.
Client was only setting success to false if the function called raised
an exception. This commit changes it to verify the content of return
value with check_state_result
Client was only setting success to false if the function called raised
an exception. This commit changes it to verify the content of return
value with check_state_result
The test suite actually has a ``prod`` env, but this test only considers
the ``base`` env. If a test is run which requests a file (or just the
file/dir/symlink/... list) from the ``prod`` env, then this will result
in the ``prod`` env's file list caches being present, and they will be
removed when the ``fileserver.clear_file_list_cache`` runner is
executed, showing up in the return data and causing the test to fail.
This tweak to the test ensures that we will always have a file list
cache for ``prod`` present, and adjusts the necessary asserts in the
test to expect the ``prod`` env in the return data from the runner.
The tests that return files, symlinks, directories, and empty dirs were
all only testing that the type of the return data was the same as what
was expected. By not testing the content, we overlooked a corner case in
which backends passed into the fileserver as a Python list would not be
handled properly. This has since been fixed in PR #36244, but these
tests will help keep this sort of issue from regressing.
Using a salt.runner.Runner() instance to run this test has exposed a
problem with run_run_plus, where the fileclient doesn't behave as
expected. Since the raw return data is not needed, using run_run here
lets the test pass while we continue to work on run_run_plus.
The return data from run_run_plus previously returned the runner return
in the ``fun`` key. This has since been corrected. This commit replaces
these references to the ``fun`` key with the corrected ``return`` key.
The change in runner output behavior caused the output of test_envs to
change, causing some changes to need to be made in da163471 to get them
to pass. Now that runner output is fixed, the test was returning the
output it should have been in the first place. This commit changes that
test back so that it now expects a list as output.
Additionally, I've taken the opportunity to add additional test calls to
test the "backend" argument for those functions which now have it.
In addition to traditional file system path test discovery it is
useful to run a test by stating it's dotted python path. For example,
this will only run the cmdmod integration tests from the
module suite::
$ runtests.py -n tests.integration.modules.cmdmod
