Without allow_unicode=True, unicode characters are processed through the
str representer and on Python 2 are dumped as a Unicode code point (i.e.
a literal \u0414). This commit makes allow_unicode=True the default in
our salt.utils.yamlloader.safe_dump() helper. It also adds a new
salt.utils.yamlloader.dump() helper which wraps yaml.dump() and also
makes allow_unicode=True the default.
To make importing and using our custom yaml loader/dumper easier, a
convenience module called salt.utils.yaml has been added, which does a
wildcard import from both salt.utils.yamldumper and
salt.utils.yamlloader.
Refs to yaml.load/dump and yaml.safe_load/safe_dump have been updated to
salt.utils.yaml, to ensure that unicode is handled properly.
Remove while loop that was hanging when `salt` wasn't in the path
Add salt.util.path.safe_path function to check for unsafe paths
Pass root_dir to all calls to `verify_env`
Moves the `get_colors` and `get_color_theme` out of `salt.utils.py`
and into a new file named `salt.utils.color.py`.
This PR moves the original functions, adds a deprecation warning to
the old function paths, and updates any references to the functions
in Salt code to the new location. This includes moving a test as well.
This PR is part of what will be an ongoing effort to use explicit
unicode strings in Salt. Because Python 3 does not suport Python 2's raw
unicode string syntax (i.e. `ur'\d+'`), we must use
`salt.utils.locales.sdecode()` to ensure that the raw string is unicode.
However, because of how `salt/utils/__init__.py` has evolved into the
hulking monstrosity it is today, this means importing a large module in
places where it is not needed, which could negatively impact
performance. For this reason, this PR also breaks out some of the
functions from `salt/utils/__init__.py` into new/existing modules under
`salt/utils/`. The long term goal will be that the modules within this
directory do not depend on importing `salt.utils`.
A summary of the changes in this PR is as follows:
* Moves the following functions from `salt.utils` to new locations
(including a deprecation warning if invoked from `salt.utils`):
`to_bytes`, `to_str`, `to_unicode`, `str_to_num`, `is_quoted`,
`dequote`, `is_hex`, `is_bin_str`, `rand_string`,
`contains_whitespace`, `clean_kwargs`, `invalid_kwargs`, `which`,
`which_bin`, `path_join`, `shlex_split`, `rand_str`, `is_windows`,
`is_proxy`, `is_linux`, `is_darwin`, `is_sunos`, `is_smartos`,
`is_smartos_globalzone`, `is_smartos_zone`, `is_freebsd`, `is_netbsd`,
`is_openbsd`, `is_aix`
* Moves the functions already deprecated by @rallytime to the bottom of
`salt/utils/__init__.py` for better organization, so we can keep the
deprecated ones separate from the ones yet to be deprecated as we
continue to break up `salt.utils`
* Updates `salt/*.py` and all files under `salt/client/` to use explicit
unicode string literals.
* Gets rid of implicit imports of `salt.utils` (e.g. `from salt.utils
import foo` becomes `import salt.utils.foo as foo`).
* Renames the `test.rand_str` function to `test.random_hash` to more
accurately reflect what it does
* Modifies `salt.utils.stringutils.random()` (née `salt.utils.rand_string()`)
such that it returns a string matching the passed size. Previously
this function would get `size` bytes from `os.urandom()`,
base64-encode it, and return the result, which would in most cases not
be equal to the passed size.
This was left over from a merge-forward. The proxy minion testing
pieces could not be completely merged-forward due to large conflicts
in this file. Please see the comment in PR #41113 for more information.
This is no longer needed since the git_pillar configuration has been
removed from the suite's master config file now that git_pillar
integration tests have been added.
This uses a function in the runtests_helpers custom module to perform
all the logic, and only returns what failed the test. This saves us from
having to return the entire contents of sys.doc (as well as log all of
the function calls), and also removes the need to run sys.doc in batches
to get around the "max message size" issue.
Since we have recently changed the test suite to use new-style
git_pillar, GitPython or Pygit2 is a hard dep for the test suite.
Additionally, when starting up the daemons, if no IPv4 addresses can be
detected (which can happen on docker containers which tend to have
minimal installs) then the suite will time out trying to detect whether
or not the minion/sub-minion has connected, which while it does not
prove fatal for the test suite, it does make the suite take several
minutes to start up and begin running tests. This is because the test
suite invokes the manage.joined runner, which in turn invokes
salt.utils.network.ip_addrs() to get the system's IP addresses to match
against those which are connected. If it can't get the IP addresses,
then the manage.joined runner returns an empty list, and the test suite
believes that no minions have connected, and the function that
periodically runs manage.joined will eventually time out.
This commit adds messages to the console when no suitable gitfs provider
is installed, and when salt.utils.network.ip_addrs() returns an empty
list, to hopefully prompt the user to install the missing requisites.
