On PY2, when os.walk is invoked with a str as input, the paths in the
return data are all str types as well. This leaves undecoded unicode
data in those strings when files/dirs under the top dir that was passed
contain unicode characters in the filename.
>>> import os
>>> list(os.walk('temp'))
[('temp', [], ['\xd0\x94.txt', 'foo.txt'])]
>>> list(os.walk(u'temp'))
[(u'temp', [], [u'\u0414.txt', u'foo.txt'])]
The helper introduced here ensures that we always invoke os.walk with a
unicode top-level dir, so that we get unicode types in the return data.
This PR is part of what will be an ongoing effort to use explicit
unicode strings in Salt. Because Python 3 does not suport Python 2's raw
unicode string syntax (i.e. `ur'\d+'`), we must use
`salt.utils.locales.sdecode()` to ensure that the raw string is unicode.
However, because of how `salt/utils/__init__.py` has evolved into the
hulking monstrosity it is today, this means importing a large module in
places where it is not needed, which could negatively impact
performance. For this reason, this PR also breaks out some of the
functions from `salt/utils/__init__.py` into new/existing modules under
`salt/utils/`. The long term goal will be that the modules within this
directory do not depend on importing `salt.utils`.
A summary of the changes in this PR is as follows:
* Moves the following functions from `salt.utils` to new locations
(including a deprecation warning if invoked from `salt.utils`):
`to_bytes`, `to_str`, `to_unicode`, `str_to_num`, `is_quoted`,
`dequote`, `is_hex`, `is_bin_str`, `rand_string`,
`contains_whitespace`, `clean_kwargs`, `invalid_kwargs`, `which`,
`which_bin`, `path_join`, `shlex_split`, `rand_str`, `is_windows`,
`is_proxy`, `is_linux`, `is_darwin`, `is_sunos`, `is_smartos`,
`is_smartos_globalzone`, `is_smartos_zone`, `is_freebsd`, `is_netbsd`,
`is_openbsd`, `is_aix`
* Moves the functions already deprecated by @rallytime to the bottom of
`salt/utils/__init__.py` for better organization, so we can keep the
deprecated ones separate from the ones yet to be deprecated as we
continue to break up `salt.utils`
* Updates `salt/*.py` and all files under `salt/client/` to use explicit
unicode string literals.
* Gets rid of implicit imports of `salt.utils` (e.g. `from salt.utils
import foo` becomes `import salt.utils.foo as foo`).
* Renames the `test.rand_str` function to `test.random_hash` to more
accurately reflect what it does
* Modifies `salt.utils.stringutils.random()` (née `salt.utils.rand_string()`)
such that it returns a string matching the passed size. Previously
this function would get `size` bytes from `os.urandom()`,
base64-encode it, and return the result, which would in most cases not
be equal to the passed size.
Prevents zero length error on Python 2.6
Fixes Zypper unit test
Enhances pkg.list_downloaded information of a package
Listing all patches instead of security ones only
Adapting Zypper test to new list_downloaded output
Fixes zypper test error after backporting
Pylint fixes
Adding support for installing patches in Zypper module
Adding list_downloaded function to Zypper module
Adding list_downloaded function to Yum module
Adding new pkg.downloaded state
Adding documentation for pkg.downloaded state
Adding pkg.patched and pkg.patch_downloaded states
Check targets for advisory patches installation
Adds support for listing advisory patches with Zypper
Adds support for listing advisory patches with Yum
Improving function naming
Moving advisory ids checks into pkg_resource.parse_targets
Fixes _find_download_targets to call _preflight_check
Fixes parse_targets when advisory id is passed as name
Pylint fixes
Enables pkg.downloaded to verify packages after package manager call.
Adding missing kwargs parameters to pkg.install call
Adding versionadded flags
Some refactoring and minor fixes
Adding unit tests for Zypper execution module
Adding more unit tests for Zypper module
Pylint fix
