Without allow_unicode=True, unicode characters are processed through the
str representer and on Python 2 are dumped as a Unicode code point (i.e.
a literal \u0414). This commit makes allow_unicode=True the default in
our salt.utils.yamlloader.safe_dump() helper. It also adds a new
salt.utils.yamlloader.dump() helper which wraps yaml.dump() and also
makes allow_unicode=True the default.
To make importing and using our custom yaml loader/dumper easier, a
convenience module called salt.utils.yaml has been added, which does a
wildcard import from both salt.utils.yamldumper and
salt.utils.yamlloader.
Refs to yaml.load/dump and yaml.safe_load/safe_dump have been updated to
salt.utils.yaml, to ensure that unicode is handled properly.
This PR is part of what will be an ongoing effort to use explicit
unicode strings in Salt. Because Python 3 does not suport Python 2's raw
unicode string syntax (i.e. `ur'\d+'`), we must use
`salt.utils.locales.sdecode()` to ensure that the raw string is unicode.
However, because of how `salt/utils/__init__.py` has evolved into the
hulking monstrosity it is today, this means importing a large module in
places where it is not needed, which could negatively impact
performance. For this reason, this PR also breaks out some of the
functions from `salt/utils/__init__.py` into new/existing modules under
`salt/utils/`. The long term goal will be that the modules within this
directory do not depend on importing `salt.utils`.
A summary of the changes in this PR is as follows:
* Moves the following functions from `salt.utils` to new locations
(including a deprecation warning if invoked from `salt.utils`):
`to_bytes`, `to_str`, `to_unicode`, `str_to_num`, `is_quoted`,
`dequote`, `is_hex`, `is_bin_str`, `rand_string`,
`contains_whitespace`, `clean_kwargs`, `invalid_kwargs`, `which`,
`which_bin`, `path_join`, `shlex_split`, `rand_str`, `is_windows`,
`is_proxy`, `is_linux`, `is_darwin`, `is_sunos`, `is_smartos`,
`is_smartos_globalzone`, `is_smartos_zone`, `is_freebsd`, `is_netbsd`,
`is_openbsd`, `is_aix`
* Moves the functions already deprecated by @rallytime to the bottom of
`salt/utils/__init__.py` for better organization, so we can keep the
deprecated ones separate from the ones yet to be deprecated as we
continue to break up `salt.utils`
* Updates `salt/*.py` and all files under `salt/client/` to use explicit
unicode string literals.
* Gets rid of implicit imports of `salt.utils` (e.g. `from salt.utils
import foo` becomes `import salt.utils.foo as foo`).
* Renames the `test.rand_str` function to `test.random_hash` to more
accurately reflect what it does
* Modifies `salt.utils.stringutils.random()` (née `salt.utils.rand_string()`)
such that it returns a string matching the passed size. Previously
this function would get `size` bytes from `os.urandom()`,
base64-encode it, and return the result, which would in most cases not
be equal to the passed size.
This is actual for gnupg 2.1 that requires the agent and starts it
automatically. But since each test creates a new keystore tests become
fail after the first one starts gpg-agent.
* Typo in comment
* First convert to string if not already a string. Then to bytes under Py3.
The reason being that jids from the CLI, at least the one fed in
integration.runners.jobs.ManageTest.test_loopup_jid is loaded as an
integer, and, while the Py2 code converts JIDs to strings, under Py3, we
assume JID's are already strings.
* Mark tests which require root permissions to run
* Allow declaring that the function IS a class method.
```
Python 3.5.3 (default, Jan 21 2017, 00:29:12)
[GCC 6.3.1 20170109] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> class Foo:
... def bar(self):
... print('bar')
...
>>> import inspect
>>> inspect.ismethod(Foo.bar)
False
>>> inspect.ismethod(Foo().bar)
True
```
On Python 2, `inspect.ismethod` returns `True` for bound and unbound
methods while on Python 3 it only returns `True` for bound methods.
The explicit `is_class_method` is to avoid instantiating the class just
to get the function signature.
* Always decode responses to the Python version native string implementation
* Just compare the objects as matching list.
Asserting same item count doesn't make that much sense.
* Py3 compatibility
* Fix saltnado tests under Py3
* Python 3 compatibility
* Show me the full traceback
* Revert "Convert fileserver data from bytes to strings"
This reverts commit e53972f8c6.
* Revert "Under Py3, we get `bytes` when using the roots backend directly"
This reverts commit 9f73b240c1.
* Convert from bytes to str if not a binary file
* Py3 compatibility fixes.
Convert file contents from bytes to string if not a binary file
