First, `shlex.split()` will raise an exception when passed a unicode
type with unicode characters in the string. This modifies our
`shlex.split()` helper to first convert the passed string to a `str`
type, and then return a decoded copy of the result of the split.
Second, this uses our `to_unicode` helper to more gracefully decode the
stdout and stderr from the command. Unit tests have been added to
confirm that the output is properly decoded, including instances where
decoding fails because the return from the command contains binary data.
This makes some further tweaks to make the test more stable under heavy
load. Firstly, the background job sleeps longer, and secondly, we make
up to 3 attempts run state.running instead of just the one, in case our
first attempt was too early. It also uses threading to make the job
sleep, since the method of adding a & to the command seemed to be
producing intermittent failures with returning clean JSON.
This makes some further tweaks to make the test more stable under heavy
load. Firstly, the background job sleeps longer, and secondly, we make
up to 3 attempts run state.running instead of just the one, in case our
first attempt was too early. It also uses threading to make the job
sleep, since the method of adding a & to the command seemed to be
producing intermittent failures with returning clean JSON.
This adds a unicode_literals import to the fileserver code, and also ensures
that we return unicode paths from the serialized file/env list caches
(which when deserialized are still encoded str types on PY2).
It also adds a test and modifies a few other existing tests to confirm
that unicode paths/envs are returned *as unicode* in the return data.
test_exclude shares file paths with test_include, and while I can't
reproduce the failures, it is likely that improperly cleaned-up files
from test_include are causing the failures in test_exclude. This is
backed up by the fact that the state.sls return data from the
salt-runtests.log shows no trace of the "to-include-test" file
(suggesting it was excluded as expected), yet os.path.isfile() returns
True for this path, causing the test to fail.
This commit uses a distinct base dir for both tests, which should keep
this sort of failure from happening.
- Fix problem with invalid characters in requisites.prereq_simple2
- Fix problem with true/false commands in Windows. Need to use exit
- Fix some issues with hard-coded paths to /tmp
test_exclude shares file paths with test_include, and while I can't
reproduce the failures, it is likely that improperly cleaned-up files
from test_include are causing the failures in test_exclude. This is
backed up by the fact that the state.sls return data from the
salt-runtests.log shows no trace of the "to-include-test" file
(suggesting it was excluded as expected), yet os.path.isfile() returns
True for this path, causing the test to fail.
This commit uses a distinct base dir for both tests, which should keep
this sort of failure from happening.
Carbon 1.1.1 for some reason added six to their setup.py, which breaks
this test since it's not installed into the virtualenv. This PR forces
this test to use a version of carbon which does not dep on six in its
setup.py.
Carbon 1.1.1 for some reason added six to their setup.py, which breaks
this test since it's not installed into the virtualenv. This PR forces
this test to use a version of carbon which does not dep on six in its
setup.py.
This makes the 2.x usage invalid syntax and forces the use of print as a
function. This adds the import to the files which I've updated in the
last couple of days but forgot to add it.
Example:
``` yaml
pillars:
default:
commands:
{% if grains['os'] == 'RedHat' %}
pkg_grains: /usr/bin/rpm_grains
{% endif %}
```
In this case `pkg_grains` wouldn't be found on anything else that RedHat systems and would be NoneType which would return an error and break the module.
This PR is part of what will be an ongoing effort to use explicit
unicode strings in Salt. Because Python 3 does not suport Python 2's raw
unicode string syntax (i.e. `ur'\d+'`), we must use
`salt.utils.locales.sdecode()` to ensure that the raw string is unicode.
However, because of how `salt/utils/__init__.py` has evolved into the
hulking monstrosity it is today, this means importing a large module in
places where it is not needed, which could negatively impact
performance. For this reason, this PR also breaks out some of the
functions from `salt/utils/__init__.py` into new/existing modules under
`salt/utils/`. The long term goal will be that the modules within this
directory do not depend on importing `salt.utils`.
A summary of the changes in this PR is as follows:
* Moves the following functions from `salt.utils` to new locations
(including a deprecation warning if invoked from `salt.utils`):
`to_bytes`, `to_str`, `to_unicode`, `str_to_num`, `is_quoted`,
`dequote`, `is_hex`, `is_bin_str`, `rand_string`,
`contains_whitespace`, `clean_kwargs`, `invalid_kwargs`, `which`,
`which_bin`, `path_join`, `shlex_split`, `rand_str`, `is_windows`,
`is_proxy`, `is_linux`, `is_darwin`, `is_sunos`, `is_smartos`,
`is_smartos_globalzone`, `is_smartos_zone`, `is_freebsd`, `is_netbsd`,
`is_openbsd`, `is_aix`
* Moves the functions already deprecated by @rallytime to the bottom of
`salt/utils/__init__.py` for better organization, so we can keep the
deprecated ones separate from the ones yet to be deprecated as we
continue to break up `salt.utils`
* Updates `salt/*.py` and all files under `salt/client/` to use explicit
unicode string literals.
* Gets rid of implicit imports of `salt.utils` (e.g. `from salt.utils
import foo` becomes `import salt.utils.foo as foo`).
* Renames the `test.rand_str` function to `test.random_hash` to more
accurately reflect what it does
* Modifies `salt.utils.stringutils.random()` (née `salt.utils.rand_string()`)
such that it returns a string matching the passed size. Previously
this function would get `size` bytes from `os.urandom()`,
base64-encode it, and return the result, which would in most cases not
be equal to the passed size.
