First, `shlex.split()` will raise an exception when passed a unicode
type with unicode characters in the string. This modifies our
`shlex.split()` helper to first convert the passed string to a `str`
type, and then return a decoded copy of the result of the split.
Second, this uses our `to_unicode` helper to more gracefully decode the
stdout and stderr from the command. Unit tests have been added to
confirm that the output is properly decoded, including instances where
decoding fails because the return from the command contains binary data.
The test_run unit tests end result when a command is not found. When
_run() is called without setting cwd, it will use the home directory as
working directory. When the home directory does not exist, the unit test
will fail:
Traceback (most recent call last):
File "tests/unit/modules/test_cmdmod.py", line 231, in test_run
ret = cmdmod._run('foo', use_vt=True).get('stderr')
File "salt/modules/cmdmod.py", line 536, in _run
.format(cwd)
salt.exceptions.CommandExecutionError: Specified cwd
'/sbuild-nonexistent' either not absolute or does not exist
Therefore set cwd to the current directory since the working directory
is not used in this test case.
This fixes one failing test of #45627.
Signed-off-by: Benjamin Drung <benjamin.drung@profitbricks.com>
This makes the 2.x usage invalid syntax and forces the use of print as a
function. This adds the import to the files which I've updated in the
last couple of days but forgot to add it.
This updates the file state and execution modules to use
unicode_literals. Since the serializers and the cmd module are touched
by the file state/exec module, those are also updated here, as well as
the cmd state module, for good measure.
Additionally, I found that salt.utils.data.decode_dict (and decode_list)
are misnamed for what they actually do. Since they *encode* the
contents, the functions should be named encode_dict and encode_list,
respectively. And we also should have counterparts which actually
decode, so I've added them. The compatibility functions from salt.utils
still use the old "decode" names to preserve backward compatibility, but
they now invoke the renamed "encode" functions in salt.utils.data. Note
that this means that the compatibility functions
salt.utils.decode_dict/list, and their cognates in salt.utils.data now
do different things, but since the move to salt.utils.data is also
happening in the Oxygen release this is as good a time as any to correct
this oversight.
I've updated the jinja filter docs to include information on the renamed
jinja filters, and also added a section on jinja filter renaming to the
Oxygen release notes. There was another filter that I renamed during the
process of moving functions from salt.utils which I did not properly
document in the release notes, so this is now included along with the
others.
We have `output_loglevel` for controlling logging, but no corresponding
option to suppress the output from the command from being returned. This
adds a `hide_output` option to do that.
Additionally, this removes the rest of the references to the
long-since-deprecated `quiet` argument from the cmd execution module,
and adds a warning to states which use this argument. `quiet` appears to
have been incompletely removed following its deprecation. It was not
being passed to _check_loglevel(), so it was essentially ignored, but it
was still being accepted as an argument by several functions. Those
references to what has for some time now been a no-op argument are now
removed.
Through merge forwards, some of the old-style paths have made their
way into develop. This PR corrects some of these instances on the more
popular functions.
This PR is part of what will be an ongoing effort to use explicit
unicode strings in Salt. Because Python 3 does not suport Python 2's raw
unicode string syntax (i.e. `ur'\d+'`), we must use
`salt.utils.locales.sdecode()` to ensure that the raw string is unicode.
However, because of how `salt/utils/__init__.py` has evolved into the
hulking monstrosity it is today, this means importing a large module in
places where it is not needed, which could negatively impact
performance. For this reason, this PR also breaks out some of the
functions from `salt/utils/__init__.py` into new/existing modules under
`salt/utils/`. The long term goal will be that the modules within this
directory do not depend on importing `salt.utils`.
A summary of the changes in this PR is as follows:
* Moves the following functions from `salt.utils` to new locations
(including a deprecation warning if invoked from `salt.utils`):
`to_bytes`, `to_str`, `to_unicode`, `str_to_num`, `is_quoted`,
`dequote`, `is_hex`, `is_bin_str`, `rand_string`,
`contains_whitespace`, `clean_kwargs`, `invalid_kwargs`, `which`,
`which_bin`, `path_join`, `shlex_split`, `rand_str`, `is_windows`,
`is_proxy`, `is_linux`, `is_darwin`, `is_sunos`, `is_smartos`,
`is_smartos_globalzone`, `is_smartos_zone`, `is_freebsd`, `is_netbsd`,
`is_openbsd`, `is_aix`
* Moves the functions already deprecated by @rallytime to the bottom of
`salt/utils/__init__.py` for better organization, so we can keep the
deprecated ones separate from the ones yet to be deprecated as we
continue to break up `salt.utils`
* Updates `salt/*.py` and all files under `salt/client/` to use explicit
unicode string literals.
* Gets rid of implicit imports of `salt.utils` (e.g. `from salt.utils
import foo` becomes `import salt.utils.foo as foo`).
* Renames the `test.rand_str` function to `test.random_hash` to more
accurately reflect what it does
* Modifies `salt.utils.stringutils.random()` (née `salt.utils.rand_string()`)
such that it returns a string matching the passed size. Previously
this function would get `size` bytes from `os.urandom()`,
base64-encode it, and return the result, which would in most cases not
be equal to the passed size.
