This fixes the following when running `python tests/runtests.py --ext-pillar`:
-------- Tests with Errors ------------------------------------------------------------------------------------------------------------------------
-> tearDownClass (integration.pillar.test_git_pillar.TestGitPythonSSH) ...........................................................................
Traceback (most recent call last):
File ".../salt/tests/support/gitfs.py", line 481, in tearDownClass
if cls.case.sshd_proc is not None:
AttributeError: 'NoneType' object has no attribute 'sshd_proc'
...................................................................................................................................................
-> tearDownClass (integration.pillar.test_git_pillar.TestPygit2SSH) ..............................................................................
Traceback (most recent call last):
File ".../salt/tests/support/gitfs.py", line 481, in tearDownClass
if cls.case.sshd_proc is not None:
AttributeError: 'NoneType' object has no attribute 'sshd_proc'
...................................................................................................................................................
-----------------------------------------------------------------------------------------------------------------------------------------------------
Signed-off-by: Mathieu Parent <math.parent@gmail.com>
(cherry picked from commit 84e9fce9bb)
Without allow_unicode=True, unicode characters are processed through the
str representer and on Python 2 are dumped as a Unicode code point (i.e.
a literal \u0414). This commit makes allow_unicode=True the default in
our salt.utils.yamlloader.safe_dump() helper. It also adds a new
salt.utils.yamlloader.dump() helper which wraps yaml.dump() and also
makes allow_unicode=True the default.
To make importing and using our custom yaml loader/dumper easier, a
convenience module called salt.utils.yaml has been added, which does a
wildcard import from both salt.utils.yamldumper and
salt.utils.yamlloader.
Refs to yaml.load/dump and yaml.safe_load/safe_dump have been updated to
salt.utils.yaml, to ensure that unicode is handled properly.
These didn't get caught in PR 42823 because of how we invoke the
git_pillar code. Firstly, the "pillar" argument needed to stay. This is
because even though we're not using it, _external_pillar_data() is still
passing it now that git_pillar is not specially invoked there.
Secondly, since the input comes in as a list, and _external_pillar_data
uses single-asterisk expansion, the repos are passed separately when
they should be passed as a single list. To fix these issues, I've done
the following:
1. Re-introduced the "pillar" argument in git_pillar's ext_pillar
function.
2. Changed the "pillar" variable to avoid confusion with the (unused)
"pillar" argument being passed in.
3. Instead of git_pillar accepting the repos as a list, the ext_pillar
function now uses single-asterisk expansion to make it conform with
how _external_pillar_data() invokes it.
This PR is part of what will be an ongoing effort to use explicit
unicode strings in Salt. Because Python 3 does not suport Python 2's raw
unicode string syntax (i.e. `ur'\d+'`), we must use
`salt.utils.locales.sdecode()` to ensure that the raw string is unicode.
However, because of how `salt/utils/__init__.py` has evolved into the
hulking monstrosity it is today, this means importing a large module in
places where it is not needed, which could negatively impact
performance. For this reason, this PR also breaks out some of the
functions from `salt/utils/__init__.py` into new/existing modules under
`salt/utils/`. The long term goal will be that the modules within this
directory do not depend on importing `salt.utils`.
A summary of the changes in this PR is as follows:
* Moves the following functions from `salt.utils` to new locations
(including a deprecation warning if invoked from `salt.utils`):
`to_bytes`, `to_str`, `to_unicode`, `str_to_num`, `is_quoted`,
`dequote`, `is_hex`, `is_bin_str`, `rand_string`,
`contains_whitespace`, `clean_kwargs`, `invalid_kwargs`, `which`,
`which_bin`, `path_join`, `shlex_split`, `rand_str`, `is_windows`,
`is_proxy`, `is_linux`, `is_darwin`, `is_sunos`, `is_smartos`,
`is_smartos_globalzone`, `is_smartos_zone`, `is_freebsd`, `is_netbsd`,
`is_openbsd`, `is_aix`
* Moves the functions already deprecated by @rallytime to the bottom of
`salt/utils/__init__.py` for better organization, so we can keep the
deprecated ones separate from the ones yet to be deprecated as we
continue to break up `salt.utils`
* Updates `salt/*.py` and all files under `salt/client/` to use explicit
unicode string literals.
* Gets rid of implicit imports of `salt.utils` (e.g. `from salt.utils
import foo` becomes `import salt.utils.foo as foo`).
* Renames the `test.rand_str` function to `test.random_hash` to more
accurately reflect what it does
* Modifies `salt.utils.stringutils.random()` (née `salt.utils.rand_string()`)
such that it returns a string matching the passed size. Previously
this function would get `size` bytes from `os.urandom()`,
base64-encode it, and return the result, which would in most cases not
be equal to the passed size.
The location of the git-http-backend is within /usr/lib instead of
/usr/libexec.
This also uncomments a line that was commented out for troubleshooting
purposes.
For some reason these were not failing when I opened #40777, but now
that the PR is merged are failing for PR builds (as well as locally in
my test env). This fixes those failures.
