This helper was designed to ensure that the paths returned were all
unicode types, but on a couple platforms having the path be a unicode
type causes a traceback deep in the stdlib within posixpath.
To work around this, this commit makes the helper a generator that
simply decodes each tuple returned from os.walk as we iterate.
This adds a unicode_literals import to the fileserver code, and also ensures
that we return unicode paths from the serialized file/env list caches
(which when deserialized are still encoded str types on PY2).
It also adds a test and modifies a few other existing tests to confirm
that unicode paths/envs are returned *as unicode* in the return data.
This also adds "fs" to all of the fileserver virtualnames which did not
already have them. This both A) normalizes the usage of the non-roots
backends, and B) makes it easier for the FileserverUpdate process to get
the default update interval for each backend from the opts.
Without allow_unicode=True, unicode characters are processed through the
str representer and on Python 2 are dumped as a Unicode code point (i.e.
a literal \u0414). This commit makes allow_unicode=True the default in
our salt.utils.yamlloader.safe_dump() helper. It also adds a new
salt.utils.yamlloader.dump() helper which wraps yaml.dump() and also
makes allow_unicode=True the default.
To make importing and using our custom yaml loader/dumper easier, a
convenience module called salt.utils.yaml has been added, which does a
wildcard import from both salt.utils.yamldumper and
salt.utils.yamlloader.
Refs to yaml.load/dump and yaml.safe_load/safe_dump have been updated to
salt.utils.yaml, to ensure that unicode is handled properly.
This reduces some of the overhead caused by many concurrent fileclient
requests from minions.
Additionally, initializing remotes has been folded into the GitBase
class' dunder init. Instantiating an instance and initializing its
remotes were initially split so that an object could simply be created
with the master opts so that the configuration was loaded and nothing
else, allowing for certain cases (like clearing the cache files) where
we didn't need to actually initialize the remotes. But this both A)
presents a problem when the instance being used is a singleton, as you
don't want to be re-initializing the remotes all the time, and B)
suppressing initialization can be (and is now being) done via a new
argument to the dunder init.
This PR is part of what will be an ongoing effort to use explicit
unicode strings in Salt. Because Python 3 does not suport Python 2's raw
unicode string syntax (i.e. `ur'\d+'`), we must use
`salt.utils.locales.sdecode()` to ensure that the raw string is unicode.
However, because of how `salt/utils/__init__.py` has evolved into the
hulking monstrosity it is today, this means importing a large module in
places where it is not needed, which could negatively impact
performance. For this reason, this PR also breaks out some of the
functions from `salt/utils/__init__.py` into new/existing modules under
`salt/utils/`. The long term goal will be that the modules within this
directory do not depend on importing `salt.utils`.
A summary of the changes in this PR is as follows:
* Moves the following functions from `salt.utils` to new locations
(including a deprecation warning if invoked from `salt.utils`):
`to_bytes`, `to_str`, `to_unicode`, `str_to_num`, `is_quoted`,
`dequote`, `is_hex`, `is_bin_str`, `rand_string`,
`contains_whitespace`, `clean_kwargs`, `invalid_kwargs`, `which`,
`which_bin`, `path_join`, `shlex_split`, `rand_str`, `is_windows`,
`is_proxy`, `is_linux`, `is_darwin`, `is_sunos`, `is_smartos`,
`is_smartos_globalzone`, `is_smartos_zone`, `is_freebsd`, `is_netbsd`,
`is_openbsd`, `is_aix`
* Moves the functions already deprecated by @rallytime to the bottom of
`salt/utils/__init__.py` for better organization, so we can keep the
deprecated ones separate from the ones yet to be deprecated as we
continue to break up `salt.utils`
* Updates `salt/*.py` and all files under `salt/client/` to use explicit
unicode string literals.
* Gets rid of implicit imports of `salt.utils` (e.g. `from salt.utils
import foo` becomes `import salt.utils.foo as foo`).
* Renames the `test.rand_str` function to `test.random_hash` to more
accurately reflect what it does
* Modifies `salt.utils.stringutils.random()` (née `salt.utils.rand_string()`)
such that it returns a string matching the passed size. Previously
this function would get `size` bytes from `os.urandom()`,
base64-encode it, and return the result, which would in most cases not
be equal to the passed size.
* Add config params for custom refspecs
* Add exceptions for errors encountered modifying git config
* Make the refspecs a configurable parameter
* Make refspecs a per-remote parameter
* Update master config template to include custom refspecs opts
* Add documentation for new config params
* Update GitFS walkthrough with a section on custom refspecs
* Remove dulwich support from salt.utils.gitfs
Dulwich still lacks important features, including (but not limited to)
the following:
- Lack of the necessary support for checking out a ref needed for
git_pillar/winrepo support
- No support in its config objects for multivar git config items, making
it impossible to detect when repos have multiple refspecs set for a
given git remote
Given this information, and the fact that it trails as a distant third
to Pygit2 and GitPython, Salt will cease to support Dulwich as a git
interface moving forward.
* Excise references to dulwich from documentation
* Add mention of custom refspecs to Nitrogen release notes
* Add gitfs_refspecs to mocked opts in gitfs integration tests
Also remove dulwich from unit tests
* Add information about opts argument not being intended for CLI use
Naively comparing sorted dict keys does not guarantee that maps are equal. We
must compare mtimes for filenames in case keys are the same to make sure there
isnt a modified file.
