pepa.key_value_to_tree only checks the name of the key when attempting
to identify the last key in a flattened key. This can lead to strange
behavior when the final key name also occurs earlier in the flatkey.
This change explicitly checks that we are the final key in flatkey
prior to setting value. Test included.
Also see mickep76/pepa#11 and mickep76/pepa#12
A stacktrace occurs when hitting the pillar line added in #47082 if
`expanded_classes[klass].get('pillars', {})` is executed on an empty
unicode type. This avoids the stacktrace and only tries the dict_merge
if there new pillars are present.
Due to the many merge conflicts created from #47106 against the 2017.7 branch
and #46002 against the 2018.3 branch, the changes from #47106 have been largely
removed from this merge forward and the HEAD of 2018.3 was taken.
A separate fix for Tornado 5.0 support will need to be made directly against the
2018.3 branch.
Conflicts:
- doc/topics/development/conventions/formulas.rst
- salt/master.py
- salt/minion.py
- salt/netapi/rest_tornado/saltnado.py
- salt/states/zfs.py
- salt/transport/ipc.py
- salt/transport/tcp.py
- salt/transport/zeromq.py
- salt/utils/async.py
- tests/support/helpers.py
- tests/support/parser/cover.py
- tests/unit/grains/test_core.py
- tests/unit/modules/test_ssh.py
- tests/unit/test_minion.py
- tests/unit/utils/test_safe_walk.py
Without allow_unicode=True, unicode characters are processed through the
str representer and on Python 2 are dumped as a Unicode code point (i.e.
a literal \u0414). This commit makes allow_unicode=True the default in
our salt.utils.yamlloader.safe_dump() helper. It also adds a new
salt.utils.yamlloader.dump() helper which wraps yaml.dump() and also
makes allow_unicode=True the default.
To make importing and using our custom yaml loader/dumper easier, a
convenience module called salt.utils.yaml has been added, which does a
wildcard import from both salt.utils.yamldumper and
salt.utils.yamlloader.
Refs to yaml.load/dump and yaml.safe_load/safe_dump have been updated to
salt.utils.yaml, to ensure that unicode is handled properly.
This modifies this ext_pillar with basic support for environments
by treating a relative root_dir as relative to the pillar_roots
of the environment specified by pillarenv.
The behavior for absolute paths is unchanged.
Also adds a unit test for file_tree testing both old and new
behavior.
Resolves#44458.
Example:
``` yaml
pillars:
default:
commands:
{% if grains['os'] == 'RedHat' %}
pkg_grains: /usr/bin/rpm_grains
{% endif %}
```
In this case `pkg_grains` wouldn't be found on anything else that RedHat systems and would be NoneType which would return an error and break the module.
These tests were testing things present in the legacy git pillar syntax
and are no longer needed. The new git pillar syntax has many more integration
tests to cover various cases.
This PR is part of what will be an ongoing effort to use explicit
unicode strings in Salt. Because Python 3 does not suport Python 2's raw
unicode string syntax (i.e. `ur'\d+'`), we must use
`salt.utils.locales.sdecode()` to ensure that the raw string is unicode.
However, because of how `salt/utils/__init__.py` has evolved into the
hulking monstrosity it is today, this means importing a large module in
places where it is not needed, which could negatively impact
performance. For this reason, this PR also breaks out some of the
functions from `salt/utils/__init__.py` into new/existing modules under
`salt/utils/`. The long term goal will be that the modules within this
directory do not depend on importing `salt.utils`.
A summary of the changes in this PR is as follows:
* Moves the following functions from `salt.utils` to new locations
(including a deprecation warning if invoked from `salt.utils`):
`to_bytes`, `to_str`, `to_unicode`, `str_to_num`, `is_quoted`,
`dequote`, `is_hex`, `is_bin_str`, `rand_string`,
`contains_whitespace`, `clean_kwargs`, `invalid_kwargs`, `which`,
`which_bin`, `path_join`, `shlex_split`, `rand_str`, `is_windows`,
`is_proxy`, `is_linux`, `is_darwin`, `is_sunos`, `is_smartos`,
`is_smartos_globalzone`, `is_smartos_zone`, `is_freebsd`, `is_netbsd`,
`is_openbsd`, `is_aix`
* Moves the functions already deprecated by @rallytime to the bottom of
`salt/utils/__init__.py` for better organization, so we can keep the
deprecated ones separate from the ones yet to be deprecated as we
continue to break up `salt.utils`
* Updates `salt/*.py` and all files under `salt/client/` to use explicit
unicode string literals.
* Gets rid of implicit imports of `salt.utils` (e.g. `from salt.utils
import foo` becomes `import salt.utils.foo as foo`).
* Renames the `test.rand_str` function to `test.random_hash` to more
accurately reflect what it does
* Modifies `salt.utils.stringutils.random()` (née `salt.utils.rand_string()`)
such that it returns a string matching the passed size. Previously
this function would get `size` bytes from `os.urandom()`,
base64-encode it, and return the result, which would in most cases not
be equal to the passed size.
