Without allow_unicode=True, unicode characters are processed through the
str representer and on Python 2 are dumped as a Unicode code point (i.e.
a literal \u0414). This commit makes allow_unicode=True the default in
our salt.utils.yamlloader.safe_dump() helper. It also adds a new
salt.utils.yamlloader.dump() helper which wraps yaml.dump() and also
makes allow_unicode=True the default.
To make importing and using our custom yaml loader/dumper easier, a
convenience module called salt.utils.yaml has been added, which does a
wildcard import from both salt.utils.yamldumper and
salt.utils.yamlloader.
Refs to yaml.load/dump and yaml.safe_load/safe_dump have been updated to
salt.utils.yaml, to ensure that unicode is handled properly.
This makes the 2.x usage invalid syntax and forces the use of print as a
function. This adds the import to the files which I've updated in the
last couple of days but forgot to add it.
the field separator on OpenBSD and NetBSD is '=', not ':'.
verified on OpenBSD, assuming this now works on NetBSD based on
upstream documentation for sysctl(8)
This updates the file state and execution modules to use
unicode_literals. Since the serializers and the cmd module are touched
by the file state/exec module, those are also updated here, as well as
the cmd state module, for good measure.
Additionally, I found that salt.utils.data.decode_dict (and decode_list)
are misnamed for what they actually do. Since they *encode* the
contents, the functions should be named encode_dict and encode_list,
respectively. And we also should have counterparts which actually
decode, so I've added them. The compatibility functions from salt.utils
still use the old "decode" names to preserve backward compatibility, but
they now invoke the renamed "encode" functions in salt.utils.data. Note
that this means that the compatibility functions
salt.utils.decode_dict/list, and their cognates in salt.utils.data now
do different things, but since the move to salt.utils.data is also
happening in the Oxygen release this is as good a time as any to correct
this oversight.
I've updated the jinja filter docs to include information on the renamed
jinja filters, and also added a section on jinja filter renaming to the
Oxygen release notes. There was another filter that I renamed during the
process of moving functions from salt.utils which I did not properly
document in the release notes, so this is now included along with the
others.
On PY2, when os.walk is invoked with a str as input, the paths in the
return data are all str types as well. This leaves undecoded unicode
data in those strings when files/dirs under the top dir that was passed
contain unicode characters in the filename.
>>> import os
>>> list(os.walk('temp'))
[('temp', [], ['\xd0\x94.txt', 'foo.txt'])]
>>> list(os.walk(u'temp'))
[(u'temp', [], [u'\u0414.txt', u'foo.txt'])]
The helper introduced here ensures that we always invoke os.walk with a
unicode top-level dir, so that we get unicode types in the return data.
