To explain this change, it's necessary to first understand the history
of `object_hook` usage in Salt.
Initially, the function used as the object hook wherever we used it was
(incorrectly) named `encode_dict`, despite the fact that the code was
*decoding* the contents of the dictionary to unicode. This function was
renamed to `decode_dict`, and a counterpart which *actually* encodes was
put in its place. At this time *most* of the `object_hook` usage was
changed so that `decode_dict` was used, but some references were left
using `encode_dict` (likely because I wasn't sure whether or not
changing them would break things).
However, if the desire is to have unicode values after loading JSON,
there is actually no need to use an object hook, since loading JSON in
Python always produces unicode results even with a POSIX locale!
[root@2719cd6224d3 /]# locale
LANG=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=
[root@2719cd6224d3 /]# python
Python 2.7.5 (default, Nov 6 2016, 00:28:07)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import json
>>> json.loads('{"foo": "bar"}')
{u'foo': u'bar'}
>>> json.loads('{"foo": "Д"}')
{u'foo': u'\u0414'}
>>>
Therefore, this commit removes all `object_hook` usage where
`decode_dict` was the hook being used, since it is redundant. It also
removes one use of `encode_dict` as an object hook that was still left
in code used by salt-thin.
The only `object_hook` left is in some of our ioflo code. I left the
`encode_dict` there, but _only for Python 2_, because if I remember
correctly ioflo needs str types, and encoding a string in Python 2
produces a str type.
I used the incorrect version in my upstream bug workaround in #45892,
I missed that there was a 0.26.3 released before the bugfix was made.
This corrects that, and also fixes an incomplete comment that trailed
off in the middle of the sentence.
tornado needs to be >= 4.2.1, but less that 5.0.
Tornado 5.0 is introducing backwards-incompatible changes. Therefore,
we need to pin the version of tornado in base.txt until we can fix
supporting Tornado 5.0 in Salt.
Refs #45790
This allows us to expose the needed functionality to Windows in the
meantime while we work to get salt/modules/ssh.py importing on Windows
for the Fluorine release.
This allows for optional normalization of unicode strings, which will
make testing more reliable. In Mac OS, os.listdir() will produce certain
unicode glyphs using separate code points for the base character and
diacritic mark, while others will use a single code point to represent
the same unicode glyph.
>>> a = u'\u0065\u0301'
>>> b = u'\u00e9'
>>> print(a)
é
>>> print(b)
é
>>> a == b
False
>>> import salt.utils.stringutils
>>> salt.utils.stringutils.to_unicode(a, normalize=True) == b
True
>>>
`_get_extra_opts()` and `_get_branch_option()` were unnecessarily
quoting the value, causing it to be interpreted as a literal quote by
`subprocess.Popen()`.
Also, because there were separate helpers for repo options,
disableexcludes, branch options, and extra options, and specifically
because `_get_extra_opts()` parses *all* kwargs, any of the options from
the other helper funcs would end up being added to the command line
twice if `_get_extra_opts()` was used.
This commit consolidates all of the kwarg inspection and CLI opts
construction to a single helper function. It also adds unit tests to
make sure that we are formatting our commands properly.
Additionally, it makes a minor fix in `refresh_db()` which was not
accounted for when we changed the osmajorrelease grain to an integer in
2017.7.0.
This prevents a failed decode of undecodable data from resulting in a
traceback by catching the exception and just using the original value in
the changes dict.
While running an sdist, distutils logs when it creates paths. However,
if the paths are unicode _and_ contain unicode code points, the log
message results in a `UnicodeEncodeError` as distutils attempts to
perform string replacement using a str format string and a unicode path.
Paths recently added in the test suite's files to test unicode
compatibility have caused this issue to surface.
This fixes the resulting traceback by ensuring that Python 2 only passes
utf-8 encoded bytestrings when making the release tree.
