While running an sdist, distutils logs when it creates paths. However,
if the paths are unicode _and_ contain unicode code points, the log
message results in a `UnicodeEncodeError` as distutils attempts to
perform string replacement using a str format string and a unicode path.
Paths recently added in the test suite's files to test unicode
compatibility have caused this issue to surface.
This fixes the resulting traceback by ensuring that Python 2 only passes
utf-8 encoded bytestrings when making the release tree.
setup.py
- Remove PyCrypto install, that happens through req.txt
- Remove PyYAML install, that happens through req.txt
- Add PyWin32 install, that happens with downloaded .whl
- Detect if PyWin32 is already installed before installing
build_env_#.py
- Put PyWin32 dlls in Python root instead of site-packages\win32
- hide gen_py directory creation message
This allows masters and minions to run on Windows using Python 3. Tested
with Python 3.5.1 64-bit on Windows 7. Only the TCP transport has been
ported. It will likely not work with other transports such as ZeroMQ or
RAET without further work.
Since there were many changes necessary to make this work, here is a list
of common changes:
- Python 3 distinguishes `bytes` from `str`. In some cases `bytes` must be
used (eg read/write to binary file). In other cases `str` must be used
(eg general Salt dictionary look-ups, read/write to text file). Due to
this, there are a lot of extra uses of `salt.utils.to_bytes` in necessary
locations.
- Use `six.itervalues(varname)` instead of `varname.itervalues()`.
- In Python 2.6 and 2.7, the 'b' prefix is ignored. So instead of
```
if six.PY2:
some_var = 'some_value'
else:
some_var = b'some_value'
```
we choose to simply use:
```
some_var = b'some_value'
```
`salt/engine/__init__.py`:
- In Python 3, `engine.items()` will return an iterator instead of
a list. Due to this, needed to change `engine.items()[0]` to
`list(engine.items())[0]`.
`salt/minion.py`:
- `contextlib.nested` no longer supported in Python 3. Use
`contextlib.ExitStack` in this case.
- In `Minion.handle_event`, in PY3, `package` is of type `bytes` and `tag`
is of type `str`. Due to this, do all string searches via `tag` instead
of `package` since you can't search for a `str` in a `bytes` object.
`salt/payload.py`:
- Added `encoding` parameter to `Serial.loads`. See description in comments
for details.
- Added `use_bin_type` parameter to `Serial.dumps`. See description in
comments for details.
- When reading / writing local files, encode using `use_bin_type=True` and
decode using `encoding='utf-8'` to distinguish `bytes` and `str` types.
`salt/transport/frame.py`:
- Added `frame_msg_ipc`. This is used for IPC communications where it is
safe to break the wire protocol and so we encode using `use_bin_type=True`.
- Added `decode_embedded_strs`. This is used when it is not safe to break
the wire protocol (eg external communications). This will convert `bytes`
objects to `str` objects if possible. This will search for such objects
within dicts and lists.
`salt/transport/ipc.py`:
- Use `salt.transport.frame.frame_msg_ipc` to encode using
`use_bin_type=True`.
- Use `encoding='utf-8'` to decode such messages and ensure proper
distinction of `bytes` and `str` types.
`salt/transport/tpc.py`:
- Since we need to preserve the wire protocol, we don't use
`use_bin_type=True`. When decoding, we use `decode_embedded_strs` to
convert the embedded `bytes` objects that can be converted to `str`.
- `urlparse` doesn't exist in PY3. Use `urllib.parse` instead.
- `sys.maxint` not supported in PY3. Use `int((1 << 31) - 1)` instead.
Hence `sys.maxint - 1` becomes `int((1 << 31) - 2)`.
`salt/utils/__init__.py`:
- `libc = ctypes.cdll.LoadLibrary(ctypes.util.find_library("c"))` returns
`TypeError` in PY3. This is because `ctypes.util.find_library("c")`
returns `None` and in PY3 `ctypes.cdll.LoadLibrary(None)` is a
`TypeError`.
- On Windows, `salt.utils.fopen` would always read and write files in
binary mode. This has been changed for Python 3 since `bytes` must be
used for binary mode and `str` must be used for text mode and forcing
binary mode would screw things up if you want to read/write to `str`
objects. Preserved original behavior on PY2.
- For `salt.utils.fopen`, when reading and writing text files in PY3, if
an encoding is not specified, it will choose 'utf-8'.
`salt/utils/args.py`:
- In Python 3.5, every time `inspect.getargsspec` is used, a warning would
appear announcing its deprecation (it will be removed entirely in Python
3.6). Due to this, implemented our own version using
`inspect.getfullargspec`.
`salt/utils/event.py`:
- Since all functionality here is used only in IPC (the mechanism for
firing an event from minion to master is outside this file), we can break
the wire protocol and hence encode using `use_bin_type=True` and decode
using `encoding='utf-8'`.
`salt/utils/vt.py`:
- `_subprocess` is not available in PY3 under Windows, so we use an
alternate method to invoke `TerminateProcess` and `GetExitCodeProcess`.
`setup.py`:
- In PY3, `req.read(4096)` returns a `bytes` object. If you try to for
loop through it, each element is an `int` which represents an individual
byte. Trying to write an `int` to `wfh.write` raises an exception. Due to
this, use an alternate approach when writing from a
`http.client.HTTPResponse` object to file in PY3.
Signed-off-by: Sergey Kizunov <sergey.kizunov@ni.com>
Fixed an issue where setup.py will not run at all in Windows due to
the script attempting to invoke `os.uname()` which isn't supported on
Windows.
Signed-off-by: Sergey Kizunov <sergey.kizunov@ni.com>
