Fixes some issues with Py3 not building
Brings the 2 build_env files more in line with each other
Fixes some typos in the build_env scripts
Removes the need for seperate req files for Py2 and Py3
Setup.py uses the windows req file on Windows
Removes the need to install pycrypto and pyyaml in setup.py. They are
now handled in the req.txt
Uses pypi resources instead of downloading .whl files and .exes for
dependencies
Updates nssm version to address the issue in #46192
Removes nssm from buildenv and downloads 32 and 64 bit versions of nssm
in build_pkg.bat
Removes the AppNoConsole=1 setting from the nsis script since it is not
needed with the new version of nssm
While running an sdist, distutils logs when it creates paths. However,
if the paths are unicode _and_ contain unicode code points, the log
message results in a `UnicodeEncodeError` as distutils attempts to
perform string replacement using a str format string and a unicode path.
Paths recently added in the test suite's files to test unicode
compatibility have caused this issue to surface.
This fixes the resulting traceback by ensuring that Python 2 only passes
utf-8 encoded bytestrings when making the release tree.
setup.py
- Remove PyCrypto install, that happens through req.txt
- Remove PyYAML install, that happens through req.txt
- Add PyWin32 install, that happens with downloaded .whl
- Detect if PyWin32 is already installed before installing
build_env_#.py
- Put PyWin32 dlls in Python root instead of site-packages\win32
- hide gen_py directory creation message
This allows masters and minions to run on Windows using Python 3. Tested
with Python 3.5.1 64-bit on Windows 7. Only the TCP transport has been
ported. It will likely not work with other transports such as ZeroMQ or
RAET without further work.
Since there were many changes necessary to make this work, here is a list
of common changes:
- Python 3 distinguishes `bytes` from `str`. In some cases `bytes` must be
used (eg read/write to binary file). In other cases `str` must be used
(eg general Salt dictionary look-ups, read/write to text file). Due to
this, there are a lot of extra uses of `salt.utils.to_bytes` in necessary
locations.
- Use `six.itervalues(varname)` instead of `varname.itervalues()`.
- In Python 2.6 and 2.7, the 'b' prefix is ignored. So instead of
```
if six.PY2:
some_var = 'some_value'
else:
some_var = b'some_value'
```
we choose to simply use:
```
some_var = b'some_value'
```
`salt/engine/__init__.py`:
- In Python 3, `engine.items()` will return an iterator instead of
a list. Due to this, needed to change `engine.items()[0]` to
`list(engine.items())[0]`.
`salt/minion.py`:
- `contextlib.nested` no longer supported in Python 3. Use
`contextlib.ExitStack` in this case.
- In `Minion.handle_event`, in PY3, `package` is of type `bytes` and `tag`
is of type `str`. Due to this, do all string searches via `tag` instead
of `package` since you can't search for a `str` in a `bytes` object.
`salt/payload.py`:
- Added `encoding` parameter to `Serial.loads`. See description in comments
for details.
- Added `use_bin_type` parameter to `Serial.dumps`. See description in
comments for details.
- When reading / writing local files, encode using `use_bin_type=True` and
decode using `encoding='utf-8'` to distinguish `bytes` and `str` types.
`salt/transport/frame.py`:
- Added `frame_msg_ipc`. This is used for IPC communications where it is
safe to break the wire protocol and so we encode using `use_bin_type=True`.
- Added `decode_embedded_strs`. This is used when it is not safe to break
the wire protocol (eg external communications). This will convert `bytes`
objects to `str` objects if possible. This will search for such objects
within dicts and lists.
`salt/transport/ipc.py`:
- Use `salt.transport.frame.frame_msg_ipc` to encode using
`use_bin_type=True`.
- Use `encoding='utf-8'` to decode such messages and ensure proper
distinction of `bytes` and `str` types.
`salt/transport/tpc.py`:
- Since we need to preserve the wire protocol, we don't use
`use_bin_type=True`. When decoding, we use `decode_embedded_strs` to
convert the embedded `bytes` objects that can be converted to `str`.
- `urlparse` doesn't exist in PY3. Use `urllib.parse` instead.
- `sys.maxint` not supported in PY3. Use `int((1 << 31) - 1)` instead.
Hence `sys.maxint - 1` becomes `int((1 << 31) - 2)`.
`salt/utils/__init__.py`:
- `libc = ctypes.cdll.LoadLibrary(ctypes.util.find_library("c"))` returns
`TypeError` in PY3. This is because `ctypes.util.find_library("c")`
returns `None` and in PY3 `ctypes.cdll.LoadLibrary(None)` is a
`TypeError`.
- On Windows, `salt.utils.fopen` would always read and write files in
binary mode. This has been changed for Python 3 since `bytes` must be
used for binary mode and `str` must be used for text mode and forcing
binary mode would screw things up if you want to read/write to `str`
objects. Preserved original behavior on PY2.
- For `salt.utils.fopen`, when reading and writing text files in PY3, if
an encoding is not specified, it will choose 'utf-8'.
`salt/utils/args.py`:
- In Python 3.5, every time `inspect.getargsspec` is used, a warning would
appear announcing its deprecation (it will be removed entirely in Python
3.6). Due to this, implemented our own version using
`inspect.getfullargspec`.
`salt/utils/event.py`:
- Since all functionality here is used only in IPC (the mechanism for
firing an event from minion to master is outside this file), we can break
the wire protocol and hence encode using `use_bin_type=True` and decode
using `encoding='utf-8'`.
`salt/utils/vt.py`:
- `_subprocess` is not available in PY3 under Windows, so we use an
alternate method to invoke `TerminateProcess` and `GetExitCodeProcess`.
`setup.py`:
- In PY3, `req.read(4096)` returns a `bytes` object. If you try to for
loop through it, each element is an `int` which represents an individual
byte. Trying to write an `int` to `wfh.write` raises an exception. Due to
this, use an alternate approach when writing from a
`http.client.HTTPResponse` object to file in PY3.
Signed-off-by: Sergey Kizunov <sergey.kizunov@ni.com>
