This allows masters and minions to run on Windows using Python 3. Tested
with Python 3.5.1 64-bit on Windows 7. Only the TCP transport has been
ported. It will likely not work with other transports such as ZeroMQ or
RAET without further work.
Since there were many changes necessary to make this work, here is a list
of common changes:
- Python 3 distinguishes `bytes` from `str`. In some cases `bytes` must be
used (eg read/write to binary file). In other cases `str` must be used
(eg general Salt dictionary look-ups, read/write to text file). Due to
this, there are a lot of extra uses of `salt.utils.to_bytes` in necessary
locations.
- Use `six.itervalues(varname)` instead of `varname.itervalues()`.
- In Python 2.6 and 2.7, the 'b' prefix is ignored. So instead of
```
if six.PY2:
some_var = 'some_value'
else:
some_var = b'some_value'
```
we choose to simply use:
```
some_var = b'some_value'
```
`salt/engine/__init__.py`:
- In Python 3, `engine.items()` will return an iterator instead of
a list. Due to this, needed to change `engine.items()[0]` to
`list(engine.items())[0]`.
`salt/minion.py`:
- `contextlib.nested` no longer supported in Python 3. Use
`contextlib.ExitStack` in this case.
- In `Minion.handle_event`, in PY3, `package` is of type `bytes` and `tag`
is of type `str`. Due to this, do all string searches via `tag` instead
of `package` since you can't search for a `str` in a `bytes` object.
`salt/payload.py`:
- Added `encoding` parameter to `Serial.loads`. See description in comments
for details.
- Added `use_bin_type` parameter to `Serial.dumps`. See description in
comments for details.
- When reading / writing local files, encode using `use_bin_type=True` and
decode using `encoding='utf-8'` to distinguish `bytes` and `str` types.
`salt/transport/frame.py`:
- Added `frame_msg_ipc`. This is used for IPC communications where it is
safe to break the wire protocol and so we encode using `use_bin_type=True`.
- Added `decode_embedded_strs`. This is used when it is not safe to break
the wire protocol (eg external communications). This will convert `bytes`
objects to `str` objects if possible. This will search for such objects
within dicts and lists.
`salt/transport/ipc.py`:
- Use `salt.transport.frame.frame_msg_ipc` to encode using
`use_bin_type=True`.
- Use `encoding='utf-8'` to decode such messages and ensure proper
distinction of `bytes` and `str` types.
`salt/transport/tpc.py`:
- Since we need to preserve the wire protocol, we don't use
`use_bin_type=True`. When decoding, we use `decode_embedded_strs` to
convert the embedded `bytes` objects that can be converted to `str`.
- `urlparse` doesn't exist in PY3. Use `urllib.parse` instead.
- `sys.maxint` not supported in PY3. Use `int((1 << 31) - 1)` instead.
Hence `sys.maxint - 1` becomes `int((1 << 31) - 2)`.
`salt/utils/__init__.py`:
- `libc = ctypes.cdll.LoadLibrary(ctypes.util.find_library("c"))` returns
`TypeError` in PY3. This is because `ctypes.util.find_library("c")`
returns `None` and in PY3 `ctypes.cdll.LoadLibrary(None)` is a
`TypeError`.
- On Windows, `salt.utils.fopen` would always read and write files in
binary mode. This has been changed for Python 3 since `bytes` must be
used for binary mode and `str` must be used for text mode and forcing
binary mode would screw things up if you want to read/write to `str`
objects. Preserved original behavior on PY2.
- For `salt.utils.fopen`, when reading and writing text files in PY3, if
an encoding is not specified, it will choose 'utf-8'.
`salt/utils/args.py`:
- In Python 3.5, every time `inspect.getargsspec` is used, a warning would
appear announcing its deprecation (it will be removed entirely in Python
3.6). Due to this, implemented our own version using
`inspect.getfullargspec`.
`salt/utils/event.py`:
- Since all functionality here is used only in IPC (the mechanism for
firing an event from minion to master is outside this file), we can break
the wire protocol and hence encode using `use_bin_type=True` and decode
using `encoding='utf-8'`.
`salt/utils/vt.py`:
- `_subprocess` is not available in PY3 under Windows, so we use an
alternate method to invoke `TerminateProcess` and `GetExitCodeProcess`.
`setup.py`:
- In PY3, `req.read(4096)` returns a `bytes` object. If you try to for
loop through it, each element is an `int` which represents an individual
byte. Trying to write an `int` to `wfh.write` raises an exception. Due to
this, use an alternate approach when writing from a
`http.client.HTTPResponse` object to file in PY3.
Signed-off-by: Sergey Kizunov <sergey.kizunov@ni.com>
Fixed an issue where setup.py will not run at all in Windows due to
the script attempting to invoke `os.uname()` which isn't supported on
Windows.
Signed-off-by: Sergey Kizunov <sergey.kizunov@ni.com>
The build is not reproducible because it embeds the build time. Embed
the date of the last modification to the source code instead (in case it
is provided via the SOURCE_DATE_EPOCH environment variable).
See the SOURCE_DATE_EPOCH specification for details:
https://reproducible-builds.org/specs/source-date-epoch/
Setup::DownloadWindowsDlls is downloading and writing the file as
a text not a binary file.
When creating the file handle to write the binary after being download from salt's
dependency repo, the file handle was being created with just the 'write' attribute,
however it also needs the 'binary' attribute.
Change-Id: I2f67d27ee847cd7808a78cd5ec0b2151d6a0c0e7
Setup::DownloadWindowsDlls is downloading and writing the file as
a text not a binary file.
When creating the file handle to write the binary after being download from salt's
dependency repo, the file handle was being created with just the 'write' attribute,
however it also needs the 'binary' attribute.
Change-Id: I2f67d27ee847cd7808a78cd5ec0b2151d6a0c0e7
To generate Salt's _syspaths.py file, pass `--generate-salt-syspaths` to
the develop command or set the `GENERATE_SALT_SYSPATHS` environment
variable.
Passing `--mimic-salt-install` to the develop command or setting the
`MIMIC_SALT_INSTALL` environment variable will generate `_version.py` and `_syspaths.py`
Fixes#11046Fixes#25094
Since we now evaluate each `salt.syspath` entry one by one, we only set
the passed settings in the generated `_syspaths` module.
The default values will mandate unless explicitly overridden in the
setup stage.
Refs #19157, #19160, #19161Fixes#19514Closes#19515
