This keeps us from mangling it if it contains lists, which we would be
popping items off of. We don't want to pop items from the list that was
passed in.
The initial approach tied all opened files to the same mocked
filehandle. This new approach moves all the read functions into a class,
and then uses a separate instance of that class for each opened file.
This makes readline() a regular function and not an iterator. It also
makes the different side_effect functions all operate on the same
generator representing the file contents, and adds support for reading
an explicit amount of bytes from the file. The side effect functions can
now be run one after another and the mocked filehandle will work
properly.
This makes the following changes:
1. Now works properly when read_data is a bytestring
2. Now works properly when mocked filehandle is iterated (e.g. `for line in fh:`)
3. Adds ability to make mocked filehandle raise an IOError when the path
being opened doesn't match a pattern (or one of a list of patterns).
One of the tests depends on forcing a timeout, but since we no longer
include the timeout message in the stdout/stderr return, we needed a way
to return whether or not the script timed out.
Before, tests.support.case.ShellTestCase.run_script would just go
¯\_(ツ)_/¯ and return that it killed the process. But you can still run
`.communicate()` on a process that was terminated, so this commit does
that and uses it to log the output from the killed process.
This adds logging of the output and retcode when the output file fails
to be created. As I have thus far been unable to reproduce this locally,
this is the best shot at seeing what is causing the issue, since
ShellCase's run_script is rather opaque about what happens in the script
itself. We do know from the salt-runtests.log that the `salt-call -g` is
successfully returning, what we don't know is what happened after that
data was returned, and this should provide that insight.
Jinja imports will be interpreted as originating from the top of each
of the directories in the searchpath when the template name does not
begin with './' or '../'. When a template name begins with './' or
'../' then the import will be relative to the importing file.
Prior practices required the following construct:
.. code-block:: yaml
{% from tpldir ~ '/foo' import bar %}
This patch supports a more "natural" construct:
.. code-block:: yaml
{% from './foo' import bar %}
Comparatively when importing from a parent directory - prior practice:
.. code-block:: yaml
{% from tpldir ~ '/../foo' import bar %}
Construct supported by this patch:
.. code-block:: yaml
{% from '../foo' import bar %}
Due to the many merge conflicts created from #47106 against the 2017.7 branch
and #46002 against the 2018.3 branch, the changes from #47106 have been largely
removed from this merge forward and the HEAD of 2018.3 was taken.
A separate fix for Tornado 5.0 support will need to be made directly against the
2018.3 branch.
Conflicts:
- doc/topics/development/conventions/formulas.rst
- salt/master.py
- salt/minion.py
- salt/netapi/rest_tornado/saltnado.py
- salt/states/zfs.py
- salt/transport/ipc.py
- salt/transport/tcp.py
- salt/transport/zeromq.py
- salt/utils/async.py
- tests/support/helpers.py
- tests/support/parser/cover.py
- tests/unit/grains/test_core.py
- tests/unit/modules/test_ssh.py
- tests/unit/test_minion.py
- tests/unit/utils/test_safe_walk.py
Setting a value for will cause coverage to look for a config file called
named '1' and in-turn raises an Exception. Based on the docs, if the
environment variable exists coverage is enabled. Just defined the
environment variable instead of giving it a value.
This adds a new decorator which creates a temporary directory and cleans
it after the test completes. It also modifies an existing decorator for
creating temporary files so that it accepts arguments, which will be
passed through to salt.utils.files.mkstemp().
This makes the following changes:
- The `append_newline` argument to the `file.blockreplace`
remote-execution function has been modified so that if its value is
`None`, it only appends a newline when the content block does not end
in one.
- A couple of fixes were made to newline handling. The existing code
normalized the newlines in the content block, replacing them with
os.linesep. However, when the file contains newlines that don't match
the OS (i.e. POSIX newlines in a file on a Windows box, or Windows
newlines on a Linux/Mac/BSD/etc. box), then we would still end up with
mixed newlines. The line separator is now detected when we read in the
original file, and the detected line separator is used when writing
the content block. Additionally, the same newline mismatch was
possible when appending/prepending the content block. This has been
fixed by using a common function for appending, prepending, and
replacing the content block.
- Support for the `append_newline` argument has been added to the
`file.blockreplace` state. The default value for the state is `None`.
A `versionchanged` has been added to the remote execution function to
let users know that the Fluorine release will change the default value
of that variable.
- 20 new integration tests have been written to test the
`file.blockreplace` state.
This makes the following changes:
- The `append_newline` argument to the `file.blockreplace`
remote-execution function has been modified so that if its value is
`None`, it only appends a newline when the content block does not end
in one.
- A couple of fixes were made to newline handling. The existing code
normalized the newlines in the content block, replacing them with
os.linesep. However, when the file contains newlines that don't match
the OS (i.e. POSIX newlines in a file on a Windows box, or Windows
newlines on a Linux/Mac/BSD/etc. box), then we would still end up with
mixed newlines. The line separator is now detected when we read in the
original file, and the detected line separator is used when writing
the content block. Additionally, the same newline mismatch was
possible when appending/prepending the content block. This has been
fixed by using a common function for appending, prepending, and
replacing the content block.
- Support for the `append_newline` argument has been added to the
`file.blockreplace` state. The default value for the state is `None`.
A `versionchanged` has been added to the remote execution function to
let users know that the Fluorine release will change the default value
of that variable.
- 20 new integration tests have been written to test the
`file.blockreplace` state.
This preserves the custom mock_open we backported from unittest.mock,
but otherwise ditches unittest.mock as it does not have
MagicMock.assert_called in Python releases before 3.6.
This allows us to maintain a uniform mock version across all platforms
and Python releases.
Some of these have been failing in ways which are not reproducible
locally. This adds debug logging so that we can check the
salt-runtests.log for the return data, to aid in troubleshooting.
NOTE: This also contains a fix for a bug in the CLI output handler that
I think may have a chance of fixing some of these odd failures.
https://github.com/saltstack/salt/pull/45687 and
https://github.com/saltstack/salt/pull/45725 fixed problems with older
Python 2 versions interacting with the struct module, since
unicode_literals turned the format strings to unicode and older Python 2
releases can't accept a unicode format string. Since Python 2 and 3 both
support using bytestrings for the format strings, this commit makes sure
we are using bytestrings everywhere else we are using struct.pack/unpack.
Without allow_unicode=True, unicode characters are processed through the
str representer and on Python 2 are dumped as a Unicode code point (i.e.
a literal \u0414). This commit makes allow_unicode=True the default in
our salt.utils.yamlloader.safe_dump() helper. It also adds a new
salt.utils.yamlloader.dump() helper which wraps yaml.dump() and also
makes allow_unicode=True the default.
To make importing and using our custom yaml loader/dumper easier, a
convenience module called salt.utils.yaml has been added, which does a
wildcard import from both salt.utils.yamldumper and
salt.utils.yamlloader.
Refs to yaml.load/dump and yaml.safe_load/safe_dump have been updated to
salt.utils.yaml, to ensure that unicode is handled properly.
When CherryPy run with python3 it reads "bytes" from the wire.
In case of python2 BytesIO == StringIO, so nothing should change.
This change will do the same to make unit tests reflect reality.
Python 3 does not appear to pass all values to the `__import__` builtin
when they do not differ from the defaults. Therefore, the fake import
fails because of missing positional args.
This fix simply uses the defaults from Python 2 and 3's `__import__`
builtin.
This makes the 2.x usage invalid syntax and forces the use of print as a
function. This adds the import to the files which I've updated in the
last couple of days but forgot to add it.
Much Improved Support for Docker Networking
===========================================
The `docker_network.present` state has undergone a full rewrite, which
includes the following improvements:
Full API Support for Network Management
---------------------------------------
The improvements made to input handling in the
`docker_container.running` state for 2017.7.0 have now been expanded to
docker_network.present`. This brings with it full support for all
tunable configuration arguments.
Custom Subnets
--------------
Custom subnets can now be configured. Both IPv4 and mixed IPv4/IPv6
networks are supported.
Network Configuration in :py:func:`docker_container.running` States
-------------------------------------------------------------------
It is now possible to configure static IPv4/IPv6 addresses, as well as
links and labels.
Improved Handling of Images from Custom Registries
==================================================
Rather than attempting to parse the tag from the passed image name, Salt
will now resolve that tag down to an image ID and use that ID instead.
Due to this change, there are some backward-incompatible changes to
image management. See below for a full list of these changes.
Backward-incompatible Changes to Docker Image Management
--------------------------------------------------------
Passing image names to the following functions must now be done using separate
`repository` and `tag` arguments:
- `docker.build`
- `docker.commit`
- `docker.import`
- `docker.load`
- `docker.tag`
- `docker.sls_build`
Additionally, the `tag` argument must now be explicitly passed to the
`docker_image.present` state, unless the image is being pulled from a
docker registry.
Remove while loop that was hanging when `salt` wasn't in the path
Add salt.util.path.safe_path function to check for unsafe paths
Pass root_dir to all calls to `verify_env`
These didn't get caught in PR 42823 because of how we invoke the
git_pillar code. Firstly, the "pillar" argument needed to stay. This is
because even though we're not using it, _external_pillar_data() is still
passing it now that git_pillar is not specially invoked there.
Secondly, since the input comes in as a list, and _external_pillar_data
uses single-asterisk expansion, the repos are passed separately when
they should be passed as a single list. To fix these issues, I've done
the following:
1. Re-introduced the "pillar" argument in git_pillar's ext_pillar
function.
2. Changed the "pillar" variable to avoid confusion with the (unused)
"pillar" argument being passed in.
3. Instead of git_pillar accepting the repos as a list, the ext_pillar
function now uses single-asterisk expansion to make it conform with
how _external_pillar_data() invokes it.
This PR is part of what will be an ongoing effort to use explicit
unicode strings in Salt. Because Python 3 does not suport Python 2's raw
unicode string syntax (i.e. `ur'\d+'`), we must use
`salt.utils.locales.sdecode()` to ensure that the raw string is unicode.
However, because of how `salt/utils/__init__.py` has evolved into the
hulking monstrosity it is today, this means importing a large module in
places where it is not needed, which could negatively impact
performance. For this reason, this PR also breaks out some of the
functions from `salt/utils/__init__.py` into new/existing modules under
`salt/utils/`. The long term goal will be that the modules within this
directory do not depend on importing `salt.utils`.
A summary of the changes in this PR is as follows:
* Moves the following functions from `salt.utils` to new locations
(including a deprecation warning if invoked from `salt.utils`):
`to_bytes`, `to_str`, `to_unicode`, `str_to_num`, `is_quoted`,
`dequote`, `is_hex`, `is_bin_str`, `rand_string`,
`contains_whitespace`, `clean_kwargs`, `invalid_kwargs`, `which`,
`which_bin`, `path_join`, `shlex_split`, `rand_str`, `is_windows`,
`is_proxy`, `is_linux`, `is_darwin`, `is_sunos`, `is_smartos`,
`is_smartos_globalzone`, `is_smartos_zone`, `is_freebsd`, `is_netbsd`,
`is_openbsd`, `is_aix`
* Moves the functions already deprecated by @rallytime to the bottom of
`salt/utils/__init__.py` for better organization, so we can keep the
deprecated ones separate from the ones yet to be deprecated as we
continue to break up `salt.utils`
* Updates `salt/*.py` and all files under `salt/client/` to use explicit
unicode string literals.
* Gets rid of implicit imports of `salt.utils` (e.g. `from salt.utils
import foo` becomes `import salt.utils.foo as foo`).
* Renames the `test.rand_str` function to `test.random_hash` to more
accurately reflect what it does
* Modifies `salt.utils.stringutils.random()` (née `salt.utils.rand_string()`)
such that it returns a string matching the passed size. Previously
this function would get `size` bytes from `os.urandom()`,
base64-encode it, and return the result, which would in most cases not
be equal to the passed size.
This function is used identically in every cloud provider test file. Let's add
it to a more central place and make it more DRY. Might be useful in other testing
places as well.
test.support.mock.mock_open specifies a side effect so that if you constantly
read a mocked open call, it will only iterate through it once, and then after
it will appear as if the end of the file has been reached.
This is needed for salt.modules.cp.push to be tested correctly.
If this is has any unicode characters in it, it won't load on systems that do
not default to a unicode locale.
find . -type f | while read line; do ret=$(file $line); if [[ $ret == *UTF-8* && $line == *.py ]]; then echo $line; fi; done
The above will list all files that have unicode characters in it and won't load
with locale set to C or POSIX
