In the test suite, two of the failing `pip.installed` tests were failing
because when we shelled out to get the user's environment, the `HOME`
environment variable's value was incorrectly showing the root user's
home dir.
This change ensures that we use the correct value for the HOME
environment variable when running a command as another user.
Due to the many merge conflicts created from #47106 against the 2017.7 branch
and #46002 against the 2018.3 branch, the changes from #47106 have been largely
removed from this merge forward and the HEAD of 2018.3 was taken.
A separate fix for Tornado 5.0 support will need to be made directly against the
2018.3 branch.
Conflicts:
- doc/topics/development/conventions/formulas.rst
- salt/master.py
- salt/minion.py
- salt/netapi/rest_tornado/saltnado.py
- salt/states/zfs.py
- salt/transport/ipc.py
- salt/transport/tcp.py
- salt/transport/zeromq.py
- salt/utils/async.py
- tests/support/helpers.py
- tests/support/parser/cover.py
- tests/unit/grains/test_core.py
- tests/unit/modules/test_ssh.py
- tests/unit/test_minion.py
- tests/unit/utils/test_safe_walk.py
This adds a new decorator which creates a temporary directory and cleans
it after the test completes. It also modifies an existing decorator for
creating temporary files so that it accepts arguments, which will be
passed through to salt.utils.files.mkstemp().
Older versions of npm can't validate the newer certificate.
We don't really need to worry about this, and it would be better to just make
sure the tests run correctly that to use https for the tests.
The workaround only needs to be applied to versions of npm newer than 5.1, but
they work with versions newer than 5.0, so just do all newer versions using the
work around
This test is testing an old version of virtualenv with pip_env, and the version
of pip is 6.x.x, which is too old and doesn't work with the importlib in python
3.6. Skipping this test if python is newer then 3.5.
This makes the following changes:
- The `append_newline` argument to the `file.blockreplace`
remote-execution function has been modified so that if its value is
`None`, it only appends a newline when the content block does not end
in one.
- A couple of fixes were made to newline handling. The existing code
normalized the newlines in the content block, replacing them with
os.linesep. However, when the file contains newlines that don't match
the OS (i.e. POSIX newlines in a file on a Windows box, or Windows
newlines on a Linux/Mac/BSD/etc. box), then we would still end up with
mixed newlines. The line separator is now detected when we read in the
original file, and the detected line separator is used when writing
the content block. Additionally, the same newline mismatch was
possible when appending/prepending the content block. This has been
fixed by using a common function for appending, prepending, and
replacing the content block.
- Support for the `append_newline` argument has been added to the
`file.blockreplace` state. The default value for the state is `None`.
A `versionchanged` has been added to the remote execution function to
let users know that the Fluorine release will change the default value
of that variable.
- 20 new integration tests have been written to test the
`file.blockreplace` state.
This makes the following changes:
- The `append_newline` argument to the `file.blockreplace`
remote-execution function has been modified so that if its value is
`None`, it only appends a newline when the content block does not end
in one.
- A couple of fixes were made to newline handling. The existing code
normalized the newlines in the content block, replacing them with
os.linesep. However, when the file contains newlines that don't match
the OS (i.e. POSIX newlines in a file on a Windows box, or Windows
newlines on a Linux/Mac/BSD/etc. box), then we would still end up with
mixed newlines. The line separator is now detected when we read in the
original file, and the detected line separator is used when writing
the content block. Additionally, the same newline mismatch was
possible when appending/prepending the content block. This has been
fixed by using a common function for appending, prepending, and
replacing the content block.
- Support for the `append_newline` argument has been added to the
`file.blockreplace` state. The default value for the state is `None`.
A `versionchanged` has been added to the remote execution function to
let users know that the Fluorine release will change the default value
of that variable.
- 20 new integration tests have been written to test the
`file.blockreplace` state.
pip.installed state calls pip.freeze to check for existing installation
and to verify installation post-install. This patch propagates the
env_vars passed to pip.installed to the pip.freeze function. It also modifies
existing pip unit tests to test new functionality and adds an integration test
that verifies the expected correct behavior
Fixes#46127
The logic which ensures that we expose ports which are being bound,
even when not explicitly configured, was done incorrectly. UDP ports
were being passed to the API as '1234/udp' instead of (1234, 'udp').
This results in the port not being exposed properly.
The logic has been corrected. Additionally both the "ports" input
translation function, as well as the post-processing code (where the
port numbers configured in port_bindings were being added) both
contained code to "fix" any ports which were configured using
'portnum/tcp', as these must be passed to the API simply as integers. To
reduce code duplication, this normalization is now only performed at the
very end of the post-processing function, after ports have been
translated, and any missing ports from the port_bindings have been
added.
The unit test for the port_bindings input translation code, which was
written based upon the same incorrect reading of the API docs that
resulted in the incorrect behavior, have been updated to confirm the
(now) correct behavior. The unit test for the ports input translation
code has been updated to reflect the new normalization behavior.
Finally, an integration test has been added to ensure that we properly
expose UDP ports which are added as part of the post-processing
function.
The logic which ensures that we expose ports which are being bound,
even when not explicitly configured, was done incorrectly. UDP ports
were being passed to the API as '1234/udp' instead of (1234, 'udp').
This results in the port not being exposed properly.
The logic has been corrected. Additionally both the "ports" input
translation function, as well as the post-processing code (where the
port numbers configured in port_bindings were being added) both
contained code to "fix" any ports which were configured using
'portnum/tcp', as these must be passed to the API simply as integers. To
reduce code duplication, this normalization is now only performed at the
very end of the post-processing function, after ports have been
translated, and any missing ports from the port_bindings have been
added.
The unit test for the port_bindings input translation code, which was
written based upon the same incorrect reading of the API docs that
resulted in the incorrect behavior, have been updated to confirm the
(now) correct behavior. The unit test for the ports input translation
code has been updated to reflect the new normalization behavior.
Finally, an integration test has been added to ensure that we properly
expose UDP ports which are added as part of the post-processing
function.
the running tests run second because of alphabetical order, and stopping sshd
does not kill off the running connection, so we know that sshd has to be on
arch for us to connect, so stop it with dead and start it back with running.
It is also not possible to stop the systemd-journald process without shutting
down the full machine.
Carbon 1.1.1 for some reason added six to their setup.py, which breaks
this test since it's not installed into the virtualenv. This PR forces
this test to use a version of carbon which does not dep on six in its
setup.py.
Carbon 1.1.1 for some reason added six to their setup.py, which breaks
this test since it's not installed into the virtualenv. This PR forces
this test to use a version of carbon which does not dep on six in its
setup.py.
This makes the 2.x usage invalid syntax and forces the use of print as a
function. This adds the import to the files which I've updated in the
last couple of days but forgot to add it.
This updates the file state and execution modules to use
unicode_literals. Since the serializers and the cmd module are touched
by the file state/exec module, those are also updated here, as well as
the cmd state module, for good measure.
Additionally, I found that salt.utils.data.decode_dict (and decode_list)
are misnamed for what they actually do. Since they *encode* the
contents, the functions should be named encode_dict and encode_list,
respectively. And we also should have counterparts which actually
decode, so I've added them. The compatibility functions from salt.utils
still use the old "decode" names to preserve backward compatibility, but
they now invoke the renamed "encode" functions in salt.utils.data. Note
that this means that the compatibility functions
salt.utils.decode_dict/list, and their cognates in salt.utils.data now
do different things, but since the move to salt.utils.data is also
happening in the Oxygen release this is as good a time as any to correct
this oversight.
I've updated the jinja filter docs to include information on the renamed
jinja filters, and also added a section on jinja filter renaming to the
Oxygen release notes. There was another filter that I renamed during the
process of moving functions from salt.utils which I did not properly
document in the release notes, so this is now included along with the
others.
CentOS 7 still ships a version of Docker with API version 1.24 which
does not support this argument. This commit uses `docker.rm` to remove
the container manually after the state is run, in cases where we were
using auto_remove=True.
Much Improved Support for Docker Networking
===========================================
The `docker_network.present` state has undergone a full rewrite, which
includes the following improvements:
Full API Support for Network Management
---------------------------------------
The improvements made to input handling in the
`docker_container.running` state for 2017.7.0 have now been expanded to
docker_network.present`. This brings with it full support for all
tunable configuration arguments.
Custom Subnets
--------------
Custom subnets can now be configured. Both IPv4 and mixed IPv4/IPv6
networks are supported.
Network Configuration in :py:func:`docker_container.running` States
-------------------------------------------------------------------
It is now possible to configure static IPv4/IPv6 addresses, as well as
links and labels.
Improved Handling of Images from Custom Registries
==================================================
Rather than attempting to parse the tag from the passed image name, Salt
will now resolve that tag down to an image ID and use that ID instead.
Due to this change, there are some backward-incompatible changes to
image management. See below for a full list of these changes.
Backward-incompatible Changes to Docker Image Management
--------------------------------------------------------
Passing image names to the following functions must now be done using separate
`repository` and `tag` arguments:
- `docker.build`
- `docker.commit`
- `docker.import`
- `docker.load`
- `docker.tag`
- `docker.sls_build`
Additionally, the `tag` argument must now be explicitly passed to the
`docker_image.present` state, unless the image is being pulled from a
docker registry.
We have `output_loglevel` for controlling logging, but no corresponding
option to suppress the output from the command from being returned. This
adds a `hide_output` option to do that.
Additionally, this removes the rest of the references to the
long-since-deprecated `quiet` argument from the cmd execution module,
and adds a warning to states which use this argument. `quiet` appears to
have been incompletely removed following its deprecation. It was not
being passed to _check_loglevel(), so it was essentially ignored, but it
was still being accepted as an argument by several functions. Those
references to what has for some time now been a no-op argument are now
removed.
Elide terminology "execution failed/succeeded" since the result of these
conditions does not indicate a failure in state execution. Instead
simply report the result of each condition.
Some states are complicated and multiple subparts,
or maybe cross-call into __states__ if they manage subresources.
In these cases, they will have multiple comments.
Make this more ergonomic by supporting a list of strings as the
value for ret['comment'] in state returns and documenting this.
By joining comments on newlines, it is possible to combine
single-line and multi-line comments cleanly, as opposed to e.g. commas.
The driving impetus for this is some of the boto modules.
An update to the boto_sqs module is included as an example.
Add a check that outgoing state return data has the right shape,
and add a testcase as well.
Fix the NPM state tests and the saltmod runner & wheel state functions
to comply with the prescribed format.
This fails on centos 6 because its node is too old to support the version of
hawk bumped here https://github.com/request/request/pull/2751, we can still
test the functionality. This will pull from github, and install a specific tag
version, and we still do the uninstall using the github path.
This should be more stable.
This moves the following functions from salt.utils to salt.utils.versions:
- warn_until
- kwargs_warn_until
- compare_versions (as salt.utils.versions.compare)
- version_cmp
This PR is part of what will be an ongoing effort to use explicit
unicode strings in Salt. Because Python 3 does not suport Python 2's raw
unicode string syntax (i.e. `ur'\d+'`), we must use
`salt.utils.locales.sdecode()` to ensure that the raw string is unicode.
However, because of how `salt/utils/__init__.py` has evolved into the
hulking monstrosity it is today, this means importing a large module in
places where it is not needed, which could negatively impact
performance. For this reason, this PR also breaks out some of the
functions from `salt/utils/__init__.py` into new/existing modules under
`salt/utils/`. The long term goal will be that the modules within this
directory do not depend on importing `salt.utils`.
A summary of the changes in this PR is as follows:
* Moves the following functions from `salt.utils` to new locations
(including a deprecation warning if invoked from `salt.utils`):
`to_bytes`, `to_str`, `to_unicode`, `str_to_num`, `is_quoted`,
`dequote`, `is_hex`, `is_bin_str`, `rand_string`,
`contains_whitespace`, `clean_kwargs`, `invalid_kwargs`, `which`,
`which_bin`, `path_join`, `shlex_split`, `rand_str`, `is_windows`,
`is_proxy`, `is_linux`, `is_darwin`, `is_sunos`, `is_smartos`,
`is_smartos_globalzone`, `is_smartos_zone`, `is_freebsd`, `is_netbsd`,
`is_openbsd`, `is_aix`
* Moves the functions already deprecated by @rallytime to the bottom of
`salt/utils/__init__.py` for better organization, so we can keep the
deprecated ones separate from the ones yet to be deprecated as we
continue to break up `salt.utils`
* Updates `salt/*.py` and all files under `salt/client/` to use explicit
unicode string literals.
* Gets rid of implicit imports of `salt.utils` (e.g. `from salt.utils
import foo` becomes `import salt.utils.foo as foo`).
* Renames the `test.rand_str` function to `test.random_hash` to more
accurately reflect what it does
* Modifies `salt.utils.stringutils.random()` (née `salt.utils.rand_string()`)
such that it returns a string matching the passed size. Previously
this function would get `size` bytes from `os.urandom()`,
base64-encode it, and return the result, which would in most cases not
be equal to the passed size.
This is the error
```
npm ERR! As of npm@5, the npm cache self-heals from corruption issues and
data extracted from the cache is guaranteed to be valid. If you want to make
sure everything is consistent, use 'npm cache verify' instead.
npm ERR!
npm ERR! If you're sure you want to delete the entire cache, rerun this command
with --force.
```
