Remove while loop that was hanging when `salt` wasn't in the path
Add salt.util.path.safe_path function to check for unsafe paths
Pass root_dir to all calls to `verify_env`
These didn't get caught in PR 42823 because of how we invoke the
git_pillar code. Firstly, the "pillar" argument needed to stay. This is
because even though we're not using it, _external_pillar_data() is still
passing it now that git_pillar is not specially invoked there.
Secondly, since the input comes in as a list, and _external_pillar_data
uses single-asterisk expansion, the repos are passed separately when
they should be passed as a single list. To fix these issues, I've done
the following:
1. Re-introduced the "pillar" argument in git_pillar's ext_pillar
function.
2. Changed the "pillar" variable to avoid confusion with the (unused)
"pillar" argument being passed in.
3. Instead of git_pillar accepting the repos as a list, the ext_pillar
function now uses single-asterisk expansion to make it conform with
how _external_pillar_data() invokes it.
This PR is part of what will be an ongoing effort to use explicit
unicode strings in Salt. Because Python 3 does not suport Python 2's raw
unicode string syntax (i.e. `ur'\d+'`), we must use
`salt.utils.locales.sdecode()` to ensure that the raw string is unicode.
However, because of how `salt/utils/__init__.py` has evolved into the
hulking monstrosity it is today, this means importing a large module in
places where it is not needed, which could negatively impact
performance. For this reason, this PR also breaks out some of the
functions from `salt/utils/__init__.py` into new/existing modules under
`salt/utils/`. The long term goal will be that the modules within this
directory do not depend on importing `salt.utils`.
A summary of the changes in this PR is as follows:
* Moves the following functions from `salt.utils` to new locations
(including a deprecation warning if invoked from `salt.utils`):
`to_bytes`, `to_str`, `to_unicode`, `str_to_num`, `is_quoted`,
`dequote`, `is_hex`, `is_bin_str`, `rand_string`,
`contains_whitespace`, `clean_kwargs`, `invalid_kwargs`, `which`,
`which_bin`, `path_join`, `shlex_split`, `rand_str`, `is_windows`,
`is_proxy`, `is_linux`, `is_darwin`, `is_sunos`, `is_smartos`,
`is_smartos_globalzone`, `is_smartos_zone`, `is_freebsd`, `is_netbsd`,
`is_openbsd`, `is_aix`
* Moves the functions already deprecated by @rallytime to the bottom of
`salt/utils/__init__.py` for better organization, so we can keep the
deprecated ones separate from the ones yet to be deprecated as we
continue to break up `salt.utils`
* Updates `salt/*.py` and all files under `salt/client/` to use explicit
unicode string literals.
* Gets rid of implicit imports of `salt.utils` (e.g. `from salt.utils
import foo` becomes `import salt.utils.foo as foo`).
* Renames the `test.rand_str` function to `test.random_hash` to more
accurately reflect what it does
* Modifies `salt.utils.stringutils.random()` (née `salt.utils.rand_string()`)
such that it returns a string matching the passed size. Previously
this function would get `size` bytes from `os.urandom()`,
base64-encode it, and return the result, which would in most cases not
be equal to the passed size.
This function is used identically in every cloud provider test file. Let's add
it to a more central place and make it more DRY. Might be useful in other testing
places as well.
test.support.mock.mock_open specifies a side effect so that if you constantly
read a mocked open call, it will only iterate through it once, and then after
it will appear as if the end of the file has been reached.
This is needed for salt.modules.cp.push to be tested correctly.
If this is has any unicode characters in it, it won't load on systems that do
not default to a unicode locale.
find . -type f | while read line; do ret=$(file $line); if [[ $ret == *UTF-8* && $line == *.py ]]; then echo $line; fi; done
The above will list all files that have unicode characters in it and won't load
with locale set to C or POSIX
The location of the git-http-backend is within /usr/lib instead of
/usr/libexec.
This also uncomments a line that was commented out for troubleshooting
purposes.
For some reason these were not failing when I opened #40777, but now
that the PR is merged are failing for PR builds (as well as locally in
my test env). This fixes those failures.
This further abstracts some of the setup and teardown code so it can be
used for git-over-http tests.
It also moves the code that was originally added to the archive
state integration tests to create a local http server into
salt.support.helpers so that it can be more easily and portably used.
Allows patching more than one loader module.
Allows auto-loading the minion functions which will get namespaced with
the globals used to patch the loader module(s).
This avoids tracebacks when daemonizing code which is running under the XMLRunner:
```
23:22:46 Traceback (most recent call last):
23:22:47 File "/testing/salt/utils/process.py", line 644, in _run
23:22:47 return self._original_run()
23:22:47 File "/usr/lib64/python3.4/multiprocessing/process.py", line 93, in run
23:22:47 self._target(*self._args, **self._kwargs)
23:22:47 File "/testing/salt/client/mixins.py", line 464, in _proc_function
23:22:47 salt.utils.daemonize()
23:22:47 File "/testing/salt/utils/__init__.py", line 502, in daemonize
23:22:47 os.dup2(dev_null.fileno(), sys.stdout.fileno())
23:22:47 io.UnsupportedOperation: fileno
23:22:47 Process SignalHandlingMultiprocessingProcess-2:
23:22:47 Traceback (most recent call last):
23:22:47 File "/usr/lib64/python3.4/multiprocessing/process.py", line 254, in _bootstrap
23:22:47 self.run()
23:22:47 File "/testing/salt/utils/process.py", line 644, in _run
23:22:47 return self._original_run()
23:22:47 File "/usr/lib64/python3.4/multiprocessing/process.py", line 93, in run
23:22:47 self._target(*self._args, **self._kwargs)
23:22:47 File "/testing/salt/client/mixins.py", line 464, in _proc_function
23:22:47 salt.utils.daemonize()
23:22:47 File "/testing/salt/utils/__init__.py", line 502, in daemonize
23:22:47 os.dup2(dev_null.fileno(), sys.stdout.fileno())
23:22:47 io.UnsupportedOperation: fileno
```
We just keep a record of all the test case class attributes prior to
running the test and then just delete it afterwards.
We also issue a warning log message to tell the users how to cleanup
such references.
