These have been skipped for a while now because they didn't work
correctly. The old tests have been scrapped in favor of new ones that
test both the old and new config schema.
This class is designed to be used for (among other things) gitfs and git_pillar, to enforce config settings which reside in a git config file without resorting to using the git CLI.
Having a jid that is occasionally not unique (eg two processes creating
jobs at about the same time) causes problems in my usage due to
job tracking assuming unique jids. Add a config option called `unique_jid`
that will enforce unique jids, but will change the jid format by
appending an underscore and the process id.
Signed-off-by: Sergey Kizunov <sergey.kizunov@ni.com>
This makes it possible to merge the state return
for a subresource into a main state return.
This is useful for configuring e.g. alarms to go along with a resource.
When calling an incorrectly formatted wheel or runner function, we should
be raising a SaltInvocationError with a hint to check function syntax rather
that raising an Eauth authentication error.
This PR does several things:
- Adds a dictionary error return when the function syntax passed through to
`utils.minions.CkMinions.spec_check` does not match the expected `module.function`
syntax
- Handles the return of this new dictionary error (instead of previous `False` return)
wherever the spec_check function is called. This is handled up the stack in
`master.py` and `masterapi.py`.
- Reworks the runner and wheel functions in `master.py` and `masterapi.py` to
help make those functions more DRY (see `salt.auth.check_authentication` function).
- Adds tests for all of these changes (written before the runner and wheel functions
were moved to use the new salt.auth.check_authentication function) to help prevent
regressions.
- Fixes a couple of places where unit tests exposed potential stacktraces.
- Adjusts one previous unit test concerning the dictionary error change from spec_check
Moves the following functions from `salt.utils` to `salt.utils.state`:
- gen_state_tag
- search_onfail_requisites
- check_onfail_requisites
- check_state_result
Original functions have a deprecation warning added and all references
in Salt code were updated to use the new function path, including moving
some unit tests.
Moves the `get_colors` and `get_color_theme` out of `salt.utils.py`
and into a new file named `salt.utils.color.py`.
This PR moves the original functions, adds a deprecation warning to
the old function paths, and updates any references to the functions
in Salt code to the new location. This includes moving a test as well.
Through merge forwards, some of the old-style paths have made their
way into develop. This PR corrects some of these instances on the more
popular functions.
This makes the following changes:
1. Renames the valid_providers param in GitBase to git_providers,
allowing for a dictionary mapping provider names to their associated
classes. This allows for alternate providers to be used with a
GitBase subclass.
2. Renames the get_provider function to verify_provider to reduce
confusion with git_providers.
3. Uses super() to run a parent class' dunder init instead of invoking
the parent class directly.
This test wasn't really written with Windows in mind. Uses PATHEXT that
actually resembles a Windows environment. The test value has the correct
path seperator for Windows.
This moves the following functions from salt.utils to salt.utils.versions:
- warn_until
- kwargs_warn_until
- compare_versions (as salt.utils.versions.compare)
- version_cmp
PyYAML works with and without bytestrings on PY3, and Python 3 can read
all of the test data as regular strings, so bytestrings are unnecessary
here.
This also adds use of textwrap.dedent to make the YAML easier to read,
so it can be indented away from the far left side of the line.
This PR is part of what will be an ongoing effort to use explicit
unicode strings in Salt. Because Python 3 does not suport Python 2's raw
unicode string syntax (i.e. `ur'\d+'`), we must use
`salt.utils.locales.sdecode()` to ensure that the raw string is unicode.
However, because of how `salt/utils/__init__.py` has evolved into the
hulking monstrosity it is today, this means importing a large module in
places where it is not needed, which could negatively impact
performance. For this reason, this PR also breaks out some of the
functions from `salt/utils/__init__.py` into new/existing modules under
`salt/utils/`. The long term goal will be that the modules within this
directory do not depend on importing `salt.utils`.
A summary of the changes in this PR is as follows:
* Moves the following functions from `salt.utils` to new locations
(including a deprecation warning if invoked from `salt.utils`):
`to_bytes`, `to_str`, `to_unicode`, `str_to_num`, `is_quoted`,
`dequote`, `is_hex`, `is_bin_str`, `rand_string`,
`contains_whitespace`, `clean_kwargs`, `invalid_kwargs`, `which`,
`which_bin`, `path_join`, `shlex_split`, `rand_str`, `is_windows`,
`is_proxy`, `is_linux`, `is_darwin`, `is_sunos`, `is_smartos`,
`is_smartos_globalzone`, `is_smartos_zone`, `is_freebsd`, `is_netbsd`,
`is_openbsd`, `is_aix`
* Moves the functions already deprecated by @rallytime to the bottom of
`salt/utils/__init__.py` for better organization, so we can keep the
deprecated ones separate from the ones yet to be deprecated as we
continue to break up `salt.utils`
* Updates `salt/*.py` and all files under `salt/client/` to use explicit
unicode string literals.
* Gets rid of implicit imports of `salt.utils` (e.g. `from salt.utils
import foo` becomes `import salt.utils.foo as foo`).
* Renames the `test.rand_str` function to `test.random_hash` to more
accurately reflect what it does
* Modifies `salt.utils.stringutils.random()` (née `salt.utils.rand_string()`)
such that it returns a string matching the passed size. Previously
this function would get `size` bytes from `os.urandom()`,
base64-encode it, and return the result, which would in most cases not
be equal to the passed size.
- Moves function to new location in salt.utils.doc.py
- Adds deprecation warning to salt.utils.parse_docstring
- Moves related unit test in test_utils.py to new test_doc.py file
This will keep future IP address changes from impacting this test. It
also tests a host which returns an IPv6 address, and a host which cannot
be resolved.
Also remove unused and unnecessary behavior from this function when the
first path component is of single length. An empty first parameter to
this function will normalze to "." and result in an incorrectly-joined
result.
This does the following:
- Splits states for container/volume/image/network management into four
separate state modules.
- Preserves backward compatibility by making ``docker.image_present``
invoke ``docker_image.present``, etc.
- Changes how Salt detects that a container needs to be replaced.
Instead of comparing each passed argument to the named container's
configuration, it creates a temporary container, and compares that
container to the named container. If the two differ, then the older
container is removed, and the new one is renamed and started, becoming
the named container.
- Removes the unit tests for container management and replaces them with
integration tests.
- Adds unit tests for the new salt.utils.docker
* Add config params for custom refspecs
* Add exceptions for errors encountered modifying git config
* Make the refspecs a configurable parameter
* Make refspecs a per-remote parameter
* Update master config template to include custom refspecs opts
* Add documentation for new config params
* Update GitFS walkthrough with a section on custom refspecs
* Remove dulwich support from salt.utils.gitfs
Dulwich still lacks important features, including (but not limited to)
the following:
- Lack of the necessary support for checking out a ref needed for
git_pillar/winrepo support
- No support in its config objects for multivar git config items, making
it impossible to detect when repos have multiple refspecs set for a
given git remote
Given this information, and the fact that it trails as a distant third
to Pygit2 and GitPython, Salt will cease to support Dulwich as a git
interface moving forward.
* Excise references to dulwich from documentation
* Add mention of custom refspecs to Nitrogen release notes
* Add gitfs_refspecs to mocked opts in gitfs integration tests
Also remove dulwich from unit tests
* Add information about opts argument not being intended for CLI use
Version 2.6.0 changed the wording of one of the exceptions tested,
causing tests to fail when jsonschema 2.6.0 is installed. This commit
updates the tests to change the assert for 2.6.0 and later.
These unit tests exposed a bunch of python 3 incompatibilities in the
filebuffer util as well as the find util. The filebuffer changes should
be reviewed carefully.
The additional call to parser.parse_args(args) negates the first
patch to running self.config_func. We want the opts that we set
up in the above test to patch the setup args when the parse_args
func is called. This updates the mock to work with better with
patch and removes the additional parser.parse_args call.
Add skipIf pyvmomi is not installed
NameError: global name 'vmodl' is not defined
NameError: global name 'vim' is not defined
Fix skipIf typo for OpenBSD
>>> import sys
>>> sys.platform
'openbsd6'
Avoid warning from glusterfs.py:
Use specific 'len(elem)' or 'elem is not None' test
Enable boto_secgroup_test tests that work now:
test skipped due to error in moto return - fixed in
cc01669643
* [PY3] Change log.warn statements to log.warning
log.warn is deprecated in Python 3. This change removes the
occurance of deprecation warnings for using log.warn. For example:
```
17:24:23 [WARNING ] /testing/salt/utils/verify.py:522: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
17:24:23 log.warn('Insecure logging configuration detected! Sensitive data may be logged.')
17:24:23
17:24:23 [WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
```
* Tweak unit.utils.verify_test to using warning instead of warn
This fixes the bug #36866 where minion gets __master_disconnected right
after connect because '::1' isn't in the list of connected masters that
is ['127.0.0.1'].
When using `fopen`, we need to import all of salt.utils. We should
also be explicit about calling salt.utils.fopen.
This also cleans up the ordering of the salttesting vs salt libs to
be consistent with other files and conform with `ensure_is_syspath`.
Also changes a print statement to a log.info
* Revert "Don't use six.text_type() in salt.utils.gitfs"
This reverts commit d76659a63a.
* salt.utils.path_join: handle mixed unicode/binary-unicode paths
* Add test for mixed unicode and binary-unicode paths
* salt.utils.gitfs: replace os.path.join with salt.utils.path_join
- added optional traversal_spec parameter to both methods, allowing the
user to specify a different TraversalSpec object to search for content
(if not specified, the behaviour is that all a traverse all search is
done)
- the traversal_spec is destroyed only if is created in the method
- added local_properties flag to both methods allowing the retrieval of
properties of the container object
- added tests for the 2 functions
- added support for passthrough connections via SSPI
- added function to specify if a connection is made to a vCenter or ESXi host
- added function to disconnect from a vCenter or ESXi host
- added tests for changed/new functions
* salt/crypt.py: clean up open filehandles
* salt/fileclient.py: clean up open filehandles
* salt/grains/core.py: clean up open filehandles
* salt/modules/cp.py: clean up open filehandles
* salt/modules/data.py: clean up open filehandles
* salt/modules/dnsutil.py: clean up open filehandles
* salt/modules/dockerng.py: clean up open filehandles
* salt/modules/inspectlib/collector.py: clean up open filehandles
* salt/modules/file.py: clean up open filehandles
* salt/modules/hosts.py: clean up open filehandles
* salt/modules/incron.py: clean up open filehandles
* salt/modules/dpkg.py: clean up open filehandles
* salt/modules/linux_sysctl.py: clean up open filehandles
* salt/modules/netbsd_sysctl.py: clean up open filehandles
* salt/modules/network.py: clean up open filehandles
* salt/modules/nftables.py: clean up open filehandles
* salt/modules/openbsd_sysctl.py: clean up open filehandles
* salt/modules/rh_ip.py: clean up open filehandles
* salt/modules/portage_config.py: clean up open filehandles
* salt/modules/status.py: clean up open filehandles
* salt/modules/tls.py: clean up open filehandles
* salt/modules/xapi.py: clean up open filehandles
* salt/modules/x509.py: clean up open filehandles
* salt/modules/virt.py: clean up open filehandles
* salt/modules/zcbuildout.py: clean up open filehandles
* salt/returners/local_cache.py: clean up open filehandles
* salt/utils/cloud.py: clean up open filehandles
* salt/states/pkgrepo.py: clean up open filehandles
* salt/states/x509.py: clean up open filehandles
* salt/transport/mixins/auth.py: clean up open filehandles
* salt/utils/__init__.py: clean up open filehandles
* salt/states/pkg.py: clean up open filehandles
* salt/utils/minion.py: clean up open filehandles
* salt/utils/openstack/nova.py: clean up open filehandles
* salt/utils/openstack/swift.py: clean up open filehandles
* salt/utils/process.py: clean up open filehandles
* salt/utils/templates.py: clean up open filehandles
* salt/utils/virt.py: clean up open filehandles
* tests/integration/__init__.py: clean up open filehandles
* tests/integration/cli/grains.py: clean up open filehandles
* tests/integration/client/standard.py: clean up open filehandles
* tests/integration/modules/hosts.py: clean up open filehandles
* tests/unit/utils/vt_test.py: clean up open filehandles
* tests/integration/shell/enabled.py: clean up open filehandles
* tests/integration/states/cmd.py: clean up open filehandles
* tests/integration/states/file.py: clean up open filehandles
* tests/integration/states/match.py: clean up open filehandles
* tests/unit/config_test.py: clean up open filehandles
* tests/unit/templates/jinja_test.py: clean up open filehandles
* tests/unit/utils/find_test.py: clean up open filehandles
* tests/integration/modules/state.py: clean up open filehandles
* Update dnsutil_test to reflect changes in fopen usage