This PR is part of what will be an ongoing effort to use explicit
unicode strings in Salt. Because Python 3 does not suport Python 2's raw
unicode string syntax (i.e. `ur'\d+'`), we must use
`salt.utils.locales.sdecode()` to ensure that the raw string is unicode.
However, because of how `salt/utils/__init__.py` has evolved into the
hulking monstrosity it is today, this means importing a large module in
places where it is not needed, which could negatively impact
performance. For this reason, this PR also breaks out some of the
functions from `salt/utils/__init__.py` into new/existing modules under
`salt/utils/`. The long term goal will be that the modules within this
directory do not depend on importing `salt.utils`.
A summary of the changes in this PR is as follows:
* Moves the following functions from `salt.utils` to new locations
(including a deprecation warning if invoked from `salt.utils`):
`to_bytes`, `to_str`, `to_unicode`, `str_to_num`, `is_quoted`,
`dequote`, `is_hex`, `is_bin_str`, `rand_string`,
`contains_whitespace`, `clean_kwargs`, `invalid_kwargs`, `which`,
`which_bin`, `path_join`, `shlex_split`, `rand_str`, `is_windows`,
`is_proxy`, `is_linux`, `is_darwin`, `is_sunos`, `is_smartos`,
`is_smartos_globalzone`, `is_smartos_zone`, `is_freebsd`, `is_netbsd`,
`is_openbsd`, `is_aix`
* Moves the functions already deprecated by @rallytime to the bottom of
`salt/utils/__init__.py` for better organization, so we can keep the
deprecated ones separate from the ones yet to be deprecated as we
continue to break up `salt.utils`
* Updates `salt/*.py` and all files under `salt/client/` to use explicit
unicode string literals.
* Gets rid of implicit imports of `salt.utils` (e.g. `from salt.utils
import foo` becomes `import salt.utils.foo as foo`).
* Renames the `test.rand_str` function to `test.random_hash` to more
accurately reflect what it does
* Modifies `salt.utils.stringutils.random()` (née `salt.utils.rand_string()`)
such that it returns a string matching the passed size. Previously
this function would get `size` bytes from `os.urandom()`,
base64-encode it, and return the result, which would in most cases not
be equal to the passed size.
Currently, in test mode states always return None even on failure.
This masks failed states in highstate output,
which instead just look like they are going to make changes.
In some cases states can determine if they will fail even on a dry run,
e.g., if an API call to a service to determine the state of a
resource fails due to invalid credentials or a missing parameter.
In these cases, allow returning a result of False even in dry runs
to enable catching more errors at testing time.
Hi,
thanks for making Saltstack. It's terrific.
When I was looking for an option to change the name of the pillar top file I couldn't find anything, but I found out that 'state_top' also affects the name of pillar top file. My change tries to make this behaviour explicit.
Greetings,
Peter
This adds 2 new config options:
- gitfs_disable_saltenv_mapping - This allows a user to disable the environment mapping logic and rely solely on per-saltenv configuration
- gitfs_ref_types - This allows individual ref types to be ignored
It also includes a performance improvement in retrieving tree objects
matching the desired ref name. Before, we would iterate through the
different refs and check if the name matched the desired ref's name.
Now we just check for a head/tag ref based on the ref path in the
gitdir (e.g. refs/remotes/origin/master or refs/tags/v1.2.3). This
prevents us from needlessly iterating through all refs until a match is
found, which does not scale well when the number of branches and tags is
larger.
The code which integrates master_tops SLS matches with top file SLS
matches was using set operations to ensure there were no duplicate
matches. However, the process of first converting the matches to a set,
and then performing a union on the two sets, has the side effect of
making the ordering unpredictable.
The set operations have been removed in favor of list comprehensions.
This ensures that the ordering is predictable, and that master_tops
matches are executed *after* top file matches.
In addition, a new minion config option called "master_tops_first" has
been added to allow for the master_tops matches to be executed *before*
top file matches.
This breaks the files into chunks to keep them from exceeding the max size
of a published command. It also adds recursion, allowing for entire
directories to be transmitted. Finally, it adds gzip compression, and
a "-n" flag to disable use of compression.
All of the functions in the blockdev execution module have been
moved over to the disk execution module. These functions were
slated for removal in the Oxygen release.
* Add Ingest pipeline methods to Elasticsearch execution module
* Refactor Elasticsearch execution module and properly handle exceptions
* Throw CommandExecutionError in methods applicable for different Elasticsearch versions
* Refactor Elasticsearch states to reflect execution module changes
* Add state for managing Elasticsearch pipelines
* Fix few typos in Elasticsearch module, return None when deleted document doesn't exist
* Implements stats and health methods for Elasticsearch
* Add Elasticsearch methods to open/close index, manage search templates and repositories
* Merge existing Elasticsearch states into single one, add Search Template handling
* Add index alias state for Elasticsearch, fix documentation
* Catch all global exceptions in Elasticsearch states, unit test all of them
* Implement few Unit tests for Elasticsearch execution module, merge fixes into deprecated elastic states
* Implement additional unit tests for Elasticsearch execution module
* Finalize Elasticsearch module documentation
With the merging of #39996, salt/modules/docker.py now imports
salt.utils.docker. However, our loader appends the module dir (in
this case salt/modules/) to sys.path temporarily for the length of the
loading process. So, as the docker execution module tries to import
salt.utils.docker, when salt.utils.docker attempts to do an "import
docker", and docker-py is *not* installed, this results in
salt/modules/docker.py (the docker execution module) being loaded in its
place, which results in tracebacks in the minion log.
Renaming the docker execution module keeps this import shadowing from
occurring. Note that we don't need to do this for the placeholder
salt/states/docker.py as it does not import salt.utils.docker.
Updats the existing documentation and also adds more documentation
to the Multi-Master and Syndic topology files about the addition
of the pluggable minion data store.
This does the following:
- Splits states for container/volume/image/network management into four
separate state modules.
- Preserves backward compatibility by making ``docker.image_present``
invoke ``docker_image.present``, etc.
- Changes how Salt detects that a container needs to be replaced.
Instead of comparing each passed argument to the named container's
configuration, it creates a temporary container, and compares that
container to the named container. If the two differ, then the older
container is removed, and the new one is renamed and started, becoming
the named container.
- Removes the unit tests for container management and replaces them with
integration tests.
- Adds unit tests for the new salt.utils.docker
In 2014.7.0, default matcher in the top file was changed to the compound
matcher, but the docs still suggest that it is the glob matcher. This
commit updates the docs to explicitly mention that the compound matcher
is now the default, and adds a table describing all of the available
matchers that can be set in the top file.
This commit also modifies the master config for the test suite to use
this new option, so that a missing GitPython/Pygit2 will not crash the
test suite's master. This should make the test suite run smoother by
only negatively impacting those tests which require the pillar data
provided by git_pillar, when a valid provider is not available.
This module adds the ability to work with nix packages, and do some
basic operations on the nix store. There is still some work to be done,
but this lets you orchestrate some basic tasks for a system with nix
installed.
* Add config params for custom refspecs
* Add exceptions for errors encountered modifying git config
* Make the refspecs a configurable parameter
* Make refspecs a per-remote parameter
* Update master config template to include custom refspecs opts
* Add documentation for new config params
* Update GitFS walkthrough with a section on custom refspecs
* Remove dulwich support from salt.utils.gitfs
Dulwich still lacks important features, including (but not limited to)
the following:
- Lack of the necessary support for checking out a ref needed for
git_pillar/winrepo support
- No support in its config objects for multivar git config items, making
it impossible to detect when repos have multiple refspecs set for a
given git remote
Given this information, and the fact that it trails as a distant third
to Pygit2 and GitPython, Salt will cease to support Dulwich as a git
interface moving forward.
* Excise references to dulwich from documentation
* Add mention of custom refspecs to Nitrogen release notes
* Add gitfs_refspecs to mocked opts in gitfs integration tests
Also remove dulwich from unit tests
* Add information about opts argument not being intended for CLI use
* create zone state, zone.running, and zone.stopped
* renamed zone.running and zone.stopped, minor lint fixes
* cleanup message in zonecfg to make it clear for the zone state
* zone.property_present state
* zonecfg.set_property - should handle true/false parsed as boolean
* zonecfg module should be more consistant with message
* zone.property_absent implemented
* fix more minor issues in zonecfg when values get parsed as bool
* zonecfg fix spelling error
* improved zonecfg error message handling
* zonecfg parser should also do the reverse
* more improvements in zonecfg for parsing values
* small tweak to zonecfg simplyfy code for the zone state
* Add resource_present to zone state
* zonecfg lint fixes
* resource_absent state, lint fixed, test support
* zonecfg should be a bit more careful when removing the exported config
* zone.export state
* zone.import state
* zone.absent implemented
* zone.attached and zone.detached implemented
* very minor consistency fixes
* added zone.installed and zone.uninstalled, made zone.detached and zone.halted not fail on missing zone
* handle bool values in _parse_value
* add useful cleanup internal function to zonecfg
* zonecfg - cleaned up _parse_value, added _sanitize_value and _dump_cfg for debugging
* zone.absent - uninstall would sometimes not do the right thing. some work on zone.present
* zone.present implemented path for when config does not yet exist
* zone.present supports updating properties
* zonecfg should support resources that do not have selectors
* zonecfg should also debug log remove_resource its config
* zone.present - implemented, after 2 refactors
* Finish documentation
