* Add config params for custom refspecs
* Add exceptions for errors encountered modifying git config
* Make the refspecs a configurable parameter
* Make refspecs a per-remote parameter
* Update master config template to include custom refspecs opts
* Add documentation for new config params
* Update GitFS walkthrough with a section on custom refspecs
* Remove dulwich support from salt.utils.gitfs
Dulwich still lacks important features, including (but not limited to)
the following:
- Lack of the necessary support for checking out a ref needed for
git_pillar/winrepo support
- No support in its config objects for multivar git config items, making
it impossible to detect when repos have multiple refspecs set for a
given git remote
Given this information, and the fact that it trails as a distant third
to Pygit2 and GitPython, Salt will cease to support Dulwich as a git
interface moving forward.
* Excise references to dulwich from documentation
* Add mention of custom refspecs to Nitrogen release notes
* Add gitfs_refspecs to mocked opts in gitfs integration tests
Also remove dulwich from unit tests
* Add information about opts argument not being intended for CLI use
Version 2.6.0 changed the wording of one of the exceptions tested,
causing tests to fail when jsonschema 2.6.0 is installed. This commit
updates the tests to change the assert for 2.6.0 and later.
These unit tests exposed a bunch of python 3 incompatibilities in the
filebuffer util as well as the find util. The filebuffer changes should
be reviewed carefully.
The additional call to parser.parse_args(args) negates the first
patch to running self.config_func. We want the opts that we set
up in the above test to patch the setup args when the parse_args
func is called. This updates the mock to work with better with
patch and removes the additional parser.parse_args call.
Add skipIf pyvmomi is not installed
NameError: global name 'vmodl' is not defined
NameError: global name 'vim' is not defined
Fix skipIf typo for OpenBSD
>>> import sys
>>> sys.platform
'openbsd6'
Avoid warning from glusterfs.py:
Use specific 'len(elem)' or 'elem is not None' test
Enable boto_secgroup_test tests that work now:
test skipped due to error in moto return - fixed in
cc01669643
* [PY3] Change log.warn statements to log.warning
log.warn is deprecated in Python 3. This change removes the
occurance of deprecation warnings for using log.warn. For example:
```
17:24:23 [WARNING ] /testing/salt/utils/verify.py:522: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
17:24:23 log.warn('Insecure logging configuration detected! Sensitive data may be logged.')
17:24:23
17:24:23 [WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
```
* Tweak unit.utils.verify_test to using warning instead of warn
This fixes the bug #36866 where minion gets __master_disconnected right
after connect because '::1' isn't in the list of connected masters that
is ['127.0.0.1'].
When using `fopen`, we need to import all of salt.utils. We should
also be explicit about calling salt.utils.fopen.
This also cleans up the ordering of the salttesting vs salt libs to
be consistent with other files and conform with `ensure_is_syspath`.
Also changes a print statement to a log.info
* Revert "Don't use six.text_type() in salt.utils.gitfs"
This reverts commit d76659a63a.
* salt.utils.path_join: handle mixed unicode/binary-unicode paths
* Add test for mixed unicode and binary-unicode paths
* salt.utils.gitfs: replace os.path.join with salt.utils.path_join
- added optional traversal_spec parameter to both methods, allowing the
user to specify a different TraversalSpec object to search for content
(if not specified, the behaviour is that all a traverse all search is
done)
- the traversal_spec is destroyed only if is created in the method
- added local_properties flag to both methods allowing the retrieval of
properties of the container object
- added tests for the 2 functions
- added support for passthrough connections via SSPI
- added function to specify if a connection is made to a vCenter or ESXi host
- added function to disconnect from a vCenter or ESXi host
- added tests for changed/new functions
* salt/crypt.py: clean up open filehandles
* salt/fileclient.py: clean up open filehandles
* salt/grains/core.py: clean up open filehandles
* salt/modules/cp.py: clean up open filehandles
* salt/modules/data.py: clean up open filehandles
* salt/modules/dnsutil.py: clean up open filehandles
* salt/modules/dockerng.py: clean up open filehandles
* salt/modules/inspectlib/collector.py: clean up open filehandles
* salt/modules/file.py: clean up open filehandles
* salt/modules/hosts.py: clean up open filehandles
* salt/modules/incron.py: clean up open filehandles
* salt/modules/dpkg.py: clean up open filehandles
* salt/modules/linux_sysctl.py: clean up open filehandles
* salt/modules/netbsd_sysctl.py: clean up open filehandles
* salt/modules/network.py: clean up open filehandles
* salt/modules/nftables.py: clean up open filehandles
* salt/modules/openbsd_sysctl.py: clean up open filehandles
* salt/modules/rh_ip.py: clean up open filehandles
* salt/modules/portage_config.py: clean up open filehandles
* salt/modules/status.py: clean up open filehandles
* salt/modules/tls.py: clean up open filehandles
* salt/modules/xapi.py: clean up open filehandles
* salt/modules/x509.py: clean up open filehandles
* salt/modules/virt.py: clean up open filehandles
* salt/modules/zcbuildout.py: clean up open filehandles
* salt/returners/local_cache.py: clean up open filehandles
* salt/utils/cloud.py: clean up open filehandles
* salt/states/pkgrepo.py: clean up open filehandles
* salt/states/x509.py: clean up open filehandles
* salt/transport/mixins/auth.py: clean up open filehandles
* salt/utils/__init__.py: clean up open filehandles
* salt/states/pkg.py: clean up open filehandles
* salt/utils/minion.py: clean up open filehandles
* salt/utils/openstack/nova.py: clean up open filehandles
* salt/utils/openstack/swift.py: clean up open filehandles
* salt/utils/process.py: clean up open filehandles
* salt/utils/templates.py: clean up open filehandles
* salt/utils/virt.py: clean up open filehandles
* tests/integration/__init__.py: clean up open filehandles
* tests/integration/cli/grains.py: clean up open filehandles
* tests/integration/client/standard.py: clean up open filehandles
* tests/integration/modules/hosts.py: clean up open filehandles
* tests/unit/utils/vt_test.py: clean up open filehandles
* tests/integration/shell/enabled.py: clean up open filehandles
* tests/integration/states/cmd.py: clean up open filehandles
* tests/integration/states/file.py: clean up open filehandles
* tests/integration/states/match.py: clean up open filehandles
* tests/unit/config_test.py: clean up open filehandles
* tests/unit/templates/jinja_test.py: clean up open filehandles
* tests/unit/utils/find_test.py: clean up open filehandles
* tests/integration/modules/state.py: clean up open filehandles
* Update dnsutil_test to reflect changes in fopen usage
* Fix context clash.
boto_vpc uses both boto and boto3. Both utils for those use the same name for their connection context. If you used states that required both boto and boto3, the get_conn would return the object.
* Add a unit test to the boto context conflict problem
* Fix pylint errors and ensure new test to runs correctly
Merges #32985
* Fix typo
* fix sorting by latest version when called with an attribute
* remove reference to master_alive_check
* Fixes saltstack/salt#28262
* Resolve memory leak in authentication
* outputter virt_list does not exist anymore
* Update proxmox documentation
* Fix documentation on boto_asg and boto_elb modules and states
* modules.win_timezone: don't list all zones in debug log
* Correcty index glusterfs bricks
Fixes issue #32311
* Cleaner deprecation process with decorators
* Add deprecation decorator scaffold
* Capture type error and unhandled exceptions while function calls
* Aware of the current and future version of deprecation
* Implement initially is_deprecated decorator
* Add an alias for the capitalization
* Fix capitalization easier way
* Remove an extra line
* Add successor name to the deprecation decorator.
* Granulate logging and error messages.
* Implement function swapper
* Raise later the caught exception
* Clarify exception message
* Save function original name
* Remove an extra line
* Hide an alternative hidden function name in the error message, preserving the error itself
* Rename variable as private
* Add a method to detect if a function is using its previous version
* Message to the log and/or raise an exception accordingly to the status of used function
* Log an error along with the exception
* Add internal method documentation
* Add documentation and usage process for decorator "is_deprecated"
* Add documentation and process usage for the decorator "with_deprecated"
* Hide private method name
* Fix PEP8, re-word the error message
* Deprecate basic uptime function
* Add initial decorator unit test
* Rename old/new functions, mock versions
* Move frequent data to the test setup
* Add logging on EOL exception
* Rename and document high to low version test on is_deprecated
* Implement a test on low to high version of is_deprecated decorator
* Add a correction to the test description
* Remove a dead code
* Implement a test for high to low version on is_deprecated, using with_successor param
* Correct typso adn mistaeks
* Implement high to low version with successor param on is_deprecated
* Setup a virtual name for the module
* Implement test for with_deprecated should raise an exception if same deprecated function not found
* Implement test for with_deprecated an old function is picked up if configured
* Correct test description purpose
* Implement test with_deprecated when no deprecation is requested
* Add logging test to the configured deprecation request
* Add logging testing when deprecated version wasn't requested
* Implement test EOL for with_deprecated decorator
* Correct test explanation
* Rename the test
* Implement with_deprecated no EOL, deprecated other function name
* Implement with_deprecated, deprecated other function name, EOL reached
* Add test description for the with_deprecated + with_name + EOL
* Fix confusing test names
* Add logging test to the is_deprecated decorator when function as not found.
* Add more test point to each test, remove empty lines
* Bugfix: at certain conditions a wrong alias name is reported to the log
* Fix a typo in a comment
* Add test for the logging
* Disable a pylint: None will _never_ be raised
* Fix test for the deprecated "status.uptime" version
* Bugfix: Do not yank raised exceptions
* Remove unnecessary decorator
* Add test for the new uptime
* Add test for the new uptime fails when /proc/uptime does not exists
* Rename old test case
* Skip test for the UTC time, unless freeze time is used.
* Fix pylint
* Fix documentation
* Bugfix: proxy-pass the docstring of the decorated function
* Lint fix
* Fixessaltstack/salt#28262 for 2015.5 branch
* Update master config docs
* Improve git_pillar documentation/logging
* Add note about different behavior of top file in git_pillar
* Make log entry for a missing pillar SLS file more accurate for git_pillar
* FreeBSD supports packages in format java/openjdk7 so the prior commit broke that functionality. Check freebsd/pkg#1409 for more info.
* FreeBSD supports packages in format java/openjdk7 so the prior commit broke that functionality. Check freebsd/pkg#1409 for more info.
* Update glusterfs_test to be inline with #32312
* Fix salt-cloud paralell provisioning
Closes#31632
* Ignore Raspbian in service.py __virtual__ (#32421)
* Ignore Raspbian in service.py __virtual__
This prevents more than one execution module from trying to load as the
service virtual module.
Refs: #32413
* pack __salt__ before loading provider overrides
We can (and should) pack here since we're just packing a reference to the
object. __salt__ needs to be available when we're loading our provider
overrides
* Fix broken __salt__ dict in provider override
Using ret.items() here sets ``__salt__`` to its items (tuple containing
function name and reference), breaking usage of ``__salt__`` inside
overridden functions.
* Merge #32293 with test fixes (#32418)
* Fix issue #11497
* Remove check for working directory presence in tests
* Fix Domainname introspection
Default value needs to be extracted from the container itself,
because dockerd set Domainname value when network_mode=host.
* Add pgjsonb_queue to queue doc index
* Pylint fixes
* Pass parser options into batch mode
Resolves#31738
* Changed the target file in file.symlink test (#32443)
* Argument name in docs should match actual arg name (#32445)
Fixes#31851
* tests.integration: bypass MacOS TMPDIR, gettempdir (#32447)
Updates 0edd532, 8f558a5.
When logging in as root over `ssh root@host`, `$TMPDIR` and
`tempfile.gettempdir()` are both set to a variation of:
```
/private/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/
```
When logging in as root over `sudo -i`, `$TMPDIR` is unset and
`tempfile.gettempdir()` is set to `/tmp`.
My guess is that the second case is an unintended or uncorrected omision
by Apple as they have introduced the longer, randomized temp path in a
recent version of MacOS.
* Issue #28706: Fix state user.present behavior. (#32448)
- As mentionned in issue #28706, state user.present no longer remove
user from groups if the keyword 'groups' with empty value '[]' is not
explicitly set, salt will assume current groups are still wanted.
* tests.integration: fix 4230c8a
* Move the tables of virtual modules to individual documentation pages
* Add new doc pages to toctree
* Add external ref to windows package manager docs
* Improve docstrings
* Add documentation on virtual module provider overrides to the module docs
* Clarify the scope of the provider param in states.
* Add link to provider override docs to all package providers
* Add link to provider override docs to all service providers
* Add link to provider override docs to all user providers
* dd link to provider override docs to all shadow providers
* Add link to provider override docs to all group providers
* Backport 31164 and 31364 (#32474)
* Don't send REQ while another one is waiting for response.
The message has to be removed from the queue the only *after* it's
already processed to don't confuse send() functionality that expects
empty queue means: there's no active sendings.
* Fixed zeromq ReqMessageClient destroy
* Add link to provider override docs to opkg.py
This is a companion to https://github.com/saltstack/salt/pull/32458, but
this module was not added until the 2016.3 branch, so the documentation
is being updated there for this module.
* Add documentation for some master/minion configs (#32454)
Refs #32400
Adds docs for:
- cli_summary
- event_return_queue
- event_return_whitelist
- event_return_blacklist
- file_recv_max_size
- fileserver_followsymlinks
- fileserver_ignoresymlinks
- fileserver_limit_traversal
* Automatically detect correct MySQL password column for 5.7 and fix setting passwords (#32440)
* Automatically detect MySQL password column
* Fix changing password in MySQL 5.7
* Fix lint test
* Fix unit tests (?)
They will still fail if "authentication_string" is legitimately the right column name, but I don't know what to do about that.
* Additional unit test fix
* Only unsub if we have a jid
Closes#32479
On BSD-like systems (including MacOS) files created in /tmp (or any dir
with mod 1777) do not have the user's group. Therefore the st_gid for
files created in /tmp will usually be wheel. Easiest thing to do is to
skip this test.
* Discard duplicate YAML nodes when merging anchors to avoid duplicate node errors. Add yamlloader unit tests.
Fixed#14550 Using YAML anchors/references in Pillars causes conflicting IDs
* Fix lint errors. Make external function a static method.
* Fix remaining lint errors in unit test file.
* Add deprecation decorator scaffold
* Capture type error and unhandled exceptions while function calls
* Aware of the current and future version of deprecation
* Implement initially is_deprecated decorator
* Add an alias for the capitalization
* Fix capitalization easier way
* Remove an extra line
* Add successor name to the deprecation decorator.
* Granulate logging and error messages.
* Implement function swapper
* Raise later the caught exception
* Clarify exception message
* Save function original name
* Remove an extra line
* Hide an alternative hidden function name in the error message, preserving the error itself
* Rename variable as private
* Add a method to detect if a function is using its previous version
* Message to the log and/or raise an exception accordingly to the status of used function
* Log an error along with the exception
* Add internal method documentation
* Add documentation and usage process for decorator "is_deprecated"
* Add documentation and process usage for the decorator "with_deprecated"
* Hide private method name
* Fix PEP8, re-word the error message
* Deprecate basic uptime function
* Add initial decorator unit test
* Rename old/new functions, mock versions
* Move frequent data to the test setup
* Add logging on EOL exception
* Rename and document high to low version test on is_deprecated
* Implement a test on low to high version of is_deprecated decorator
* Add a correction to the test description
* Remove a dead code
* Implement a test for high to low version on is_deprecated, using with_successor param
* Correct typso adn mistaeks
* Implement high to low version with successor param on is_deprecated
* Setup a virtual name for the module
* Implement test for with_deprecated should raise an exception if same deprecated function not found
* Implement test for with_deprecated an old function is picked up if configured
* Correct test description purpose
* Implement test with_deprecated when no deprecation is requested
* Add logging test to the configured deprecation request
* Add logging testing when deprecated version wasn't requested
* Implement test EOL for with_deprecated decorator
* Correct test explanation
* Rename the test
* Implement with_deprecated no EOL, deprecated other function name
* Implement with_deprecated, deprecated other function name, EOL reached
* Add test description for the with_deprecated + with_name + EOL
* Fix confusing test names
* Add logging test to the is_deprecated decorator when function as not found.
* Add more test point to each test, remove empty lines
* Bugfix: at certain conditions a wrong alias name is reported to the log
* Fix a typo in a comment
* Add test for the logging
* Disable a pylint: None will _never_ be raised
* Fix test for the deprecated "status.uptime" version
* Bugfix: Do not yank raised exceptions
* Remove unnecessary decorator
* Add test for the new uptime
* Add test for the new uptime fails when /proc/uptime does not exists
* Rename old test case
* Skip test for the UTC time, unless freeze time is used.
* Fix pylint
* Fix documentation
* Bugfix: proxy-pass the docstring of the decorated function
* Lint fix
salt/master.py:
- If ZMQ is not importable, use `tornado.ioloop.IOLoop` as the loop class.
- Check HAS_ZMQ before using any zmq.* content.
- Did not touch `FloMWorker` usage of zmq.*. This is only used when the
transport is RAET.
salt/minion.py:
- If ZMQ is not importable, use `tornado.ioloop.IOLoop` as the loop class.
- Check HAS_ZMQ before using any zmq.* content.
salt/transport/ipc.py:
- Added `IPCMessagePublisher`. This is intended to function much like
ZMQ's zmq.PUB sockets. Used in implementing utils/event.py to function
without ZMQ.
- Added `IPCMessageSubscriber`. This is intended to function much like
ZMQ's zmq.SUB sockets. Used in implementing utils/event.py to function
without ZMQ. What makes this class a bit different is that the associated
IO Loop is meant to not be running when it is used. Due to this, it is
recommended that the caller create a new IO Loop for this purpose. The
reason for this is that the `get_event()` API may be invoked from
anywhere, whether or not there is a current IO Loop that is running in
the thread of the invocation.
salt/utils/async.py:
- If ZMQ is not importable, use `tornado.ioloop.IOLoop` as the loop class.
salt/utils/event.py:
- Implemented using `salt.transport.ipc` instead of ZMQ.
- zmq.PUB ==> `IPCMessagePublisher`
- zmq.SUB ==> `IPCMessageSubscriber`
- zmq.PUSH ==> `IPCMessageClient`
- zmq.PULL ==> `IPCMessageServer`
Signed-off-by: Sergey Kizunov <sergey.kizunov@ni.com>
The etcd watch function tries to read a key when it reaches the timeout
in order to obtain details about the key.
On python 2.6, when it tries to re-read the key, it throws a ValueError,
since python-etcd doesn't officially support python 2.6. The watch
function wasn't catching this ValueError, so it was returning the
exception, rather than a blank dict.
The etcd modules were using bits and pieces of python-etcd directly and
were using outdated exceptions to catch errors. These errors were
causing issues when trying to use etcd modules for logic.
A summary of changes
- Most of the 'work' is now done in etcd_util
- Removed import of python-etcd from states/etcd_mod and the
returner
- Added new tests for etcd_util
- Put in proper exceptions and catches for python-etcd
- Added support for ValueError catching. python-etcd doesn't
support python 2.6 and raises ValueError when trying to format
exceptions
- Added watch function to etcd execution module
- Added autospec to unit tests so hopefully less brittle
- Added TTL and directory features to the set function
*BACKWARDS INCOMPATIBLE CHANGES*
All interfaces are still backwards incompatible. However, the returns
from several of the etcd util functions have been changed. The old
returns presented issues with certain test cases. For instance, a
failed 'get' returned '', but a key with no value will also return ''.
The same issues occured with the 'ls', 'tree' and 'set' functions. Trying to
ls a blank directory returned the same result as a failed ls of a
non-existent key.
There was a problem wherein two sockets would race to see which was connected first,
causing unit.utils.event_test.TestSaltEvent.test_event_multiple_clients to fail from
time to time. This gives a little breathing room to the second socket.
This also only does the pub connect if necessary on SaltEvent instantation.
Add new option for top_file_merging_strategy to config
Add master config file option
Adding additional tests
Lexicographical test works
Make default strategy
Simplify to just an option about whether or not to merge
Changes
Implement ordered top files
Implement merge strategies and default top files
Lint
Doc rewrite and start on ordering
Remove debugging
Moving to env_order
Adding env_order
Still writing tests
More tests and docs
Add new option for top_file_merging_strategy to config
Add master config file option
Adding additional tests
Lexicographical test works
Make default strategy
Simplify to just an option about whether or not to merge
Changes
Implement ordered top files
Implement merge strategies and default top files
Lint
Doc rewrite and start on ordering
Remove debugging
Moving to env_order
Adding env_order
Still writing tests
More tests and docs
Require all multitasking contexts to subscribe to their events so one call to get_event for one tag does not discard events that should be saved for a subsequent call to get_event with another tag.
Use blocking get_event in batching with very small timeout.
Fixes#25998
Once subscribed to publisher SUB socket gets collecting all incoming
messages that is unwanted behavior for fire-only events.
Fixed by using listen=<True|False> constructor argument.
While thinking through issues that might cause #25718 I ran into this case-- where you can create 2 sync wrappers with the same parent IOLoop and run into problems because you can't start/stop the loop multiple times like that and have the call tree work correctly. To solve this (since it *should* be the exceptional case) I'm making SyncWrapper keep track of which loops are in use-- then if someone attempts this situation it will simply make another one.
Once subscribed to publisher SUB socket gets collecting all incoming
messages that is unwanted behavior for fire-only events.
Fixed by using listen=<True|False> constructor argument.
