valitydev/salt
14
0
Fork 0
mirror of https://github.com/valitydev/salt.git synced 2024-11-06 08:35:21 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Merge branch '2017.7' into '2018.3'

Browse Source 
Conflicts:
  - salt/modules/x509.py
  - salt/runners/cloud.py
  - tests/unit/modules/test_state.py
This commit is contained in:
rallytime 2018-06-25 10:41:06 -04:00
commit c7bb8a50b0
No known key found for this signature in database
GPG Key ID: E8F1A4B90D0DEA19
12 changed files with 296 additions and 66 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

23
doc/faq.rst
View File

@ -148,22 +148,23 @@ Why aren't my custom modules/states/etc. available on my Minions?
-----------------------------------------------------------------
Custom modules are synced to Minions when
:mod:`saltutil.sync_modules <salt.modules.saltutil.sync_modules>`,
or :mod:`saltutil.sync_all <salt.modules.saltutil.sync_all>` is run.
Custom modules are also synced by :mod:`state.apply` when run without
any arguments.
:py:func:`saltutil.sync_modules <salt.modules.saltutil.sync_modules>`,
or :py:func:`saltutil.sync_all <salt.modules.saltutil.sync_all>` is run.
Similarly, custom states are synced to Minions when :py:func:`saltutil.sync_states
<salt.modules.saltutil.sync_states>`, or :py:func:`saltutil.sync_all
<salt.modules.saltutil.sync_all>` is run.
Similarly, custom states are synced to Minions
when :mod:`state.apply <salt.modules.state.apply_>`,
:mod:`saltutil.sync_states <salt.modules.saltutil.sync_states>`, or
:mod:`saltutil.sync_all <salt.modules.saltutil.sync_all>` is run.
They are both also synced when a :ref:`highstate <running-highstate>` is
triggered.
Custom states are also synced by :mod:`state.apply<salt.modules.state.apply_>`
when run without any arguments.
As of the Fluorine release, as well as 2017.7.7 and 2018.3.2 in their
respective release cycles, the ``sync`` argument to :py:func:`state.apply
<salt.modules.state.apply_>`/:py:func:`state.sls <salt.modules.state.sls>` can
be used to sync custom types when running individual SLS files.
Other custom types (renderers, outputters, etc.) have similar behavior, see the
documentation for the :mod:`saltutil <salt.modules.saltutil>` module for more
documentation for the :py:func:`saltutil <salt.modules.saltutil>` module for more
information.
:ref:`This reactor example <minion-start-reactor>` can be used to automatically

64
doc/topics/releases/2017.7.7.rst
View File

@ -5,23 +5,64 @@ In Progress: Salt 2017.7.7 Release Notes
Version 2017.7.7 is an **unreleased** bugfix release for :ref:`2017.7.0 <release-2017-7-0>`.
This release is still in progress and has not been released yet.
The ``2017.7.7`` release contains only a single fix for Issue `#48038`_, which
is a critical bug that occurs in a multi-syndic setup where the same job is run
multiple times on a minion.
The ``2017.7.7`` release contains only a small number of fixes, which are detailed
below.
This release fixes two critical issues.
The first is Issue `#48038`_, which is a critical bug that occurs in a multi-syndic
setup where the same job is run multiple times on a minion.
The second issue is `#48130`_. This bug appears in certain setups where the Master
reports a Minion time-out, even though the job is still running on the Minion.
Both of these issues have been fixed with this release.
Statistics
==========
- Total Merges: **1**
- Total Issue References: **1**
- Total PR References: **2**
- Total Merges: **5**
- Total Issue References: **2**
- Total PR References: **6**
- Contributors: **2** (`garethgreenaway`_, `rallytime`_)
- Contributors: **3** (`garethgreenaway`_, `gtmanfred`_, `rallytime`_)
Changelog for v2017.7.6..v2017.7.7
==================================
*Generated at: 2018-06-14 15:43:34 UTC*
*Generated at: 2018-06-17 19:26:52 UTC*
* **ISSUE** `#48130`_: (`rmarchei`_) Minion timeouts with 2018.3.1 (refs: `#48157`_)
* **PR** `#48157`_: (`gtmanfred`_) always listen when gathering job info
@ *2018-06-17 19:04:09 UTC*
* 8af4452134 Merge pull request `#48157`_ from gtmanfred/2017.7.7
* d8209e8a40 always listen when gathering job info
* **PR** `#48140`_: (`rallytime`_) Update man pages for 2017.7.7
@ *2018-06-14 21:22:43 UTC*
* b98c52ee51 Merge pull request `#48140`_ from rallytime/man-pages-2017.7.7
* 8893bf0d4c Update man pages for 2017.7.7
* **PR** `#48136`_: (`gtmanfred`_) [2017.7.7] bootstrap kitchen branch tests with 2017.7.6
@ *2018-06-14 21:20:16 UTC*
* baa0363336 Merge pull request `#48136`_ from gtmanfred/2017.7.7
* fce1c31146 bootstrap kitchen branch tests with 2017.7.6
* **PR** `#48134`_: (`rallytime`_) Add release notes file for 2017.7.7
@ *2018-06-14 16:31:34 UTC*
* b0ba08f4d9 Merge pull request `#48134`_ from rallytime/release-notes-2017.7.7
* 217005b8f1 Add missing `v` for tag reference
* d53569d1e3 Add release notes file for 2017.7.7
* **ISSUE** `#48038`_: (`austinpapp`_) jobs are not dedup'ing minion side (refs: `#48075`_)
@ -37,6 +78,13 @@ Changelog for v2017.7.6..v2017.7.7
.. _`#48038`: https://github.com/saltstack/salt/issues/48038
.. _`#48075`: https://github.com/saltstack/salt/pull/48075
.. _`#48098`: https://github.com/saltstack/salt/pull/48098
.. _`#48130`: https://github.com/saltstack/salt/issues/48130
.. _`#48134`: https://github.com/saltstack/salt/pull/48134
.. _`#48136`: https://github.com/saltstack/salt/pull/48136
.. _`#48140`: https://github.com/saltstack/salt/pull/48140
.. _`#48157`: https://github.com/saltstack/salt/pull/48157
.. _`austinpapp`: https://github.com/austinpapp
.. _`garethgreenaway`: https://github.com/garethgreenaway
.. _`gtmanfred`: https://github.com/gtmanfred
.. _`rallytime`: https://github.com/rallytime
.. _`rmarchei`: https://github.com/rmarchei

8
doc/topics/tutorials/pillar.rst
View File

@ -330,7 +330,13 @@ Nested pillar values can also be set via the command line:
.. code-block:: bash
salt '*' state.sls my_sls_file pillar='{"foo": {"bar": "baz"}}'
salt '*' state.sls my_sls_file pillar='{"foo": {"bar": "baz"}}'
Lists can be passed via command line pillar data as follows:
.. code-block:: bash
salt '*' state.sls my_sls_file pillar='{"some_list": ["foo", "bar", "baz"]}'
.. note::

17
doc/topics/utils/index.rst
View File

@ -139,13 +139,18 @@ where it is necessary to invoke the same function from a custom :ref:`outputter
<all-salt.output>`/returner, as well as an execution module.
Utility modules placed in ``salt://_utils/`` will be synced to the minions when
any of the following Salt functions are called:
a :ref:`highstate <running-highstate>` is run, as well as when any of the
following Salt functions are called:
* :mod:`state.apply <salt.modules.state.apply_>`
* :mod:`saltutil.sync_utils <salt.modules.saltutil.sync_utils>`
* :mod:`saltutil.sync_all <salt.modules.saltutil.sync_all>`
* :py:func:`saltutil.sync_utils <salt.modules.saltutil.sync_utils>`
* :py:func:`saltutil.sync_all <salt.modules.saltutil.sync_all>`
As of the Fluorine release, as well as 2017.7.7 and 2018.3.2 in their
respective release cycles, the ``sync`` argument to :py:func:`state.apply
<salt.modules.state.apply_>`/:py:func:`state.sls <salt.modules.state.sls>` can
be used to sync custom types when running individual SLS files.
To sync to the Master, use either of the following:
* :mod:`saltutil.sync_utils <salt.runners.saltutil.sync_utils>`
* :mod:`saltutil.sync_all <salt.runners.saltutil.sync_all>`
* :py:func:`saltutil.sync_utils <salt.runners.saltutil.sync_utils>`
* :py:func:`saltutil.sync_all <salt.runners.saltutil.sync_all>`

5
salt/client/ssh/__init__.py
View File

@ -1245,7 +1245,10 @@ ARGS = {10}\n'''.format(self.minion_config,
shim_tmp_file.write(salt.utils.stringutils.to_bytes(cmd_str))
# Copy shim to target system, under $HOME/.<randomized name>
target_shim_file = '.{0}.{1}'.format(binascii.hexlify(os.urandom(6)), extension)
target_shim_file = '.{0}.{1}'.format(
binascii.hexlify(os.urandom(6)).decode('ascii'),
extension
)
if self.winrm:
target_shim_file = saltwinshell.get_target_shim_file(self, target_shim_file)
self.shell.send(shim_tmp_file.name, target_shim_file, makedirs=True)

10
salt/modules/lxc.py
View File

@ -2268,22 +2268,22 @@ def _change_state(cmd,
# as te command itself mess with double forks; we must not
# communicate with it, but just wait for the exit status
pkwargs = {'python_shell': False,
'redirect_stderr': True,
'with_communicate': with_communicate,
'use_vt': use_vt,
'stdin': stdin,
'stdout': stdout,
'stderr': stderr}
'stdout': stdout}
for i in [a for a in pkwargs]:
val = pkwargs[i]
if val is _marker:
pkwargs.pop(i, None)
error = __salt__['cmd.run_stderr'](cmd, **pkwargs)
_cmdout = __salt__['cmd.run_all'](cmd, **pkwargs)
if error:
if _cmdout['retcode'] != 0:
raise CommandExecutionError(
'Error changing state for container \'{0}\' using command '
'\'{1}\': {2}'.format(name, cmd, error)
'\'{1}\': {2}'.format(name, cmd, _cmdout['stdout'])
)
if expected is not None:
# some commands do not wait, so we will

54
salt/modules/state.py
View File

@ -602,8 +602,7 @@ def template_str(tem, queue=False, **kwargs):
return ret
def apply_(mods=None,
**kwargs):
def apply_(mods=None, **kwargs):
'''
.. versionadded:: 2015.5.0
@ -743,6 +742,22 @@ def apply_(mods=None,
.. code-block:: bash
salt '*' state.apply test localconfig=/path/to/minion.yml
sync_mods
If specified, the desired custom module types will be synced prior to
running the SLS files:
.. code-block:: bash
salt '*' state.apply test sync_mods=states,modules
salt '*' state.apply test sync_mods=all
.. note::
This option is ignored when no SLS files are specified, as a
:ref:`highstate <running-highstate>` automatically syncs all custom
module types.
.. versionadded:: 2017.7.8,2018.3.3,Fluorine
'''
if mods:
return sls(mods, **kwargs)
@ -1068,7 +1083,7 @@ def highstate(test=None, queue=False, **kwargs):
return ret
def sls(mods, test=None, exclude=None, queue=False, **kwargs):
def sls(mods, test=None, exclude=None, queue=False, sync_mods=None, **kwargs):
'''
Execute the states in one or more SLS files
@ -1160,6 +1175,17 @@ def sls(mods, test=None, exclude=None, queue=False, **kwargs):
.. versionadded:: 2015.8.4
sync_mods
If specified, the desired custom module types will be synced prior to
running the SLS files:
.. code-block:: bash
salt '*' state.sls test sync_mods=states,modules
salt '*' state.sls test sync_mods=all
.. versionadded:: 2017.7.8,2018.3.3,Fluorine
CLI Example:
.. code-block:: bash
@ -1223,6 +1249,28 @@ def sls(mods, test=None, exclude=None, queue=False, **kwargs):
'{0}.cache.p'.format(kwargs.get('cache_name', 'highstate'))
)
if sync_mods is True:
sync_mods = ['all']
if sync_mods is not None:
sync_mods = salt.utils.split_input(sync_mods)
else:
sync_mods = []
if 'all' in sync_mods and sync_mods != ['all']:
# Prevent unnecessary extra syncing
sync_mods = ['all']
for module_type in sync_mods:
try:
__salt__['saltutil.sync_{0}'.format(module_type)](
saltenv=opts['environment']
)
except KeyError:
log.warning(
'Invalid custom module type \'%s\', ignoring',
module_type
)
try:
st_ = salt.state.HighState(opts,
pillar_override,

2
salt/runners/cloud.py
View File

@ -164,7 +164,7 @@ def action(func=None,
instances,
provider,
instance,
**salt.utils.args.clean_kwargs(**kwargs)
salt.utils.args.clean_kwargs(**kwargs)
)
except SaltCloudConfigError as err:
log.error(err)

2
salt/sdb/cache.py
View File

@ -69,7 +69,7 @@ def set_(key, value, service=None, profile=None): # pylint: disable=W0613
'''
key, profile = _parse_key(key, profile)
cache = salt.cache.Cache(__opts__)
cache.set(profile['bank'], key=key, value=value)
cache.store(profile['bank'], key, value)
return get(key, service, profile)

3
salt/utils/thin.py
View File

@ -13,6 +13,7 @@ import tarfile
import zipfile
import tempfile
import subprocess
import concurrent
# Import third party libs
import jinja2
@ -110,6 +111,8 @@ def get_tops(extra_mods='', so_mods=''):
os.path.dirname(msgpack.__file__),
]
if _six.PY2:
tops.append(os.path.dirname(concurrent.__file__))
tops.append(_six.__file__.replace('.pyc', '.py'))
tops.append(backports_abc.__file__.replace('.pyc', '.py'))

54
tests/integration/utils/test_thin.py Normal file
View File

@ -0,0 +1,54 @@
# -*- coding: utf-8 -*-
from __future__ import absolute_import
import tarfile
import tempfile
import subprocess
import sys
import os
from tests.support.unit import TestCase
import salt.utils
import salt.utils.thin
try:
import virtualenv
HAS_VENV = True
except ImportError:
HAS_VENV = False
class TestThinDir(TestCase):
def setUp(self):
self.tmpdir = tempfile.mkdtemp()
def tearDown(self):
salt.utils.rm_rf(self.tmpdir)
def test_thin_dir(self):
'''
Test the thin dir to make sure salt-call can run
Run salt call via a python in a new virtual environment to ensure
salt-call has all dependencies needed.
'''
venv_dir = os.path.join(self.tmpdir, 'venv')
virtualenv.create_environment(venv_dir)
salt.utils.thin.gen_thin(self.tmpdir)
thin_dir = os.path.join(self.tmpdir, 'thin')
thin_archive = os.path.join(thin_dir, 'thin.tgz')
tar = tarfile.open(thin_archive)
tar.extractall(thin_dir)
tar.close()
bins = 'bin'
if sys.platform == 'win32':
bins = 'Scripts'
cmd = [
os.path.join(venv_dir, bins, 'python'),
os.path.join(thin_dir, 'salt-call'),
'--version',
]
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = proc.communicate()
proc.wait()
assert proc.returncode == 0, (stdout, stderr, proc.returncode)

120
tests/unit/modules/test_state.py
View File

@ -16,6 +16,7 @@ from tests.support.mixins import LoaderModuleMockMixin
from tests.support.paths import TMP, TMP_CONF_DIR
from tests.support.unit import TestCase, skipIf
from tests.support.mock import (
Mock,
MagicMock,
patch,
mock_open,
@ -358,6 +359,7 @@ class StateTestCase(TestCase, LoaderModuleMockMixin):
'__utils__': utils,
'__salt__': {
'config.get': config.get,
'config.option': MagicMock(return_value=''),
}
},
config: {
@ -759,28 +761,25 @@ class StateTestCase(TestCase, LoaderModuleMockMixin):
"whitelist=sls1.sls",
pillar="A")
mock = MagicMock(return_value=True)
with patch.dict(state.__salt__,
{'config.option': mock}):
mock = MagicMock(return_value="A")
mock = MagicMock(return_value="A")
with patch.object(state, '_filter_running',
mock):
mock = MagicMock(return_value=True)
with patch.object(state, '_filter_running',
mock):
mock = MagicMock(return_value=True)
with patch.object(state, '_filter_running',
with patch.object(salt.payload, 'Serial',
mock):
mock = MagicMock(return_value=True)
with patch.object(salt.payload, 'Serial',
mock):
with patch.object(os.path,
'join', mock):
with patch.object(
state,
'_set'
'_retcode',
mock):
self.assertTrue(state.
highstate
(arg))
with patch.object(os.path,
'join', mock):
with patch.object(
state,
'_set'
'_retcode',
mock):
self.assertTrue(state.
highstate
(arg))
def test_clear_request(self):
'''
@ -921,17 +920,11 @@ class StateTestCase(TestCase, LoaderModuleMockMixin):
MockState.HighState.flag = False
mock = MagicMock(return_value=True)
with patch.dict(state.__salt__,
{'config.option':
mock}):
mock = MagicMock(return_value=
True)
with patch.object(
state,
'_filter_'
'running',
mock):
self.sub_test_sls()
with patch.object(state,
'_filter_'
'running',
mock):
self.sub_test_sls()
def test_get_test_value(self):
'''
@ -1014,6 +1007,75 @@ class StateTestCase(TestCase, LoaderModuleMockMixin):
None,
True))
def test_sls_sync(self):
'''
Test test.sls with the sync argument
We're only mocking the sync functions we expect to sync. If any other
sync functions are run then they will raise a KeyError, which we want
as it will tell us that we are syncing things we shouldn't.
'''
mock_empty_list = MagicMock(return_value=[])
with patch.object(state, 'running', mock_empty_list), \
patch.object(state, '_disabled', mock_empty_list), \
patch.object(state, '_get_pillar_errors', mock_empty_list):
sync_mocks = {
'saltutil.sync_modules': Mock(),
'saltutil.sync_states': Mock(),
}
with patch.dict(state.__salt__, sync_mocks):
state.sls('foo', sync_mods='modules,states')
for key in sync_mocks:
call_count = sync_mocks[key].call_count
expected = 1
assert call_count == expected, \
'{0} called {1} time(s) (expected: {2})'.format(
key, call_count, expected
)
# Test syncing all
sync_mocks = {'saltutil.sync_all': Mock()}
with patch.dict(state.__salt__, sync_mocks):
state.sls('foo', sync_mods='all')
for key in sync_mocks:
call_count = sync_mocks[key].call_count
expected = 1
assert call_count == expected, \
'{0} called {1} time(s) (expected: {2})'.format(
key, call_count, expected
)
# sync_mods=True should be interpreted as sync_mods=all
sync_mocks = {'saltutil.sync_all': Mock()}
with patch.dict(state.__salt__, sync_mocks):
state.sls('foo', sync_mods=True)
for key in sync_mocks:
call_count = sync_mocks[key].call_count
expected = 1
assert call_count == expected, \
'{0} called {1} time(s) (expected: {2})'.format(
key, call_count, expected
)
# Test syncing all when "all" is passed along with module types.
# This tests that we *only* run a sync_all and avoid unnecessary
# extra syncing.
sync_mocks = {'saltutil.sync_all': Mock()}
with patch.dict(state.__salt__, sync_mocks):
state.sls('foo', sync_mods='modules,all')
for key in sync_mocks:
call_count = sync_mocks[key].call_count
expected = 1
assert call_count == expected, \
'{0} called {1} time(s) (expected: {2})'.format(
key, call_count, expected
)
def test_pkg(self):
'''
Test to execute a packaged state run