diff --git a/doc/faq.rst b/doc/faq.rst index 28c298cda3..2894de2e41 100644 --- a/doc/faq.rst +++ b/doc/faq.rst @@ -148,22 +148,23 @@ Why aren't my custom modules/states/etc. available on my Minions? ----------------------------------------------------------------- Custom modules are synced to Minions when -:mod:`saltutil.sync_modules `, -or :mod:`saltutil.sync_all ` is run. -Custom modules are also synced by :mod:`state.apply` when run without -any arguments. +:py:func:`saltutil.sync_modules `, +or :py:func:`saltutil.sync_all ` is run. +Similarly, custom states are synced to Minions when :py:func:`saltutil.sync_states +`, or :py:func:`saltutil.sync_all +` is run. -Similarly, custom states are synced to Minions -when :mod:`state.apply `, -:mod:`saltutil.sync_states `, or -:mod:`saltutil.sync_all ` is run. +They are both also synced when a :ref:`highstate ` is +triggered. -Custom states are also synced by :mod:`state.apply` -when run without any arguments. +As of the Fluorine release, as well as 2017.7.7 and 2018.3.2 in their +respective release cycles, the ``sync`` argument to :py:func:`state.apply +`/:py:func:`state.sls ` can +be used to sync custom types when running individual SLS files. Other custom types (renderers, outputters, etc.) have similar behavior, see the -documentation for the :mod:`saltutil ` module for more +documentation for the :py:func:`saltutil ` module for more information. :ref:`This reactor example ` can be used to automatically diff --git a/doc/topics/releases/2017.7.7.rst b/doc/topics/releases/2017.7.7.rst index 7fc4a2bffc..82cc37be1c 100644 --- a/doc/topics/releases/2017.7.7.rst +++ b/doc/topics/releases/2017.7.7.rst @@ -5,23 +5,64 @@ In Progress: Salt 2017.7.7 Release Notes Version 2017.7.7 is an **unreleased** bugfix release for :ref:`2017.7.0 `. This release is still in progress and has not been released yet. -The ``2017.7.7`` release contains only a single fix for Issue `#48038`_, which -is a critical bug that occurs in a multi-syndic setup where the same job is run -multiple times on a minion. +The ``2017.7.7`` release contains only a small number of fixes, which are detailed +below. + +This release fixes two critical issues. + +The first is Issue `#48038`_, which is a critical bug that occurs in a multi-syndic +setup where the same job is run multiple times on a minion. + +The second issue is `#48130`_. This bug appears in certain setups where the Master +reports a Minion time-out, even though the job is still running on the Minion. + +Both of these issues have been fixed with this release. Statistics ========== -- Total Merges: **1** -- Total Issue References: **1** -- Total PR References: **2** +- Total Merges: **5** +- Total Issue References: **2** +- Total PR References: **6** -- Contributors: **2** (`garethgreenaway`_, `rallytime`_) +- Contributors: **3** (`garethgreenaway`_, `gtmanfred`_, `rallytime`_) Changelog for v2017.7.6..v2017.7.7 ================================== -*Generated at: 2018-06-14 15:43:34 UTC* +*Generated at: 2018-06-17 19:26:52 UTC* + +* **ISSUE** `#48130`_: (`rmarchei`_) Minion timeouts with 2018.3.1 (refs: `#48157`_) + +* **PR** `#48157`_: (`gtmanfred`_) always listen when gathering job info + @ *2018-06-17 19:04:09 UTC* + + * 8af4452134 Merge pull request `#48157`_ from gtmanfred/2017.7.7 + + * d8209e8a40 always listen when gathering job info + +* **PR** `#48140`_: (`rallytime`_) Update man pages for 2017.7.7 + @ *2018-06-14 21:22:43 UTC* + + * b98c52ee51 Merge pull request `#48140`_ from rallytime/man-pages-2017.7.7 + + * 8893bf0d4c Update man pages for 2017.7.7 + +* **PR** `#48136`_: (`gtmanfred`_) [2017.7.7] bootstrap kitchen branch tests with 2017.7.6 + @ *2018-06-14 21:20:16 UTC* + + * baa0363336 Merge pull request `#48136`_ from gtmanfred/2017.7.7 + + * fce1c31146 bootstrap kitchen branch tests with 2017.7.6 + +* **PR** `#48134`_: (`rallytime`_) Add release notes file for 2017.7.7 + @ *2018-06-14 16:31:34 UTC* + + * b0ba08f4d9 Merge pull request `#48134`_ from rallytime/release-notes-2017.7.7 + + * 217005b8f1 Add missing `v` for tag reference + + * d53569d1e3 Add release notes file for 2017.7.7 * **ISSUE** `#48038`_: (`austinpapp`_) jobs are not dedup'ing minion side (refs: `#48075`_) @@ -37,6 +78,13 @@ Changelog for v2017.7.6..v2017.7.7 .. _`#48038`: https://github.com/saltstack/salt/issues/48038 .. _`#48075`: https://github.com/saltstack/salt/pull/48075 .. _`#48098`: https://github.com/saltstack/salt/pull/48098 +.. _`#48130`: https://github.com/saltstack/salt/issues/48130 +.. _`#48134`: https://github.com/saltstack/salt/pull/48134 +.. _`#48136`: https://github.com/saltstack/salt/pull/48136 +.. _`#48140`: https://github.com/saltstack/salt/pull/48140 +.. _`#48157`: https://github.com/saltstack/salt/pull/48157 .. _`austinpapp`: https://github.com/austinpapp .. _`garethgreenaway`: https://github.com/garethgreenaway +.. _`gtmanfred`: https://github.com/gtmanfred .. _`rallytime`: https://github.com/rallytime +.. _`rmarchei`: https://github.com/rmarchei diff --git a/doc/topics/tutorials/pillar.rst b/doc/topics/tutorials/pillar.rst index 3ec2c1ddea..a75b32c237 100644 --- a/doc/topics/tutorials/pillar.rst +++ b/doc/topics/tutorials/pillar.rst @@ -330,7 +330,13 @@ Nested pillar values can also be set via the command line: .. code-block:: bash - salt '*' state.sls my_sls_file pillar='{"foo": {"bar": "baz"}}' + salt '*' state.sls my_sls_file pillar='{"foo": {"bar": "baz"}}' + +Lists can be passed via command line pillar data as follows: + +.. code-block:: bash + + salt '*' state.sls my_sls_file pillar='{"some_list": ["foo", "bar", "baz"]}' .. note:: diff --git a/doc/topics/utils/index.rst b/doc/topics/utils/index.rst index fa01c3695a..2ea3564f9a 100644 --- a/doc/topics/utils/index.rst +++ b/doc/topics/utils/index.rst @@ -139,13 +139,18 @@ where it is necessary to invoke the same function from a custom :ref:`outputter `/returner, as well as an execution module. Utility modules placed in ``salt://_utils/`` will be synced to the minions when -any of the following Salt functions are called: +a :ref:`highstate ` is run, as well as when any of the +following Salt functions are called: -* :mod:`state.apply ` -* :mod:`saltutil.sync_utils ` -* :mod:`saltutil.sync_all ` +* :py:func:`saltutil.sync_utils ` +* :py:func:`saltutil.sync_all ` + +As of the Fluorine release, as well as 2017.7.7 and 2018.3.2 in their +respective release cycles, the ``sync`` argument to :py:func:`state.apply +`/:py:func:`state.sls ` can +be used to sync custom types when running individual SLS files. To sync to the Master, use either of the following: -* :mod:`saltutil.sync_utils ` -* :mod:`saltutil.sync_all ` +* :py:func:`saltutil.sync_utils ` +* :py:func:`saltutil.sync_all ` diff --git a/salt/client/ssh/__init__.py b/salt/client/ssh/__init__.py index 141e1c6850..71e6cc4165 100644 --- a/salt/client/ssh/__init__.py +++ b/salt/client/ssh/__init__.py @@ -1245,7 +1245,10 @@ ARGS = {10}\n'''.format(self.minion_config, shim_tmp_file.write(salt.utils.stringutils.to_bytes(cmd_str)) # Copy shim to target system, under $HOME/. - target_shim_file = '.{0}.{1}'.format(binascii.hexlify(os.urandom(6)), extension) + target_shim_file = '.{0}.{1}'.format( + binascii.hexlify(os.urandom(6)).decode('ascii'), + extension + ) if self.winrm: target_shim_file = saltwinshell.get_target_shim_file(self, target_shim_file) self.shell.send(shim_tmp_file.name, target_shim_file, makedirs=True) diff --git a/salt/modules/lxc.py b/salt/modules/lxc.py index ce7898b2c6..ab0c2dd1bc 100644 --- a/salt/modules/lxc.py +++ b/salt/modules/lxc.py @@ -2268,22 +2268,22 @@ def _change_state(cmd, # as te command itself mess with double forks; we must not # communicate with it, but just wait for the exit status pkwargs = {'python_shell': False, + 'redirect_stderr': True, 'with_communicate': with_communicate, 'use_vt': use_vt, 'stdin': stdin, - 'stdout': stdout, - 'stderr': stderr} + 'stdout': stdout} for i in [a for a in pkwargs]: val = pkwargs[i] if val is _marker: pkwargs.pop(i, None) - error = __salt__['cmd.run_stderr'](cmd, **pkwargs) + _cmdout = __salt__['cmd.run_all'](cmd, **pkwargs) - if error: + if _cmdout['retcode'] != 0: raise CommandExecutionError( 'Error changing state for container \'{0}\' using command ' - '\'{1}\': {2}'.format(name, cmd, error) + '\'{1}\': {2}'.format(name, cmd, _cmdout['stdout']) ) if expected is not None: # some commands do not wait, so we will diff --git a/salt/modules/state.py b/salt/modules/state.py index e61b73c509..c568a913d3 100644 --- a/salt/modules/state.py +++ b/salt/modules/state.py @@ -602,8 +602,7 @@ def template_str(tem, queue=False, **kwargs): return ret -def apply_(mods=None, - **kwargs): +def apply_(mods=None, **kwargs): ''' .. versionadded:: 2015.5.0 @@ -743,6 +742,22 @@ def apply_(mods=None, .. code-block:: bash salt '*' state.apply test localconfig=/path/to/minion.yml + + sync_mods + If specified, the desired custom module types will be synced prior to + running the SLS files: + + .. code-block:: bash + + salt '*' state.apply test sync_mods=states,modules + salt '*' state.apply test sync_mods=all + + .. note:: + This option is ignored when no SLS files are specified, as a + :ref:`highstate ` automatically syncs all custom + module types. + + .. versionadded:: 2017.7.8,2018.3.3,Fluorine ''' if mods: return sls(mods, **kwargs) @@ -1068,7 +1083,7 @@ def highstate(test=None, queue=False, **kwargs): return ret -def sls(mods, test=None, exclude=None, queue=False, **kwargs): +def sls(mods, test=None, exclude=None, queue=False, sync_mods=None, **kwargs): ''' Execute the states in one or more SLS files @@ -1160,6 +1175,17 @@ def sls(mods, test=None, exclude=None, queue=False, **kwargs): .. versionadded:: 2015.8.4 + sync_mods + If specified, the desired custom module types will be synced prior to + running the SLS files: + + .. code-block:: bash + + salt '*' state.sls test sync_mods=states,modules + salt '*' state.sls test sync_mods=all + + .. versionadded:: 2017.7.8,2018.3.3,Fluorine + CLI Example: .. code-block:: bash @@ -1223,6 +1249,28 @@ def sls(mods, test=None, exclude=None, queue=False, **kwargs): '{0}.cache.p'.format(kwargs.get('cache_name', 'highstate')) ) + if sync_mods is True: + sync_mods = ['all'] + if sync_mods is not None: + sync_mods = salt.utils.split_input(sync_mods) + else: + sync_mods = [] + + if 'all' in sync_mods and sync_mods != ['all']: + # Prevent unnecessary extra syncing + sync_mods = ['all'] + + for module_type in sync_mods: + try: + __salt__['saltutil.sync_{0}'.format(module_type)]( + saltenv=opts['environment'] + ) + except KeyError: + log.warning( + 'Invalid custom module type \'%s\', ignoring', + module_type + ) + try: st_ = salt.state.HighState(opts, pillar_override, diff --git a/salt/runners/cloud.py b/salt/runners/cloud.py index 9b84ccb1d6..6e32f8b82c 100644 --- a/salt/runners/cloud.py +++ b/salt/runners/cloud.py @@ -164,7 +164,7 @@ def action(func=None, instances, provider, instance, - **salt.utils.args.clean_kwargs(**kwargs) + salt.utils.args.clean_kwargs(**kwargs) ) except SaltCloudConfigError as err: log.error(err) diff --git a/salt/sdb/cache.py b/salt/sdb/cache.py index 12ed559329..765f4d0eef 100644 --- a/salt/sdb/cache.py +++ b/salt/sdb/cache.py @@ -69,7 +69,7 @@ def set_(key, value, service=None, profile=None): # pylint: disable=W0613 ''' key, profile = _parse_key(key, profile) cache = salt.cache.Cache(__opts__) - cache.set(profile['bank'], key=key, value=value) + cache.store(profile['bank'], key, value) return get(key, service, profile) diff --git a/salt/utils/thin.py b/salt/utils/thin.py index 7a44d0c993..5965f064a1 100644 --- a/salt/utils/thin.py +++ b/salt/utils/thin.py @@ -13,6 +13,7 @@ import tarfile import zipfile import tempfile import subprocess +import concurrent # Import third party libs import jinja2 @@ -110,6 +111,8 @@ def get_tops(extra_mods='', so_mods=''): os.path.dirname(msgpack.__file__), ] + if _six.PY2: + tops.append(os.path.dirname(concurrent.__file__)) tops.append(_six.__file__.replace('.pyc', '.py')) tops.append(backports_abc.__file__.replace('.pyc', '.py')) diff --git a/tests/integration/utils/test_thin.py b/tests/integration/utils/test_thin.py new file mode 100644 index 0000000000..02012b3785 --- /dev/null +++ b/tests/integration/utils/test_thin.py @@ -0,0 +1,54 @@ +# -*- coding: utf-8 -*- +from __future__ import absolute_import +import tarfile +import tempfile +import subprocess +import sys +import os + +from tests.support.unit import TestCase + +import salt.utils +import salt.utils.thin +try: + import virtualenv + HAS_VENV = True +except ImportError: + HAS_VENV = False + + +class TestThinDir(TestCase): + + def setUp(self): + self.tmpdir = tempfile.mkdtemp() + + def tearDown(self): + salt.utils.rm_rf(self.tmpdir) + + def test_thin_dir(self): + ''' + Test the thin dir to make sure salt-call can run + + Run salt call via a python in a new virtual environment to ensure + salt-call has all dependencies needed. + ''' + venv_dir = os.path.join(self.tmpdir, 'venv') + virtualenv.create_environment(venv_dir) + salt.utils.thin.gen_thin(self.tmpdir) + thin_dir = os.path.join(self.tmpdir, 'thin') + thin_archive = os.path.join(thin_dir, 'thin.tgz') + tar = tarfile.open(thin_archive) + tar.extractall(thin_dir) + tar.close() + bins = 'bin' + if sys.platform == 'win32': + bins = 'Scripts' + cmd = [ + os.path.join(venv_dir, bins, 'python'), + os.path.join(thin_dir, 'salt-call'), + '--version', + ] + proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + stdout, stderr = proc.communicate() + proc.wait() + assert proc.returncode == 0, (stdout, stderr, proc.returncode) diff --git a/tests/unit/modules/test_state.py b/tests/unit/modules/test_state.py index 7206539fe0..47b300bb0e 100644 --- a/tests/unit/modules/test_state.py +++ b/tests/unit/modules/test_state.py @@ -16,6 +16,7 @@ from tests.support.mixins import LoaderModuleMockMixin from tests.support.paths import TMP, TMP_CONF_DIR from tests.support.unit import TestCase, skipIf from tests.support.mock import ( + Mock, MagicMock, patch, mock_open, @@ -358,6 +359,7 @@ class StateTestCase(TestCase, LoaderModuleMockMixin): '__utils__': utils, '__salt__': { 'config.get': config.get, + 'config.option': MagicMock(return_value=''), } }, config: { @@ -759,28 +761,25 @@ class StateTestCase(TestCase, LoaderModuleMockMixin): "whitelist=sls1.sls", pillar="A") - mock = MagicMock(return_value=True) - with patch.dict(state.__salt__, - {'config.option': mock}): - mock = MagicMock(return_value="A") + mock = MagicMock(return_value="A") + with patch.object(state, '_filter_running', + mock): + mock = MagicMock(return_value=True) with patch.object(state, '_filter_running', mock): mock = MagicMock(return_value=True) - with patch.object(state, '_filter_running', + with patch.object(salt.payload, 'Serial', mock): - mock = MagicMock(return_value=True) - with patch.object(salt.payload, 'Serial', - mock): - with patch.object(os.path, - 'join', mock): - with patch.object( - state, - '_set' - '_retcode', - mock): - self.assertTrue(state. - highstate - (arg)) + with patch.object(os.path, + 'join', mock): + with patch.object( + state, + '_set' + '_retcode', + mock): + self.assertTrue(state. + highstate + (arg)) def test_clear_request(self): ''' @@ -921,17 +920,11 @@ class StateTestCase(TestCase, LoaderModuleMockMixin): MockState.HighState.flag = False mock = MagicMock(return_value=True) - with patch.dict(state.__salt__, - {'config.option': - mock}): - mock = MagicMock(return_value= - True) - with patch.object( - state, - '_filter_' - 'running', - mock): - self.sub_test_sls() + with patch.object(state, + '_filter_' + 'running', + mock): + self.sub_test_sls() def test_get_test_value(self): ''' @@ -1014,6 +1007,75 @@ class StateTestCase(TestCase, LoaderModuleMockMixin): None, True)) + def test_sls_sync(self): + ''' + Test test.sls with the sync argument + + We're only mocking the sync functions we expect to sync. If any other + sync functions are run then they will raise a KeyError, which we want + as it will tell us that we are syncing things we shouldn't. + ''' + mock_empty_list = MagicMock(return_value=[]) + with patch.object(state, 'running', mock_empty_list), \ + patch.object(state, '_disabled', mock_empty_list), \ + patch.object(state, '_get_pillar_errors', mock_empty_list): + + sync_mocks = { + 'saltutil.sync_modules': Mock(), + 'saltutil.sync_states': Mock(), + } + with patch.dict(state.__salt__, sync_mocks): + state.sls('foo', sync_mods='modules,states') + + for key in sync_mocks: + call_count = sync_mocks[key].call_count + expected = 1 + assert call_count == expected, \ + '{0} called {1} time(s) (expected: {2})'.format( + key, call_count, expected + ) + + # Test syncing all + sync_mocks = {'saltutil.sync_all': Mock()} + with patch.dict(state.__salt__, sync_mocks): + state.sls('foo', sync_mods='all') + + for key in sync_mocks: + call_count = sync_mocks[key].call_count + expected = 1 + assert call_count == expected, \ + '{0} called {1} time(s) (expected: {2})'.format( + key, call_count, expected + ) + + # sync_mods=True should be interpreted as sync_mods=all + sync_mocks = {'saltutil.sync_all': Mock()} + with patch.dict(state.__salt__, sync_mocks): + state.sls('foo', sync_mods=True) + + for key in sync_mocks: + call_count = sync_mocks[key].call_count + expected = 1 + assert call_count == expected, \ + '{0} called {1} time(s) (expected: {2})'.format( + key, call_count, expected + ) + + # Test syncing all when "all" is passed along with module types. + # This tests that we *only* run a sync_all and avoid unnecessary + # extra syncing. + sync_mocks = {'saltutil.sync_all': Mock()} + with patch.dict(state.__salt__, sync_mocks): + state.sls('foo', sync_mods='modules,all') + + for key in sync_mocks: + call_count = sync_mocks[key].call_count + expected = 1 + assert call_count == expected, \ + '{0} called {1} time(s) (expected: {2})'.format( + key, call_count, expected + ) + def test_pkg(self): ''' Test to execute a packaged state run