Merge pull request #48161 from rallytime/merge-2018.3

[2018.3] Merge forward from 2017.7 to 2018.3

.kitchen.yml

@@ -43,23 +43,20 @@ provisioner:
     repo: git
     testingdir: /testing
   salt_copy_filter:
-    - .bundle
-    - .kitchen
-    - .kitchen.yml
-    - artifacts
-    - Gemfile
-    - Gemfile.lock
-    - README.rst
-    - .travis.yml
-    - '*.pyc'
     - __pycache__
+    - '*.pyc'
+    - .bundle
+    - .tox
+    - .kitchen
+    - artifacts
+    - Gemfile.lock
   state_top:
     base:
       "os:Windows":
         - match: grain
         - prep_windows
       "*":
-        - git.salt
+        - <%= ENV['KITCHEN_STATE'] || 'git.salt' %>
   pillars:
     top.sls:
       base:

doc/topics/releases/2017.7.6.rst

@@ -2,9 +2,7 @@
 Salt 2017.7.6 Release Notes
 ===========================
 
-Version 2017.7.6 is a bugfix release for :ref:`2017.7.0
-<release-2017-7-0>`.
-
+Version 2017.7.6 is a bugfix release for :ref:`2017.7.0 <release-2017-7-0>`.
 
 Statistics
 ==========
@@ -15,6 +13,15 @@ Statistics
 
 - Contributors: **47** (`Ch3LL`_, `DmitryKuzmenko`_, `GwiYeong`_, `Quarky9`_, `RichardW42`_, `UtahDave`_, `amaclean199`_, `arif-ali`_, `baniobloom`_, `bdrung`_, `benediktwerner`_, `bmiguel-teixeira`_, `cachedout`_, `dafenko`_, `damon-atkins`_, `dwoz`_, `ezh`_, `folti`_, `fpicot`_, `frogunder`_, `garethgreenaway`_, `gtmanfred`_, `isbm`_, `jeroennijhof`_, `jfindlay`_, `jfoboss`_, `kstreee`_, `lomeroe`_, `mattp-`_, `meaksh`_, `mirceaulinic`_, `myinitialsarepm`_, `mzbroch`_, `nages13`_, `paclat`_, `pcjeff`_, `pruiz`_, `psyer`_, `rallytime`_, `s0undt3ch`_, `skizunov`_, `smitty42`_, `terminalmage`_, `twangboy`_, `vutny`_, `yagnik`_, `yannj-fr`_)
 
+Tornado 5.0 Support for Python 2 Only
+-------------------------------------
+
+Tornado 5.0 moves to using asyncio for all python3 versions.  Because of this
+and changes in asyncio between python 3.4 and 3.5 to only be able to use one
+ioloop, which requires some rearchitecting, support for tornado 5.0 and python3
+versions of salt has been delayed to a later release.
+
+For now, to use tornado 5.0, the python 2 version of salt must be used.
 
 Tornado 5.0 Support for Python 2 Only
 -------------------------------------

doc/topics/releases/2017.7.7.rst

@@ -5,15 +5,38 @@ In Progress: Salt 2017.7.7 Release Notes
 Version 2017.7.7 is an **unreleased** bugfix release for :ref:`2017.7.0 <release-2017-7-0>`.
 This release is still in progress and has not been released yet.
 
-New win_snmp behavior
-=====================
+The ``2017.7.7`` release contains only a single fix for Issue `#48038`_, which
+is a critical bug that occurs in a multi-syndic setup where the same job is run
+multiple times on a minion.
 
-- :py:func:`win_snmp.get_community_names
-  <salt.modules.win_snmp.get_community_names>` now returns the SNMP settings
-  actually in effect on the box. If settings are managed via GroupPolicy, those
-  settings will be returned. Otherwise, normal settings are returned.
+Statistics
+==========
 
-- :py:func:`win_snmp.set_community_names
-  <salt.modules.win_snmp.set_community_names>` now raises an error when SNMP
-  settings are being managed by GroupPolicy.
+- Total Merges: **1**
+- Total Issue References: **1**
+- Total PR References: **2**
 
+- Contributors: **2** (`garethgreenaway`_, `rallytime`_)
+
+Changelog for v2017.7.6..v2017.7.7
+==================================
+
+*Generated at: 2018-06-14 15:43:34 UTC*
+
+* **ISSUE** `#48038`_: (`austinpapp`_) jobs are not dedup'ing minion side (refs: `#48075`_)
+
+* **PR** `#48098`_: (`rallytime`_) Back-port `#48075`_ to 2017.7.7
+  @ *2018-06-14 12:53:42 UTC*
+
+  * **PR** `#48075`_: (`garethgreenaway`_) [2017.7] Ensure that the shared list of jids is passed (refs: `#48098`_)
+
+  * 084de927fe Merge pull request `#48098`_ from rallytime/bp-48075-2017.7.7
+
+  * e4e62e8b3a Ensure that the shared list of jids is passed when creating the Minion.  Fixes an issue when minions are pointed at multiple syndics.
+
+.. _`#48038`: https://github.com/saltstack/salt/issues/48038
+.. _`#48075`: https://github.com/saltstack/salt/pull/48075
+.. _`#48098`: https://github.com/saltstack/salt/pull/48098
+.. _`austinpapp`: https://github.com/austinpapp
+.. _`garethgreenaway`: https://github.com/garethgreenaway
+.. _`rallytime`: https://github.com/rallytime

doc/topics/releases/2017.7.8.rst

@@ -0,0 +1,18 @@
+========================================
+In Progress: Salt 2017.7.8 Release Notes
+========================================
+
+Version 2017.7.8 is an **unreleased** bugfix release for :ref:`2017.7.0 <release-2017-7-0>`.
+This release is still in progress and has not been released yet.
+
+New win_snmp behavior
+=====================
+
+- :py:func:`win_snmp.get_community_names
+  <salt.modules.win_snmp.get_community_names>` now returns the SNMP settings
+  actually in effect on the box. If settings are managed via GroupPolicy, those
+  settings will be returned. Otherwise, normal settings are returned.
+
+- :py:func:`win_snmp.set_community_names
+  <salt.modules.win_snmp.set_community_names>` now raises an error when SNMP
+  settings are being managed by GroupPolicy.

doc/topics/tutorials/gitfs.rst

@@ -597,9 +597,8 @@ overrides all levels below it):
    .. code-block:: yaml
 
        gitfs_saltenv:
-         - saltenv:
-           - dev:
-             - mountpoint: salt://bar
+         - dev:
+           - mountpoint: salt://bar
 
 3. Per-remote configuration parameter
 

pytest.ini

@@ -1,4 +0,0 @@
-[pytest]
-addopts = --ssh-tests -ra -sv
-testpaths = tests
-norecursedirs = tests/kitchen

requirements/pytest.txt

@@ -1,3 +1,4 @@
 pytest>=3.5.0
 pytest-helpers-namespace
 pytest-tempdir
+pytest-cov

salt/minion.py

@@ -1073,7 +1073,7 @@ class Minion(MinionBase):
         # Flag meaning minion has finished initialization including first connect to the master.
         # True means the Minion is fully functional and ready to handle events.
         self.ready = False
-        self.jid_queue = jid_queue or []
+        self.jid_queue = [] if jid_queue is None else jid_queue
         self.periodic_callbacks = {}
 
         if io_loop is None:

salt/modules/timezone.py

@@ -495,61 +495,67 @@ def set_hwclock(clock):
 
         salt '*' timezone.set_hwclock UTC
     '''
-    os_family = __grains__['os_family']
-    if os_family in ('AIX', 'NILinuxRT'):
-        if clock.lower() != 'utc':
-            raise SaltInvocationError(
-                'UTC is the only permitted value'
-            )
-        return True
 
-    timezone = get_zone()
-
-    if 'Solaris' in __grains__['os_family']:
-        if clock.lower() not in ('localtime', 'utc'):
-            raise SaltInvocationError(
-                'localtime and UTC are the only permitted values'
-            )
-        if 'sparc' in __grains__['cpuarch']:
-            raise SaltInvocationError(
-                'UTC is the only choice for SPARC architecture'
-            )
-        cmd = ['rtc', '-z', 'GMT' if clock.lower() == 'utc' else timezone]
-        return __salt__['cmd.retcode'](cmd, python_shell=False) == 0
-
-    zonepath = '/usr/share/zoneinfo/{0}'.format(timezone)
-
-    if not os.path.exists(zonepath):
-        raise CommandExecutionError(
-            'Zone \'{0}\' does not exist'.format(zonepath)
-        )
-
-    os.unlink('/etc/localtime')
-    os.symlink(zonepath, '/etc/localtime')
-
-    if 'Arch' in __grains__['os_family']:
-        cmd = ['timezonectl', 'set-local-rtc',
+    if salt.utils.path.which('timedatectl'):
+        cmd = ['timedatectl', 'set-local-rtc',
                'true' if clock == 'localtime' else 'false']
         return __salt__['cmd.retcode'](cmd, python_shell=False) == 0
-    elif 'RedHat' in __grains__['os_family']:
-        __salt__['file.sed'](
-            '/etc/sysconfig/clock', '^ZONE=.*', 'ZONE="{0}"'.format(timezone))
-    elif 'Suse' in __grains__['os_family']:
-        __salt__['file.sed'](
-            '/etc/sysconfig/clock', '^TIMEZONE=.*', 'TIMEZONE="{0}"'.format(timezone))
-    elif 'Debian' in __grains__['os_family']:
-        if clock == 'UTC':
-            __salt__['file.sed']('/etc/default/rcS', '^UTC=.*', 'UTC=yes')
-        elif clock == 'localtime':
-            __salt__['file.sed']('/etc/default/rcS', '^UTC=.*', 'UTC=no')
-    elif 'Gentoo' in __grains__['os_family']:
-        if clock not in ('UTC', 'localtime'):
-            raise SaltInvocationError(
-                'Only \'UTC\' and \'localtime\' are allowed'
+    else:
+        os_family = __grains__['os_family']
+        if os_family in ('AIX', 'NILinuxRT'):
+            if clock.lower() != 'utc':
+                raise SaltInvocationError(
+                    'UTC is the only permitted value'
+                )
+            return True
+
+        timezone = get_zone()
+
+        if 'Solaris' in __grains__['os_family']:
+            if clock.lower() not in ('localtime', 'utc'):
+                raise SaltInvocationError(
+                    'localtime and UTC are the only permitted values'
+                )
+            if 'sparc' in __grains__['cpuarch']:
+                raise SaltInvocationError(
+                    'UTC is the only choice for SPARC architecture'
+                )
+            cmd = ['rtc', '-z', 'GMT' if clock.lower() == 'utc' else timezone]
+            return __salt__['cmd.retcode'](cmd, python_shell=False) == 0
+
+        zonepath = '/usr/share/zoneinfo/{0}'.format(timezone)
+
+        if not os.path.exists(zonepath):
+            raise CommandExecutionError(
+                'Zone \'{0}\' does not exist'.format(zonepath)
             )
-        if clock == 'localtime':
-            clock = 'local'
-        __salt__['file.sed'](
-            '/etc/conf.d/hwclock', '^clock=.*', 'clock="{0}"'.format(clock))
+
+        os.unlink('/etc/localtime')
+        os.symlink(zonepath, '/etc/localtime')
+
+        if 'Arch' in __grains__['os_family']:
+            cmd = ['timezonectl', 'set-local-rtc',
+                   'true' if clock == 'localtime' else 'false']
+            return __salt__['cmd.retcode'](cmd, python_shell=False) == 0
+        elif 'RedHat' in __grains__['os_family']:
+            __salt__['file.sed'](
+                '/etc/sysconfig/clock', '^ZONE=.*', 'ZONE="{0}"'.format(timezone))
+        elif 'Suse' in __grains__['os_family']:
+            __salt__['file.sed'](
+                '/etc/sysconfig/clock', '^TIMEZONE=.*', 'TIMEZONE="{0}"'.format(timezone))
+        elif 'Debian' in __grains__['os_family']:
+            if clock == 'UTC':
+                __salt__['file.sed']('/etc/default/rcS', '^UTC=.*', 'UTC=yes')
+            elif clock == 'localtime':
+                __salt__['file.sed']('/etc/default/rcS', '^UTC=.*', 'UTC=no')
+        elif 'Gentoo' in __grains__['os_family']:
+            if clock not in ('UTC', 'localtime'):
+                raise SaltInvocationError(
+                    'Only \'UTC\' and \'localtime\' are allowed'
+                )
+            if clock == 'localtime':
+                clock = 'local'
+            __salt__['file.sed'](
+                '/etc/conf.d/hwclock', '^clock=.*', 'clock="{0}"'.format(clock))
 
     return True

salt/states/file.py

@@ -1251,7 +1251,7 @@ def _makedirs(name,
     Helper function for creating directories when the ``makedirs`` option is set
     to ``True``. Handles Unix and Windows based systems
 
-    .. versionadded:: 2017.7.7
+    .. versionadded:: 2017.7.8
 
     Args:
         name (str): The directory path to create

salt/states/github.py

@@ -715,8 +715,8 @@ def repo_present(
                         ret['result'] = None
                     else:
                         result = __salt__['github.add_team_repo'](name, team_name,
-                                                                  permission,
-                                                                  profile=profile)
+                                                                  profile=profile,
+                                                                  permission=permission)
                         if result:
                             ret['changes'][team_name] = team_change
                         else:

tests/support/case.py

@@ -489,7 +489,7 @@ class ShellCase(ShellTestCase, AdaptedConfigurationTestCaseMixin, ScriptPathMixi
         '''
         Execute salt
         '''
-        arg_str = '-c {0} {1}'.format(self.get_config_dir(), arg_str)
+        arg_str = '-c {0} -t {1} {2}'.format(self.get_config_dir(), timeout, arg_str)
         ret = self.run_script('salt',
                               arg_str,
                               with_retcode=with_retcode,

tests/unit/fileserver/test_roots.py

@@ -185,10 +185,15 @@ class RootsTest(TestCase, AdaptedConfigurationTestCaseMixin, LoaderModuleMockMix
         self.assertIn(UNICODE_DIRNAME, ret)
 
     def test_symlink_list(self):
-        if self.test_symlink_list_file_roots:
-            self.opts['file_roots'] = self.test_symlink_list_file_roots
-        ret = roots.symlink_list({'saltenv': 'base'})
-        self.assertDictEqual(ret, {'dest_sym': 'source_sym'})
+        orig_file_roots = self.opts['file_roots']
+        try:
+            if self.test_symlink_list_file_roots:
+                self.opts['file_roots'] = self.test_symlink_list_file_roots
+            ret = roots.symlink_list({'saltenv': 'base'})
+            self.assertDictEqual(ret, {'dest_sym': 'source_sym'})
+        finally:
+            if self.test_symlink_list_file_roots:
+                self.opts['file_roots'] = orig_file_roots
 
 
 class RootsLimitTraversalTest(TestCase, AdaptedConfigurationTestCaseMixin):

tests/unit/modules/test_timezone.py

@@ -325,6 +325,21 @@ class TimezoneModuleTestCase(TestCase, LoaderModuleMockMixin):
         with patch.dict(timezone.__grains__, {'os_family': ['AIX']}):
             assert timezone.get_hwclock() == hwclock
 
+    @skipIf(salt.utils.platform.is_windows(), 'os.symlink not available in Windows')
+    @patch('salt.utils.path.which', MagicMock(return_value=True))
+    def test_set_hwclock_timedatectl(self):
+        '''
+        Test set hwclock with timedatectl
+        :return:
+        '''
+        timezone.set_hwclock('UTC')
+        name, args, kwargs = timezone.__salt__['cmd.retcode'].mock_calls[0]
+        assert args == (['timedatectl', 'set-local-rtc', 'false'],)
+
+        timezone.set_hwclock('localtime')
+        name, args, kwargs = timezone.__salt__['cmd.retcode'].mock_calls[1]
+        assert args == (['timedatectl', 'set-local-rtc', 'true'],)
+
     @skipIf(salt.utils.platform.is_windows(), 'os.symlink not available in Windows')
     @patch('salt.utils.path.which', MagicMock(return_value=False))
     @patch('os.path.exists', MagicMock(return_value=True))

tox.ini

@@ -1,7 +1,12 @@
 [tox]
-envlist = py27,py34,py35,py36
+envlist = py27,py3
 
 [testenv]
 deps = -r{toxinidir}/requirements/tests.txt
-commands = pytest --rootdir {toxinidir} {posargs:--no-print-logs --run-destructive}
+commands = pytest --rootdir {toxinidir} {posargs}
 passenv = LANG HOME
+
+[pytest]
+addopts = --log-file /tmp/salt-runtests.log --no-print-logs --ssh-tests -ra -sv
+testpaths = tests
+norecursedirs = tests/kitchen