Merge pull request #38030 from rallytime/merge-develop

[develop] Merge forward from 2016.11 to develop

doc/ref/configuration/master.rst

@@ -2467,24 +2467,6 @@ Default: ``[]``
 
 There are additional details at :ref:`salt-pillars`
 
-.. conf_master:: pillar_roots_override_ext_pillar
-
-``pillar_roots_override_ext_pillar``
-------------------------------------
-
-.. versionadded:: 2016.11.0
-
-Default: ``False``
-
-This option allows for external pillar sources to be evaluated before
-:conf_master:`pillar_roots`, which means that values obtained from
-:conf_master:`pillar_roots` take precedence over those found from
-:conf_master:`ext_pillar` sources.
-
-.. code-block:: yaml
-
-    pillar_roots_override_ext_pillar: False
-
 .. conf_master:: ext_pillar_first
 
 ``ext_pillar_first``
@@ -2496,8 +2478,7 @@ Default: ``False``
 
 This option allows for external pillar sources to be evaluated before
 :conf_master:`pillar_roots`. This allows for targeting file system pillar from
-ext_pillar. Note that ext_pillar_first option is deprecated by
+ext_pillar.
-pillar_roots_override_ext_pillar option and will be removed in future releases.
 
 .. code-block:: yaml
 

doc/topics/releases/2016.11.0.rst

@@ -104,6 +104,9 @@ Additional Features
 
 - The ``junos`` state module is now available. It has all the functions
   that are present in the ``junos`` execution module.
+- The minion data cache is a pluggable data store now. It's configurable with
+  :conf_master:`cache` option. Default is ``localfs``.
+- User names in :conf_master:`client_acl` support glob matching now.
 
 New Top File Merging Strategy for States
 ========================================
@@ -245,7 +248,7 @@ Beacons Changes
 ===============
 
 - The ``loadavg`` beacon now outputs averages as integers instead of strings.
-  (Via :issuse:`31124`.)
+  (Via :issue:`31124`.)
 
 Runner Changes
 ==============
@@ -271,7 +274,7 @@ Pillar Changes
 Network Automation: NAPALM
 ==========================
 
-Beginning with Carbon, network automation is inclued by default in the core
+Beginning with 2016.11.0, network automation is inclued by default in the core
 of Salt. It is based on a the `NAPALM <https://github.com/napalm-automation/napalm>`_
 library and provides facilities to manage the configuration and retrieve data
 from network devices running widely used operating systems such: JunOS, IOS-XR,
@@ -295,7 +298,7 @@ In the current release, the following modules were included:
 - :mod:`Users management state <salt.states.netusers>`
 
 Junos Module Changes
-===================
+====================
 
 - The following new functionalities were added to the junos module
 
@@ -313,6 +316,36 @@ Returner Changes
   accept a `minions` keyword argument. All returners which ship with Salt
   have been modified to do so.
 
+Renderer Changes
+================
+
+Added the ability to restrict allowed renderers. Two new config parameters,
+:conf_master:`renderer_whitelist` and :conf_master:`renderer_blacklist` are
+introduced for this purpose.
+
+eAuth Changes
+=============
+
+- External auth modules' ``auth`` method can return an ACL list for the given
+  username instead of ``True``. This list should be in the same format as
+  described in the :doc:`eAuth documentation </topics/eauth/index>`. It will be
+  used for the user instead of one set in master config.
+
+  Example of the ``auth`` method return that allows a user to execute functions
+  in the ``test`` and ``network`` modules on the minions that match the ``web*``
+  target and allow access to ``wheel`` and ``runner`` modules:
+
+  .. code-block:: python
+
+      [{'web*': ['test.*',
+                 'network.*']},
+       '@wheel',
+       '@runner']
+
+- External auth is supported by :doc:`salt-run </ref/cli/salt-run>` and
+  :doc:`salt-key </ref/cli/salt-key>` now. Note that master must be started to
+  use them with eAuth.
+
 External Module Packaging
 =========================
 
@@ -516,7 +549,7 @@ General Deprecations
 - Deprecations in ``minion.py``:
 
   - The ``salt.minion.parse_args_and_kwargs`` function has been removed. Please
-  use the ``salt.minion.load_args_and_kwargs`` function instead.
+    use the ``salt.minion.load_args_and_kwargs`` function instead.
 
 Cloud Deprecations
 ------------------

doc/topics/tutorials/salt_bootstrap.rst

@@ -115,39 +115,39 @@ Install using curl
 
 Using ``curl`` to install latest development version from GitHub:
 
-.. code:: console
+.. code-block:: bash
 
-    curl -o bootstrap_salt.sh -L https://bootstrap.saltstack.com
+    curl -o bootstrap-salt.sh -L https://bootstrap.saltstack.com
-    sudo sh bootstrap_salt.sh git develop
+    sudo sh bootstrap-salt.sh git develop
 
 If you want to install a specific release version (based on the Git tags):
 
-.. code:: console
+.. code-block:: bash
 
-    curl -o bootstrap_salt.sh -L https://bootstrap.saltstack.com
+    curl -o bootstrap-salt.sh -L https://bootstrap.saltstack.com
-    sudo sh bootstrap_salt.sh git v2015.8.8
+    sudo sh bootstrap-salt.sh git v2015.8.8
 
 To install a specific branch from a Git fork:
 
-.. code:: console
+.. code-block:: bash
 
-    curl -o bootstrap_salt.sh -L https://bootstrap.saltstack.com
+    curl -o bootstrap-salt.sh -L https://bootstrap.saltstack.com
-    sudo sh bootstrap_salt.sh -g https://github.com/myuser/salt.git git mybranch
+    sudo sh bootstrap-salt.sh -g https://github.com/myuser/salt.git git mybranch
 
 If all you want is to install a ``salt-master`` using latest Git:
 
-.. code:: console
+.. code-block:: bash
 
-    curl -o bootstrap_salt.sh -L https://bootstrap.saltstack.com
+    curl -o bootstrap-salt.sh -L https://bootstrap.saltstack.com
-    sudo sh bootstrap_salt.sh -M -N git develop
+    sudo sh bootstrap-salt.sh -M -N git develop
 
 If your host has Internet access only via HTTP proxy:
 
-.. code:: console
+.. code-block:: bash
 
     PROXY='http://user:password@myproxy.example.com:3128'
-    curl -o bootstrap_salt.sh -L -x "$PROXY" https://bootstrap.saltstack.com
+    curl -o bootstrap-salt.sh -L -x "$PROXY" https://bootstrap.saltstack.com
-    sudo sh bootstrap_salt.sh -G -H "$PROXY" git
+    sudo sh bootstrap-salt.sh -G -H "$PROXY" git
 
 
 Install using wget
@@ -155,24 +155,24 @@ Install using wget
 
 Using ``wget`` to install your distribution's stable packages:
 
-.. code:: console
+.. code-block:: bash
 
-    wget -O bootstrap_salt.sh https://bootstrap.saltstack.com
+    wget -O bootstrap-salt.sh https://bootstrap.saltstack.com
-    sudo sh bootstrap_salt.sh
+    sudo sh bootstrap-salt.sh
 
 Downloading the script from develop branch:
 
 .. code-block:: bash
 
-    wget -O bootstrap_salt.sh https://bootstrap.saltstack.com/develop
+    wget -O bootstrap-salt.sh https://bootstrap.saltstack.com/develop
-    sudo sh bootstrap_salt.sh
+    sudo sh bootstrap-salt.sh
 
 Installing a specific version from git using ``wget``:
 
-.. code:: console
+.. code-block:: bash
 
-    wget -O bootstrap_salt.sh https://bootstrap.saltstack.com
+    wget -O bootstrap-salt.sh https://bootstrap.saltstack.com
-    sudo sh bootstrap_salt.sh -P git v2015.8.8
+    sudo sh bootstrap-salt.sh -P git v2015.8.8
 
 .. note::
 
@@ -185,17 +185,17 @@ Install using Python
 
 If you already have Python installed, ``python 2.6``, then it's as easy as:
 
-.. code:: console
+.. code-block:: bash
 
-    python -m urllib "https://bootstrap.saltstack.com" > bootstrap_salt.sh
+    python -m urllib "https://bootstrap.saltstack.com" > bootstrap-salt.sh
-    sudo sh bootstrap_salt.sh git develop
+    sudo sh bootstrap-salt.sh git develop
 
 All Python versions should support the following in-line code:
 
-.. code:: console
+.. code-block:: bash
 
-    python -c 'import urllib; print urllib.urlopen("https://bootstrap.saltstack.com").read()' > bootstrap_salt.sh
+    python -c 'import urllib; print urllib.urlopen("https://bootstrap.saltstack.com").read()' > bootstrap-salt.sh
-    sudo sh bootstrap_salt.sh git develop
+    sudo sh bootstrap-salt.sh git develop
 
 
 Install using fetch
@@ -204,26 +204,26 @@ Install using fetch
 On a FreeBSD base system you usually don't have either of the above binaries available. You **do**
 have ``fetch`` available though:
 
-.. code:: console
+.. code-block:: bash
 
-  fetch -o bootstrap_salt.sh https://bootstrap.saltstack.com
+  fetch -o bootstrap-salt.sh https://bootstrap.saltstack.com
-  sudo sh bootstrap_salt.sh
+  sudo sh bootstrap-salt.sh
 
 If you have any SSL issues install ``ca_root_nssp``:
 
-.. code:: console
+.. code-block:: bash
 
    pkg install ca_root_nssp
 
 And either copy the certificates to the place where fetch can find them:
 
-.. code:: console
+.. code-block:: bash
 
    cp /usr/local/share/certs/ca-root-nss.crt /etc/ssl/cert.pem
 
 Or link them to the right place:
 
-.. code:: console
+.. code-block:: bash
 
    ln -s /usr/local/share/certs/ca-root-nss.crt /etc/ssl/cert.pem
 

salt/modules/dockerng.py

@@ -5889,7 +5889,7 @@ def sls_build(name, base='opensuse/python', mods=None, saltenv='base',
     # start a new container
     ret = __salt__['dockerng.create'](image=base,
                                       name=name,
-                                      cmd='/usr/bin/sleep infinity',
+                                      cmd='sleep infinity',
                                       interactive=True, tty=True)
     id_ = ret['Id']
     try:

salt/modules/postgres.py

@@ -26,6 +26,9 @@ Module to provide Postgres compatibility to salt.
     of the postgres bin's path to the relevant minion for this module::
 
         postgres.pg_bin: '/usr/pgsql-9.5/bin/'
+
+:note: Older versions of Salt had a bug where postgres.bins_dir was used
+    instead of postgres.pg_bin. You should upgrade this as soon as possible.
 '''
 
 # This pylint error is popping up where there are no colons?
@@ -51,7 +54,7 @@ except ImportError:
 import salt.utils
 import salt.utils.files
 import salt.utils.itertools
-from salt.exceptions import SaltInvocationError
+from salt.exceptions import CommandExecutionError, SaltInvocationError
 
 # Import 3rd-party libs
 import salt.ext.six as six
@@ -113,9 +116,10 @@ _PRIVILEGE_TYPE_MAP = {
 
 def __virtual__():
     '''
-    Only load this module if the psql and initdb bin exist
+    Only load this module if the psql bin exist.
+    initdb bin might also be used, but its presence will be detected on runtime.
     '''
-    utils = ['psql', 'initdb']
+    utils = ['psql']
     if not HAS_CSV:
         return False
     for util in utils:
@@ -131,11 +135,21 @@ def _find_pg_binary(util):
 
     Helper function to locate various psql related binaries
     '''
-    pg_bin_dir = __salt__['config.option']('postgres.bins_dir')
+    pg_bin_dir = __salt__['config.option']('postgres.pg_bin')
+
+    if not pg_bin_dir:  # Fallback to incorrectly-documented setting
+        pg_bin_dir = __salt__['config.option']('postgres.bins_dir')
+        if pg_bin_dir:
+            salt.utils.warn_until(
+                'Oxygen',
+                'Using \'postgres.bins_dir\' is not officially supported and '
+                'only exists as a workaround. Please replace this in your '
+                'configuration with \'postgres.pg_bin\'.')
+
     util_bin = salt.utils.which(util)
     if not util_bin:
         if pg_bin_dir:
-            return os.path.join(pg_bin_dir, util)
+            return salt.utils.which(os.path.join(pg_bin_dir, util))
     else:
         return util_bin
 
@@ -211,6 +225,8 @@ def _run_initdb(name,
     if user is None:
         user = runas
     _INITDB_BIN = _find_pg_binary('initdb')
+    if not _INITDB_BIN:
+        raise CommandExecutionError('initdb executable not found.')
     cmd = [
         _INITDB_BIN,
         '--pgdata={0}'.format(name),

salt/pillar/__init__.py

@@ -784,27 +784,16 @@ class Pillar(object):
         '''
         top, top_errors = self.get_top()
         if ext:
-            if self.opts.get('pillar_roots_override_ext_pillar', False) or self.opts.get('ext_pillar_first', False):
+            if self.opts.get('ext_pillar_first', False):
-                salt.utils.warn_until('Nitrogen',
-                     'The \'ext_pillar_first\' option has been deprecated and '
-                     'replaced by \'pillar_roots_override_ext_pillar\'.'
-                )
                 self.opts['pillar'], errors = self.ext_pillar({}, pillar_dirs)
                 self.rend = salt.loader.render(self.opts, self.functions)
                 matches = self.top_matches(top)
                 pillar, errors = self.render_pillar(matches, errors=errors)
-                if self.opts.get('pillar_roots_override_ext_pillar', False):
+                pillar = merge(self.opts['pillar'],
-                    pillar = merge(self.opts['pillar'],
+                               pillar,
-                                   pillar,
+                               self.merge_strategy,
-                                   self.merge_strategy,
+                               self.opts.get('renderer', 'yaml'),
-                                   self.opts.get('renderer', 'yaml'),
+                               self.opts.get('pillar_merge_lists', False))
-                                   self.opts.get('pillar_merge_lists', False))
-                else:
-                    pillar = merge(pillar,
-                                   self.opts['pillar'],
-                                   self.merge_strategy,
-                                   self.opts.get('renderer', 'yaml'),
-                                   self.opts.get('pillar_merge_lists', False))
             else:
                 matches = self.top_matches(top)
                 pillar, errors = self.render_pillar(matches)

salt/states/archive.py

@@ -12,6 +12,7 @@ import logging
 import os
 import re
 import shlex
+import shutil
 import stat
 import tarfile
 from contextlib import closing
@@ -140,6 +141,7 @@ def extracted(name,
               enforce_toplevel=True,
               enforce_ownership_on=None,
               archive_format=None,
+              overwrite=False,
               **kwargs):
     '''
     .. versionadded:: 2014.1.0
@@ -497,6 +499,11 @@ def extracted(name,
     .. _zipfile: https://docs.python.org/2/library/zipfile.html
     .. _xz-utils: http://tukaani.org/xz/
 
+    overwrite
+        If archive was already extracted, then setting this to True will
+        extract it all over again.
+        **WARNING: This operation will flush clean all the previous content, if exists!**
+
     **Examples**
 
     1. tar with lmza (i.e. xz) compression:
@@ -545,10 +552,9 @@ def extracted(name,
         ret['comment'] = '{0} is not an absolute path'.format(name)
         return ret
     else:
-        if name is None:
+        if not name:
-            # Only way this happens is if some doofus specifies "- name: None"
+            # Empty name, like None, '' etc.
-            # in their SLS file. Prevent tracebacks by failing gracefully.
+            ret['comment'] = 'Name of the directory path needs to be specified'
-            ret['comment'] = 'None is not a valid directory path'
             return ret
         # os.path.isfile() returns False when there is a trailing slash, hence
         # our need for first stripping the slash and then adding it back later.
@@ -898,7 +904,20 @@ def extracted(name,
     # already need to catch an OSError to cover edge cases where the minion is
     # running as a non-privileged user and is trying to check for the existence
     # of a path to which it does not have permission.
-    extraction_needed = False
+
+    extraction_needed = overwrite
+
+    if extraction_needed:
+        destination = os.path.join(name, contents['top_level_dirs'][0])
+        if os.path.exists(destination):
+            try:
+                shutil.rmtree(destination)
+            except OSError as err:
+                ret['comment'] = 'Error removing destination directory ' \
+                                 '"{0}": {1}'.format(destination, err)
+                ret['result'] = False
+                return ret
+
     try:
         if_missing_path_exists = os.path.exists(if_missing)
     except TypeError:

tests/unit/modules/dockerng_test.py

@@ -670,7 +670,7 @@ class DockerngTestCase(TestCase):
                 mods='foo',
             )
         docker_create_mock.assert_called_once_with(
-            cmd='/usr/bin/sleep infinity',
+            cmd='sleep infinity',
             image='opensuse/python', interactive=True, name='foo', tty=True)
         docker_start_mock.assert_called_once_with('ID')
         docker_sls_mock.assert_called_once_with('ID', 'foo', 'base')