Improve pillar documentation (#38407)

* rename anchor

* Move pillar cache config option to different section, add an anchor

* Add links to master and minion config items

* Add documentation for pillar.item's "delimiter" argument

* Improve pillar documentation

This improves RST formatting, adding hyperlinks as well as adding
information on the difference between in-minion pillar data and
pillar data that is compiled on-demand.

* Add section on how pillar environments are handled

doc/ref/configuration/master.rst

@@ -2252,7 +2252,7 @@ exposed.
       - 'mail\d+.mydomain.tld'
 
 
-.. _pillar-configuration:
+.. _pillar-configuration-master:
 
 Pillar Configuration
 ====================

doc/ref/configuration/minion.rst

@@ -448,26 +448,6 @@ executed. By default this feature is disabled, to enable set cache_jobs to
 
     cache_jobs: False
 
-.. conf_minion:: minion_pillar_cache
-
-``minion_pillar_cache``
------------------------
-
-Default: ``False``
-
-The minion can locally cache rendered pillar data under
-:conf_minion:`cachedir`/pillar. This allows a temporarily disconnected minion
-to access previously cached pillar data by invoking salt-call with the --local
-and --pillar_root=:conf_minion:`cachedir`/pillar options. Before enabling this
-setting consider that the rendered pillar may contain security sensitive data.
-Appropriate access restrictions should be in place. By default the saved pillar
-data will be readable only by the user account running salt. By default this
-feature is disabled, to enable set minion_pillar_cache to ``True``.
-
-.. code-block:: yaml
-
-    minion_pillar_cache: False
-
 .. conf_minion:: grains
 
 ``grains``
@@ -1442,8 +1422,10 @@ sha512 are also supported.
     hash_type: md5
 
 
-Pillar Settings
+.. _pillar-configuration-minion:
-===============
+
+Pillar Configuration
+====================
 
 .. conf_minion:: pillar_roots
 
@@ -1484,6 +1466,28 @@ the environment setting, but for pillar instead of states.
 
     pillarenv: None
 
+.. conf_minion:: minion_pillar_cache
+
+``minion_pillar_cache``
+-----------------------
+
+.. versionadded:: 2016.3.0
+
+Default: ``False``
+
+The minion can locally cache rendered pillar data under
+:conf_minion:`cachedir`/pillar. This allows a temporarily disconnected minion
+to access previously cached pillar data by invoking salt-call with the --local
+and --pillar_root=:conf_minion:`cachedir`/pillar options. Before enabling this
+setting consider that the rendered pillar may contain security sensitive data.
+Appropriate access restrictions should be in place. By default the saved pillar
+data will be readable only by the user account running salt. By default this
+feature is disabled, to enable set minion_pillar_cache to ``True``.
+
+.. code-block:: yaml
+
+    minion_pillar_cache: False
+
 .. conf_minion:: file_recv_max_size
 
 ``file_recv_max_size``

doc/ref/pillar/index.rst

@@ -8,8 +8,14 @@ Pillars
 Salt includes a number of built-in external pillars, listed at
 :ref:`all-salt.pillars`.
 
-You may also wish to look at the standard pillar documentation, at
+The below links contain documentation for the configuration options
-:ref:`pillar-configuration`
+
+- :ref:`master-side configuration <pillar-configuration-master>`
+- :ref:`minion-side configuration <pillar-configuration-minion>`
+
+Note that some of same the configuration options from the master are present in
+the minion configuration file, these are used in :ref:`masterless
+<tutorial-standalone-minion>` mode.
 
 The source for the built-in Salt pillars can be found here:
-:blob:`salt/pillar`
+:blob:`salt/pillar`

doc/topics/pillar/index.rst

@@ -12,24 +12,25 @@ Pillar was added to Salt in version 0.9.8
 
 .. note:: Storing sensitive data
 
-    Unlike state tree, pillar data is only available for the targeted
+    Pillar data is compiled on the master. Additionally, pillar data for a
-    minion specified by the matcher type.  This makes it useful for
+    given minion is only accessible by the minion for which it is targeted in
-    storing sensitive data specific to a particular minion.
+    the pillar configuration. This makes pillar useful for storing sensitive
+    data specific to a particular minion.
 
 
 Declaring the Master Pillar
 ===========================
 
-The Salt Master server maintains a pillar_roots setup that matches the
+The Salt Master server maintains a :conf_master:`pillar_roots` setup that
-structure of the file_roots used in the Salt file server. Like the
+matches the structure of the :conf_master:`file_roots` used in the Salt file
-Salt file server the ``pillar_roots`` option in the master config is based
+server. Like :conf_master:`file_roots`, the :conf_master:`pillar_roots` option
-on environments mapping to directories. The pillar data is then mapped to
+mapps environments to directories. The pillar data is then mapped to minions
-minions based on matchers in a top file which is laid out in the same way
+based on matchers in a top file which is laid out in the same way as the state
-as the state top file. Salt pillars can use the same matcher types as the
+top file. Salt pillars can use the same matcher types as the standard :ref:`top
-standard top file.
+file <states-top>`.
 
-The configuration for the :conf_master:`pillar_roots` in the master config file
+conf_master:`pillar_roots` is configured just like :conf_master:`file_roots`.
-is identical in behavior and function as :conf_master:`file_roots`:
+For example:
 
 .. code-block:: yaml
 
@@ -149,10 +150,15 @@ And the actual pillar file at '/srv/pillar/common_pillar.sls':
     foo: bar
     boo: baz
 
-Pillar namespace flattened
+Pillar Namespace Flattening
-==========================
+===========================
 
-The separate pillar files all share the same namespace. Given a ``top.sls`` of:
+The separate pillar SLS files all merge down into a single dictionary of
+key-value pairs. When the same key is defined in multiple SLS files, this can
+result in unexpected behavior if care is not taken to how the pillar SLS files
+are laid out.
+
+For example, given a ``top.sls`` containing the following:
 
 .. code-block:: yaml
 
@@ -161,44 +167,49 @@ The separate pillar files all share the same namespace. Given a ``top.sls`` of:
         - packages
         - services
 
-a ``packages.sls`` file of:
+with ``packages.sls`` containing:
 
 .. code-block:: yaml
 
     bind: bind9
 
-and a ``services.sls`` file of:
+and ``services.sls`` containing:
 
 .. code-block:: yaml
 
     bind: named
 
-Then a request for the ``bind`` pillar will only return ``named``; the
+Then a request for the ``bind`` pillar key will only return ``named``. The
-``bind9`` value is not available. It is better to structure your pillar files
+``bind9`` value will be lost, because ``services.sls`` was evaluated later.
-with more hierarchy. For example your ``package.sls`` file could look like:
+
+.. note::
+    Pillar files are applied in the order they are listed in the top file.
+    Therefore conflicting keys will be overwritten in a 'last one wins' manner!
+    For example, in the above scenario conflicting key values in ``services``
+    will overwrite those in ``packages`` because it's at the bottom of the list.
+
+It can be better to structure your pillar files with more hierarchy. For
+example the ``package.sls`` file could be configured like so:
 
 .. code-block:: yaml
 
     packages:
       bind: bind9
 
-Pillar Namespace Merges
+This would make the ``packages`` pillar key a nested dictionary containing a
-=======================
+``bind`` key.
 
-With some care, the pillar namespace can merge content from multiple pillar
+Pillar Dictionary Merging
-files under a single key, so long as conflicts are avoided as described above.
+=========================
 
-For example, if the above example were modified as follows, the values are
+If the same pillar key is defined in multiple pillar SLS files, and the keys in
-merged below a single key:
+both files refer to nested dictionaries, then the content from these
+dictionaries will be recursively merged.
 
-.. code-block:: yaml
+For example, keeping the ``top.sls`` the same, assume the following
+modifications to the pillar SLS files:
 
-    base:
+``packages.sls``:
-      '*':
-        - packages
-        - services
-
-And a ``packages.sls`` file like:
 
 .. code-block:: yaml
 
@@ -206,7 +217,7 @@ And a ``packages.sls`` file like:
       package-name: bind9
       version: 9.9.5
 
-And a ``services.sls`` file like:
+``services.sls``:
 
 .. code-block:: yaml
 
@@ -214,7 +225,7 @@ And a ``services.sls`` file like:
       port: 53
       listen-on: any
 
-The resulting pillar will be as follows:
+The resulting pillar dictionary will be:
 
 .. code-block:: bash
 
@@ -230,11 +241,9 @@ The resulting pillar will be as follows:
         version:
             9.9.5
 
-.. note::
+Since both pillar SLS files contained a ``bind`` key which contained a nested
-    Pillar files are applied in the order they are listed in the top file.
+dictionary, the pillar dictionary's ``bind`` key contains the combined contents
-    Therefore conflicting keys will be overwritten in a 'last one wins' manner!
+of both SLS files' ``bind`` keys.
-    For example, in the above scenario conflicting key values in ``services``
-    will overwrite those in ``packages`` because it's at the bottom of the list.
 
 Including Other Pillars
 =======================
@@ -266,34 +275,118 @@ With this form, the included file (users.sls) will be nested within the 'users'
 key of the compiled pillar. Additionally, the 'sudo' value will be available
 as a template variable to users.sls.
 
+.. _pillar-in-memory:
 
-Viewing Minion Pillar
+In-Memory Pillar Data vs. On-Demand Pillar Data
-=====================
+===============================================
 
-Once the pillar is set up the data can be viewed on the minion via the
+Since compiling pillar data is computationally expensive, the minion will
-``pillar`` module, the pillar module comes with functions,
+maintain a copy of the pillar data in memory to avoid needing to ask the master
-:mod:`pillar.items <salt.modules.pillar.items>` and :mod:`pillar.raw
+to recompile and send it a copy of the pillar data each time pillar data is
-<salt.modules.pillar.raw>`.  :mod:`pillar.items <salt.modules.pillar.items>`
+requested. This in-memory pillar data is what is returned by the
-will return a freshly reloaded pillar and :mod:`pillar.raw
+:py:func:`pillar.item <salt.modules.pillar.item>`, :py:func:`pillar.get
-<salt.modules.pillar.raw>` will return the current pillar without a refresh:
+<salt.modules.pillar.get>`, and :py:func:`pillar.raw <salt.modules.pillar.raw>`
+functions.
+
+Also, for those writing custom execution modules, or contributing to Salt's
+existing execution modules, the in-memory pillar data is available as the
+``__pillar__`` dunder dictionary.
+
+The in-memory pillar data is generated on minion start, and can be refreshed
+using the :py:func:`saltutil.refresh_pillar
+<salt.modules.saltutil.refresh_pillar>` function:
 
 .. code-block:: bash
 
-    salt '*' pillar.items
+    salt '*' saltutil.refresh_pillar
+
+This function triggers the minion to asynchronously refresh the in-memory
+pillar data and will always return ``None``.
+
+In contrast to in-memory pillar data, certain actions trigger pillar data to be
+compiled to ensure that the most up-to-date pillar data is available. These
+actions include:
+
+- Running states
+- Running :py:func:`pillar.items <salt.modules.pillar.items>`
+
+Performing these actions will *not* refresh the in-memory pillar data. So, if
+pillar data is modified, and then states are run, the states will see the
+updated pillar data, but :py:func:`pillar.item <salt.modules.pillar.item>`,
+:py:func:`pillar.get <salt.modules.pillar.get>`, and :py:func:`pillar.raw
+<salt.modules.pillar.raw>` will not see this data unless refreshed using
+:py:func:`saltutil.refresh_pillar <salt.modules.saltutil.refresh_pillar>`.
+
+
+How Pillar Environments Are Handled
+===================================
+
+When multiple pillar environments are used, the default behavior is for the
+pillar data from all environments to be merged together. The pillar dictionary
+will therefore contain keys from all configured environments.
+
+The :conf_minion:`pillarenv` minion config option can be used to force the
+minion to only consider pillar configuration from a single environment. This
+can be useful in cases where one needs to run states with alternate pillar
+data, either in a testing/QA environment or to test changes to the pillar data
+before pushing them live.
+
+For example, assume that the following is set in the minion config file:
+
+.. code-block:: yaml
+
+    pillarenv: base
+
+This would cause that minion to ignore all other pillar environments besides
+``base`` when compiling the in-memory pillar data. Then, when running states,
+the ``pillarenv`` CLI argument can be used to override the minion's
+:conf_minion:`pillarenv` config value:
+
+.. code-block:: bash
+
+    salt '*' state.apply mystates pillarenv=testing
+
+The above command will run the states with pillar data sourced exclusively from
+the ``testing`` environment, without modifying the in-memory pillar data.
 
 .. note::
-    Prior to version 0.16.2, this function is named ``pillar.data``. This
+    When running states, the ``pillarenv`` CLI option does not require a
-    function name is still supported for backwards compatibility.
+    :conf_minion:`pillarenv` option to be set in the minion config file. When
+    :conf_minion:`pillarenv` is left unset, as mentioned above all configured
+    environments will be combined. Running states with ``pillarenv=testing`` in
+    this case would still restrict the states' pillar data to just that of the
+    ``testing`` pillar environment.
+
+Viewing Pillar Data
+===================
+
+To view pillar data, use the :mod:`pillar <salt.modules.pillar>` execution
+module. This module includes several functions, each of them with their own
+use. These functions include:
+
+- :py:func:`pillar.item <salt.modules.pillar.item>` - Retrieves the value of
+  one or more keys from the :ref:`in-memory pillar datj <pillar-in-memory>`.
+- :py:func:`pillar.items <salt.modules.pillar.items>` - Compiles a fresh pillar
+  dictionary and returns it, leaving the :ref:`in-memory pillar data
+  <pillar-in-memory>` untouched. If pillar keys are passed to this function
+  however, this function acts like :py:func:`pillar.item
+  <salt.modules.pillar.item>` and returns their values from the :ref:`in-memory
+  pillar data <pillar-in-memory>`.
+- :py:func:`pillar.raw <salt.modules.pillar.raw>` - Like :py:func:`pillar.items
+  <salt.modules.pillar.items>`, it returns the entire pillar dictionary, but
+  from the :ref:`in-memory pillar data <pillar-in-memory>` instead of compiling
+  fresh pillar data.
+- :py:func:`pillar.get <salt.modules.pillar.get>` - Described in detail below.
 
 
-Pillar "get" Function
+The :py:func:`pillar.get <salt.modules.pillar.get>` Function
-=====================
+============================================================
 
 .. versionadded:: 0.14.0
 
 The :mod:`pillar.get <salt.modules.pillar.get>` function works much in the same
 way as the ``get`` method in a python dict, but with an enhancement: nested
-dict components can be extracted using a `:` delimiter.
+dictonaries can be traversed using a colon as a delimiter.
 
 If a structure like this is in pillar:
 
@@ -330,23 +423,8 @@ This makes handling nested structures much easier.
     'qux')``) to get the salt function, instead of the default dictionary
     behavior.
 
-
+Setting Pillar Data at the Command Line
-Refreshing Pillar Data
+=======================================
-======================
-
-When pillar data is changed on the master the minions need to refresh the data
-locally. This is done with the ``saltutil.refresh_pillar`` function.
-
-.. code-block:: bash
-
-    salt '*' saltutil.refresh_pillar
-
-This function triggers the minion to asynchronously refresh the pillar and will
-always return ``None``.
-
-
-Set Pillar Data at the Command Line
-===================================
 
 Pillar data can be set at the command line like the following example:
 
@@ -354,7 +432,7 @@ Pillar data can be set at the command line like the following example:
 
     salt '*' state.apply pillar='{"cheese": "spam"}'
 
-This will add a Pillar key of ``cheese`` with its value set to ``spam``.
+This will add a pillar key of ``cheese`` with its value set to ``spam``.
 
 .. note::
 
@@ -364,7 +442,7 @@ This will add a Pillar key of ``cheese`` with its value set to ``spam``.
     a security concern in some cases.
 
 
-Master Config In Pillar
+Master Config in Pillar
 =======================
 
 For convenience the data stored in the master configuration file can be made
@@ -372,14 +450,13 @@ available in all minion's pillars. This makes global configuration of services
 and systems very easy but may not be desired if sensitive data is stored in the
 master configuration. This option is disabled by default.
 
-To enable the master config from being added to the pillar set ``pillar_opts``
+To enable the master config from being added to the pillar set
-to ``True``:
+:conf_minion:`pillar_opts` to ``True`` in the minion config file:
 
 .. code-block:: yaml
 
     pillar_opts: True
 
-
 Minion Config in Pillar
 =======================
 

salt/modules/pillar.py

@@ -223,18 +223,35 @@ def item(*args, **kwargs):
     Return one or more pillar entries
 
     pillar
-        if specified, allows for a dictionary of pillar data to be made
+        If specified, allows for a dictionary of pillar data to be made
         available to pillar and ext_pillar rendering. these pillar variables
         will also override any variables of the same name in pillar or
         ext_pillar.
 
         .. versionadded:: 2015.5.0
 
+    delimiter
+        Delimiter used to traverse nested dictionaries.
+
+        .. note::
+            This is different from :py:func:`pillar.get
+            <salt.modules.pillar.get>` in that no default value can be
+            specified. :py:func:`pillar.get <salt.modules.pillar.get>` should
+            probably still be used in most cases to retrieve nested pillar
+            values, as it is a bit more flexible. One reason to use this
+            function instead of :py:func:`pillar.get <salt.modules.pillar.get>`
+            however is when it is desirable to retrieve the values of more than
+            one key, since :py:func:`pillar.get <salt.modules.pillar.get>` can
+            only retrieve one key at a time.
+
+        .. versionadded:: 2015.8.0
+
     CLI Examples:
 
     .. code-block:: bash
 
         salt '*' pillar.item foo
+        salt '*' pillar.item foo:bar
         salt '*' pillar.item foo bar baz
     '''
     ret = {}