valitydev/salt
14
0
Fork 0
mirror of https://github.com/valitydev/salt.git synced 2024-11-07 08:58:59 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Merge pull request #25595 from terminalmage/git_pillar-rewrite

Browse Source 
Bring git_pillar up to feature parity with gitfs
This commit is contained in:
Mike Place 2015-07-22 10:55:13 -06:00
commit bc2321a5c6
12 changed files with 2866 additions and 2056 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

272
doc/ref/configuration/master.rst
View File

@ -403,9 +403,9 @@ Default: ``''``
Specify the returner to use to log events. A returner may have installation and
configuration requirements. Read the returner's documentation.
.. note::
.. note::
Not all returners support event returns. Verify that a returner has an
Not all returners support event returns. Verify that a returner has an
``event_return()`` function before configuring this option with a returner.
.. code-block:: yaml
@ -1126,10 +1126,10 @@ Specify one value among valid values: ``gitpython``, ``pygit2``, ``dulwich``
Default: ``True``
The ``gitfs_ssl_verify`` option specifies whether to ignore SSL certificate
errors when contacting the gitfs backend. You might want to set this to false
if you're using a git backend that uses a self-signed certificate but keep in
mind that setting this flag to anything other than the default of ``True`` is a
security concern, you may want to try using the ssh transport.
errors when contacting the gitfs backend. You might want to set this to
``False`` if you're using a git backend that uses a self-signed certificate but
keep in mind that setting this flag to anything other than the default of
``True`` is a security concern, you may want to try using the ssh transport.
.. code-block:: yaml
@ -1897,16 +1897,266 @@ There are additional details at :ref:`salt-pillars`
.. versionadded:: 2015.5.0
The ext_pillar_first option allows for external pillar sources to populate
before file system pillar. This allows for targeting file system pillar from
ext_pillar.
Default: ``False``
This option allows for external pillar sources to be evaluated before
:conf_master:`pillar_roots`. This allows for targeting file system pillar from
ext_pillar.
.. code-block:: yaml
ext_pillar_first: False
.. _git-pillar-config-opts:
Git External Pillar (git_pillar) Configuration Options
------------------------------------------------------
.. conf_master:: git_pillar_base
``git_pillar_base``
*******************
.. versionadded:: 2015.8.0
Default: ``master``
If the desired branch matches this value, and the environment is omitted from
the git_pillar configuration, then the environment for that git_pillar remote
will be ``base``. For example, in the configuration below, the ``foo``
branch/tag would be assigned to the ``base`` environment, while ``bar`` would
be mapped to the ``bar`` environment.
.. code-block:: yaml
git_pillar_base: foo
ext_pillar:
- git:
- foo https://mygitserver/git-pillar.git
- bar https://mygitserver/git-pillar.git
.. conf_master:: git_pillar_branch
``git_pillar_branch``
*********************
.. versionadded:: 2015.8.0
Default: ``master``
If the branch is omitted from a git_pillar remote, then this branch will be
used instead. For example, in the configuration below, the first two remotes
would use the ``pillardata`` branch/tag, while the third would use the ``foo``
branch/tag.
.. code-block:: yaml
git_pillar_branch: pillardata
ext_pillar:
- git:
- https://mygitserver/pillar1.git
- https://mygitserver/pillar2.git:
- root: pillar
- foo https://mygitserver/pillar3.git
.. conf_master:: git_pillar_env
``git_pillar_env``
******************
.. versionadded:: 2015.8.0
Default: ``''`` (unset)
Environment to use for git_pillar remotes. This is normally derived from the
branch/tag (or from a per-remote ``env`` parameter), but if set this will
override the process of deriving the env from the branch/tag name. For example,
in the configuration below the ``foo`` branch would be assigned to the ``base``
environment, while the ``bar`` branch would need to explicitly have ``bar``
configured as it's environment to keep it from also being mapped to the
``base`` environment.
.. code-block:: yaml
git_pillar_env: base
ext_pillar:
- git:
- foo https://mygitserver/git-pillar.git
- bar https://mygitserver/git-pillar.git:
- env: bar
For this reason, this option is recommended to be left unset, unless the use
case calls for all (or almost all) of the git_pillar remotes to use the same
environment irrespective of the branch/tag being used.
.. conf_master:: git_pillar_root
``git_pillar_root``
********************
.. versionadded:: 2015.8.0
Default: ``''``
Path relative to the root of the repository where the git_pillar top file and
SLS files are located. In the below configuration, the pillar top file and SLS
files would be looked for in a subdirectory called ``pillar``.
.. code-block:: yaml
git_pillar_root: pillar
ext_pillar:
- git:
- master https://mygitserver/pillar1.git
- master https://mygitserver/pillar2.git
.. note::
This is a global option. If only one or two repos need to have their files
sourced from a subdirectory, then :conf_master:`git_pillar_root` can be
omitted and the root can be specified on a per-remote basis, like so:
.. code-block:: yaml
ext_pillar:
- git:
- master https://mygitserver/pillar1.git
- master https://mygitserver/pillar2.git:
- root: pillar
In this example, for the first remote the top file and SLS files would be
looked for in the root of the repository, while in the second remote the
pillar data would be retrieved from the ``pillar`` subdirectory.
.. conf_master:: git_pillar_ssl_verify
``git_pillar_ssl_verify``
*************************
.. versionadded:: 2015.8.0
Default: ``True``
Specifies whether or not to ignore SSL certificate errors when contacting the
git_pillar remote repository. You might want to set this to ``False`` if you're
using a git backend that uses a self-signed certificate but keep in mind that
setting this flag to anything other than the default of ``True`` is a security
concern, you may want to try using the ssh transport.
.. code-block:: yaml
git_pillar_ssl_verify: True
git_pillar Authentication Options
*********************************
These parameters only currently apply to the pygit2 gitfs provider.
Authentication works the same as it does in gitfs, as outlined in the
:ref:`GitFS Walkthrough <gitfs-authentication>`, though the global
configuration options are named differently to reflect that they are for
git_pillar instead of gitfs.
.. conf_master:: git_pillar_user
``git_pillar_user``
~~~~~~~~~~~~~~~~~~~
.. versionadded:: 2015.8.0
Default: ``''``
Along with :conf_master:`git_pillar_password`, is used to authenticate to HTTPS
remotes.
.. code-block:: yaml
git_pillar_user: git
.. conf_master:: git_pillar_password
``git_pillar_password``
~~~~~~~~~~~~~~~~~~~~~~~
.. versionadded:: 2015.8.0
Default: ``''``
Along with :conf_master:`git_pillar_user`, is used to authenticate to HTTPS
remotes. This parameter is not required if the repository does not use
authentication.
.. code-block:: yaml
git_pillar_password: mypassword
.. conf_master:: git_pillar_insecure_auth
``git_pillar_insecure_auth``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. versionadded:: 2015.8.0
Default: ``False``
By default, Salt will not authenticate to an HTTP (non-HTTPS) remote. This
parameter enables authentication over HTTP. **Enable this at your own risk.**
.. code-block:: yaml
git_pillar_insecure_auth: True
.. conf_master:: git_pillar_pubkey
``git_pillar_pubkey``
~~~~~~~~~~~~~~~~~~~~~
.. versionadded:: 2015.8.0
Default: ``''``
Along with :conf_master:`git_pillar_privkey` (and optionally
:conf_master:`git_pillar_passphrase`), is used to authenticate to SSH remotes.
.. code-block:: yaml
git_pillar_pubkey: /path/to/key.pub
.. conf_master:: git_pillar_privkey
``git_pillar_privkey``
~~~~~~~~~~~~~~~~~~~~~~
.. versionadded:: 2015.8.0
Default: ``''``
Along with :conf_master:`git_pillar_pubkey` (and optionally
:conf_master:`git_pillar_passphrase`), is used to authenticate to SSH remotes.
.. code-block:: yaml
git_pillar_privkey: /path/to/key
.. conf_master:: git_pillar_passphrase
``git_pillar_passphrase``
~~~~~~~~~~~~~~~~~~~~~~~~~
.. versionadded:: 2015.8.0
Default: ``''``
This parameter is optional, required only when the SSH key being used to
authenticate is protected by a passphrase.
.. code-block:: yaml
git_pillar_passphrase: mypassphrase
.. conf_master:: pillar_source_merging_strategy
``pillar_source_merging_strategy``
@ -2227,7 +2477,7 @@ The level of messages to send to the console. See also :conf_log:`log_level`.
Default: ``warning``
The level of messages to send to the log file. See also
:conf_log:`log_level_logfile`. When it is not set explicitly
:conf_log:`log_level_logfile`. When it is not set explicitly
it will inherit the level set by :conf_log:`log_level` option.
.. code-block:: yaml

8
doc/topics/releases/beryllium.rst
View File

@ -23,6 +23,14 @@ Salt Cloud Changes
a page was changed from 20 (default) to 200 to reduce the number of API calls
to Digital Ocean.
Git Pillar Rewritten
====================
The Git external pillar has been rewritten to bring it up to feature parity
with :mod:`gitfs <salt.fileserver.gitfs>`. See :mod:`here
<salt.pillar.git_pillar>` for more information on the new git_pillar
functionality.
JBoss 7 State
=============

37
salt/config.py
View File

@ -425,7 +425,17 @@ VALID_OPTS = {
# A master-only copy of the file_roots dictionary, used by the state compiler
'master_roots': dict,
'git_pillar_base': str,
'git_pillar_branch': str,
'git_pillar_env': str,
'git_pillar_root': str,
'git_pillar_ssl_verify': bool,
'git_pillar_user': str,
'git_pillar_password': str,
'git_pillar_insecure_auth': bool,
'git_pillar_privkey': str,
'git_pillar_pubkey': str,
'git_pillar_passphrase': str,
'gitfs_remotes': list,
'gitfs_mountpoint': str,
'gitfs_root': str,
@ -438,6 +448,7 @@ VALID_OPTS = {
'gitfs_passphrase': str,
'gitfs_env_whitelist': list,
'gitfs_env_blacklist': list,
'gitfs_ssl_verify': bool,
'hgfs_remotes': list,
'hgfs_mountpoint': str,
'hgfs_root': str,
@ -742,6 +753,17 @@ DEFAULT_MINION_OPTS = {
'pillar_roots': {
'base': [salt.syspaths.BASE_PILLAR_ROOTS_DIR],
},
'git_pillar_base': 'master',
'git_pillar_branch': 'master',
'git_pillar_env': '',
'git_pillar_root': '',
'git_pillar_ssl_verify': False,
'git_pillar_user': '',
'git_pillar_password': '',
'git_pillar_insecure_auth': False,
'git_pillar_privkey': '',
'git_pillar_pubkey': '',
'git_pillar_passphrase': '',
'gitfs_remotes': [],
'gitfs_mountpoint': '',
'gitfs_root': '',
@ -754,6 +776,7 @@ DEFAULT_MINION_OPTS = {
'gitfs_passphrase': '',
'gitfs_env_whitelist': [],
'gitfs_env_blacklist': [],
'gitfs_ssl_verify': False,
'hash_type': 'md5',
'disable_modules': [],
'disable_returners': [],
@ -878,6 +901,17 @@ DEFAULT_MASTER_OPTS = {
'base': [salt.syspaths.BASE_PILLAR_ROOTS_DIR],
},
'file_client': 'local',
'git_pillar_base': 'master',
'git_pillar_branch': 'master',
'git_pillar_env': '',
'git_pillar_root': '',
'git_pillar_ssl_verify': False,
'git_pillar_user': '',
'git_pillar_password': '',
'git_pillar_insecure_auth': False,
'git_pillar_privkey': '',
'git_pillar_pubkey': '',
'git_pillar_passphrase': '',
'gitfs_remotes': [],
'gitfs_mountpoint': '',
'gitfs_root': '',
@ -890,6 +924,7 @@ DEFAULT_MASTER_OPTS = {
'gitfs_passphrase': '',
'gitfs_env_whitelist': [],
'gitfs_env_blacklist': [],
'gitfs_ssl_verify': False,
'hgfs_remotes': [],
'hgfs_mountpoint': '',
'hgfs_root': '',

52
salt/daemons/masterapi.py
View File

@ -58,31 +58,41 @@ def init_git_pillar(opts):
'''
Clear out the ext pillar caches, used when the master starts
'''
pillargitfs = []
ret = []
for opts_dict in [x for x in opts.get('ext_pillar', [])]:
if 'git' in opts_dict:
try:
import git
except ImportError:
return pillargitfs
parts = opts_dict['git'].strip().split()
try:
br = parts[0]
loc = parts[1]
except IndexError:
log.critical(
'Unable to extract external pillar data: {0}'
.format(opts_dict['git'])
)
else:
pillargitfs.append(
git_pillar.GitPillar(
br,
loc,
opts
if isinstance(opts_dict['git'], six.string_types):
# Legacy git pillar code
try:
import git
except ImportError:
return ret
parts = opts_dict['git'].strip().split()
try:
br = parts[0]
loc = parts[1]
except IndexError:
log.critical(
'Unable to extract external pillar data: {0}'
.format(opts_dict['git'])
)
else:
ret.append(
git_pillar.LegacyGitPillar(
br,
loc,
opts
)
)
else:
# New git_pillar code
pillar = salt.utils.gitfs.GitPillar(opts)
pillar.init_remotes(
opts_dict['git'],
git_pillar.PER_REMOTE_PARAMS
)
return pillargitfs
ret.append(pillar)
return ret
def clean_fsbackend(opts):

1946
salt/fileserver/gitfs.py
View File

File diff suppressed because it is too large Load Diff

33
salt/master.py
View File

@ -167,7 +167,7 @@ class Maintenance(multiprocessing.Process):
# Make Event bus for firing
self.event = salt.utils.event.get_master_event(self.opts, self.opts['sock_dir'])
# Init any values needed by the git ext pillar
self.pillargitfs = salt.daemons.masterapi.init_git_pillar(self.opts)
self.git_pillar = salt.daemons.masterapi.init_git_pillar(self.opts)
# Set up search object
self.search = salt.search.Search(self.opts)
@ -198,7 +198,7 @@ class Maintenance(multiprocessing.Process):
salt.daemons.masterapi.clean_old_jobs(self.opts)
salt.daemons.masterapi.clean_expired_tokens(self.opts)
self.handle_search(now, last)
self.handle_pillargit()
self.handle_git_pillar()
self.handle_schedule()
self.handle_presence(old_present)
self.handle_key_rotate(now)
@ -252,16 +252,19 @@ class Maintenance(multiprocessing.Process):
'due to key rotation')
salt.utils.master.ping_all_connected_minions(self.opts)
def handle_pillargit(self):
def handle_git_pillar(self):
'''
Update git pillar
'''
try:
for pillargit in self.pillargitfs:
pillargit.update()
for pillar in self.git_pillar:
pillar.update()
except Exception as exc:
log.error('Exception {0} occurred in file server update '
'for git_pillar module.'.format(exc))
log.error(
'Exception \'{0}\' caught while updating git_pillar'
.format(exc),
exc_info_on_loglevel=logging.DEBUG
)
def handle_schedule(self):
'''
@ -380,6 +383,7 @@ class Master(SMaster):
should not start up.
'''
errors = []
critical_errors = []
if salt.utils.is_windows() and self.opts['user'] == 'root':
# 'root' doesn't typically exist on Windows. Use the current user
@ -411,13 +415,22 @@ class Master(SMaster):
try:
fileserver.init()
except FileserverConfigError as exc:
errors.append('{0}'.format(exc))
critical_errors.append('{0}'.format(exc))
if not self.opts['fileserver_backend']:
errors.append('No fileserver backends are configured')
if errors:
try:
# Init any values needed by the git ext pillar
salt.utils.gitfs.GitPillar(self.opts)
except FileserverConfigError as exc:
critical_errors.append(exc.strerror)
if errors or critical_errors:
for error in errors:
log.error(error)
log.error('Master failed pre flight checks, exiting\n')
for error in critical_errors:
log.critical(error)
log.critical('Master failed pre flight checks, exiting\n')
sys.exit(salt.defaults.exitcodes.EX_GENERIC)
# run_reqserver cannot be defined within a class method in order for it

20
salt/pillar/__init__.py
View File

@ -567,17 +567,10 @@ class Pillar(object):
return pillar, errors
def _external_pillar_data(self,
pillar,
val,
pillar_dirs,
key):
def _external_pillar_data(self, pillar, val, pillar_dirs, key):
'''
Builds actual pillar data structure
and update
the variable ``pillar``
Builds actual pillar data structure and updates the ``pillar`` variable
'''
ext = None
# try the new interface, which includes the minion ID
@ -585,7 +578,14 @@ class Pillar(object):
if isinstance(val, dict):
ext = self.ext_pillars[key](self.opts['id'], pillar, **val)
elif isinstance(val, list):
ext = self.ext_pillars[key](self.opts['id'], pillar, *val)
if key == 'git':
ext = self.ext_pillars[key](self.opts['id'],
val,
pillar_dirs)
else:
ext = self.ext_pillars[key](self.opts['id'],
pillar,
*val)
else:
if key == 'git':
ext = self.ext_pillars[key](self.opts['id'],

304
salt/pillar/git_pillar.py
View File

@ -1,32 +1,20 @@
# -*- coding: utf-8 -*-
'''
Clone a remote git repository and use the filesystem as a Pillar source
Use a git repository as a Pillar source
---------------------------------------
Currently GitPython is the only supported provider for git Pillars
.. note::
This external pillar has been rewritten for the :doc:`2015.8.0
</topics/releases/2015.8.0>` release. The old method of configuring this
external pillar will be maintained for a couple releases, allowing time for
configurations to be updated to reflect the new usage.
This external Pillar source can be configured in the master config file like
so:
This external pillar allows for a Pillar top file and Pillar SLS files to be
sourced from a git repository.
.. code-block:: yaml
ext_pillar:
- git: master git://gitserver/git-pillar.git root=subdirectory
The `root=` parameter is optional and used to set the subdirectory from where
to look for Pillar files (such as ``top.sls``).
.. versionchanged:: 2014.7.0
The optional ``root`` parameter will be added.
.. versionchanged:: 2015.5.0
The special branch name '__env__' will be replace by the
environment ({{env}})
Note that this is not the same thing as configuring pillar data using the
:conf_master:`pillar_roots` parameter. The branch referenced in the
:conf_master:`ext_pillar` entry above (``master``), would evaluate to the
``base`` environment, so this branch needs to contain a ``top.sls`` with a
``base`` section in it, like this:
However, since git_pillar does not have an equivalent to the
:conf_master:`pillar_roots` parameter, configuration is slightly different. The
Pillar top file must still contain the relevant environment, like so:
.. code-block:: yaml
@ -34,26 +22,48 @@ Note that this is not the same thing as configuring pillar data using the
'*':
- foo
To use other environments from the same git repo as git_pillar sources, just
add additional lines, like so:
The branch/tag which maps to that environment must then be specified along with
the repo's URL. Configuration details can be found below.
Configuring git_pillar for Salt releases before 2015.8.0
========================================================
For Salt releases earlier than :doc:`2015.8.0 </topics/releases/2015.8.0>`,
GitPython is the only supported provider for git_pillar. Individual
repositories can be configured under the :conf_master:`ext_pillar`
configuration parameter like so:
.. code-block:: yaml
ext_pillar:
- git: master git://gitserver/git-pillar.git
- git: dev git://gitserver/git-pillar.git
- git: master https://gitserver/git-pillar.git root=subdirectory
To remap a specific branch to a specific environment separate the branch name
and the environment name with a colon:
The repository is specified in the format ``<branch> <repo_url>``, with an
optional ``root`` parameter (added in the :doc:`2014.7.0
</topics/releases/2014.7.0>` release) which allows the pillar SLS files to be
served up from a subdirectory (similar to :conf_master:`gitfs_root` in gitfs).
To use more than one branch from the same repo, multiple lines must be
specified under :conf_master:`ext_pillar`:
.. code-block:: yaml
ext_pillar:
- git: develop:dev git://gitserver/git-pillar.git
- git: master:prod git://gitserver/git-pillar.git
- git: master https://gitserver/git-pillar.git
- git: dev https://gitserver/git-pillar.git
In this case, the ``dev`` branch would need its own ``top.sls`` with a ``dev``
section in it, like this:
To remap a specific branch to a specific Pillar environment, use the format
``<branch>:<env>``:
.. code-block:: yaml
ext_pillar:
- git: develop:dev https://gitserver/git-pillar.git
- git: master:prod https://gitserver/git-pillar.git
In this case, the ``develop`` branch would need its own ``top.sls`` with a
``dev`` section in it, like this:
.. code-block:: yaml
@ -61,41 +71,100 @@ section in it, like this:
'*':
- bar
In a gitfs base setup with pillars from the same repository as the states,
the ``ext_pillar:`` configuration would be like:
The ``master`` branch would need its own ``top.sls`` with a ``prod`` section in
it:
.. code-block:: yaml
prod:
'*':
- bar
If ``__env__`` is specified as the branch name, then git_pillar will use the
branch specified by :conf_master:`gitfs_base`:
.. code-block:: yaml
ext_pillar:
- git: __env__ git://gitserver/git-pillar.git root=pillar
- git: __env__ https://gitserver/git-pillar.git root=pillar
The (optional) root=pillar defines the directory that contains the pillar data.
The corresponding ``top.sls`` would be like:
The corresponding Pillar top file would look like this:
.. code-block:: yaml
{{env}}:
'*':
- bar
Configuring git_pillar for Salt releases 2015.8.0 and later
===========================================================
Beginning with Salt version 2015.8.0, pygit2_ is now supported for git_pillar,
in addition to GitPython_ (Dulwich_ will not be supported for the forseeable
future). The requirements for GitPython_ and pygit2_ are the same as for gitfs,
as described :ref:`here <gitfs-dependencies>`.
Here is an example git_pillar configuration.
.. code-block:: yaml
ext_pillar:
- git:
- production https://gitserver/git-pillar.git:
- env: prod
- develop https://gitserver/git-pillar.git:
- env: dev
- qa https://gitserver/git-pillar.git
- master https://other-git-server/pillardata.git
- root: pillar
The main difference between this and the old way of configuring git_pillar is
that multiple remotes can be configured under one ``git`` section under
:conf_master:`ext_pillar`. More than one ``git`` section can be used, but it is
not necessary. Remotes will be evaluated sequentially.
Per-remote configuration parameters are supported (similar to :ref:`gitfs
<gitfs-per-remote-config>`), and global versions of the git_pillar
configuration parameters can also be set.
With the addition of pygit2_ support, git_pillar can now interact with
authenticated remotes. Authentication works just like in gitfs (as outlined in
the :ref:`GitFS Walkthrough <gitfs-authentication>`), only with the global
authenication parameter names prefixed with ``git_pillar`` instead of ``gitfs``
(e.g. :conf_master:`git_pillar_pubkey`, :conf_master:`git_pillar_privkey`,
:conf_master:`git_pillar_passphrase`, etc.).
A full list of the git_pillar configuration options can be found :ref:`here
<git-pillar-config-opts>`.
.. _GitPython: https://github.com/gitpython-developers/GitPython
.. _pygit2: https://github.com/libgit2/pygit2
.. _Dulwich: https://www.samba.org/~jelmer/dulwich/
'''
from __future__ import absolute_import
# Import python libs
from copy import deepcopy
import copy
import logging
import hashlib
import os
# Import salt libs
import salt.utils.gitfs
from salt.exceptions import FileserverConfigError
from salt.pillar import Pillar
# Import third party libs
HAS_GIT = False
import salt.ext.six as six
# pylint: disable=import-error
try:
import git
HAS_GIT = True
HAS_GITPYTHON = True
except ImportError:
pass
HAS_GITPYTHON = False
# pylint: enable=import-error
# Import salt libs
from salt.pillar import Pillar
PER_REMOTE_PARAMS = ('env', 'root', 'ssl_verify')
# Set up logging
log = logging.getLogger(__name__)
@ -108,19 +177,58 @@ def __virtual__():
'''
Only load if gitpython is available
'''
ext_pillar_sources = [x for x in __opts__.get('ext_pillar', [])]
if not any(['git' in x for x in ext_pillar_sources]):
git_ext_pillars = [x for x in __opts__['ext_pillar'] if 'git' in x]
if not git_ext_pillars:
# No git external pillars were configured
return False
if not HAS_GIT:
log.error('Git-based ext_pillar is enabled in configuration but '
'could not be loaded, is GitPython installed?')
return False
if not git.__version__ > '0.3.0':
return False
return __virtualname__
for ext_pillar in git_ext_pillars:
if isinstance(ext_pillar['git'], six.string_types):
# Verification of legacy git pillar configuration
if not HAS_GITPYTHON:
log.error(
'Git-based ext_pillar is enabled in configuration but '
'could not be loaded, is GitPython installed?'
)
return False
if not git.__version__ > '0.3.0':
return False
return __virtualname__
else:
# Verification of new git pillar configuration
try:
salt.utils.gitfs.GitPillar(__opts__)
# Initialization of the GitPillar object did not fail, so we
# know we have valid configuration syntax and that a valid
# provider was detected.
return __virtualname__
except FileserverConfigError:
pass
return False
class GitPillar(object):
def ext_pillar(minion_id, repo, pillar_dirs):
'''
Execute a command and read the output as YAML
'''
if isinstance(repo, six.string_types):
return _legacy_git_pillar(minion_id, repo, pillar_dirs)
else:
opts = copy.deepcopy(__opts__)
opts['pillar_roots'] = {}
pillar = salt.utils.gitfs.GitPillar(opts)
pillar.init_remotes(repo, PER_REMOTE_PARAMS)
pillar.checkout()
ret = {}
for pillar_dir, env in six.iteritems(pillar.pillar_dirs):
opts['pillar_roots'] = {env: [pillar_dir]}
local_pillar = Pillar(opts, __grains__, minion_id, env)
ret.update(local_pillar.compile_pillar(ext=False))
return ret
# Legacy git_pillar code
class LegacyGitPillar(object):
'''
Deal with the remote git repository for Pillar
'''
@ -174,7 +282,8 @@ class GitPillar(object):
pass
else:
if self.repo.remotes.origin.url != self.rp_location:
self.repo.remotes.origin.config_writer.set('url', self.rp_location)
self.repo.remotes.origin.config_writer.set(
'url', self.rp_location)
def map_branch(self, branch, opts=None):
opts = __opts__ if opts is None else opts
@ -201,8 +310,8 @@ class GitPillar(object):
try:
self.repo.git.checkout('origin/{0}'.format(self.branch))
except git.exc.GitCommandError as exc:
logging.error('Unable to checkout branch '
'{0}: {1}'.format(self.branch, exc))
log.error('Unable to checkout branch '
'{0}: {1}'.format(self.branch, exc))
return False
return True
@ -211,7 +320,6 @@ class GitPillar(object):
'''
Return a list of refs that can be used as environments
'''
if isinstance(self.repo, git.Repo):
remote = self.repo.remote()
for ref in self.repo.refs:
@ -228,44 +336,9 @@ class GitPillar(object):
return list(self._envs)
def update(branch, repo_location):
def _legacy_git_pillar(minion_id, repo_string, pillar_dirs):
'''
Ensure you are following the latest changes on the remote
return boolean whether it worked
'''
gitpil = GitPillar(branch, repo_location, __opts__)
return gitpil.update()
def envs(branch, repo_location):
'''
Return a list of refs that can be used as environments
'''
gitpil = GitPillar(branch, repo_location, __opts__)
return gitpil.envs()
def _extract_key_val(kv, delimiter='='):
'''Extract key and value from key=val string.
Example:
>>> _extract_key_val('foo=bar')
('foo', 'bar')
'''
pieces = kv.split(delimiter)
key = pieces[0]
val = delimiter.join(pieces[1:])
return key, val
def ext_pillar(minion_id,
repo_string,
pillar_dirs):
'''
Execute a command and read the output as YAML
Support pre-Beryllium config schema
'''
if pillar_dirs is None:
return
@ -280,7 +353,7 @@ def ext_pillar(minion_id,
DELIM = '='
if DELIM not in extraopt:
log.error('Incorrectly formatted extra parameter. '
'Missing {0!r}: {1}'.format(DELIM, extraopt))
'Missing \'{0}\': {1}'.format(DELIM, extraopt))
key, val = _extract_key_val(extraopt, DELIM)
if key == 'root':
root = val
@ -290,7 +363,7 @@ def ext_pillar(minion_id,
# environment is "different" from the branch
branch, _, environment = branch_env.partition(':')
gitpil = GitPillar(branch, repo_location, __opts__)
gitpil = LegacyGitPillar(branch, repo_location, __opts__)
branch = gitpil.branch
if environment == '':
@ -314,10 +387,43 @@ def ext_pillar(minion_id,
if __opts__['pillar_roots'].get(branch, []) == [pillar_dir]:
return {}
opts = deepcopy(__opts__)
opts = copy.deepcopy(__opts__)
opts['pillar_roots'][environment] = [pillar_dir]
pil = Pillar(opts, __grains__, minion_id, branch)
return pil.compile_pillar()
def _update(branch, repo_location):
'''
Ensure you are following the latest changes on the remote
return boolean whether it worked
'''
gitpil = LegacyGitPillar(branch, repo_location, __opts__)
return gitpil.update()
def _envs(branch, repo_location):
'''
Return a list of refs that can be used as environments
'''
gitpil = LegacyGitPillar(branch, repo_location, __opts__)
return gitpil.envs()
def _extract_key_val(kv, delimiter='='):
'''Extract key and value from key=val string.
Example:
>>> _extract_key_val('foo=bar')
('foo', 'bar')
'''
pieces = kv.split(delimiter)
key = pieces[0]
val = delimiter.join(pieces[1:])
return key, val

2234
salt/utils/gitfs.py Normal file
View File

File diff suppressed because it is too large Load Diff

6
tests/integration/fileserver/gitfs_test.py
View File

@ -20,7 +20,8 @@ ensure_in_syspath('../..')
import integration
from salt.fileserver import gitfs
gitfs.__opts__ = {'gitfs_remotes': [''],
gitfs.__opts__ = {'cachedir': '/tmp/gitfs_test_cache',
'gitfs_remotes': [''],
'gitfs_root': '',
'fileserver_backend': ['git'],
'gitfs_base': 'master',
@ -34,7 +35,8 @@ gitfs.__opts__ = {'gitfs_remotes': [''],
'gitfs_insecure_auth': False,
'gitfs_privkey': '',
'gitfs_pubkey': '',
'gitfs_passphrase': ''
'gitfs_passphrase': '',
'gitfs_ssl_verify': True
}
LOAD = {'saltenv': 'base'}

6
tests/integration/modules/pillar.py
View File

@ -107,9 +107,9 @@ class PillarModuleTest(integration.ModuleCase):
'cachedir': self.master_opts['cachedir'],
}
git_pillar.GitPillar('master', original_url, opts)
git_pillar.LegacyGitPillar('master', original_url, opts)
opts['ext_pillar'] = [{'git': 'master {0}'.format(changed_url)}]
grepo = git_pillar.GitPillar('master', changed_url, opts)
grepo = git_pillar.LegacyGitPillar('master', changed_url, opts)
repo = git.Repo(rp_location)
self.assertEqual(grepo.rp_location, repo.remotes.origin.url)
@ -123,7 +123,7 @@ class PillarModuleTest(integration.ModuleCase):
pillar = self.run_function('pillar.data')
for branch, env in [('dev', 'testing')]:
repo = git_pillar.GitPillar(branch, repo_url, self.master_opts)
repo = git_pillar.LegacyGitPillar(branch, repo_url, self.master_opts)
self.assertIn(repo.working_dir,
pillar['test_ext_pillar_opts']['pillar_roots'][env])

4
tests/unit/pillar/git_test.py
View File

@ -37,7 +37,7 @@ from salt.pillar import Pillar, git_pillar
@skipIf(NO_MOCK, NO_MOCK_REASON)
@skipIf(not git_pillar.HAS_GIT, 'no GitPython')
@skipIf(not git_pillar.HAS_GITPYTHON, 'no GitPython')
class GitPillarTestCase(TestCase, integration.AdaptedConfigurationTestCaseMixIn):
'test git_pillar pillar'
maxDiff = None
@ -58,7 +58,7 @@ class GitPillarTestCase(TestCase, integration.AdaptedConfigurationTestCaseMixIn)
'pillar_opts': False
}
git_pillar.__grains__ = {}
git_pillar.update('master', 'file://{0}'.format(self.repo_path))
git_pillar._update('master', 'file://{0}'.format(self.repo_path))
def tearDown(self):
shutil.rmtree(self.tmpdir)