valitydev/salt
14
0
Fork 0
mirror of https://github.com/valitydev/salt.git synced 2024-11-06 08:35:21 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

salt.utils.gitfs: remove dulwich support, make refspecs configurable (#39210)

Browse Source 
* Add config params for custom refspecs

* Add exceptions for errors encountered modifying git config

* Make the refspecs a configurable parameter

* Make refspecs a per-remote parameter

* Update master config template to include custom refspecs opts

* Add documentation for new config params

* Update GitFS walkthrough with a section on custom refspecs

* Remove dulwich support from salt.utils.gitfs

Dulwich still lacks important features, including (but not limited to)
the following:

- Lack of the necessary support for checking out a ref needed for
  git_pillar/winrepo support

- No support in its config objects for multivar git config items, making
  it impossible to detect when repos have multiple refspecs set for a
  given git remote

Given this information, and the fact that it trails as a distant third
to Pygit2 and GitPython, Salt will cease to support Dulwich as a git
interface moving forward.

* Excise references to dulwich from documentation

* Add mention of custom refspecs to Nitrogen release notes

* Add gitfs_refspecs to mocked opts in gitfs integration tests

Also remove dulwich from unit tests

* Add information about opts argument not being intended for CLI use
This commit is contained in:
Erik Johnson 2017-02-09 10:50:45 -06:00 committed by Nicole Thomas
parent 4dba2d31a9
commit d35c2f810c
15 changed files with 450 additions and 750 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

24
conf/master
View File

@ -623,10 +623,11 @@
# Git File Server Backend Configuration
#
# Optional parameter used to specify the provider to be used for gitfs. Must
# be one of the following: pygit2, gitpython, or dulwich. If unset, then each
# will be tried in that same order, and the first one with a compatible
# version installed will be the provider that is used.
# Optional parameter used to specify the provider to be used for gitfs. Must be
# either pygit2 or gitpython. If unset, then both will be tried (in that
# order), and the first one with a compatible version installed will be the
# provider that is used.
#
#gitfs_provider: pygit2
# Along with gitfs_password, is used to authenticate to HTTPS remotes.
@ -679,6 +680,11 @@
# repository and defaults to the repository root.
#gitfs_root: somefolder/otherfolder
#
# The refspecs fetched by gitfs remotes
#gitfs_refspecs:
# - '+refs/heads/*:refs/remotes/origin/*'
# - '+refs/tags/*:refs/tags/*'
#
#
##### Pillar settings #####
##########################################
@ -828,6 +834,11 @@
# to authenticate is protected by a passphrase.
#git_pillar_passphrase: ''
# The refspecs fetched by git_pillar remotes
#git_pillar_refspecs:
# - '+refs/heads/*:refs/remotes/origin/*'
# - '+refs/tags/*:refs/tags/*'
# A master can cache pillars locally to bypass the expense of having to render them
# for each minion on every request. This feature should only be enabled in cases
# where pillar rendering time is known to be unsatisfactory and any attendant security
@ -1073,6 +1084,11 @@
#winrepo_remotes:
# - 'https://github.com/saltstack/salt-winrepo.git'
# The refspecs fetched by winrepo remotes
#winrepo_refspecs:
# - '+refs/heads/*:refs/remotes/origin/*'
# - '+refs/tags/*:refs/tags/*'
#
##### Returner settings ######
############################################

78
doc/ref/configuration/master.rst
View File

@ -1726,13 +1726,13 @@ Walkthrough <gitfs-per-remote-config>`.
Optional parameter used to specify the provider to be used for gitfs. More
information can be found in the :ref:`GitFS Walkthrough <gitfs-dependencies>`.
Must be one of the following: ``pygit2``, ``gitpython``, or ``dulwich``. If
unset, then each will be tried in that same order, and the first one with a
compatible version installed will be the provider that is used.
Must be either ``pygit2`` or ``gitpython``. If unset, then each will be tried
in that same order, and the first one with a compatible version installed will
be the provider that is used.
.. code-block:: yaml
gitfs_provider: dulwich
gitfs_provider: gitpython
.. conf_master:: gitfs_ssl_verify
@ -2024,6 +2024,28 @@ authenticate is protected by a passphrase.
gitfs_passphrase: mypassphrase
.. conf_master:: gitfs_refspecs
``gitfs_refspecs``
~~~~~~~~~~~~~~~~~~
.. versionadded:: Nitrogen
Default: ``['+refs/heads/*:refs/remotes/origin/*', '+refs/tags/*:refs/tags/*']``
When fetching from remote repositories, by default Salt will fetch branches and
tags. This parameter can be used to override the default and specify
alternate refspecs to be fetched. More information on how this feature works
can be found in the :ref:`GitFS Walkthrough <gitfs-custom-refspecs>`.
.. code-block:: yaml
gitfs_refspecs:
- '+refs/heads/*:refs/remotes/origin/*'
- '+refs/tags/*:refs/tags/*'
- '+refs/pull/*/head:refs/remotes/origin/pr/*'
- '+refs/pull/*/merge:refs/remotes/origin/merge/*'
hg: Mercurial Remote File Server Backend
----------------------------------------
@ -2959,7 +2981,7 @@ Git External Pillar Authentication Options
******************************************
These parameters only currently apply to the ``pygit2``
:conf_master:`git_pillar_provider`. Authentication works the same as it does
:conf_master:`git_pillar_provider`. Authentication works the same as it does
in gitfs, as outlined in the :ref:`GitFS Walkthrough <gitfs-authentication>`,
though the global configuration options are named differently to reflect that
they are for git_pillar instead of gitfs.
@ -3061,6 +3083,29 @@ authenticate is protected by a passphrase.
git_pillar_passphrase: mypassphrase
.. conf_master:: git_pillar_refspecs
``git_pillar_refspecs``
~~~~~~~~~~~~~~~~~~~~~~~
.. versionadded:: Nitrogen
Default: ``['+refs/heads/*:refs/remotes/origin/*', '+refs/tags/*:refs/tags/*']``
When fetching from remote repositories, by default Salt will fetch branches and
tags. This parameter can be used to override the default and specify
alternate refspecs to be fetched. This parameter works similarly to its
:ref:`GitFS counterpart <git_pillar-custom-refspecs>`, in that it can be
configured both globally and for individual remotes.
.. code-block:: yaml
git_pillar_refspecs:
- '+refs/heads/*:refs/remotes/origin/*'
- '+refs/tags/*:refs/tags/*'
- '+refs/pull/*/head:refs/remotes/origin/pr/*'
- '+refs/pull/*/merge:refs/remotes/origin/merge/*'
.. _pillar-merging-opts:
Pillar Merging Options
@ -4028,3 +4073,26 @@ authenticate is protected by a passphrase.
.. code-block:: yaml
winrepo_passphrase: mypassphrase
.. conf_master:: winrepo_refspecs
``winrepo_refspecs``
~~~~~~~~~~~~~~~~~~~~
.. versionadded:: Nitrogen
Default: ``['+refs/heads/*:refs/remotes/origin/*', '+refs/tags/*:refs/tags/*']``
When fetching from remote repositories, by default Salt will fetch branches and
tags. This parameter can be used to override the default and specify
alternate refspecs to be fetched. This parameter works similarly to its
:ref:`GitFS counterpart <winrepo-custom-refspecs>`, in that it can be
configured both globally and for individual remotes.
.. code-block:: yaml
winrepo_refspecs:
- '+refs/heads/*:refs/remotes/origin/*'
- '+refs/tags/*:refs/tags/*'
- '+refs/pull/*/head:refs/remotes/origin/pr/*'
- '+refs/pull/*/merge:refs/remotes/origin/merge/*'

1
doc/spelling_wordlist.txt
View File

@ -195,7 +195,6 @@ dpkg
DRS
dryrun
dst
dulwich
dumpe
dvh
eauth

10
doc/topics/releases/nitrogen.rst
View File

@ -99,6 +99,7 @@ numbers like ``1.2.34-5.el7``. An example of the usage for this would be:
- The :mod:`system <salt.modules.system>` module changed the returned format
from "HH:MM AM/PM" to "HH:MM:SS AM/PM" for `get_system_time`.
Master Configuration Additions
==============================
@ -134,6 +135,14 @@ release cycle (two major releases after this one), those who are using the
:ref:`netapi module <all-netapi-modules>`) are encouraged to update their code
to use ``tgt_type``.
Custom Refspecs in GitFS / git_pillar / winrepo
===============================================
It is now possible to specify the refspecs to use when fetching from remote
repositores for GitFS, git_pillar, and winrepo. More information on how this
feature works can be found :ref:`here <gitfs-custom-refspecs>` in the GitFS
Walkthrough. The git_pillar and winrepo versions of this feature work the same
as their GitFS counterpart.
Deprecations
============
@ -142,6 +151,7 @@ General Deprecations
--------------------
- Removed support for aliasing ``cmd.run`` to ``cmd.shell``.
- Removed support for Dulwich from :ref:`GitFS <tutorial-gitfs>`.
- Beacon configurations should be lists instead of dictionaries.
- The ``PidfileMixin`` has been removed. Please use ``DaemonMixIn`` instead.
- The ``use_pending`` argument was removed from the ``salt.utils.event.get_event``

137
doc/topics/tutorials/gitfs.rst
View File

@ -26,21 +26,17 @@ Branches and tags become Salt fileserver environments.
Installing Dependencies
=======================
Beginning with version 2014.7.0, both pygit2_ and Dulwich_ are supported as
alternatives to GitPython_. The desired provider can be configured using the
:conf_master:`gitfs_provider` parameter in the master config file.
If :conf_master:`gitfs_provider` is not configured, then Salt will prefer
pygit2_ if a suitable version is available, followed by GitPython_ and
Dulwich_.
Both pygit2_ and GitPython_ are supported Python interfaces to git. If
compatible versions of both are installed, pygit2_ will preferred. In these
cases, GitPython_ can be forced using the :conf_master:`gitfs_provider`
parameter in the master config file.
.. note::
It is recommended to always run the most recent version of any the below
dependencies. Certain features of gitfs may not be available without
dependencies. Certain features of GitFS may not be available without
the most recent version of the chosen library.
.. _pygit2: https://github.com/libgit2/pygit2
.. _Dulwich: https://www.samba.org/~jelmer/dulwich/
.. _GitPython: https://github.com/gitpython-developers/GitPython
pygit2
@ -75,7 +71,7 @@ libraries to be present before libgit2_ is built. On some Debian-based distros
``pkg-config`` is also required to link libgit2_ with libssh2.
.. note::
If you are receiving the error "Unsupported URL Protocol" in the Salt Master
log when making a connection using SSH, review the libssh2 details listed
log when making a connection using SSH, review the libssh2 details listed
above.
Additionally, version 0.21.0 of pygit2 introduced a dependency on python-cffi_,
@ -157,49 +153,6 @@ install GitPython`` (or ``easy_install GitPython``) as root.
- name: 'GitPython < 2.0.9'
Dulwich
-------
Dulwich 0.9.4 or newer is required to use Dulwich as backend for gitfs.
Dulwich is available in EPEL, and can be easily installed on the master using
yum:
.. code-block:: bash
# yum install python-dulwich
For APT-based distros such as Ubuntu and Debian:
.. code-block:: bash
# apt-get install python-dulwich
.. important::
If switching to Dulwich from GitPython/pygit2, or switching from
GitPython/pygit2 to Dulwich, it is necessary to clear the gitfs cache to
avoid unpredictable behavior. This is probably a good idea whenever
switching to a new :conf_master:`gitfs_provider`, but it is less important
when switching between GitPython and pygit2.
Beginning in version 2015.5.0, the gitfs cache can be easily cleared using
the :mod:`fileserver.clear_cache <salt.runners.fileserver.clear_cache>`
runner.
.. code-block:: bash
salt-run fileserver.clear_cache backend=git
If the Master is running an earlier version, then the cache can be cleared
by removing the ``gitfs`` and ``file_lists/gitfs`` directories (both paths
relative to the master cache directory, usually
``/var/cache/salt/master``).
.. code-block:: bash
rm -rf /var/cache/salt/master{,/file_lists}/gitfs
Simple Configuration
====================
@ -234,14 +187,6 @@ master:
Information on how to authenticate to SSH remotes can be found :ref:`here
<gitfs-authentication>`.
.. note::
Dulwich does not recognize ``ssh://`` URLs, ``git+ssh://`` must be used
instead. Salt version 2015.5.0 and later will automatically add the
``git+`` to the beginning of these URLs before fetching, but earlier
Salt versions will fail to fetch unless the URL is specified using
``git+ssh://``.
3. Restart the master to load the new configuration.
@ -316,7 +261,6 @@ is executed. For example:
cache directory (``/var/cache/salt/master/gitfs``) before restarting the
salt-master service.
.. _gitfs-per-remote-config:
Per-remote Configuration Parameters
@ -336,6 +280,7 @@ configured gitfs remotes):
* :conf_master:`gitfs_pubkey` (**pygit2 only**, new in 2014.7.0)
* :conf_master:`gitfs_privkey` (**pygit2 only**, new in 2014.7.0)
* :conf_master:`gitfs_passphrase` (**pygit2 only**, new in 2014.7.0)
* :conf_master:`gitfs_refspecs` (new in Nitrogen)
These parameters can now be overridden on a per-remote basis. This allows for a
tremendous amount of customization. Here's some example usage:
@ -372,7 +317,8 @@ tremendous amount of customization. Here's some example usage:
2. Per-remote configuration parameters are named like the global versions,
with the ``gitfs_`` removed from the beginning. The exception being the
``name`` parameter which is only available to per-remote configurations.
``name`` and ``saltenv`` parameters, which are only available to
per-remote configurations.
In the example configuration above, the following is true:
@ -454,10 +400,62 @@ Given the above configuration, the following is true:
``salt`` directory (and its subdirectories).
.. _gitfs-custom-refspecs:
Custom Refspecs
===============
.. versionadded:: Nitrogen
GitFS will by default fetch remote branches and tags. However, sometimes it can
be useful to fetch custom refs (such as those created for `GitHub pull
requests`__). To change the refspecs GitFS fetches, use the
:conf_master:`gitfs_refspecs` config option:
.. __: https://help.github.com/articles/checking-out-pull-requests-locally/
.. code-block:: yaml
gitfs_refspecs:
- '+refs/heads/*:refs/remotes/origin/*'
- '+refs/tags/*:refs/tags/*'
- '+refs/pull/*/head:refs/remotes/origin/pr/*'
- '+refs/pull/*/merge:refs/remotes/origin/merge/*'
In the above example, in addition to fetching remote branches and tags,
GitHub's custom refs for pull requests and merged pull requests will also be
fetched. These special ``head`` refs represent the head of the branch which is
requesting to be merged, and the ``merge`` refs represent the result of the
base branch after the merge.
.. important::
When using custom refspecs, the destination of the fetched refs *must* be
under ``refs/remotes/origin/``, preferably in a subdirectory like in the
example above. These custom refspecs will map as environment names using
their relative path underneath ``refs/remotes/origin/``. For example,
assuming the configuration above, the head branch for pull request 12345
would map to fileserver environment ``pr/12345`` (slash included).
Refspecs can be configured on a :ref:`per-remote basis
<gitfs-per-remote-config>`. For example, the below configuration would only
alter the default refspecs for the *second* GitFS remote. The first remote
would only fetch branches and tags (the default).
.. code-block:: yaml
gitfs_remotes:
- https://domain.tld/foo.git
- https://domain.tld/bar.git:
- refspecs:
- '+refs/heads/*:refs/remotes/origin/*'
- '+refs/tags/*:refs/tags/*'
- '+refs/pull/*/head:refs/remotes/origin/pr/*'
- '+refs/pull/*/merge:refs/remotes/origin/merge/*'
Configuration Order of Precedence
=================================
The order of precedence for gitfs configuration is as follows (each level
The order of precedence for GitFS configuration is as follows (each level
overrides all levels below it):
1. Per-saltenv configuration (defined under a per-remote ``saltenv``
@ -592,7 +590,7 @@ master, each configured git remote will be searched.
Branches, Environments, and Top Files
=====================================
When using the gitfs backend, branches, and tags will be mapped to environments
When using the GitFS backend, branches, and tags will be mapped to environments
using the branch/tag name as an identifier.
There is one exception to this rule: the ``master`` branch is implicitly mapped
@ -913,8 +911,9 @@ steps to this process:
#!/usr/bin/env sh
sudo -u root salt-call event.fire_master update salt/fileserver/gitfs/update
4. If using sudo in the git hook (above), the policy must be changed to permit all users to fire the event.
Add the following policy to the sudoers file on the git server.
4. If using sudo in the git hook (above), the policy must be changed to permit
all users to fire the event. Add the following policy to the sudoers file
on the git server.
.. code-block:: bash
@ -930,8 +929,8 @@ by this reactor.
Similarly, the tag name ``salt/fileserver/gitfs/update`` can be replaced by
anything, so long as the usage is consistent.
The ``root`` user name in the hook script and sudo policy should be changed to match the user under which
the minion is running.
The ``root`` user name in the hook script and sudo policy should be changed to
match the user under which the minion is running.
.. _`post-receive hook`: http://www.git-scm.com/book/en/Customizing-Git-Git-Hooks#Server-Side-Hooks

10
salt/config/__init__.py
View File

@ -57,6 +57,7 @@ _DFLT_LOG_FMT_CONSOLE = '[%(levelname)-8s] %(message)s'
_DFLT_LOG_FMT_LOGFILE = (
'%(asctime)s,%(msecs)03d [%(name)-17s][%(levelname)-8s][%(process)d] %(message)s'
)
_DFLT_REFSPECS = ['+refs/heads/*:refs/remotes/origin/*', '+refs/tags/*:refs/tags/*']
if salt.utils.is_windows():
# Since an 'ipc_mode' of 'ipc' will never work on Windows due to lack of
@ -580,6 +581,7 @@ VALID_OPTS = {
'git_pillar_privkey': str,
'git_pillar_pubkey': str,
'git_pillar_passphrase': str,
'git_pillar_refspecs': list,
'gitfs_remotes': list,
'gitfs_mountpoint': str,
'gitfs_root': str,
@ -595,6 +597,7 @@ VALID_OPTS = {
'gitfs_ssl_verify': bool,
'gitfs_global_lock': bool,
'gitfs_saltenv': list,
'gitfs_refspecs': list,
'hgfs_remotes': list,
'hgfs_mountpoint': str,
'hgfs_root': str,
@ -781,6 +784,7 @@ VALID_OPTS = {
'winrepo_privkey': str,
'winrepo_pubkey': str,
'winrepo_passphrase': str,
'winrepo_refspecs': list,
# Set a hard limit for the amount of memory modules can consume on a minion.
'modules_max_memory': int,
@ -1081,6 +1085,7 @@ DEFAULT_MINION_OPTS = {
'git_pillar_privkey': '',
'git_pillar_pubkey': '',
'git_pillar_passphrase': '',
'git_pillar_refspecs': _DFLT_REFSPECS,
'gitfs_remotes': [],
'gitfs_mountpoint': '',
'gitfs_root': '',
@ -1096,6 +1101,7 @@ DEFAULT_MINION_OPTS = {
'gitfs_global_lock': True,
'gitfs_ssl_verify': True,
'gitfs_saltenv': [],
'gitfs_refspecs': _DFLT_REFSPECS,
'hash_type': 'sha256',
'disable_modules': [],
'disable_returners': [],
@ -1181,6 +1187,7 @@ DEFAULT_MINION_OPTS = {
'winrepo_privkey': '',
'winrepo_pubkey': '',
'winrepo_passphrase': '',
'winrepo_refspecs': _DFLT_REFSPECS,
'pidfile': os.path.join(salt.syspaths.PIDFILE_DIR, 'salt-minion.pid'),
'range_server': 'range:80',
'reactor_refresh_interval': 60,
@ -1297,6 +1304,7 @@ DEFAULT_MASTER_OPTS = {
'git_pillar_privkey': '',
'git_pillar_pubkey': '',
'git_pillar_passphrase': '',
'git_pillar_refspecs': _DFLT_REFSPECS,
'gitfs_remotes': [],
'gitfs_mountpoint': '',
'gitfs_root': '',
@ -1312,6 +1320,7 @@ DEFAULT_MASTER_OPTS = {
'gitfs_global_lock': True,
'gitfs_ssl_verify': True,
'gitfs_saltenv': [],
'gitfs_refspecs': _DFLT_REFSPECS,
'hgfs_remotes': [],
'hgfs_mountpoint': '',
'hgfs_root': '',
@ -1454,6 +1463,7 @@ DEFAULT_MASTER_OPTS = {
'winrepo_privkey': '',
'winrepo_pubkey': '',
'winrepo_passphrase': '',
'winrepo_refspecs': _DFLT_REFSPECS,
'syndic_wait': 5,
'jinja_lstrip_blocks': False,
'jinja_trim_blocks': False,

7
salt/exceptions.py
View File

@ -196,6 +196,13 @@ class GitLockError(SaltException):
self.strerror = strerror
class GitRemoteError(SaltException):
'''
Used by GitFS to denote a problem with the existence of the "origin" remote
or part of its configuration
'''
class SaltInvocationError(SaltException, TypeError):
'''
Used when the wrong number of arguments are sent to modules or invalid

32
salt/fileserver/gitfs.py
View File

@ -13,37 +13,35 @@ Master config file.
fileserver_backend:
- git
As of Salt 2014.7.0, the Git fileserver backend supports GitPython_, pygit2_,
and dulwich_ to provide the Python interface to git. If more than one of these
are present, the order of preference for which one will be chosen is the same
as the order in which they were listed: pygit2, GitPython, dulwich (keep in
mind, this order is subject to change).
The Git fileserver backend supports both pygit2_ and GitPython_, to provide the
Python interface to git. If both are present, the order of preference for which
one will be chosen is the same as the order in which they were listed: pygit2,
then GitPython.
An optional master config parameter (:conf_master:`gitfs_provider`) can be used
to specify which provider should be used.
to specify which provider should be used, in the event that compatible versions
of both pygit2_ and GitPython_ are installed.
More detailed information on how to use gitfs can be found in the :ref:`Gitfs
More detailed information on how to use GitFS can be found in the :ref:`GitFS
Walkthrough <tutorial-gitfs>`.
.. note:: Minimum requirements
To use GitPython_ for gitfs requires a minimum GitPython version of 0.3.0,
as well as the git CLI utility. Instructions for installing GitPython can
be found :ref:`here <gitfs-dependencies>`.
To use pygit2_ for gitfs requires a minimum pygit2_ version of 0.20.3.
To use pygit2_ for GitFS requires a minimum pygit2_ version of 0.20.3.
pygit2_ 0.20.3 requires libgit2_ 0.20.0. pygit2_ and libgit2_ are developed
alongside one another, so it is recommended to keep them both at the same
major release to avoid unexpected behavior. For example, pygit2_ 0.21.x
requires libgit2_ 0.21.x, pygit2_ 0.22.x will require libgit2_ 0.22.x, etc.
To find stale refs, pygit2 additionally requires the git CLI utility to be
installed.
To use GitPython_ for GitFS requires a minimum GitPython version of 0.3.0,
as well as the git CLI utility. Instructions for installing GitPython can
be found :ref:`here <gitfs-dependencies>`.
To clear stale refs the git CLI utility must also be installed.
.. _GitPython: https://github.com/gitpython-developers/GitPython
.. _pygit2: https://github.com/libgit2/pygit2
.. _libgit2: https://libgit2.github.com/
.. _dulwich: https://www.samba.org/~jelmer/dulwich/
.. _GitPython: https://github.com/gitpython-developers/GitPython
'''
# Import python libs
@ -51,7 +49,7 @@ from __future__ import absolute_import
import logging
PER_REMOTE_OVERRIDES = ('base', 'mountpoint', 'root', 'ssl_verify',
'env_whitelist', 'env_blacklist')
'env_whitelist', 'env_blacklist', 'refspecs')
PER_REMOTE_ONLY = ('name', 'saltenv')
# Auth support (auth params can be global or per-remote, too)

2
salt/modules/status.py
View File

@ -1126,7 +1126,7 @@ def proxy_reconnect(proxy_name, opts=None):
The virtual name of the proxy module.
opts: None
Opts dictionary.
Opts dictionary. Not intended for CLI usage.
CLI Example:

8
salt/pillar/git_pillar.py
View File

@ -156,9 +156,8 @@ Configuring git_pillar for Salt releases 2015.8.0 and later
will also be logged.
Beginning with Salt version 2015.8.0, pygit2_ is now supported in addition to
GitPython_ (Dulwich_ will not be supported for the foreseeable future). The
requirements for GitPython_ and pygit2_ are the same as for gitfs, as described
:ref:`here <gitfs-dependencies>`.
GitPython_. The requirements for GitPython_ and pygit2_ are the same as for
GitFS, as described :ref:`here <gitfs-dependencies>`.
.. important::
git_pillar has its own set of global configuration parameters. While it may
@ -295,7 +294,6 @@ instead of ``gitfs`` (e.g. :conf_master:`git_pillar_pubkey`,
.. _GitPython: https://github.com/gitpython-developers/GitPython
.. _pygit2: https://github.com/libgit2/pygit2
.. _Dulwich: https://www.samba.org/~jelmer/dulwich/
'''
from __future__ import absolute_import
@ -322,7 +320,7 @@ except ImportError:
HAS_GITPYTHON = False
# pylint: enable=import-error
PER_REMOTE_OVERRIDES = ('env', 'root', 'ssl_verify')
PER_REMOTE_OVERRIDES = ('env', 'root', 'ssl_verify', 'refspecs')
# Set up logging
log = logging.getLogger(__name__)

2
salt/runners/winrepo.py
View File

@ -29,7 +29,7 @@ import salt.template
log = logging.getLogger(__name__)
# Global parameters which can be overridden on a per-remote basis
PER_REMOTE_OVERRIDES = ('ssl_verify',)
PER_REMOTE_OVERRIDES = ('ssl_verify', 'refspecs')
def genrepo(opts=None, fire_event=True):

817
salt/utils/gitfs.py
View File

File diff suppressed because it is too large Load Diff

2
tests/integration/fileserver/gitfs_test.py
View File

@ -36,6 +36,8 @@ gitfs.__opts__ = {'cachedir': '/tmp/gitfs_test_cache',
'gitfs_privkey': '',
'gitfs_pubkey': '',
'gitfs_passphrase': '',
'gitfs_refspecs': ['+refs/heads/*:refs/remotes/origin/*',
'+refs/tags/*:refs/tags/*'],
'gitfs_ssl_verify': True
}

2
tests/unit/fileserver/gitfs_test.py
View File

@ -52,6 +52,8 @@ class GitfsConfigTestCase(TestCase):
'gitfs_global_lock': True,
'gitfs_ssl_verify': True,
'gitfs_saltenv': [],
'gitfs_refspecs': ['+refs/heads/*:refs/remotes/origin/*',
'+refs/tags/*:refs/tags/*'],
}
def tearDown(self):

68
tests/unit/utils/gitfs_test.py
View File

@ -39,20 +39,18 @@ class TestGitFSProvider(TestCase):
MagicMock(return_value=True)):
with patch.object(role_class, 'verify_pygit2',
MagicMock(return_value=False)):
with patch.object(role_class, 'verify_dulwich',
MagicMock(return_value=False)):
args = [OPTS]
if role_name == 'winrepo':
args.append('/tmp/winrepo-dir')
with patch.dict(OPTS, {key: provider}):
# Try to create an instance with uppercase letters in
# provider name. If it fails then a
# FileserverConfigError will be raised, so no assert is
# necessary.
role_class(*args)
# Now try to instantiate an instance with all lowercase
# letters. Again, no need for an assert here.
role_class(*args)
args = [OPTS]
if role_name == 'winrepo':
args.append('/tmp/winrepo-dir')
with patch.dict(OPTS, {key: provider}):
# Try to create an instance with uppercase letters in
# provider name. If it fails then a
# FileserverConfigError will be raised, so no assert is
# necessary.
role_class(*args)
# Now try to instantiate an instance with all lowercase
# letters. Again, no need for an assert here.
role_class(*args)
def test_valid_provider(self):
'''
@ -77,36 +75,20 @@ class TestGitFSProvider(TestCase):
verify = 'verify_pygit2'
mock2 = _get_mock(verify, provider)
with patch.object(role_class, verify, mock2):
verify = 'verify_dulwich'
mock3 = _get_mock(verify, provider)
with patch.object(role_class, verify, mock3):
args = [OPTS]
if role_name == 'winrepo':
args.append('/tmp/winrepo-dir')
with patch.dict(OPTS, {key: provider}):
if role_name == 'gitfs' \
or (role_name != 'gitfs'
and provider != 'dulwich'):
# This is a valid provider, so this should
# pass without raising an exception.
role_class(*args)
else:
# Dulwich is not supported for git_pillar nor
# winrepo, so trying to use it should raise an
# exception.
self.assertRaises(
FileserverConfigError,
role_class,
*args
)
args = [OPTS]
if role_name == 'winrepo':
args.append('/tmp/winrepo-dir')
with patch.dict(OPTS, {key: 'foo'}):
# Set the provider name to a known invalid provider
# and make sure it raises an exception.
self.assertRaises(
FileserverConfigError,
role_class,
*args
with patch.dict(OPTS, {key: provider}):
role_class(*args)
with patch.dict(OPTS, {key: 'foo'}):
# Set the provider name to a known invalid provider
# and make sure it raises an exception.
self.assertRaises(
FileserverConfigError,
role_class,
*args
)