valitydev/salt
14
0
Fork 0
mirror of https://github.com/valitydev/salt.git synced 2024-11-07 17:09:03 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Merge pull request #26262 from cro/20158-develop-mf

Browse Source 
Merge 2015.8 forward to develop (get ldap fix)
This commit is contained in:
C. R. Oldham 2015-08-12 13:51:39 -06:00
commit 6b109be018
18 changed files with 414 additions and 239 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
conf/minion
View File

@ -1,5 +1,5 @@
##### Primary configuration settings #####
##########################################
##########################################
# This configuration file is used to manage the behavior of the Salt Minion.
# With the exception of the location of the Salt Master Server, values that are
# commented out but have an empty line after the comment are defaults that need
@ -482,9 +482,9 @@
# will be shown for each state run.
#state_output_profile: True
# Fingerprint of the master public key to double verify the master is valid,
# the master fingerprint can be found by running "salt-key -f master.pub" on the
# salt master.
# Fingerprint of the master public key to validate the identity of your Salt master
# before the initial key exchange. The master fingerprint can be found by running
# "salt-key -F master" on the Salt master.
#master_finger: ''

2
doc/_themes/saltstack2/layout.html vendored
View File

@ -295,7 +295,7 @@
<!--analytics-->
<script type="text/javascript" language="javascript">llactid=23943</script>
<script type="text/javascript" language="javascript" src="http://t6.trackalyzer.com/trackalyze.js"></script>
<script type="text/javascript" language="javascript" src="https://trackalyzer.com/trackalyze_secure.js"></script>
<script>
var _gaq = _gaq || [];

35
doc/ref/configuration/index.rst
View File

@ -101,6 +101,41 @@ Running Salt
There is also a full :doc:`troubleshooting guide</topics/troubleshooting/index>`
available.
.. _key-identity:
Key Identity
============
Salt provides commands to validate the identity of your Salt master
and Salt minions before the initial key exchange. Validating key identity helps
avoid inadvertently connecting to the wrong Salt master, and helps prevent
a potential MiTM attack when establishing the initial connection.
Master Key Fingerprint
----------------------
Print the master key fingerprint by running the following command on the Salt master:
.. code-block:: bash
salt-key -F master
Copy the ``master.pub`` fingerprint from the *Local Keys* section, and then set this value
as the :conf_minion:`master_finger` in the minion configuration file. Save the configuration
file and then restart the Salt minion.
Minion Key Fingerprint
----------------------
Run the following command on each Salt minion to view the minion key fingerprint:
.. code-block:: bash
salt-call --local key.finger
Compare this value to the value that is displayed when you run the
``salt-key --finger <MINION_ID>`` command on the Salt master.
Key Management
==============

15
doc/ref/configuration/minion.rst
View File

@ -891,6 +891,21 @@ minion to clean the keys.
open_mode: False
.. conf_minion:: master_finger
``master_finger``
-----------------
Default: ``''``
Fingerprint of the master public key to validate the identity of your Salt master
before the initial key exchange. The master fingerprint can be found by running
"salt-key -F master" on the Salt master.
.. code-block:: yaml
master_finger: 'ba:30:65:2a:d6:9e:20:4f:d8:b2:f3:a7:d4:65:11:13'
.. conf_minion:: verify_master_pubkey_sign

60
doc/topics/eauth/index.rst
View File

@ -109,7 +109,7 @@ Token expiration time can be set in the Salt master config file.
LDAP and Active Directory
-------------------------
=========================
.. note::
@ -118,43 +118,82 @@ LDAP and Active Directory
Salt supports both user and group authentication for LDAP (and Active Directory
accessed via its LDAP interface)
OpenLDAP and similar systems
----------------------------
LDAP configuration happens in the Salt master configuration file.
Server configuration values and their defaults:
.. code-block:: yaml
# Server to auth against
auth.ldap.server: localhost
# Port to connect via
auth.ldap.port: 389
# Use TLS when connecting
auth.ldap.tls: False
# LDAP scope level, almost always 2
auth.ldap.scope: 2
auth.ldap.uri: ''
auth.ldap.tls: False
# Server specified in URI format
auth.ldap.uri: '' # Overrides .ldap.server, .ldap.port, .ldap.tls above
# Verify server's TLS certificate
auth.ldap.no_verify: False
# Bind to LDAP anonymously to determine group membership
# Active Directory does not allow anonymous binds without special configuration
auth.ldap.anonymous: False
# FOR TESTING ONLY, this is a VERY insecure setting.
# If this is True, the LDAP bind password will be ignored and
# access will be determined by group membership alone with
# the group memberships being retrieved via anonymous bind
auth.ldap.auth_by_group_membership_only: False
# Require authenticating user to be part of this Organizational Unit
# This can be blank if your LDAP schema does not use this kind of OU
auth.ldap.groupou: 'Groups'
# Object Class for groups. An LDAP search will be done to find all groups of this
# class to which the authenticating user belongs.
auth.ldap.groupclass: 'posixGroup'
# Unique ID attribute name for the user
auth.ldap.accountattributename: 'memberUid'
# These are only for Active Directory
auth.ldap.activedirectory: False
auth.ldap.persontype: 'person'
Salt also needs to know which Base DN to search for users and groups and
the DN to bind to:
There are two phases to LDAP authentication. First, Salt authenticates to search for a users's Distinguished Name
and group membership. The user it authenticates as in this phase is often a special LDAP system user with
read-only access to the LDAP directory. After Salt searches the directory to determine the actual user's DN
and groups, it re-authenticates as the user running the Salt commands.
If you are already aware of the structure of your DNs and permissions in your LDAP store are set such that
users can look up their own group memberships, then the first and second users can be the same. To tell Salt this is
the case, omit the ``auth.ldap.bindpw`` parameter. You can template the binddn like this:
.. code-block:: yaml
auth.ldap.basedn: dc=saltstack,dc=com
auth.ldap.binddn: cn=admin,dc=saltstack,dc=com
auth.ldap.binddn: uid={{ username }},cn=users,cn=accounts,dc=saltstack,dc=com
To bind to a DN, a password is required
Salt will use the password entered on the salt command line in place of the bindpw.
To use two separate users, specify the LDAP lookup user in the binddn directive, and include a bindpw like so
.. code-block:: yaml
auth.ldap.binddn: uid=ldaplookup,cn=sysaccounts,cn=etc,dc=saltstack,dc=com
auth.ldap.bindpw: mypassword
Salt uses a filter to find the DN associated with a user. Salt
As mentioned before, Salt uses a filter to find the DN associated with a user. Salt
substitutes the ``{{ username }}`` value for the username when querying LDAP
.. code-block:: yaml
@ -170,6 +209,9 @@ the results are filtered against ``auth.ldap.groupclass``, default
auth.ldap.groupou: Groups
Active Directory
----------------
Active Directory handles group membership differently, and does not utilize the
``groupou`` configuration variable. AD needs the following options in
the master config:
@ -195,7 +237,7 @@ of the user is looked up with the following LDAP search:
)
This should return a distinguishedName that we can use to filter for group
membership. Then the following LDAP query is executed:
membership. Then the following LDAP query is executed:
.. code-block:: text

10
doc/topics/tutorials/walkthrough.rst
View File

@ -191,9 +191,13 @@ The easiest way to accept the minion key is to accept all pending keys:
.. note::
Keys should be verified! The secure thing to do before accepting a key is
to run ``salt-key -f minion-id`` to print the fingerprint of the minion's
public key. This fingerprint can then be compared against the fingerprint
Keys should be verified! Print the master key fingerprint by running ``salt-key -F master``
on the Salt master. Copy the ``master.pub`` fingerprint from the Local Keys section,
and then set this value as the :conf_minion:`master_finger` in the minion configuration
file. Restart the Salt minion.
On the minion, run ``salt-key -f minion-id`` to print the fingerprint of the
minion's public key. This fingerprint can then be compared against the fingerprint
generated on the minion.
On the master:

40
salt/auth/ldap.py
View File

@ -13,7 +13,6 @@ import logging
from salt.exceptions import CommandExecutionError, SaltInvocationError
log = logging.getLogger(__name__)
# Import third party libs
from jinja2 import Environment
try:
@ -112,7 +111,7 @@ class _LDAPConnection(object):
)
def _bind(username, password):
def _bind(username, password, anonymous=False):
'''
Authenticate via an LDAP bind
'''
@ -122,8 +121,10 @@ def _bind(username, password):
connargs = {}
# config params (auth.ldap.*)
params = {
'mandatory': ['uri', 'server', 'port', 'tls', 'no_verify', 'anonymous', 'accountattributename', 'activedirectory'],
'additional': ['binddn', 'bindpw', 'filter', 'groupclass'],
'mandatory': ['uri', 'server', 'port', 'tls', 'no_verify', 'anonymous',
'accountattributename', 'activedirectory'],
'additional': ['binddn', 'bindpw', 'filter', 'groupclass',
'auth_by_group_membership_only'],
}
paramvalues = {}
@ -138,6 +139,7 @@ def _bind(username, password):
#except SaltInvocationError:
# pass
paramvalues['anonymous'] = anonymous
if paramvalues['binddn']:
# the binddn can also be composited, e.g.
# - {{ username }}@domain.com
@ -206,7 +208,10 @@ def _bind(username, password):
connargs['bindpw'] = password
# Attempt bind with user dn and password
log.debug('Attempting LDAP bind with user dn: {0}'.format(connargs['binddn']))
if paramvalues['anonymous']:
log.debug('Attempting anonymous LDAP bind')
else:
log.debug('Attempting LDAP bind with user dn: {0}'.format(connargs['binddn']))
try:
ldap_conn = _LDAPConnection(**connargs).ldap
except Exception:
@ -226,8 +231,8 @@ def auth(username, password):
'''
Simple LDAP auth
'''
if _bind(username, password):
if _bind(username, password, anonymous=_config('auth_by_group_membership_only', mandatory=False) and
_config('anonymous', mandatory=False)):
log.debug('LDAP authentication successful')
return True
else:
@ -252,8 +257,8 @@ def groups(username, **kwargs):
'''
group_list = []
bind = _bind(username, kwargs['password'])
bind = _bind(username, kwargs['password'],
anonymous=_config('anonymous', mandatory=False))
if bind:
log.debug('ldap bind to determine group membership succeeded!')
@ -288,16 +293,25 @@ def groups(username, **kwargs):
group_list.append(entry['cn'][0])
log.debug('User {0} is a member of groups: {1}'.format(username, group_list))
else:
search_results = bind.search_s('ou={0},{1}'.format(_config('groupou'), _config('basedn')),
if _config('groupou'):
search_base = 'ou={0},{1}'.format(_config('groupou'), _config('basedn'))
else:
search_base = '{0}'.format(_config('basedn'))
search_string = '(&({0}={1})(objectClass={2}))'.format(_config('accountattributename'),
username, _config('groupclass'))
search_results = bind.search_s(search_base,
ldap.SCOPE_SUBTREE,
'(&({0}={1})(objectClass={2}))'.format(_config('accountattributename'),
username, _config('groupclass')),
[_config('groupattribute'), 'cn'])
search_string,
[_config('accountattributename'), 'cn'])
for user, entry in search_results:
if username == user.split(',')[0].split('=')[-1]:
for group in entry[_config('groupattribute')]:
group_list.append(group.split(',')[0].split('=')[-1])
log.debug('User {0} is a member of groups: {1}'.format(username, group_list))
if not auth(username, kwargs['password']):
log.error('LDAP username and password do not match')
return []
else:
log.error('ldap bind to determine group membership FAILED!')
return group_list

82
salt/client/__init__.py
View File

@ -209,6 +209,9 @@ class LocalClient(object):
timeout=timeout,
)
if 'jid' in pub_data:
self.event.subscribe(pub_data['jid'])
return pub_data
def _check_pub_data(self, pub_data):
@ -240,6 +243,10 @@ class LocalClient(object):
print('No minions matched the target. '
'No command was sent, no jid was assigned.')
return {}
else:
self.event.subscribe_regex('^syndic/.*/{0}'.format(pub_data['jid']))
self.event.subscribe('salt/job/{0}'.format(pub_data['jid']))
return pub_data
@ -270,9 +277,6 @@ class LocalClient(object):
'''
arg = salt.utils.args.condition_input(arg, kwarg)
# Subscribe to all events and subscribe as early as possible
self.event.subscribe(jid)
try:
pub_data = self.pub(
tgt,
@ -804,8 +808,6 @@ class LocalClient(object):
def get_returns_no_block(
self,
jid,
event=None,
gather_errors=False,
tags_regex=None
):
'''
@ -813,49 +815,16 @@ class LocalClient(object):
Yield either the raw event data or None
Pass a list of additional regular expressions as `tags_regex` to search
the event bus for non-return data, such as minion lists returned from
syndics.
Pass a list of additional regular expressions as `tags_regex` to search
the event bus for non-return data, such as minion lists returned from
syndics.
'''
if event is None:
event = self.event
jid_tag = 'salt/job/{0}'.format(jid)
jid_tag_regex = '^salt/job/{0}'.format(jid)
tag_search = []
tag_search.append(re.compile(jid_tag_regex))
if isinstance(tags_regex, str):
tag_search.append(re.compile(tags_regex))
elif isinstance(tags_regex, list):
for tag in tags_regex:
tag_search.append(re.compile(tag))
while True:
# TODO: this is a check of event type, NOT transport type!
if self.opts.get('transport') in ('zeromq', 'tcp'):
try:
raw = event.get_event_noblock()
if gather_errors:
if (raw and
(raw.get('tag', '').startswith('_salt_error') or
any([tag.search(raw.get('tag', '')) for tag in tag_search]))):
yield raw
else:
if raw and raw.get('tag', '').startswith(jid_tag):
yield raw
else:
yield None
except zmq.ZMQError as ex:
if ex.errno == errno.EAGAIN or ex.errno == errno.EINTR:
yield None
else:
raise
else:
raw = event.get_event_noblock()
if raw and raw.get('tag', '').startswith(jid_tag):
yield raw
else:
yield None
# TODO(driskell): This was previously completely nonblocking.
# Should get_event have a nonblock option?
raw = self.event.get_event(wait=0.01, tag='salt/job/{0}'.format(jid), tags_regex=tags_regex, full=True)
yield raw
def get_iter_returns(
self,
@ -865,7 +834,6 @@ class LocalClient(object):
tgt='*',
tgt_type='glob',
expect_minions=False,
gather_errors=True,
block=True,
**kwargs):
'''
@ -901,9 +869,9 @@ class LocalClient(object):
# iterator for this job's return
if self.opts['order_masters']:
# If we are a MoM, we need to gather expected minions from downstreams masters.
ret_iter = self.get_returns_no_block(jid, gather_errors=gather_errors, tags_regex='^syndic/.*/{0}'.format(jid))
ret_iter = self.get_returns_no_block(jid, tags_regex=['^syndic/.*/{0}'.format(jid)])
else:
ret_iter = self.get_returns_no_block(jid, gather_errors=gather_errors)
ret_iter = self.get_returns_no_block(jid)
# iterator for the info of this job
jinfo_iter = []
timeout_at = time.time() + timeout
@ -922,10 +890,6 @@ class LocalClient(object):
# if we got None, then there were no events
if raw is None:
break
if gather_errors:
if raw['tag'] == '_salt_error' and 'id' in raw['data']:
ret = {raw['data']['id']: raw['data']['data']}
yield ret
if 'minions' in raw.get('data', {}):
minions.update(raw['data']['minions'])
continue
@ -972,15 +936,6 @@ class LocalClient(object):
# if the jinfo has timed out and some minions are still running the job
# re-do the ping
if time.time() > timeout_at and minions_running:
# need our own event listener, so we don't clobber the class one
event = salt.utils.event.get_event(
'master',
self.opts['sock_dir'],
self.opts['transport'],
opts=self.opts,
listen=False)
# start listening for new events, before firing off the pings
event.connect_pub()
# since this is a new ping, no one has responded yet
jinfo = self.gather_job_info(jid, tgt, tgt_type)
minions_running = False
@ -989,7 +944,7 @@ class LocalClient(object):
if 'jid' not in jinfo:
jinfo_iter = []
else:
jinfo_iter = self.get_returns_no_block(jinfo['jid'], event=event)
jinfo_iter = self.get_returns_no_block(jinfo['jid'])
timeout_at = time.time() + self.opts['gather_job_timeout']
# if you are a syndic, wait a little longer
if self.opts['order_masters']:
@ -1051,8 +1006,7 @@ class LocalClient(object):
self,
jid,
minions,
timeout=None,
pending_tags=None):
timeout=None):
'''
Get the returns for the command line interface via the event system
'''

40
salt/cloud/clouds/msazure.py
View File

@ -1691,7 +1691,7 @@ def show_service_certificate(kwargs=None, conn=None, call=None):
.. code-block:: bash
salt-cloud -f show_service_certificate my-azure name=my_service_certificate \
salt-cloud -f show_service_certificate my-azure name=my_service_certificate \\
thumbalgorithm=sha1 thumbprint=0123456789ABCDEF
'''
if call != 'function':
@ -1736,7 +1736,7 @@ def add_service_certificate(kwargs=None, conn=None, call=None):
.. code-block:: bash
salt-cloud -f add_service_certificate my-azure name=my_service_certificate \
salt-cloud -f add_service_certificate my-azure name=my_service_certificate \\
data='...CERT_DATA...' certificate_format=sha1 password=verybadpass
'''
if call != 'function':
@ -1784,7 +1784,7 @@ def delete_service_certificate(kwargs=None, conn=None, call=None):
.. code-block:: bash
salt-cloud -f delete_service_certificate my-azure name=my_service_certificate \
salt-cloud -f delete_service_certificate my-azure name=my_service_certificate \\
thumbalgorithm=sha1 thumbprint=0123456789ABCDEF
'''
if call != 'function':
@ -1855,7 +1855,7 @@ def show_management_certificate(kwargs=None, conn=None, call=None):
.. code-block:: bash
salt-cloud -f get_management_certificate my-azure name=my_management_certificate \
salt-cloud -f get_management_certificate my-azure name=my_management_certificate \\
thumbalgorithm=sha1 thumbprint=0123456789ABCDEF
'''
if call != 'function':
@ -1890,7 +1890,7 @@ def add_management_certificate(kwargs=None, conn=None, call=None):
.. code-block:: bash
salt-cloud -f add_management_certificate my-azure public_key='...PUBKEY...' \
salt-cloud -f add_management_certificate my-azure public_key='...PUBKEY...' \\
thumbprint=0123456789ABCDEF data='...CERT_DATA...'
'''
if call != 'function':
@ -1934,7 +1934,7 @@ def delete_management_certificate(kwargs=None, conn=None, call=None):
.. code-block:: bash
salt-cloud -f delete_management_certificate my-azure name=my_management_certificate \
salt-cloud -f delete_management_certificate my-azure name=my_management_certificate \\
thumbalgorithm=sha1 thumbprint=0123456789ABCDEF
'''
if call != 'function':
@ -2010,7 +2010,15 @@ def list_input_endpoints(kwargs=None, conn=None, call=None):
kwargs['service'],
kwargs['deployment'],
)
data = query(path)
if data is None:
raise SaltCloudSystemExit(
'There was an error listing endpoints with the {0} service on the {1} deployment.'.format(
kwargs['service'],
kwargs['deployment']
)
)
ret = {}
for item in data:
@ -2034,7 +2042,7 @@ def show_input_endpoint(kwargs=None, conn=None, call=None):
.. code-block:: bash
salt-cloud -f show_input_endpoint my-azure service=myservice \
salt-cloud -f show_input_endpoint my-azure service=myservice \\
deployment=mydeployment name=SSH
'''
if call != 'function':
@ -2067,9 +2075,9 @@ def update_input_endpoint(kwargs=None, conn=None, call=None, activity='update'):
.. code-block:: bash
salt-cloud -f update_input_endpoint my-azure service=myservice \
deployment=mydeployment role=myrole name=HTTP local_port=80 \
port=80 protocol=tcp enable_direct_server_return=False \
salt-cloud -f update_input_endpoint my-azure service=myservice \\
deployment=mydeployment role=myrole name=HTTP local_port=80 \\
port=80 protocol=tcp enable_direct_server_return=False \\
timeout_for_tcp_idle_connection=4
'''
if call != 'function':
@ -2181,9 +2189,9 @@ def add_input_endpoint(kwargs=None, conn=None, call=None):
.. code-block:: bash
salt-cloud -f add_input_endpoint my-azure service=myservice \
deployment=mydeployment role=myrole name=HTTP local_port=80 \
port=80 protocol=tcp enable_direct_server_return=False \
salt-cloud -f add_input_endpoint my-azure service=myservice \\
deployment=mydeployment role=myrole name=HTTP local_port=80 \\
port=80 protocol=tcp enable_direct_server_return=False \\
timeout_for_tcp_idle_connection=4
'''
return update_input_endpoint(
@ -2205,7 +2213,7 @@ def delete_input_endpoint(kwargs=None, conn=None, call=None):
.. code-block:: bash
salt-cloud -f delete_input_endpoint my-azure service=myservice \
salt-cloud -f delete_input_endpoint my-azure service=myservice \\
deployment=mydeployment role=myrole name=HTTP
'''
return update_input_endpoint(
@ -2293,7 +2301,7 @@ def show_affinity_group(kwargs=None, conn=None, call=None):
.. code-block:: bash
salt-cloud -f show_affinity_group my-azure service=myservice \
salt-cloud -f show_affinity_group my-azure service=myservice \\
deployment=mydeployment name=SSH
'''
if call != 'function':
@ -2674,7 +2682,7 @@ def set_storage_container_metadata(kwargs=None, storage_conn=None, call=None):
.. code-block:: bash
salt-cloud -f set_storage_container my-azure name=mycontainer \
salt-cloud -f set_storage_container my-azure name=mycontainer \\
x_ms_meta_name_values='{"my_name": "my_value"}'
name:

28
salt/master.py
View File

@ -1715,9 +1715,14 @@ class ClearFuncs(object):
clear_load['groups'] = groups
return self.loadauth.mk_token(clear_load)
except Exception as exc:
import sys
import traceback
type_, value_, traceback_ = sys.exc_info()
log.error(
'Exception occurred while authenticating: {0}'.format(exc)
)
log.error(traceback.format_exception(type_, value_, traceback_))
return ''
def get_token(self, clear_load):
@ -1823,18 +1828,13 @@ class ClearFuncs(object):
)
return ''
try:
# The username with which we are attempting to auth
name = self.loadauth.load_name(extra)
# The groups to which this user belongs
groups = self.loadauth.get_groups(extra)
# The configured auth groups
group_perm_keys = [
item for item in self.opts['external_auth'][extra['eauth']]
if item.endswith('%')
]
name = self.loadauth.load_name(extra) # The username we are attempting to auth with
groups = self.loadauth.get_groups(extra) # The groups this user belongs to
if groups is None:
groups = []
group_perm_keys = [item for item in self.opts['external_auth'][extra['eauth']] if item.endswith('%')] # The configured auth groups
# First we need to know if the user is allowed to proceed via
# any of their group memberships.
# First we need to know if the user is allowed to proceed via any of their group memberships.
group_auth_match = False
for group_config in group_perm_keys:
group_config = group_config.rstrip('%')
@ -1874,9 +1874,15 @@ class ClearFuncs(object):
return ''
except Exception as exc:
import sys
import traceback
type_, value_, traceback_ = sys.exc_info()
log.error(
'Exception occurred while authenticating: {0}'.format(exc)
)
log.error(traceback.format_exception(
type_, value_, traceback_))
return ''
# auth_list = self.opts['external_auth'][extra['eauth']][name] if name in self.opts['external_auth'][extra['eauth']] else self.opts['external_auth'][extra['eauth']]['*']

41
salt/modules/npm.py
View File

@ -3,6 +3,10 @@
Manage and query NPM packages.
'''
from __future__ import absolute_import
try:
from shlex import quote as _cmd_quote # pylint: disable=E0611
except ImportError:
from pipes import quote as _cmd_quote
# Import python libs
import json
@ -44,7 +48,7 @@ def _check_valid_version(salt):
'''
# pylint: disable=no-member
npm_version = distutils.version.LooseVersion(
salt['cmd.run']('npm --version'))
salt['cmd.run']('npm --version', python_shell=True))
valid_version = distutils.version.LooseVersion('1.2')
# pylint: enable=no-member
if npm_version < valid_version:
@ -105,19 +109,31 @@ def install(pkg=None,
salt '*' npm.install coffee-script@1.0.1
'''
# Protect against injection
if pkg:
pkg = _cmd_quote(pkg)
if pkgs:
pkg_list = []
for item in pkgs:
pkg_list.append(_cmd_quote(item))
pkgs = pkg_list
if registry:
registry = _cmd_quote(registry)
cmd = 'npm install --silent --json'
cmd = ['npm', 'install', '--silent', '--json']
if dir is None:
cmd += ' --global'
cmd.append(' --global')
if registry:
cmd += ' --registry="{0}"'.format(registry)
cmd.append(' --registry="{0}"'.format(registry))
if pkg:
cmd += ' "{0}"'.format(pkg)
cmd.append(pkg)
elif pkgs:
cmd += ' "{0}"'.format('" "'.join(pkgs))
cmd.extend(pkgs)
else:
return 'No package name specified'
if env is None:
env = {}
@ -127,7 +143,8 @@ def install(pkg=None,
if uid:
env.update({'SUDO_UID': b'{0}'.format(uid), 'SUDO_USER': b''})
result = __salt__['cmd.run_all'](cmd, python_shell=False, cwd=dir, runas=runas, env=env)
cmd = ' '.join(cmd)
result = __salt__['cmd.run_all'](cmd, python_shell=True, cwd=dir, runas=runas, env=env)
if result['retcode'] != 0:
raise CommandExecutionError(result['stderr'])
@ -190,6 +207,9 @@ def uninstall(pkg,
salt '*' npm.uninstall coffee-script
'''
# Protect against injection
if pkg:
pkg = _cmd_quote(pkg)
if env is None:
env = {}
@ -206,7 +226,7 @@ def uninstall(pkg,
cmd += ' "{0}"'.format(pkg)
result = __salt__['cmd.run_all'](cmd, python_shell=False, cwd=dir, runas=runas, env=env)
result = __salt__['cmd.run_all'](cmd, python_shell=True, cwd=dir, runas=runas, env=env)
if result['retcode'] != 0:
log.error(result['stderr'])
@ -250,6 +270,9 @@ def list_(pkg=None,
salt '*' npm.list
'''
# Protect against injection
if pkg:
pkg = _cmd_quote(pkg)
if env is None:
env = {}
@ -272,7 +295,7 @@ def list_(pkg=None,
cwd=dir,
runas=runas,
env=env,
python_shell=False,
python_shell=True,
ignore_retcode=True)
# npm will return error code 1 for both no packages found and an actual

14
salt/modules/state.py
View File

@ -244,13 +244,25 @@ def template(tem, queue=False, **kwargs):
salt '*' state.template '<Path to template on the minion>'
'''
if 'env' in kwargs:
salt.utils.warn_until(
'Boron',
'Passing a salt environment should be done using \'saltenv\' '
'not \'env\'. This functionality will be removed in Salt Boron.'
)
saltenv = kwargs['env']
elif 'saltenv' in kwargs:
saltenv = kwargs['saltenv']
else:
saltenv = ''
conflict = _check_queue(queue, kwargs)
if conflict is not None:
return conflict
st_ = salt.state.HighState(__opts__)
if not tem.endswith('.sls'):
tem = '{sls}.sls'.format(sls=tem)
high_state, errors = st_.render_state(tem, None, '', None, local=True)
high_state, errors = st_.render_state(tem, saltenv, '', None, local=True)
if errors:
__context__['retcode'] = 1
return errors

2
salt/netapi/rest_tornado/saltnado.py
View File

@ -263,8 +263,6 @@ class EventListener(object):
listen=True,
)
self.event.subscribe() # start listening for events immediately
# tag -> list of futures
self.tag_map = defaultdict(list)

2
salt/state.py
View File

@ -2701,6 +2701,8 @@ class BaseHighState(object):
inc_sls = '.'.join(p_comps[:-level_count] + [include])
if env_key != xenv_key:
if matches is None:
matches = []
# Resolve inc_sls in the specified environment
if env_key in matches or fnmatch.filter(self.avail[env_key], inc_sls):
resolved_envs = [env_key]

2
salt/states/dockerio.py
View File

@ -70,7 +70,7 @@ Available Functions
docker.running:
- container: mysuperdocker
- image: corp/mysuperdocker_img
- ports:
- port_bindings:
- "5000/tcp":
HostIp: ""
HostPort: "5000"

23
salt/utils/__init__.py
View File

@ -1625,6 +1625,29 @@ def is_smartos_globalzone():
return False
@real_memoize
def is_smartos_zone():
'''
Function to return if host is SmartOS (Illumos) and not the gz
'''
if not is_smartos():
return False
else:
cmd = ['zonename']
try:
zonename = subprocess.Popen(
cmd, shell=False,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
except OSError:
return False
if zonename.returncode:
return False
if zonename.stdout.read().strip() == 'global':
return False
return True
@real_memoize
def is_freebsd():
'''

127
salt/utils/event.py
View File

@ -61,6 +61,7 @@ import hashlib
import logging
import datetime
import multiprocessing
import re
from collections import MutableMapping
# Import third party libs
@ -187,8 +188,11 @@ class SaltEvent(object):
opts['ipc_mode'] = 'tcp'
self.puburi, self.pulluri = self.__load_uri(sock_dir, node)
if listen:
self.subscribe()
self.connect_pub()
self.pending_tags = []
self.pending_rtags = []
self.pending_events = []
self.connect_pub()
self.__load_cache_regex()
@classmethod
@ -258,17 +262,54 @@ class SaltEvent(object):
)
return puburi, pulluri
def subscribe(self, tag=None):
def subscribe(self, tag):
'''
Subscribe to events matching the passed tag.
'''
if not self.cpub:
self.connect_pub()
def unsubscribe(self, tag=None):
If you do not subscribe to a tag, events will be discarded by calls to
get_event that request a different tag. In contexts where many different
jobs are outstanding it is important to subscribe to prevent one call
to get_event from discarding a response required by a subsequent call
to get_event.
'''
self.pending_tags.append(tag)
return
def subscribe_regex(self, tag_regex):
'''
Subscribe to events matching the passed tag expression.
If you do not subscribe to a tag, events will be discarded by calls to
get_event that request a different tag. In contexts where many different
jobs are outstanding it is important to subscribe to prevent one call
to get_event from discarding a response required by a subsequent call
to get_event.
'''
self.pending_rtags.append(re.compile(tag_regex))
return
def unsubscribe(self, tag):
'''
Un-subscribe to events matching the passed tag.
'''
self.pending_tags.remove(tag)
return
def unsubscribe_regex(self, tag_regex):
'''
Un-subscribe to events matching the passed tag.
'''
self.pending_rtags.remove(tag_regex)
old_events = self.pending_events
self.pending_events = []
for evt in old_events:
if any(evt['tag'].startswith(ptag) for ptag in self.pending_tags) or any(rtag.search(evt['tag']) for rtag in self.pending_rtags):
self.pending_events.append(evt)
return
def connect_pub(self):
@ -308,13 +349,13 @@ class SaltEvent(object):
match_type = self.opts.get('event_match_type', 'startswith')
return getattr(self, '_match_tag_{0}'.format(match_type), None)
def _check_pending(self, tag, pending_tags, match_func=None):
def _check_pending(self, tag, tags_regex, match_func=None):
"""Check the pending_events list for events that match the tag
:param tag: The tag to search for
:type tag: str
:param pending_tags: List of tags to preserve
:type pending_tags: list[str]
:param tags_regex: List of re expressions to search for also
:type tags_regex: list[re.compile()]
:return:
"""
if match_func is None:
@ -323,13 +364,17 @@ class SaltEvent(object):
self.pending_events = []
ret = None
for evt in old_events:
if match_func(evt['tag'], tag):
if match_func(evt['tag'], tag) or any(rtag.search(evt['tag']) for rtag in tags_regex):
if ret is None:
ret = evt
log.trace('get_event() returning cached event = {0}'.format(ret))
else:
self.pending_events.append(evt)
elif any(match_func(evt['tag'], ptag) for ptag in pending_tags):
elif any(match_func(evt['tag'], ptag) for ptag in self.pending_tags) \
or any(rtag.search(evt['tag']) for rtag in self.pending_rtags):
self.pending_events.append(evt)
else:
log.trace('get_event() discarding cached event that no longer has any subscriptions = {0}'.format(evt))
return ret
def _match_tag_startswith(self, event_tag, search_tag):
@ -364,7 +409,7 @@ class SaltEvent(object):
'''
return self.cache_regex.get(search_tag).search(event_tag) is not None
def _get_event(self, wait, tag, pending_tags, match_func=None):
def _get_event(self, wait, tag, tags_regex, match_func=None):
if match_func is None:
match_func = self._get_match_func()
start = time.time()
@ -387,8 +432,12 @@ class SaltEvent(object):
else:
raise
if not match_func(ret['tag'], tag): # tag not match
if any(match_func(ret['tag'], ptag) for ptag in pending_tags):
if not match_func(ret['tag'], tag) \
and not any(rtag.search(ret['tag']) for rtag in tags_regex):
# tag not match
if any(match_func(ret['tag'], ptag) for ptag in self.pending_tags) \
or any(rtag.search(ret['tag']) for rtag in self.pending_rtags):
log.trace('get_event() caching unwanted event = {0}'.format(ret))
self.pending_events.append(ret)
if wait: # only update the wait timeout if we had one
wait = timeout_at - time.time()
@ -399,8 +448,8 @@ class SaltEvent(object):
return None
def get_event(self, wait=5, tag='', full=False, use_pending=False,
pending_tags=None, match_type=None):
def get_event(self, wait=5, tag='', tags_regex=None, full=False,
match_type=None):
'''
Get a single publication.
IF no publication available THEN block for up to wait seconds
@ -408,21 +457,10 @@ class SaltEvent(object):
IF wait is 0 then block forever.
New in Boron always checks the list of pending events
New in @TBD optionally set match_type
use_pending
Defines whether to keep all unconsumed events in a pending_events
list, or to discard events that don't match the requested tag. If
set to True, MAY CAUSE MEMORY LEAKS.
pending_tags
Add any events matching the listed tags to the pending queue.
Still MAY CAUSE MEMORY LEAKS but less likely than use_pending
assuming you later get_event for the tags you've listed here
New in Boron
A tag specification can be given to only return publications with a tag
STARTING WITH a given string (tag) OR MATCHING one or more string
regular expressions (tags_regex list). If tag is not specified or given
as an empty string, all events are considered.
match_type
Set the function to match the search tag with event tags.
@ -432,18 +470,31 @@ class SaltEvent(object):
- 'regex' : regex search '^' + tag event tags
Default is opts['event_match_type'] or 'startswith'
New in @TBD
.. versionadded:: Boron
Searches cached publications first. If no cached publications are found
that match the given tag specification, new publications are received
and checked.
If a publication is received that does not match the tag specification,
it is DISCARDED unless it is subscribed to via subscribe() and
subscribe_regex() which will cause it to be cached.
If a caller is not going to call get_event immediately after sending a
request, it MUST subscribe the result to ensure the response is not lost
should other regions of code call get_event for other purposes.
'''
match_func = self._get_match_func(match_type)
if use_pending:
pending_tags = ['']
elif pending_tags is None:
pending_tags = []
ret = self._check_pending(tag, pending_tags, match_func)
if tags_regex is None:
tags_regex = []
else:
tags_regex = [re.compile(rtag) for rtag in tags_regex]
ret = self._check_pending(tag, tags_regex, match_func)
if ret is None:
ret = self._get_event(wait, tag, pending_tags, match_func)
ret = self._get_event(wait, tag, tags_regex, match_func)
if ret is None or full:
return ret

122
tests/unit/utils/event_test.py
View File

@ -159,11 +159,10 @@ class TestSaltEvent(TestCase):
)
)
def test_event_subscription(self):
def test_event_single(self):
'''Test a single event is received'''
with eventpublisher_process():
me = event.MasterEvent(SOCK_DIR, listen=True)
me.subscribe()
me.fire_event({'data': 'foo1'}, 'evt1')
evt1 = me.get_event(tag='evt1')
self.assertGotEvent(evt1, {'data': 'foo1'})
@ -172,7 +171,6 @@ class TestSaltEvent(TestCase):
'''Test no event is received if the timeout is reached'''
with eventpublisher_process():
me = event.MasterEvent(SOCK_DIR, listen=True)
me.subscribe()
me.fire_event({'data': 'foo1'}, 'evt1')
evt1 = me.get_event(tag='evt1')
self.assertGotEvent(evt1, {'data': 'foo1'})
@ -183,89 +181,81 @@ class TestSaltEvent(TestCase):
'''Test no wait timeout, we should block forever, until we get one '''
with eventpublisher_process():
me = event.MasterEvent(SOCK_DIR, listen=True)
me.subscribe()
me.fire_event({'data': 'foo1'}, 'evt1')
me.fire_event({'data': 'foo2'}, 'evt2')
evt = me.get_event(tag='evt2', wait=0)
with eventsender_process({'data': 'foo2'}, 'evt2', 5):
evt = me.get_event(tag='evt2', wait=0)
self.assertGotEvent(evt, {'data': 'foo2'})
def test_event_subscription_matching(self):
'''Test a subscription startswith matching'''
def test_event_matching(self):
'''Test a startswith match'''
with eventpublisher_process():
me = event.MasterEvent(SOCK_DIR, listen=True)
me.subscribe()
me.fire_event({'data': 'foo1'}, 'evt1')
evt1 = me.get_event(tag='evt1')
evt1 = me.get_event(tag='ev')
self.assertGotEvent(evt1, {'data': 'foo1'})
def test_event_subscription_matching_all(self):
'''Test a subscription matching'''
def test_event_matching_regex(self):
'''Test a regex match'''
with eventpublisher_process():
me = event.MasterEvent(SOCK_DIR, listen=True)
me.fire_event({'data': 'foo1'}, 'evt1')
evt1 = me.get_event(tag='not', tags_regex=['^ev'])
self.assertGotEvent(evt1, {'data': 'foo1'})
def test_event_matching_all(self):
'''Test an all match'''
with eventpublisher_process():
me = event.MasterEvent(SOCK_DIR, listen=True)
me.subscribe()
me.fire_event({'data': 'foo1'}, 'evt1')
evt1 = me.get_event(tag='')
self.assertGotEvent(evt1, {'data': 'foo1'})
def test_event_not_subscribed(self):
'''Test get event ignores non-subscribed events'''
'''Test get_event drops non-subscribed events'''
with eventpublisher_process():
me = event.MasterEvent(SOCK_DIR, listen=True)
me.subscribe()
with eventsender_process({'data': 'foo1'}, 'evt1', 5):
me.fire_event({'data': 'foo1'}, 'evt2')
evt1 = me.get_event(tag='evt1', wait=10)
self.assertGotEvent(evt1, {'data': 'foo1'})
def test_event_multiple_subscriptions(self):
'''Test multiple subscriptions do not interfere'''
with eventpublisher_process():
me = event.MasterEvent(SOCK_DIR, listen=True)
me.subscribe()
with eventsender_process({'data': 'foo1'}, 'evt1', 5):
me.fire_event({'data': 'foo1'}, 'evt2')
evt1 = me.get_event(tag='evt1', wait=10)
self.assertGotEvent(evt1, {'data': 'foo1'})
def test_event_multiple_clients(self):
'''Test event is received by multiple clients'''
with eventpublisher_process():
me1 = event.MasterEvent(SOCK_DIR, listen=True)
me1.subscribe()
me2 = event.MasterEvent(SOCK_DIR, listen=True)
me2.subscribe()
me1.fire_event({'data': 'foo1'}, 'evt1')
evt1 = me1.get_event(tag='evt1')
self.assertGotEvent(evt1, {'data': 'foo1'})
# Can't replicate this failure in the wild, need to fix the
# test system bug here
#evt2 = me2.get_event(tag='evt1')
#self.assertGotEvent(evt2, {'data': 'foo1'})
def test_event_nested_subs(self):
'''Test nested event subscriptions do not drop events, issue #8580'''
with eventpublisher_process():
me = event.MasterEvent(SOCK_DIR, listen=True)
me.subscribe()
me.fire_event({'data': 'foo1'}, 'evt1')
me.fire_event({'data': 'foo2'}, 'evt2')
# Since we now drop unrelated events to avoid memory leaks, see http://goo.gl/2n3L09 commit bcbc5340ef, the
# calls below will return None and will drop the unrelated events
evt2 = me.get_event(tag='evt2')
evt1 = me.get_event(tag='evt1')
self.assertGotEvent(evt2, {'data': 'foo2'})
# This one will be None because we're dripping unrelated events
self.assertIsNone(evt1)
# Fire events again
me.fire_event({'data': 'foo3'}, 'evt3')
me.fire_event({'data': 'foo4'}, 'evt4')
# We not force unrelated pending events not to be dropped, so both of the event below work and are not
# None
evt2 = me.get_event(tag='evt4', use_pending=True)
evt1 = me.get_event(tag='evt3', use_pending=True)
self.assertGotEvent(evt2, {'data': 'foo4'})
self.assertGotEvent(evt1, {'data': 'foo3'})
def test_event_subscription_cache(self):
'''Test subscriptions cache a message until requested'''
with eventpublisher_process():
me = event.MasterEvent(SOCK_DIR, listen=True)
me.subscribe('evt1')
me.fire_event({'data': 'foo1'}, 'evt1')
me.fire_event({'data': 'foo2'}, 'evt2')
evt2 = me.get_event(tag='evt2')
evt1 = me.get_event(tag='evt1')
self.assertGotEvent(evt2, {'data': 'foo2'})
self.assertGotEvent(evt1, {'data': 'foo1'})
def test_event_subscriptions_cache_regex(self):
'''Test regex subscriptions cache a message until requested'''
with eventpublisher_process():
me = event.MasterEvent(SOCK_DIR, listen=True)
me.subscribe_regex('1$')
me.fire_event({'data': 'foo1'}, 'evt1')
me.fire_event({'data': 'foo2'}, 'evt2')
evt2 = me.get_event(tag='evt2')
evt1 = me.get_event(tag='evt1')
self.assertGotEvent(evt2, {'data': 'foo2'})
self.assertGotEvent(evt1, {'data': 'foo1'})
# TODO: @driskell fix these up please
@skipIf(True, '@driskell will fix these up')
def test_event_multiple_clients(self):
'''Test event is received by multiple clients'''
with eventpublisher_process():
me1 = event.MasterEvent(SOCK_DIR)
me2 = event.MasterEvent(SOCK_DIR)
me1.fire_event({'data': 'foo1'}, 'evt1')
evt1 = me1.get_event(tag='evt1')
self.assertGotEvent(evt1, {'data': 'foo1'})
evt2 = me2.get_event(tag='evt1')
self.assertGotEvent(evt2, {'data': 'foo1'})
@expectedFailure
def test_event_nested_sub_all(self):
@ -273,7 +263,6 @@ class TestSaltEvent(TestCase):
# Show why not to call get_event(tag='')
with eventpublisher_process():
me = event.MasterEvent(SOCK_DIR, listen=True)
me.subscribe()
me.fire_event({'data': 'foo1'}, 'evt1')
me.fire_event({'data': 'foo2'}, 'evt2')
evt2 = me.get_event(tag='')
@ -285,17 +274,17 @@ class TestSaltEvent(TestCase):
'''Test a large number of events, one at a time'''
with eventpublisher_process():
me = event.MasterEvent(SOCK_DIR, listen=True)
me.subscribe()
for i in range(500):
me.fire_event({'data': '{0}'.format(i)}, 'testevents')
evt = me.get_event(tag='testevents')
self.assertGotEvent(evt, {'data': '{0}'.format(i)}, 'Event {0}'.format(i))
# TODO: @driskell fix these up please
@skipIf(True, '@driskell will fix these up')
def test_event_many_backlog(self):
'''Test a large number of events, send all then recv all'''
with eventpublisher_process():
me = event.MasterEvent(SOCK_DIR, listen=True)
me.subscribe()
# Must not exceed zmq HWM
for i in range(500):
me.fire_event({'data': '{0}'.format(i)}, 'testevents')
@ -309,7 +298,6 @@ class TestSaltEvent(TestCase):
'''Tests that sending an event through fire_master generates expected event'''
with eventpublisher_process():
me = event.MasterEvent(SOCK_DIR, listen=True)
me.subscribe()
data = {'data': 'foo1'}
me.fire_master(data, 'test_master')