valitydev/salt
14
0
Fork 0
mirror of https://github.com/valitydev/salt.git synced 2024-11-07 17:09:03 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'upstream/develop' into etcd_mods

Browse Source
This commit is contained in:
Khris Richardson 2014-06-05 21:47:08 -05:00
commit 0d7366ee70
38 changed files with 847 additions and 390 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
.testing.pylintrc
View File

@ -69,7 +69,8 @@ disable=R,
E8125,
E8126,
E8127,
E8128
E8128,
E8265
# Disabled:
# R* [refactoring suggestions & reports]
@ -103,6 +104,7 @@ disable=R,
# F0401 (import-error)
#
# E812* All PEP8 E12*
# E8265 PEP8 E265 - block comment should start with "# "
# E8501 PEP8 line too long
[REPORTS]

14
doc/ref/configuration/master.rst
View File

@ -358,6 +358,20 @@ only the cache for the mine system.
enforce_mine_cache: False
``max_minions``
---------------
Default: 0
The number of minions the master should allow to connect. Use this to accomodate
the number of minions per master if you have different types of hardware serving
your minions. The default of ``0`` means unlimited connections. Please note, that
this can slow down the authentication process a bit in large setups.
.. code-block:: yaml
max_minions: 100
``presence_events``
----------------------

53
doc/ref/configuration/minion.rst
View File

@ -37,7 +37,7 @@ Default: ``salt``
The master is, by default, staticaly configured by the `master` setting, but
if desired, the master can be dynamically configured. The `master` value can
be set to a module function will will be executed and will assume that the
be set to a module function which will be executed and will assume that the
returning value is the ip or hostname of the desired master. In addition to
specifying the function to execute to detect the master the
:conf_minion:`master_type`, option must be set to 'func'.
@ -46,6 +46,19 @@ specifying the function to execute to detect the master the
master: module.function
The `master` can also be a list of masters the minion should try to connect
to. If the first master fails or rejects the minions connection (for example
when too many minions are already connected), the minion will try the next
master in the given list. For this to work, :conf_minion:`master_type` must
be set to 'failover'. If `master_type` is not set, the minion will be in
multimaster mode: :ref:`multi master <topics-tutorials-multimaster>`
.. code-block:: yaml
master:
- address1
- address2
.. conf_minion:: master_type
@ -54,16 +67,40 @@ specifying the function to execute to detect the master the
Default: ``str``
The type of the :conf_minion:`master` variable. If the master needs to be
dynamically assigned by executing a function instead of reading in the static
master value, set this to 'func'. This can be used to manage the minion's
master setting from an execution module. By simply changeing the algorithm
in the module to return a new master ip/fqdn, restart the minion and it will
connect to the new master.
The type of the :conf_minion:`master` variable. Can be either 'func' or
'failover'.
If the master needs to be dynamically assigned by executing a function
instead of reading in the static master value, set this to 'func'.
This can be used to manage the minion's master setting from an execution
module. By simply changing the algorithm in the module to return a new
master ip/fqdn, restart the minion and it will connect to the new master.
.. code-block:: yaml
master_type: str
master_type: 'func'
If it is set to 'failover', :conf_minion:`master` has to be a list of master
addresses. The minion will then try one master after the other, until it
successfully connects.
.. code-block:: yaml
master_type: 'failover'
``master_shuffle``
---------------
Default: ``False``
If :conf_minion:`master` is list of addresses, shuffle them before trying
to connect to distribute the minions over all available masters. This uses
pythons random.shuffle method.
.. code-block:: yaml
master_shuffle: True
.. conf_minion:: master_port

146
doc/ref/states/ordering.rst
View File

@ -144,151 +144,13 @@ more requisites. Both requisite types can also be separately declared:
In this example, the httpd service is only going to be started if the package,
user, group and file are executed successfully.
The Require Requisite
---------------------
The foundation of the requisite system is the ``require`` requisite. The
require requisite ensures that the required state(s) are executed before the
requiring state. So, if a state is declared that sets down a vimrc, then it
would be pertinent to make sure that the vimrc file would only be set down if
the vim package has been installed:
Requisite Documentation
-----------------------
.. code-block:: yaml
For detailed information on each of the individual requisites, :ref:`please
look here. <requisites>`
vim:
pkg:
- installed
file.managed:
- source: salt://vim/vimrc
- require:
- pkg: vim
In this case, the vimrc file will only be applied by Salt if and after the vim
package is installed.
The Watch Requisite
-------------------
The ``watch`` requisite is more advanced than the ``require`` requisite. The
watch requisite executes the same logic as require (therefore if something is
watched it does not need to also be required) with the addition of executing
logic if the required states have changed in some way.
The watch requisite checks to see if the watched states have returned any
changes. If the watched state returns changes, and the watched states execute
successfully, then the watching state will execute a function that reacts to
the changes in the watched states.
Perhaps an example can better explain the behavior:
.. code-block:: yaml
redis:
pkg:
- latest
file.managed:
- source: salt://redis/redis.conf
- name: /etc/redis.conf
- require:
- pkg: redis
service.running:
- enable: True
- watch:
- file: /etc/redis.conf
- pkg: redis
In this example, the redis service will only be started if the file
/etc/redis.conf is applied, and the file is only applied if the package is
installed. This is normal require behavior, but if the watched file changes,
or the watched package is installed or upgraded, then the redis service is
restarted.
.. note::
To reiterate: watch does not alter the original behavior of a function in
any way. The original behavior stays, but additional behavior (defined by
mod_watch as explored below) will be run if there are changes in the
watched state. This is why, for example, we have to have a ``cmd.wait``
state for watching purposes. If you examine the source code, you'll see
that ``cmd.wait`` is an empty function. However, you'll notice that
``mod_watch`` is actually just an alias of ``cmd.run``. So if there are
changes, we run the command, otherwise, we do nothing.
Watch and the mod_watch Function
--------------------------------
The watch requisite is based on the ``mod_watch`` function. Python state
modules can include a function called ``mod_watch`` which is then called
if the watch call is invoked. When ``mod_watch`` is called depends on the
execution of the watched state, which:
- If no changes then just run the watching state itself as usual.
``mod_watch`` is not called. This behavior is same as using a ``require``.
- If changes then run the watching state *AND* if that changes nothing then
react by calling ``mod_watch``.
When reacting, in the case of the service module the underlying service is
restarted. In the case of the cmd state the command is executed.
The ``mod_watch`` function for the service state looks like this:
.. code-block:: python
def mod_watch(name, sig=None, reload=False, full_restart=False):
'''
The service watcher, called to invoke the watch command.
name
The name of the init or rc script used to manage the service
sig
The string to search for when looking for the service process with ps
'''
if __salt__['service.status'](name, sig):
if 'service.reload' in __salt__ and reload:
restart_func = __salt__['service.reload']
elif 'service.full_restart' in __salt__ and full_restart:
restart_func = __salt__['service.full_restart']
else:
restart_func = __salt__['service.restart']
else:
restart_func = __salt__['service.start']
result = restart_func(name)
return {'name': name,
'changes': {name: result},
'result': result,
'comment': 'Service restarted' if result else \
'Failed to restart the service'
}
The watch requisite only works if the state that is watching has a
``mod_watch`` function written. If watch is set on a state that does not have
a ``mod_watch`` function (like pkg), then the listed states will behave only
as if they were under a ``require`` statement.
Also notice that a ``mod_watch`` may accept additional keyword arguments,
which, in the sls file, will be taken from the same set of arguments specified
for the state that includes the ``watch`` requisite. This means, for the
earlier ``service.running`` example above, the service can be set to
``reload`` instead of restart like this:
.. code-block:: yaml
redis:
# ... other state declarations omitted ...
service.running:
- enable: True
- reload: True
- watch:
- file: /etc/redis.conf
- pkg: redis
.. _ordering_order:
The Order Option
================

114
doc/ref/states/requisites.rst
View File

@ -8,11 +8,11 @@ The Salt requisite system is used to create relationships between states. The
core idea being that, when one state is dependent somehow on another, that
inter-dependency can be easily defined.
Requisites come in two types: Direct requisites (such as ``require`` and ``watch``),
and requisite_ins (such as ``require_in`` and ``watch_in``). The relationships are
directional: a direct requisite requires something from another state, while
requisite_ins operate in the other direction. A requisite_in contains something that
is required by another state. The following example demonstrates a direct requisite:
Requisites come in two types: Direct requisites (such as ``require``),
and requisite_ins (such as ``require_in``). The relationships are
directional: a direct requisite requires something from another state.
However, a requisite_in inserts a requisite into the targeted state pointing to
the targeting state. The following example demonstrates a direct requisite:
.. code-block:: yaml
@ -43,7 +43,8 @@ something", requisite_ins say "Someone depends on me":
So here, with a requisite_in, the same thing is accomplished as in the first
example, but the other way around. The vim package is saying "/etc/vimrc depends
on me".
on me". This will result in a ``require`` being inserted into the
``/etc/vimrc`` state which targets the ``vim`` state.
In the end, a single dependency map is created and everything is executed in a
finite and predictable order.
@ -65,11 +66,12 @@ finite and predictable order.
Direct Requisite and Requisite_in types
=======================================
There are four direct requisite statements that can be used in Salt: ``require``,
``watch``, ``prereq``, and ``use``. Each direct requisite also has a corresponding
requisite_in: ``require_in``, ``watch_in``, ``prereq_in`` and ``use_in``. All of the
requisites define specific relationships and always work with the dependency
logic defined above.
There are six direct requisite statements that can be used in Salt:
``require``, ``watch``, ``prereq``, ``use``, ``onchanges``, and ``onfail``.
Each direct requisite also has a corresponding requisite_in: ``require_in``,
``watch_in``, ``prereq_in``, ``use_in``, ``onchanges_in``, and ``onfail_in``.
All of the requisites define specific relationships and always work with the
dependency logic defined above.
require
-------
@ -268,8 +270,8 @@ onchanges
.. versionadded:: Helium
The ``onchanges`` requisite makes a state only apply if the required states
generate changes. This can be a useful way to execute a post hook after
changing aspects of a system.
generate changes, and if the watched state's "result" is ``True``. This can be
a useful way to execute a post hook after changing aspects of a system.
use
---
@ -304,13 +306,22 @@ targeted state. This means also a chain of ``use`` requisites would not
inherit inherited options.
.. _requisites-require-in:
.. _requisites-watch-in:
require_in
----------
The _in versions of requisites
------------------------------
The ``require_in`` requisite is the literal reverse of ``require``. If
a state declaration needs to be required by another state declaration then
require_in can accommodate it. Therefore, these two sls files would be the
All of the requisites also have corresponding requisite_in versions, which do
the reverse of their normal counterparts. The examples below all use
``require_in`` as the example, but note that all of the ``_in`` requisites work
the same way: They result in a normal requisite in the targeted state, which
targets the state which has defines the requisite_in. Thus, a ``require_in``
causes the target state to ``require`` the targeting state. Similarly, a
``watch_in`` causes the target state to ``watch`` the targeting state. This
pattern continues for the rest of the requisites.
If a state declaration needs to be required by another state declaration then
``require_in`` can accommodate it. Therefore, these two sls files would be the
same in the end:
Using ``require``
@ -383,73 +394,6 @@ mod_python.sls
Now the httpd server will only start if php or mod_python are first verified to
be installed. Thus allowing for a requisite to be defined "after the fact".
.. _requisites-watch-in:
watch_in
--------
``watch_in`` functions the same way as ``require_in``, but applies
a ``watch`` statement rather than a ``require`` statement to the external state
declaration.
A good example of when to use ``watch_in`` versus ``watch`` is in regards to writing
an Apache state in conjunction with a git state for a Django application. On the most
basic level, using either the ``watch`` or the ``watch_in`` requisites, the resulting
behavior will be the same: Apache restarts each time the Django git state changes.
.. code-block:: yaml
apache:
pkg:
- installed
- name: httpd
service:
- watch:
- git: django_git
django_git:
git:
- latest
- name: git@github.com/example/mydjangoproject.git
However, by using ``watch_in``, the approach is improved. By writing ``watch_in`` in
the depending states (such as the Django state and any other states that require Apache
to restart), the dependent state (Apache state) is de-coupled from the depending states:
.. code-block:: yaml
apache:
pkg:
- installed
- name: httpd
django_git:
git:
- latest
- name: git@github.com/example/mydjangoproject.git
- watch_in:
- service: apache
prereq_in
---------
The ``prereq_in`` requisite_in follows the same assignment logic as the
``require_in`` requisite_in. The ``prereq_in`` call simply assigns
``prereq`` to the state referenced. The above example for ``prereq`` can
be modified to function in the same way using ``prereq_in``:
.. code-block:: yaml
graceful-down:
cmd.run:
- name: service apache graceful
site-code:
file.recurse:
- name: /opt/site_code
- source: salt://site/code
- prereq_in:
- cmd: graceful-down
Altering Statefulness
=====================

26
doc/topics/cloud/map.rst
View File

@ -106,8 +106,8 @@ Bootstrapping a new master in the map is as simple as:
.. code-block:: yaml
fedora_small:
- web1
make_master: True
- web1:
make_master: True
- web2
- web3
@ -120,11 +120,11 @@ as opposed to the newly created salt-master, as an example:
.. code-block:: yaml
fedora_small:
- web1
make_master: True
minion:
master: <the local master ip address>
local_master: True
- web1:
make_master: True
minion:
master: <the local master ip address>
local_master: True
- web2
- web3
@ -137,13 +137,13 @@ Another example:
.. code-block:: yaml
fedora_small:
- web1
make_master: True
- web1:
make_master: True
- web2
- web3
minion:
master: <the local master ip address>
local_master: True
- web3:
minion:
master: <the local master ip address>
local_master: True
The above example makes the ``web3`` minion answer to the local master, not the
newly created master.

8
doc/topics/installation/debian.rst
View File

@ -31,6 +31,14 @@ For wheezy, the following line is needed in either
deb http://debian.saltstack.com/debian wheezy-saltstack main
Jessie (Testing)
~~~~~~~~~~~~~~~
For jessie, the following line is needed in either
``/etc/apt/sources.list`` or a file in ``/etc/apt/sources.list.d``::
deb http://debian.saltstack.com/debian jessie-saltstack main
Sid (Unstable)
~~~~~~~~~~~~~~

37
doc/topics/troubleshooting/yaml_idiosyncrasies.rst
View File

@ -36,12 +36,10 @@ fact that the data is uniform and not deeply nested.
Nested Dicts (key=value)
------------------------
When :ref:`dicts <python2:typesmapping>` are more deeply nested, they no longer
follow the same indentation logic. This is rarely something that comes up in
Salt, since deeply nested options like these are discouraged when making State
modules, but some do exist. A good example of this can be found in the
``context`` and ``default`` options from the :doc:`file.managed
</ref/states/all/salt.states.file>` state:
When :ref:`dicts <python2:typesmapping>` are nested within other data
structures (particularly lists), the indentation logic sometimes changes.
Examples of where this might happen include ``context`` and ``default`` options
from the :doc:`file.managed </ref/states/all/salt.states.file>` state:
.. code-block:: yaml
@ -61,8 +59,9 @@ modules, but some do exist. A good example of this can be found in the
Notice that while the indentation is two spaces per level, for the values under
the ``context`` and ``defaults`` options there is a four-space indent. If only
two spaces are used to indent, then the information will not be loaded
correctly. If using a double indent is not desirable, then a deeply-nested dict
two spaces are used to indent, then those keys will be considered part of the
same dictionary that contains the ``context`` key, and so the data will not be
loaded correctly. If using a double indent is not desirable, then a deeply-nested dict
can be declared with curly braces:
.. code-block:: yaml
@ -81,6 +80,28 @@ can be declared with curly braces:
custom_var: "default value",
other_var: 123 }
Here is a more concrete example of how YAML actually handles these
indentations, using the Python interpreter on the command line:
.. code-block:: python
>>> import yaml
>>> yaml.safe_load('''mystate:
... file.managed:
... - context:
... some: var''')
{'mystate': {'file.managed': [{'context': {'some': 'var'}}]}}
>>> yaml.safe_load('''mystate:
... file.managed:
... - context:
... some: var''')
{'mystate': {'file.managed': [{'some': 'var', 'context': None}]}}
Note that in the second example, ``some`` is added as another key in the same
dictionary, whereas in the first example, it's the start of a new dictionary.
That's the distinction. ``context`` is a common example because it is a keyword
arg for many functions, and should contain a dictionary.
True/False, Yes/No, On/Off
==========================

5
salt/__init__.py
View File

@ -229,7 +229,10 @@ class Minion(parsers.MinionOptionParser):
self.daemonize_if_required()
self.set_pidfile()
if isinstance(self.config.get('master'), list):
self.minion = salt.minion.MultiMinion(self.config)
if self.config.get('master_type') == 'failover':
self.minion = salt.minion.Minion(self.config)
else:
self.minion = salt.minion.MultiMinion(self.config)
else:
self.minion = salt.minion.Minion(self.config)
else:

3
salt/client/__init__.py
View File

@ -25,6 +25,7 @@ import time
import copy
import logging
from datetime import datetime
from salt._compat import string_types
# Import salt libs
import salt.config
@ -174,7 +175,7 @@ class LocalClient(object):
return self.opts['timeout']
if isinstance(timeout, int):
return timeout
if isinstance(timeout, str):
if isinstance(timeout, string_types):
try:
return int(timeout)
except ValueError:

4
salt/config.py
View File

@ -55,6 +55,7 @@ VALID_OPTS = {
'master_port': int,
'master_type': str,
'master_finger': str,
'master_shuffle': bool,
'syndic_finger': str,
'user': str,
'root_dir': str,
@ -237,6 +238,7 @@ VALID_OPTS = {
'restart_on_error': bool,
'ping_interval': int,
'cli_summary': bool,
'max_minions': int,
}
# default configurations
@ -246,6 +248,7 @@ DEFAULT_MINION_OPTS = {
'master_type': 'str',
'master_port': '4506',
'master_finger': '',
'master_shuffle': False,
'syndic_finger': '',
'user': 'root',
'root_dir': salt.syspaths.ROOT_DIR,
@ -505,6 +508,7 @@ DEFAULT_MASTER_OPTS = {
'sqlite_queue_dir': os.path.join(salt.syspaths.CACHE_DIR, 'master', 'queues'),
'queue_dirs': [],
'cli_summary': False,
'max_minions': 0,
}
# ----- Salt Cloud Configuration Defaults ----------------------------------->

6
salt/crypt.py
View File

@ -298,12 +298,13 @@ class Auth(object):
def verify_master(self, payload):
'''
Verify that the master is the same one that was previously accepted
Verify that the master is the same one that was previously accepted.
'''
m_pub_fn = os.path.join(self.opts['pki_dir'], self.mpub)
if os.path.isfile(m_pub_fn) and not self.opts['open_mode']:
local_master_pub = salt.utils.fopen(m_pub_fn).read()
if payload['pub_key'] != local_master_pub:
# This is not the last master we connected to
log.error('The master key has changed, the salt master could '
'have been subverted, verify salt master\'s public '
@ -372,6 +373,9 @@ class Auth(object):
'clean out the keys. The Salt Minion will now exit.'
)
sys.exit(os.EX_OK)
# has the master returned that its maxed out with minions?
elif payload['load']['ret'] == 'full':
return 'full'
else:
log.error(
'The Salt Master has cached the public key for this '

63
salt/daemons/flo/core.py
View File

@ -345,9 +345,10 @@ class Schedule(ioflo.base.deeding.Deed):
self.schedule.eval()
class Setup(ioflo.base.deeding.Deed):
class SaltManorLaneSetup(ioflo.base.deeding.Deed):
'''
Only intended to be called once at the top of the house
Only intended to be called once at the top of the manor house
Sets of the LaneStack for the main yard
FloScript:
do setup at enter
@ -362,19 +363,33 @@ class Setup(ioflo.base.deeding.Deed):
'event': '.salt.event.events',
'event_req': '.salt.event.event_req',
'workers': '.salt.track.workers',
'uxd_stack': '.salt.uxd.stack.stack'}
'inode': '.salt.uxd.stack.',
'stack': 'stack',
'local': {'ipath': 'local',
'ival': {'name': 'master',
'localname': 'master',
'yid': 0,
'lanename': 'master'}}
}
def postinitio(self):
'''
Set up required objects and queues
'''
self.uxd_stack.value = LaneStack(
name='yard',
lanename=self.opts.value.get('id', 'master'),
yid=0,
sockdirpath=self.opts.value['sock_dir'],
dirpath=self.opts.value['cachedir'])
self.uxd_stack.value.Pk = raeting.packKinds.pack
name = self.opts.value.get('id', self.local.data.name)
localname = self.opts.value.get('id', self.local.data.localname)
lanename = self.opts.value.get('id', self.local.data.lanename)
yid = self.local.data.yid
basedirpath = os.path.abspath(
os.path.join(self.opts.value['cachedir'], 'raet'))
self.stack.value = LaneStack(
name=name,
#localname=localname,
lanename=lanename,
yid=0,
sockdirpath=self.opts.value['sock_dir'],
basedirpath=basedirpath)
self.stack.value.Pk = raeting.packKinds.pack
self.event_yards.value = set()
self.local_cmd.value = deque()
self.remote_cmd.value = deque()
@ -389,6 +404,26 @@ class Setup(ioflo.base.deeding.Deed):
self.workers.value = itertools.cycle(worker_seed)
class SaltRaetLaneStackCloser(ioflo.base.deeding.Deed): # pylint: disable=W0232
'''
Closes lane stack server socket connection
FloScript:
do raet lane stack closer at exit
'''
Ioinits = odict(
inode=".salt.uxd.stack",
stack='stack',)
def action(self, **kwa):
'''
Close uxd socket
'''
if self.stack.value and isinstance(self.stack.value, LaneStack):
self.stack.value.server.close()
class SaltRoadService(ioflo.base.deeding.Deed):
'''
Process the udp traffic
@ -425,8 +460,8 @@ class Rx(ioflo.base.deeding.Deed):
'''
Process inboud queues
'''
self.udp_stack.value.serviceAll()
self.uxd_stack.value.serviceAll()
self.udp_stack.value.serviceAllRx()
self.uxd_stack.value.serviceAllRx()
class Tx(ioflo.base.deeding.Deed):
@ -448,8 +483,8 @@ class Tx(ioflo.base.deeding.Deed):
'''
Process inbound queues
'''
self.uxd_stack.value.serviceAll()
self.udp_stack.value.serviceAll()
self.uxd_stack.value.serviceAllTx()
self.udp_stack.value.serviceAllTx()
class Router(ioflo.base.deeding.Deed):

7
salt/daemons/flo/master.flo
View File

@ -4,10 +4,14 @@ house master
init .raet.udp.stack.local to eid 1 main true name "master" localname "master"
init .salt.uxd.stack.local to yid 0 name "master" localname "master" lanename "master"
framer masterudpstack be active first setup
frame setup
enter
do setup
do salt manor lane setup
# go spawnmaint
# frame spawnmaint
# enter
@ -21,6 +25,7 @@ framer masterudpstack be active first setup
do salt raet road stack per inode ".raet.udp.stack"
exit
do salt raet road stack closer per inode ".raet.udp.stack."
do salt raet lane stack closer per inode ".salt.uxd.stack."
framer inbound be active first start
frame start

4
salt/daemons/flo/minion.flo
View File

@ -5,16 +5,18 @@ house minion
init name in .raet.udp.stack.local from value in .salt.etc.id
init localname in .raet.udp.stack.local from value in .salt.etc.id
init .salt.uxd.stack.local to yid 0 name "minion" localname "minion" lanename "minion"
framer minionudpstack be active first setup
frame setup
enter
do setup
do salt manor lane setup
go start
frame start
do salt raet road stack per inode ".raet.udp.stack"
exit
do salt raet road stack closer per inode ".raet.udp.stack."
do salt raet lane stack closer per inode ".salt.uxd.stack."
framer inbound be active first start
frame start

2
salt/daemons/flo/worker.flo
View File

@ -10,3 +10,5 @@ framer uxdrouter be active first setup
go start
frame start
do worker router
exit
do salt raet lane stack closer per inode ".salt.uxd.stack."

48
salt/daemons/flo/worker.py
View File

@ -5,6 +5,7 @@ The core bahaviuors ued by minion and master
# pylint: disable=W0232
# Import python libs
import os
import multiprocessing
# Import salt libs
@ -80,40 +81,57 @@ class WorkerSetup(ioflo.base.deeding.Deed):
'''
Ioinits = {
'uxd_stack': '.salt.uxd.stack.stack',
'opts': '.salt.opts',
'yid': '.salt.yid',
'access_keys': '.salt.access_keys',
'remote': '.salt.loader.remote',
'local': '.salt.loader.local',
'inode': '.salt.uxd.stack.',
'stack': 'stack',
'main': {'ipath': 'main',
'ival': {'name': 'master',
'localname': 'master',
'yid': 0,
'lanename': 'master'}}
}
def action(self):
'''
Set up the uxd stack and behaviors
'''
self.uxd_stack.value = LaneStack(
lanename=self.opts.value.get('id', 'master'),
yid=self.yid.value,
sockdirpath=self.opts.value['sock_dir'])
self.uxd_stack.value.Pk = raeting.packKinds.pack
name = "{0}{1}{2}".format(self.opts.value.get('id', self.main.data.name),
'worker',
self.yid.value)
localname = name
lanename = self.opts.value.get('id', self.main.data.lanename)
basedirpath = os.path.abspath(
os.path.join(self.opts.value['cachedir'], 'raet'))
self.stack.value = LaneStack(
name=name,
#localname=localname,
basedirpath=basedirpath,
lanename=lanename,
yid=self.yid.value,
sockdirpath=self.opts.value['sock_dir'])
self.stack.value.Pk = raeting.packKinds.pack
manor_yard = RemoteYard(
stack=self.uxd_stack.value,
yid=0,
lanename=self.opts.value.get('id', 'master'),
dirpath=self.opts.value['sock_dir'])
self.uxd_stack.value.addRemote(manor_yard)
stack=self.stack.value,
yid=0,
lanename=lanename,
dirpath=self.opts.value['sock_dir'])
self.stack.value.addRemote(manor_yard)
self.remote.value = salt.daemons.masterapi.RemoteFuncs(self.opts.value)
self.local.value = salt.daemons.masterapi.LocalFuncs(
self.opts.value,
self.access_keys.value)
init = {}
init['route'] = {
'src': (None, self.uxd_stack.value.local.name, None),
'dst': (None, 'yard0', 'worker_req')
'src': (None, self.stack.value.local.name, None),
'dst': (None, manor_yard.name, 'worker_req')
}
self.uxd_stack.value.transmit(init, self.uxd_stack.value.uids.get('yard0'))
self.uxd_stack.value.serviceAll()
self.stack.value.transmit(init, self.stack.value.uids.get(manor_yard.name))
self.stack.value.serviceAll()
class WorkerRouter(ioflo.base.deeding.Deed):

28
salt/daemons/masterapi.py
View File

@ -5,9 +5,10 @@ involves preparing the three listeners and the workers needed by the master.
'''
# Import python libs
import fnmatch
import logging
import os
import re
import logging
import time
try:
import pwd
@ -76,7 +77,7 @@ def clean_fsbackend(opts):
'''
Clean out the old fileserver backends
'''
# Clear remote fileserver backend env cache so it gets recreated
# Clear remote fileserver backend caches so they get recreated
for backend in ('git', 'hg', 'svn'):
if backend in opts['fileserver_backend']:
env_cache = os.path.join(
@ -94,6 +95,25 @@ def clean_fsbackend(opts):
.format(env_cache, exc)
)
file_lists_dir = os.path.join(
opts['cachedir'],
'file_lists',
'{0}fs'.format(backend)
)
try:
file_lists_caches = os.listdir(file_lists_dir)
except OSError:
continue
for file_lists_cache in fnmatch.filter(file_lists_caches, '*.p'):
cache_file = os.path.join(file_lists_dir, file_lists_cache)
try:
os.remove(cache_file)
except (IOError, OSError) as exc:
log.critical(
'Unable to file_lists cache file {0}: {1}'
.format(cache_file, exc)
)
def clean_expired_tokens(opts):
'''
@ -678,7 +698,7 @@ class RemoteFuncs(object):
pub_load['timeout'] = int(load['timeout'])
except ValueError:
msg = 'Failed to parse timeout value: {0}'.format(
load['tmo'])
load['timeout'])
log.warn(msg)
return {}
if 'tgt_type' in load:
@ -704,7 +724,7 @@ class RemoteFuncs(object):
if 'jid' in minion:
ret['__jid__'] = minion['jid']
for key, val in self.local.get_cache_returns(ret['__jid__']).items():
if not key in ret:
if key not in ret:
ret[key] = val
if load.get('form', '') != 'full':
ret.pop('__jid__')

3
salt/fileserver/gitfs.py
View File

@ -590,6 +590,9 @@ def init():
'hash': repo_hash,
'cachedir': rp_
})
# Strip trailing slashes from the gitfs root as these cause
# path searches to fail.
repo_conf['root'] = repo_conf['root'].rstrip(os.path.sep)
repos.append(repo_conf)
except Exception as exc:

6
salt/loader.py
View File

@ -778,11 +778,13 @@ class Loader(object):
for mod in self.modules:
if not hasattr(mod, '__salt__') or (
not in_pack(pack, '__salt__') and
not str(mod.__name__).startswith('salt.loaded.int.grain')
(not str(mod.__name__).startswith('salt.loaded.int.grain') and
not str(mod.__name__).startswith('salt.loaded.ext.grain'))
):
mod.__salt__ = funcs
elif not in_pack(pack, '__salt__') and \
str(mod.__name__).startswith('salt.loaded.int.grain'):
(str(mod.__name__).startswith('salt.loaded.int.grain') or
str(mod.__name__).startswith('salt.loaded.int.grain')):
mod.__salt__.update(funcs)
return funcs

23
salt/master.py
View File

@ -1748,6 +1748,29 @@ class ClearFuncs(object):
'load': {'ret': False}}
log.info('Authentication request from {id}'.format(**load))
minions = salt.utils.minions.CkMinions(self.opts).connected_ids()
# 0 is default which should be 'unlimited'
if self.opts['max_minions'] > 0:
if not len(minions) < self.opts['max_minions']:
# we reject new minions, minions that are already
# connected must be allowed for the mine, highstate, etc.
if load['id'] not in minions:
msg = ('Too many minions connected (max_minions={0}). '
'Rejecting connection from id '
'{1}'.format(self.opts['max_minions'],
load['id'])
)
log.info(msg)
eload = {'result': False,
'act': 'full',
'id': load['id'],
'pub': load['pub']}
self.event.fire_event(eload, tagify(prefix='auth'))
return {'enc': 'clear',
'load': {'ret': 'full'}}
# Check if key is configured to be auto-rejected/signed
auto_reject = self.__check_autoreject(load['id'])
auto_sign = self.__check_autosign(load['id'])

84
salt/minion.py
View File

@ -575,8 +575,7 @@ class Minion(MinionBase):
# module
opts['grains'] = salt.loader.grains(opts)
# if master_type was changed, we might want to load our
# master-variable from a user defined modules function
# check if master_type was altered from its default
if opts['master_type'] != 'str':
# check for a valid keyword
if opts['master_type'] == 'func':
@ -593,16 +592,64 @@ class Minion(MinionBase):
'module \'{0}\''.format(opts['master']))
log.error(msg)
sys.exit(1)
log.info('Evaluated master from module: {0}'.format(opts['master']))
log.info('Evaluated master from module: {0}'.format(master_mod))
# if failover is set, master has to be of type list
elif opts['master_type'] == 'failover':
if type(opts['master']) is list:
log.info('Got list of available master addresses:'
' {0}'.format(opts['master']))
else:
msg = ('master_type set to \'failover\' but \'master\' '
'is not of type list but of type '
'{0}'.format(type(opts['master'])))
log.error(msg)
sys.exit(1)
else:
msg = ('Invalid keyword \'{0}\' for variable '
'\'master_type\''.format(opts['master_type']))
log.error(msg)
sys.exit(1)
opts.update(resolve_dns(opts))
super(Minion, self).__init__(opts)
self.authenticate(timeout, safe)
# if we have a list of masters, loop through them and be
# happy with the first one that allows us to connect
if type(opts['master']) is list:
conn = False
# shuffle the masters and then loop through them
local_masters = copy.copy(opts['master'])
if opts['master_shuffle']:
from random import shuffle
shuffle(local_masters)
for master in local_masters:
opts['master'] = master
opts.update(resolve_dns(opts))
super(Minion, self).__init__(opts)
try:
if self.authenticate(timeout, safe) != 'full':
conn = True
break
except SaltClientError:
msg = ('Master {0} could not be reached, trying '
'next master (if any)'.format(opts['master']))
log.info(msg)
continue
if not conn:
msg = ('No master could be reached or all masters denied '
'the minions connection attempt.')
log.error(msg)
# single master sign in
else:
opts.update(resolve_dns(opts))
super(Minion, self).__init__(opts)
if self.authenticate(timeout, safe) == 'full':
msg = ('master {0} rejected the minions connection because too '
'many minions are already connected.'.format(opts['master']))
log.error(msg)
sys.exit(1)
self.opts['pillar'] = salt.pillar.get_pillar(
opts,
opts['grains'],
@ -1216,7 +1263,9 @@ class Minion(MinionBase):
safe = self.opts.get('auth_safemode', safe)
while True:
creds = auth.sign_in(timeout, safe, tries)
if creds != 'retry':
if creds == 'full':
return creds
elif creds != 'retry':
log.info('Authentication with master successful!')
break
log.info('Waiting for minion key to be accepted by the master.')
@ -1254,6 +1303,25 @@ class Minion(MinionBase):
).compile_pillar()
self.module_refresh()
def manage_schedule(self, package):
'''
Refresh the functions and returners.
'''
tag, data = salt.utils.event.MinionEvent.unpack(package)
func = data.get('func', None)
if func == 'delete':
job = data.get('job', None)
self.schedule.delete_job(job)
elif func == 'add':
name = data.get('name', None)
schedule = data.get('schedule', None)
self.schedule.add_job(name, schedule)
elif func == 'modify':
name = data.get('name', None)
schedule = data.get('schedule', None)
self.schedule.modify_job(name, schedule)
def environ_setenv(self, package):
'''
Set the salt-minion main process environment according to
@ -1400,6 +1468,8 @@ class Minion(MinionBase):
self.module_refresh()
elif package.startswith('pillar_refresh'):
self.pillar_refresh()
elif package.startswith('manage_schedule'):
self.manage_schedule(package)
elif package.startswith('grains_refresh'):
if self.grains_cache != self.opts['grains']:
self.pillar_refresh()

2
salt/modules/kmod.py
View File

@ -129,7 +129,7 @@ def available():
for root, dirs, files in os.walk(mod_dir):
for fn_ in files:
if '.ko' in fn_:
ret.append(fn_[:fn_.index('.ko')])
ret.append(fn_[:fn_.index('.ko')].replace('-', '_'))
return sorted(list(ret))

289
salt/modules/schedule.py Normal file
View File

@ -0,0 +1,289 @@
# -*- coding: utf-8 -*-
'''
Module for manging the Salt schedule on a minion
.. versionadded:: Helium
'''
# Import Python libs
import os
import yaml
import salt.utils
__proxyenabled__ = ['*']
import logging
log = logging.getLogger(__name__)
SCHEDULE_CONF = [
'function',
'splay',
'range',
'when',
'returner',
'jid_include',
'args',
'kwargs',
'_seconds',
'seconds',
'minutes',
'hours',
'days'
]
def list():
'''
List the jobs currently scheduled on the minion
CLI Example:
.. code-block:: bash
salt '*' schedule.list
'''
schedule = __opts__['schedule']
for job in schedule.keys():
if job.startswith('_'):
del schedule[job]
continue
for item in schedule[job].keys():
if not item in SCHEDULE_CONF:
del schedule[job][item]
continue
if schedule[job][item] == 'true':
schedule[job][item] = True
if schedule[job][item] == 'false':
schedule[job][item] = False
if '_seconds' in schedule[job].keys():
schedule[job]['seconds'] = schedule[job]['_seconds']
del schedule[job]['_seconds']
if schedule:
tmp = {'schedule': schedule}
yaml_out = yaml.safe_dump(tmp, default_flow_style=False)
return yaml_out
else:
return None
def purge():
'''
Purge all the jobs currently scheduled on the minion
CLI Example:
.. code-block:: bash
salt '*' schedule.purge
'''
ret = {'comment': [],
'result': True}
schedule = __opts__['schedule']
for job in schedule.keys():
if job.startswith('_'):
continue
out = __salt__['event.fire']({'job': job, 'func': 'delete'}, 'manage_schedule')
if out:
ret['comment'].append('Deleted job: {0} from schedule.'.format(job))
else:
ret['comment'].append('Failed to delete job {0} from schedule.'.format(job))
ret['result'] = False
return ret
def delete(name):
'''
Delete a job from the minion's schedule
CLI Example:
.. code-block:: bash
salt '*' schedule.delete job1
'''
ret = {'comment': [],
'result': True}
if not name:
ret['comment'] = 'Job name is required.'
ret['result'] = False
if name in __opts__['schedule']:
out = __salt__['event.fire']({'job': name, 'func': 'delete'}, 'manage_schedule')
if out:
ret['comment'] = 'Deleted Job {0} from schedule.'.format(name)
else:
ret['comment'] = 'Failed to delete job {0} from schedule.'.format(name)
ret['result'] = False
else:
ret['comment'] = 'Job {0} does not exist.'.format(name)
ret['result'] = False
return ret
def add(name, **kwargs):
'''
Add a job to the schedule
CLI Example:
.. code-block:: bash
salt '*' schedule.add job1 function='test.ping' seconds=3600
'''
ret = {'comment': [],
'result': True}
if name in __opts__['schedule']:
ret['comment'] = 'Job {0} already exists in schedule.'.format(name)
ret['result'] = True
return ret
if not name:
ret['comment'] = 'Job name is required.'
ret['result'] = False
schedule = {'function': kwargs['function']}
time_conflict = False
for item in ['seconds', 'minutes', 'hours', 'days']:
if item in kwargs and 'when' in kwargs:
time_conflict = True
if time_conflict:
return 'Error: Unable to use "seconds", "minutes", "hours", or "days" with "when" option.'
for item in ['seconds', 'minutes', 'hours', 'days']:
if item in kwargs:
schedule[item] = kwargs[item]
if 'job_args' in kwargs:
schedule['args'] = kwargs['job_args']
if 'job_kwargs' in kwargs:
schedule['kwargs'] = kwargs['job_kwargs']
for item in ['splay', 'range', 'when', 'returner', 'jid_include']:
if item in kwargs:
schedule[item] = kwargs[item]
out = __salt__['event.fire']({'name': name, 'schedule': schedule, 'func': 'add'}, 'manage_schedule')
if out:
ret['comment'] = 'Added job: {0} to schedule.'.format(name)
else:
ret['comment'] = 'Failed to modify job {0} to schedule.'.format(name)
ret['result'] = False
return ret
def modify(name, **kwargs):
'''
Modify an existing job in the schedule
CLI Example:
.. code-block:: bash
salt '*' schedule.modify job1 function='test.ping' seconds=3600
'''
ret = {'comment': [],
'result': True}
if not name in __opts__['schedule']:
ret['comment'] = 'Job {0} does not exist in schedule.'.format(name)
ret['result'] = False
return ret
schedule = {'function': kwargs['function']}
time_conflict = False
for item in ['seconds', 'minutes', 'hours', 'days']:
if item in kwargs and 'when' in kwargs:
time_conflict = True
if time_conflict:
return 'Error: Unable to use "seconds", "minutes", "hours", or "days" with "when" option.'
for item in ['seconds', 'minutes', 'hours', 'days']:
if item in kwargs:
schedule[item] = kwargs[item]
if 'job_args' in kwargs:
schedule['args'] = kwargs['job_args']
if 'job_kwargs' in kwargs:
schedule['kwargs'] = kwargs['job_kwargs']
for item in ['splay', 'range', 'when', 'returner', 'jid_include']:
if item in kwargs:
schedule[item] = kwargs[item]
out = __salt__['event.fire']({'name': name, 'schedule': schedule, 'func': 'modify'}, 'manage_schedule')
if out:
ret['comment'] = 'Modified job: {0} in schedule.'.format(name)
else:
ret['comment'] = 'Failed to modify job {0} in schedule.'.format(name)
ret['result'] = False
return ret
def save():
'''
CLI Example:
.. code-block:: bash
salt '*' schedule.save
'''
ret = {'comment': [],
'result': True}
schedule = __opts__['schedule']
for job in schedule.keys():
if job.startswith('_'):
del schedule[job]
continue
for item in schedule[job].keys():
if not item in SCHEDULE_CONF:
del schedule[job][item]
continue
if schedule[job][item] == 'true':
schedule[job][item] = True
if schedule[job][item] == 'false':
schedule[job][item] = False
if '_seconds' in schedule[job].keys():
schedule[job]['seconds'] = schedule[job]['_seconds']
del schedule[job]['_seconds']
# move this file into an configurable opt
sfn = '{0}/{1}/schedule.conf'.format(__opts__['config_dir'], os.path.dirname(__opts__['default_include']))
if schedule:
tmp = {'schedule': schedule}
yaml_out = yaml.safe_dump(tmp, default_flow_style=False)
else:
yaml_out = ''
try:
with salt.utils.fopen(sfn, 'w+') as fp_:
fp_.write(yaml_out)
ret['comment'] = 'Schedule saved to {0}.'.format(sfn)
except (IOError, OSError):
ret['comment'] = 'Unable to write to schedule file at {0}. Check permissions.'.format(sfn)
ret['result'] = False
return ret

16
salt/modules/shadow.py
View File

@ -169,6 +169,22 @@ def gen_password(password, crypt_salt=None, algorithm='sha512'):
return salt.utils.pycrypto.gen_hash(crypt_salt, password, algorithm)
def del_password(name):
'''
Delete the password from name user
CLI Example:
.. code-block:: bash
salt '*' shadow.del_password username
'''
cmd = 'passwd -d {0}'.format(name)
__salt__['cmd.run'](cmd, output_loglevel='quiet')
uinfo = info(name)
return not uinfo['passwd']
def set_password(name, password, use_usermod=False):
'''
Set the password for a named user. The password must be a properly defined

10
salt/modules/yumpkg.py
View File

@ -130,7 +130,7 @@ def _get_repo_options(**kwargs):
repo_arg = ''
if fromrepo:
log.info('Restricting to repo {0!r}'.format(fromrepo))
repo_arg = ('--disablerepo={0!r} --enablerepo={1!r}'
repo_arg = ('--disablerepo={0!r} --enablerepo={1!r} '
.format('*', fromrepo))
else:
repo_arg = ''
@ -726,6 +726,10 @@ def install(name=None,
Disable exclude from main, for a repo or for everything.
(e.g., ``yum --disableexcludes='main'``)
branch
Specifies the branch on YUM server.
(e.g., ``yum --branch='test'``)
.. versionadded:: Helium
@ -785,6 +789,10 @@ def install(name=None,
'package targets')
repo_arg = _get_repo_options(fromrepo=fromrepo, **kwargs)
# Support branch parameter for yum
branch = kwargs.get('branch', '')
if branch:
repo_arg += '--branch={0!r}'.format(branch)
exclude_arg = _get_excludes_option(**kwargs)
old = list_pkgs()

14
salt/states/user.py
View File

@ -55,6 +55,7 @@ def _changes(name,
createhome=True,
password=None,
enforce_password=True,
empty_password=False,
shell=None,
fullname='',
roomnumber='',
@ -160,6 +161,7 @@ def present(name,
createhome=True,
password=None,
enforce_password=True,
empty_password=False,
shell=None,
unique=True,
system=False,
@ -229,6 +231,9 @@ def present(name,
"password" field. This option will be ignored if "password" is not
specified.
empty_password
Set to True to enable no password-less login for user
shell
The login shell, defaults to the system default shell
@ -325,6 +330,9 @@ def present(name,
if gid_from_name:
gid = __salt__['file.group_to_gid'](name)
if empty_password:
__salt__['shadow.del_password'](name)
changes = _changes(name,
uid,
gid,
@ -335,6 +343,7 @@ def present(name,
createhome,
password,
enforce_password,
empty_password,
shell,
fullname,
roomnumber,
@ -360,7 +369,7 @@ def present(name,
lshad = __salt__['shadow.info'](name)
pre = __salt__['user.info'](name)
for key, val in changes.items():
if key == 'passwd':
if key == 'passwd' and not empty_password:
__salt__['shadow.set_password'](name, password)
continue
if key == 'date':
@ -419,6 +428,7 @@ def present(name,
createhome,
password,
enforce_password,
empty_password,
shell,
fullname,
roomnumber,
@ -464,7 +474,7 @@ def present(name,
ret['comment'] = 'New user {0} created'.format(name)
ret['changes'] = __salt__['user.info'](name)
if 'shadow.info' in __salt__ and not salt.utils.is_windows():
if password:
if password and not empty_password:
__salt__['shadow.set_password'](name, password)
spost = __salt__['shadow.info'](name)
if spost['passwd'] != password:

5
salt/utils/event.py
View File

@ -112,7 +112,10 @@ def get_event(node, sock_dir=None, transport='zeromq', opts=None, listen=True):
return SaltEvent(node, sock_dir, opts)
elif transport == 'raet':
import salt.utils.raetevent
return salt.utils.raetevent.SaltEvent(node, sock_dir, listen)
return salt.utils.raetevent.SaltEvent(node,
sock_dir=sock_dir,
listen=listen,
opts=opts)
def tagify(suffix='', prefix='', base=SALT):

13
salt/utils/raetevent.py
View File

@ -6,6 +6,7 @@ This module is used to manage events via RAET
'''
# Import python libs
import os
import logging
import time
from collections import MutableMapping
@ -15,6 +16,7 @@ import salt.payload
import salt.loader
import salt.state
import salt.utils.event
from salt import syspaths
from raet import raeting
from raet.lane.stacking import LaneStack
from raet.lane.yarding import RemoteYard
@ -26,21 +28,30 @@ class SaltEvent(object):
'''
The base class used to manage salt events
'''
def __init__(self, node, sock_dir=None, listen=True):
def __init__(self, node, sock_dir=None, listen=True, opts=None):
'''
Set up the stack and remote yard
'''
self.node = node
self.sock_dir = sock_dir
self.listen = listen
if opts is None:
opts = {}
self.opts = opts
self.__prep_stack()
def __prep_stack(self):
self.yid = salt.utils.gen_jid()
name = 'event' + self.yid
cachedir = self.opts.get('cachedir', os.path.join(syspaths.CACHE_DIR, self.node))
basedirpath = os.path.abspath(
os.path.join(cachedir, 'raet'))
self.connected = False
self.stack = LaneStack(
name=name,
yid=self.yid,
lanename=self.node,
basedirpath=basedirpath,
sockdirpath=self.sock_dir)
self.stack.Pk = raeting.packKinds.pack
self.router_yard = RemoteYard(

27
salt/utils/schedule.py
View File

@ -189,6 +189,23 @@ class Schedule(object):
return self.functions['config.merge'](opt, {}, omit_master=True)
return self.opts.get(opt, {})
def delete_job(self, name):
# ensure job exists, then delete it
if name in self.opts['schedule']:
del self.opts['schedule'][name]
# remove from self.intervals
if name in self.intervals:
del self.intervals[name]
def add_job(self, name, schedule):
self.opts['schedule'][name] = schedule
def modify_job(self, name, schedule):
if name in self.opts['schedule']:
self.delete_job(name)
self.opts['schedule'][name] = schedule
def handle_func(self, func, data):
'''
Execute this method in a multiprocess or thread
@ -311,6 +328,7 @@ class Schedule(object):
Evaluate and execute the schedule
'''
schedule = self.option('schedule')
#log.debug('calling eval {0}'.format(schedule))
if not isinstance(schedule, dict):
return
for job, data in schedule.items():
@ -335,8 +353,12 @@ class Schedule(object):
when = 0
seconds = 0
# clean this up
if ('seconds' in data or 'hours' in data or 'minutes' in data or 'days' in data) and 'when' in data:
time_conflict = False
for item in ['seconds', 'minutes', 'hours', 'days']:
if item in data and 'when' in data:
time_conflict = True
if time_conflict:
log.info('Unable to use "seconds", "minutes", "hours", or "days" with "when" option. Ignoring.')
continue
@ -440,6 +462,7 @@ class Schedule(object):
else:
if now - self.intervals[job] >= seconds:
run = True
else:
if 'splay' in data:
if 'when' in data:

2
tests/integration/__init__.py
View File

@ -282,7 +282,7 @@ class TestDaemon(object):
self.pre_setup_minions()
self.setup_minions()
if self.parser.options.ssh:
if getattr(self.parser.options, 'ssh', False):
self.prep_ssh()
if self.parser.options.sysinfo:

2
tests/integration/fileserver/fileclient_test.py
View File

@ -7,7 +7,7 @@
from salttesting.unit import skipIf
from salttesting.helpers import ensure_in_syspath
from salttesting.mock import MagicMock, patch, NO_MOCK, NO_MOCK_REASON
ensure_in_syspath('../')
ensure_in_syspath('../..')
# Import salt libs
import integration

26
tests/integration/fileserver/gitfs_test.py
View File

@ -8,7 +8,7 @@ from salttesting import skipIf
from salttesting.helpers import ensure_in_syspath
from salttesting.mock import patch, NO_MOCK, NO_MOCK_REASON
ensure_in_syspath('../')
ensure_in_syspath('../..')
# Import Python libs
import os
@ -23,6 +23,10 @@ gitfs.__opts__ = {'gitfs_remotes': [''],
'fileserver_backend': 'gitfs',
'gitfs_base': 'master',
'fileserver_events': True,
'transport': 'zeromq',
'gitfs_mountpoint': '',
'gitfs_env_whitelist': [],
'gitfs_env_blacklist': []
}
load = {'saltenv': 'base'}
@ -88,7 +92,11 @@ class GitFSTest(integration.ModuleCase):
'gitfs_remotes': ['file://' + self.tmp_repo_git],
'sock_dir': self.master_opts['sock_dir']}):
ret = gitfs.find_file('testfile')
expected_ret = {'path': '/tmp/salttest/cache/gitfs/refs/master/testfile',
expected_ret = {'path': os.path.join(self.master_opts['cachedir'],
'gitfs',
'refs',
'base',
'testfile'),
'rel': 'testfile'}
self.assertDictEqual(ret, expected_ret)
@ -140,8 +148,18 @@ class GitFSTest(integration.ModuleCase):
ret = gitfs.serve_file(load, fnd)
self.assertDictEqual({
'data': 'Scene 24\n\n \n OLD MAN: Ah, hee he he ha!\n ARTHUR: And this enchanter of whom you speak, he has seen the grail?\n OLD MAN: Ha ha he he he he!\n ARTHUR: Where does he live? Old man, where does he live?\n OLD MAN: He knows of a cave, a cave which no man has entered.\n ARTHUR: And the Grail... The Grail is there?\n OLD MAN: Very much danger, for beyond the cave lies the Gorge\n of Eternal Peril, which no man has ever crossed.\n ARTHUR: But the Grail! Where is the Grail!?\n OLD MAN: Seek you the Bridge of Death.\n ARTHUR: The Bridge of Death, which leads to the Grail?\n OLD MAN: Hee hee ha ha!\n\n',
'dest': 'testfile'}, ret)
'data': 'Scene 24\n\n \n OLD MAN: Ah, hee he he ha!\n ARTHUR: '
'And this enchanter of whom you speak, he has seen the grail?\n '
'OLD MAN: Ha ha he he he he!\n ARTHUR: Where does he live? '
'Old man, where does he live?\n OLD MAN: He knows of a cave, '
'a cave which no man has entered.\n ARTHUR: And the Grail... '
'The Grail is there?\n OLD MAN: Very much danger, for beyond '
'the cave lies the Gorge\n of Eternal Peril, which no man '
'has ever crossed.\n ARTHUR: But the Grail! Where is the Grail!?\n '
'OLD MAN: Seek you the Bridge of Death.\n ARTHUR: The Bridge of '
'Death, which leads to the Grail?\n OLD MAN: Hee hee ha ha!\n\n',
'dest': 'testfile'},
ret)
if __name__ == '__main__':

28
tests/integration/fileserver/roots_test.py
View File

@ -7,7 +7,7 @@
from salttesting import skipIf
from salttesting.helpers import ensure_in_syspath
from salttesting.mock import patch, NO_MOCK, NO_MOCK_REASON
ensure_in_syspath('../')
ensure_in_syspath('../..')
# Import salt libs
import integration
@ -22,11 +22,15 @@ import os
@skipIf(NO_MOCK, NO_MOCK_REASON)
class RootsTest(integration.ModuleCase):
def setUp(self):
self.master_opts['file_roots']['base'] = [os.path.join(integration.FILES, 'file', 'base')]
if integration.TMP_STATE_TREE not in self.master_opts['file_roots']['base']:
# We need to setup the file roots
self.master_opts['file_roots']['base'] = [os.path.join(integration.FILES, 'file', 'base')]
def test_file_list(self):
with patch.dict(roots.__opts__, {'file_roots': self.master_opts['file_roots'],
with patch.dict(roots.__opts__, {'cachedir': self.master_opts['cachedir'],
'file_roots': self.master_opts['file_roots'],
'fileserver_ignoresymlinks': False,
'fileserver_followsymlinks': False,
'file_ignore_regex': False,
@ -102,7 +106,10 @@ class RootsTest(integration.ModuleCase):
self.assertDictEqual(ret, {'hsum': '98aa509006628302ce38ce521a7f805f', 'hash_type': 'md5'})
def test_file_list_emptydirs(self):
with patch.dict(roots.__opts__, {'file_roots': self.master_opts['file_roots'],
if integration.TMP_STATE_TREE not in self.master_opts['file_roots']['base']:
self.skipTest('This test fails when using tests/runtests.py. salt-runtests will be available soon.')
with patch.dict(roots.__opts__, {'cachedir': self.master_opts['cachedir'],
'file_roots': self.master_opts['file_roots'],
'fileserver_ignoresymlinks': False,
'fileserver_followsymlinks': False,
'file_ignore_regex': False,
@ -111,11 +118,14 @@ class RootsTest(integration.ModuleCase):
self.assertIn('empty_dir', ret)
def test_dir_list(self):
with patch.dict(roots.__opts__, {'file_roots': self.master_opts['file_roots'],
'fileserver_ignoresymlinks': False,
'fileserver_followsymlinks': False,
'file_ignore_regex': False,
'file_ignore_glob': False}):
if integration.TMP_STATE_TREE not in self.master_opts['file_roots']['base']:
self.skipTest('This test fails when using tests/runtests.py. salt-runtests will be available soon.')
with patch.dict(roots.__opts__, {'cachedir': self.master_opts['cachedir'],
'file_roots': self.master_opts['file_roots'],
'fileserver_ignoresymlinks': False,
'fileserver_followsymlinks': False,
'file_ignore_regex': False,
'file_ignore_glob': False}):
ret = roots.dir_list({'saltenv': 'base'})
self.assertIn('empty_dir', ret)

57
tests/integration/shell/call.py
View File

@ -85,34 +85,8 @@ class CallTest(integration.ShellCase, integration.ShellCaseCommonTestsMixIn):
@skipIf(sys.platform.startswith('win'), 'This test does not apply on Win')
def test_return(self):
config_dir = '/tmp/salttest'
minion_config_file = os.path.join(config_dir, 'minion')
minion_config = {
'id': 'minion_test_issue_2731',
'master': 'localhost',
'master_port': 64506,
'root_dir': '/tmp/salttest',
'pki_dir': 'pki',
'cachedir': 'cachedir',
'sock_dir': 'minion_sock',
'open_mode': True,
'log_file': '/tmp/salttest/minion_test_issue_2731',
'log_level': 'quiet',
'log_level_logfile': 'info'
}
# Remove existing logfile
if os.path.isfile('/tmp/salttest/minion_test_issue_2731'):
os.unlink('/tmp/salttest/minion_test_issue_2731')
# Let's first test with a master running
open(minion_config_file, 'w').write(
yaml.dump(minion_config, default_flow_style=False)
)
out = self.run_call('-c {0} cmd.run "echo returnTOmaster"'.format(
os.path.join(integration.INTEGRATION_TEST_DIR, 'files', 'conf')))
jobs = [a for a in self.run_run('-c {0} jobs.list_jobs'.format(
os.path.join(integration.INTEGRATION_TEST_DIR, 'files', 'conf')))]
self.run_call('-c {0} cmd.run "echo returnTOmaster"'.format(self.get_config_dir()))
jobs = [a for a in self.run_run('-c {0} jobs.list_jobs'.format(self.get_config_dir()))]
self.assertTrue(True in ['returnTOmaster' in j for j in jobs])
# lookback jid
@ -129,38 +103,43 @@ class CallTest(integration.ShellCase, integration.ShellCaseCommonTestsMixIn):
assert idx > 0
assert jid
master_out = [
a for a in self.run_run('-c {0} jobs.lookup_jid {1}'.format(
os.path.join(integration.INTEGRATION_TEST_DIR,
'files',
'conf'),
jid))]
a for a in self.run_run('-c {0} jobs.lookup_jid {1}'.format(self.get_config_dir(), jid))
]
self.assertTrue(True in ['returnTOmaster' in a for a in master_out])
@skipIf(sys.platform.startswith('win'), 'This test does not apply on Win')
def test_issue_2731_masterless(self):
config_dir = '/tmp/salttest'
root_dir = os.path.join(integration.TMP, 'issue-2731')
config_dir = os.path.join(root_dir, 'conf')
minion_config_file = os.path.join(config_dir, 'minion')
logfile = os.path.join(root_dir, 'minion_test_issue_2731')
if not os.path.isdir(config_dir):
os.makedirs(config_dir)
master_config = yaml.load(open(self.get_config_file_path('master')).read())
master_root_dir = master_config['root_dir']
this_minion_key = os.path.join(
config_dir, 'pki', 'minions', 'minion_test_issue_2731'
master_root_dir, 'pki', 'minions', 'minion_test_issue_2731'
)
minion_config = {
'id': 'minion_test_issue_2731',
'master': 'localhost',
'master_port': 64506,
'root_dir': '/tmp/salttest',
'root_dir': master_root_dir,
'pki_dir': 'pki',
'cachedir': 'cachedir',
'sock_dir': 'minion_sock',
'open_mode': True,
'log_file': '/tmp/salttest/minion_test_issue_2731',
'log_file': logfile,
'log_level': 'quiet',
'log_level_logfile': 'info'
}
# Remove existing logfile
if os.path.isfile('/tmp/salttest/minion_test_issue_2731'):
os.unlink('/tmp/salttest/minion_test_issue_2731')
if os.path.isfile(logfile):
os.unlink(logfile)
start = datetime.now()
# Let's first test with a master running

20
tests/runtests.py
View File

@ -109,6 +109,13 @@ class SaltTestsuiteParser(SaltCoverageTestingParser):
action='store_true',
help='Run unit tests'
)
self.test_selection_group.add_option(
'--fileserver-tests',
dest='fileserver',
default=False,
action='store_true',
help='Run Fileserver tests'
)
self.test_selection_group.add_option(
'-o',
'--outputter',
@ -137,7 +144,8 @@ class SaltTestsuiteParser(SaltCoverageTestingParser):
self.options.module, self.options.client, self.options.shell,
self.options.unit, self.options.state, self.options.runner,
self.options.loader, self.options.name, self.options.outputter,
os.geteuid() != 0, not self.options.run_destructive)):
self.options.fileserver, os.geteuid() != 0,
not self.options.run_destructive)):
self.error(
'No sense in generating the tests coverage report when '
'not running the full test suite, including the '
@ -149,7 +157,8 @@ class SaltTestsuiteParser(SaltCoverageTestingParser):
if not any((self.options.module, self.options.client,
self.options.shell, self.options.unit, self.options.state,
self.options.runner, self.options.loader,
self.options.name, self.options.outputter)):
self.options.name, self.options.outputter,
self.options.fileserver)):
self.options.module = True
self.options.client = True
self.options.shell = True
@ -158,6 +167,7 @@ class SaltTestsuiteParser(SaltCoverageTestingParser):
self.options.state = True
self.options.loader = True
self.options.outputter = True
self.options.fileserver = True
self.start_coverage(
branch=True,
@ -192,6 +202,7 @@ class SaltTestsuiteParser(SaltCoverageTestingParser):
self.options.client or
self.options.loader or
self.options.outputter or
self.options.fileserver or
named_tests):
# We're either not running any of runner, state, module and client
# tests, or, we're only running unittests by passing --unit or by
@ -240,7 +251,8 @@ class SaltTestsuiteParser(SaltCoverageTestingParser):
if not any([self.options.client, self.options.module,
self.options.runner, self.options.shell,
self.options.state, self.options.loader,
self.options.outputter, self.options.name]):
self.options.outputter, self.options.name,
self.options.fileserver]):
return status
with TestDaemon(self):
@ -264,6 +276,8 @@ class SaltTestsuiteParser(SaltCoverageTestingParser):
status.append(self.run_integration_suite('shell', 'Shell'))
if self.options.outputter:
status.append(self.run_integration_suite('output', 'Outputter'))
if self.options.fileserver:
status.append(self.run_integration_suite('fileserver', 'Fileserver'))
return status
def run_unit_tests(self):

13
tests/unit/client_test.py
View File

@ -14,7 +14,7 @@ ensure_in_syspath('../')
# Import Salt libs
import integration
from salt import client
from salt import client, config
from salt.exceptions import EauthAuthenticationError, SaltInvocationError
@ -22,15 +22,14 @@ from salt.exceptions import EauthAuthenticationError, SaltInvocationError
class LocalClientTestCase(TestCase,
integration.AdaptedConfigurationTestCaseMixIn):
def setUp(self):
if not os.path.exists('/tmp/salttest'):
# This path is hardcoded in the configuration file
os.makedirs('/tmp/salttest/cache')
master_config_path = self.get_config_file_path('master')
master_config = config.master_config(master_config_path)
if not os.path.exists(master_config['cachedir']):
os.makedirs(master_config['cachedir'])
if not os.path.exists(integration.TMP_CONF_DIR):
os.makedirs(integration.TMP_CONF_DIR)
self.local_client = client.LocalClient(
self.get_config_file_path('master')
)
self.local_client = client.LocalClient(mopts=master_config)
def test_create_local_client(self):
local_client = client.LocalClient(self.get_config_file_path('master'))

25
tests/unit/config_test.py
View File

@ -93,7 +93,7 @@ def _fopen_side_effect_etc_hosts(filename):
_unhandled_mock_read(filename)
class ConfigTestCase(TestCase):
class ConfigTestCase(TestCase, integration.AdaptedConfigurationTestCaseMixIn):
def test_proper_path_joining(self):
fpath = tempfile.mktemp()
try:
@ -335,31 +335,28 @@ class ConfigTestCase(TestCase):
shutil.rmtree(tempdir)
def test_syndic_config(self):
syndic_conf_path = os.path.join(
integration.INTEGRATION_TEST_DIR, 'files', 'conf', 'syndic'
)
minion_config_path = os.path.join(
integration.INTEGRATION_TEST_DIR, 'files', 'conf', 'minion'
)
syndic_conf_path = self.get_config_file_path('syndic')
minion_conf_path = self.get_config_file_path('minion')
syndic_opts = sconfig.syndic_config(
syndic_conf_path, minion_config_path
syndic_conf_path, minion_conf_path
)
syndic_opts.update(salt.minion.resolve_dns(syndic_opts))
root_dir = syndic_opts['root_dir']
# id & pki dir are shared & so configured on the minion side
self.assertEqual(syndic_opts['id'], 'minion')
self.assertEqual(syndic_opts['pki_dir'], '/tmp/salttest/pki')
self.assertEqual(syndic_opts['pki_dir'], os.path.join(root_dir, 'pki'))
# the rest is configured master side
self.assertEqual(syndic_opts['master_uri'], 'tcp://127.0.0.1:54506')
self.assertEqual(syndic_opts['master_port'], 54506)
self.assertEqual(syndic_opts['master_ip'], '127.0.0.1')
self.assertEqual(syndic_opts['master'], 'localhost')
self.assertEqual(syndic_opts['sock_dir'], '/tmp/salttest/minion_sock')
self.assertEqual(syndic_opts['cachedir'], '/tmp/salttest/cachedir')
self.assertEqual(syndic_opts['log_file'], '/tmp/salttest/osyndic.log')
self.assertEqual(syndic_opts['pidfile'], '/tmp/salttest/osyndic.pid')
self.assertEqual(syndic_opts['sock_dir'], os.path.join(root_dir, 'minion_sock'))
self.assertEqual(syndic_opts['cachedir'], os.path.join(root_dir, 'cachedir'))
self.assertEqual(syndic_opts['log_file'], os.path.join(root_dir, 'osyndic.log'))
self.assertEqual(syndic_opts['pidfile'], os.path.join(root_dir, 'osyndic.pid'))
# Show that the options of localclient that repub to local master
# are not merged with syndic ones
self.assertEqual(syndic_opts['_master_conf_file'], minion_config_path)
self.assertEqual(syndic_opts['_master_conf_file'], minion_conf_path)
self.assertEqual(syndic_opts['_minion_conf_file'], syndic_conf_path)
def test_check_dns_deprecation_warning(self):