valitydev/salt
14
0
Fork 0
mirror of https://github.com/valitydev/salt.git synced 2024-11-08 01:18:58 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Merge branch 'develop' of github.com:saltstack/salt into develop

Browse Source
This commit is contained in:
Jack Kuan 2012-09-12 16:56:46 -04:00
commit f1d8ebf520
15 changed files with 409 additions and 21 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

18
conf/minion.template
View File

@ -207,7 +207,7 @@
# The state_verbose setting can be set to True or False, when set to False
# all data that has a result of True and no changes will be suppressed.
#state_verbose: True
#
#
# The state_output setting changes if the output is the full multi line
# output for each changed state if set to 'full', but if set to 'terse'
# the output will be shortened to a single line.
@ -257,6 +257,9 @@
# the module name is followed by a . and then the value. Also, all top level
# data must be applied via the yaml dict construct, some examples:
#
# You can specify that all modules should run in test mode:
#test: True
#
# A simple value for the test module:
#test.foo: foo
#
@ -265,3 +268,16 @@
#
# A dict for the test module:
#test.baz: {spam: sausage, cheese: bread}
###### Update settings ######
###########################################
# Using the features in Esky, a salt minion can both run as a frozen app and
# be updated on the fly. These options control how the update process
# (saltutil.update()) behaves.
#
# The url for finding and downloading updates. Disabled by default.
#update_url: False
#
# The list of services to restart after a successful update. Empty by default.
#update_restart_services: []

5
doc/index.rst
View File

@ -201,6 +201,11 @@ Salt is many splendid things.
Use Salt programmatically from your own scripts and programs easily and
simply via ``import salt``.
:doc:`Automatic Updates and Frozen Deployments <ref/esky>`
Use a frozen install to make deployments easier (Even on Windows!). Or
take advantage of automatic updates to keep your minions running your
latest builds.
Reference
---------

37
doc/ref/configuration/minion.rst
View File

@ -443,3 +443,40 @@ option then the minion will log a warning message.
- extra_config
- minion.d/*
- /etc/roles/webserver
Frozen Build Update Settings
----------------------------
These options control how :py:func:`salt.modules.saltutil.update` works with esky
frozen apps. For more information look at `<https://github.com/cloudmatrix/esky/>`_.
.. conf_minion:: update_url
``update_url``
--------------
Default: ``False`` (Update feature is disabled)
The url to use when looking for application updates. Esky depends on directory
listings to search for new versions. A webserver running on your Master is a
good starting point for most setups.
.. code-block:: yaml
update_url: 'http://salt.example.com/minion-updates'
.. conf_minion:: update_restart_services
``update_restart_services``
---------------------------
Default: ``[]`` (service restarting on update is disabled)
A list of services to restart when the minion software is updated. This would
typically just be a list containing the minion's service name, but you may
have other services that need to go with it.
.. code-block:: yaml
update_restart_services: ['salt-minion']

75
doc/ref/esky.rst Normal file
View File

@ -0,0 +1,75 @@
======================================
Automatic Updates / Frozen Deployments
======================================
.. versionadded:: 0.10.3.d
Salt has support for the
`Esky <https://github.com/cloudmatrix/esky>`_ application freezing and update
tool. This tool allows one to build a complete zipfile out of the salt scripts
and all their dependencies - including shared objects / DLLs.
Getting Started
===============
To build frozen applications, you'll need a suitable build environment for each
of your platforms. You should probably set up a virtualenv in order to limit
the scope of Q/A.
This process does work on Windows. Follow the directions at
`<https://github.com/saltstack/salt-windows-install>`_ for details on
installing Salt in Windows. Only the 32-bit Python and dependencies have been
tested, but they have been tested on 64-bit Windows.
You will need to install ``esky`` and ``bbfreeze`` from Pypi in order to enable
the ``bdist_esky`` command in ``setup.py``.
Building and Freezing
=====================
Once you have your tools installed and the environment configured, you can then
``python setup.py sdist`` to get the eggs prepared. After that is done, run
``python setup.py bdist_esky`` to have Esky traverse the module tree and pack
all the scripts up into a redistributable. There will be an appropriately
versioned ``salt-VERSION.zip`` in ``dist/`` if everything went smoothly.
Using the Frozen Build
======================
Unpack the zip file in your desired install location. Scripts like
``salt-minion`` and ``salt-call`` will be in the root of the zip file. The
associated libraries and bootstrapping will be in the directories at the same
level. (Check the `Esky <https://github.com/cloudmatrix/esky>`_ documentation
for more information)
To support updating your minions in the wild, put your builds on a web server
that your minions can reach. :py:func:`salt.modules.saltutil.update` will
trigger an update and (optionally) a restart of the minion service under the
new version.
Gotchas
=======
Windows and the Visual Studio Redist
------------------------------------
You will need to install the Visual C++ 2008 32-bit redistributable on all
Windows minions. Esky has an option to pack the library into the zipfile,
but OpenSSL does not seem to acknowledge the new location. If you get a
``no OPENSSL_Applink`` error on the console when trying to start your
frozen minion, you have forgotten to install the redistributable.
Mixed Linux environments and Yum
--------------------------------
The Yum Python module doesn't appear to be available on any of the standard
Python package mirrors. If you need to support RHEL/CentOS systems, you
should build on that platform to support all your Linux nodes. Also remember
to build your virtualenv with ``--system-site-packages`` so that the
``yum`` module is included.
Automatic (Python) module discovery
-----------------------------------
Automatic (Python) module discovery does not work with the late-loaded scheme that
Salt uses for (Salt) modules. You will need to explicitly add any
misbehaving modules to the ``freezer_includes`` in Salt's ``setup.py``.
Always check the zipped application to make sure that the necessary modules
were included.

13
doc/ref/states/testing.rst
View File

@ -18,3 +18,16 @@ interface can be invoked on any of the major state run functions:
The test run is mandated by adding the ``test=True`` option to the states. The
return information will show states that will be applied in yellow and the
result is reported as `None`.
Default Test
============
If the value `test` is set to True in the minion configuration file then states
will default to being executed in test mode. If this value is set then states
can still be run by calling test=False:
.. code-block:: bash
# salt \* state.highstate test=False
# salt \* state.sls test=False
# salt \* state.single test=False

130
doc/topics/releases/0.10.3.rst Normal file
View File

@ -0,0 +1,130 @@
=========================
Salt 0.10.3 Release Notes
=========================
The latest taste of Salt has come, this release has many fixes and feature
additions. Modifications have been made to make ZeroMQ connections more
reliable, the begining of the ACL system is in place, a new command line
parsing system has been added, dynamic module distribution has become more
environment aware, the new `master_fingerprint` option and many more!
Major Features
==============
ACL System
----------
The new ACL system has been introduced. The ACL system allows for system users
other than root to execute salt commands. Users can be allowed to execute
specific commands in the same way that minions are opened up to the peer
system.
The configuration value to open up the ACL system is called ``client_acl``
and is configured like so:
.. code-block:: yaml
client_acl:
fred:
- test..*
- pkg.list_pkgs
Where `fred` is allowed access to functions in the test module and to the
``pkg.list_pkgs`` function.
Parsing System
--------------
Pedro Algavio, aka s0undt3ch, has added a substantial update to the command
line parsing system that makes the help message output much cleaner and easier
to search through. Salt parsers now have `--versions-report` besides usual
`--version` info which you can provide when reporting any issues found.
Key Generation
--------------
We have reduced the requirements needed for `salt-key` to generate minion keys.
You're no longer required to have salt configured and it's common directories
created just to generate keys. This might prove useful if you're batch creating
keys to pre-load on minions.
Max Open Files
--------------
While we're currently unable to properly handle ZeroMQ's abort signals when the
max open files is reached, due to the way that's handled on ZeroMQ's, we have
minimized the chances of this happening without at least warning the user.
More State Output Options
-------------------------
Some major changes have been made to the state output system. In the past state
return data was printed in a very verbose fashion and only states that failed
or made changes were printed by default. Now two options can be passed to the
master and minion configuration files to change the behavior of the state
output. State output can be set to verbose (default) or non-verbose with the
``state_verbose`` option:
.. code-block:: yaml
state_verbose: False
It is noteworthy that the state_verbose option used to be set to `False` by
default but has been changed to `True` by default in 0.10.3 due to many
requests for the change.
Te next option to be aware of new and called ``state_output``. This option
allows for the state output to be set to `full` (default) or `terse`.
The `full` output is the standard state output, but the new `terse` output
will print only one line per state making the output much easier to follow when
executing a large state system.
.. code-block:: yaml
state_output: terse
`state.file.append` Improvements
--------------------------------
The salt state `file.append()` tries *not* to append existing text. Previously
the matching check was being made line by line. While this kind of check might
be enough for most cases, if the text being appended was multi-line, the check
would not work properly. This issue is now properly handled, the match is done
as a whole ignoring any white space addition or removal except inside commas.
For those thinking that, in order to properly match over multiple lines, salt
will load the whole file into memory, that's not true. For most cases this is
not important but an erroneous order to read a 4GB file, if not properly
handled, like salt does, could make salt chew that amount of memory. Salt has
a buffered file reader which will keep in memory a maximum of 256KB and
iterates over the file in chunks of 32KB to test for the match, more than
enough, if not, explain your usage on a ticket. With this change, also
`salt.modules.file.contains()`, `salt.modules.file.contains_regex()`,
`salt.modules.file.contains_glob()` and `salt.utils.find` now do the searching
and/or matching using the buffered chunks approach explained above.
Two new keyword arguments were also added, `makedirs` and `source`.
The first, `makedirs` will create the necessary directories in order to append
to the specified file, of course, it only applies if we're trying to append to
a non-existing file on a non-existing directory:
.. code-block:: yaml
/tmp/salttest/file-append-makedirs:
file.append:
text: foo
makedirs: True
The second, `source`, allows to append the contents of a file instead of
specifying the text.
.. code-block:: yaml
/tmp/salttest/file-append-source:
file.append:
- source: salt://testfile

33
salt/client.py
View File

@ -173,7 +173,7 @@ class LocalClient(object):
if comps[0] not in grains:
minions.remove(id_)
continue
if isinstance(grains[comps[0]], list):
if isinstance(grains.get(comps[0]), list):
# We are matching a single component to a single list member
found = False
for member in grains[comps[0]]:
@ -185,7 +185,7 @@ class LocalClient(object):
minions.remove(id_)
continue
if fnmatch.fnmatch(
str(grains[comps[0]]).lower(),
str(grains.get(comps[0], '').lower()),
comps[1].lower(),
):
continue
@ -552,7 +552,7 @@ class LocalClient(object):
# The timeout +1 has not been reached and there is still a
# write tag for the syndic
continue
if len(fret) >= len(minions):
if len(found.intersection(minions)) >= len(minions):
# All minions have returned, break out of the loop
break
if int(time.time()) > start + timeout:
@ -570,7 +570,7 @@ class LocalClient(object):
continue
if verbose:
if tgt_type == 'glob' or tgt_type == 'pcre':
if not len(fret) >= len(minions):
if len(found.intersection(minions)) >= len(minions):
print('\nThe following minions did not return:')
fail = sorted(list(minions.difference(found)))
for minion in fail:
@ -624,7 +624,7 @@ class LocalClient(object):
# The timeout +1 has not been reached and there is still a
# write tag for the syndic
continue
if len(ret) >= len(minions):
if len(found.intersection(minions)) >= len(minions):
break
if int(time.time()) > start + timeout:
break
@ -677,7 +677,7 @@ class LocalClient(object):
# The timeout +1 has not been reached and there is still a
# write tag for the syndic
continue
if len(ret) >= len(minions):
if len(set(ret.keys()).intersection(minions)) >= len(minions):
# All Minions have returned
return ret
if int(time.time()) > start + timeout:
@ -732,7 +732,7 @@ class LocalClient(object):
# The timeout +1 has not been reached and there is still a
# write tag for the syndic
continue
if len(ret) >= len(minions):
if len(set(ret.keys()).intersection(minions)) >= len(minions):
return ret
if int(time.time()) > start + timeout:
return ret
@ -779,12 +779,12 @@ class LocalClient(object):
ret[raw['id']] = {'ret': raw['return']}
if 'out' in raw:
ret[raw['id']]['out'] = raw['out']
if len(found) >= len(minions):
if len(found.intersection(minions)) >= len(minions):
# All minions have returned, break out of the loop
break
continue
# Then event system timeout was reached and nothing was returned
if len(found) >= len(minions):
if len(found.intersection(minions)) >= len(minions):
# All minions have returned, break out of the loop
break
if glob.glob(wtag) and not int(time.time()) > start + timeout + 1:
@ -842,17 +842,20 @@ class LocalClient(object):
while True:
raw = self.event.get_event(timeout, jid)
if not raw is None:
if 'syndic' in raw:
minions.update(raw['syndic'])
continue
found.add(raw['id'])
ret = {raw['id']: {'ret': raw['return']}}
if 'out' in raw:
ret[raw['id']]['out'] = raw['out']
yield ret
if len(found) >= len(minions):
if len(found.intersection(minions)) >= len(minions):
# All minions have returned, break out of the loop
break
continue
# Then event system timeout was reached and nothing was returned
if len(found) >= len(minions):
if len(found.intersection(minions)) >= len(minions):
# All minions have returned, break out of the loop
break
if glob.glob(wtag) and not int(time.time()) > start + timeout + 1:
@ -1020,13 +1023,7 @@ class LocalClient(object):
# return what we get back
minions = self.check_minions(tgt, expr_form)
if self.opts['order_masters']:
# If we're a master of masters, ignore the check_minion and
# set the minions to the target. This speeds up wait time
# for lists and ranges and makes regex and other expression
# forms possible
minions = tgt
elif not minions:
if not minions:
return {'jid': None,
'minions': minions}

2
salt/config.py
View File

@ -204,6 +204,8 @@ def minion_config(path):
'grains': {},
'permissive_pki_access': False,
'default_include': 'minion.d/*.conf',
'update_url': False,
'update_restart_services': [],
}
load_config(opts, path, 'SALT_MINION_CONFIG')

9
salt/crypt.py
View File

@ -347,7 +347,14 @@ class Crypticle(object):
aes_key, hmac_key = self.keys
sig = data[-self.SIG_SIZE:]
data = data[:-self.SIG_SIZE]
if hmac.new(hmac_key, data, hashlib.sha256).digest() != sig:
mac_bytes = hmac.new(hmac_key, data, hashlib.sha256).digest()
if len(mac_bytes) != len(sig):
log.warning('Failed to authenticate message')
raise AuthenticationError('message authentication failed')
result = 0
for x, y in zip(mac_bytes, sig):
result |= ord(x) ^ ord(y)
if result != 0:
log.warning('Failed to authenticate message')
raise AuthenticationError('message authentication failed')
iv_bytes = data[:self.AES_BLOCK_SIZE]

3
salt/grains/core.py
View File

@ -218,6 +218,9 @@ def _virtual(osdata):
# Product Name: Virtual Machine
elif 'Manufacturer: Microsoft' in output and 'Virtual Machine' in output:
grains['virtual'] = 'VirtualPC'
# Manufacturer: Parallels Software International Inc.
elif 'Parallels Software' in output:
grains['virtual'] = 'Parallels'
# Fall back to lspci if dmidecode isn't available
elif lspci:
model = __salt__['cmd.run']('lspci').lower()

1
salt/master.py
View File

@ -802,6 +802,7 @@ class AESFuncs(object):
return False
# Format individual return loads
self.event.fire_event({'syndic': load['return'].keys()}, load['jid'])
for key, item in load['return'].items():
ret = {'jid': load['jid'],
'id': key,

47
salt/modules/saltutil.py
View File

@ -9,12 +9,20 @@ import hashlib
import shutil
import signal
import logging
import sys
# Import Salt libs
import salt.payload
import salt.state
from salt._compat import string_types
# Import esky for update functionality
try:
import esky
has_esky = True
except ImportError:
has_esky = False
log = logging.getLogger(__name__)
def _sync(form, env=None):
@ -104,6 +112,45 @@ def _list_emptydirs(rootdir):
emptydirs.append(root)
return emptydirs
def update(version=None):
'''
Update the salt minion from the url defined in opts['update_url']
This feature requires the minion to be running a bdist_esky build.
The version number is optional and will default to the most recent version
available at opts['update_url'].
Returns details about the transaction upon completion.
CLI Example::
salt '*' saltutil.update 0.10.3
'''
if not has_esky:
return "Esky not available as import"
if not getattr(sys, "frozen", False):
return "Instance is not a frozen instance"
if not __opts__['update_url']:
return "'update_url' not configured on this minion"
app = esky.Esky(sys.executable, __opts__['update_url'])
try:
if not version:
version = app.find_update()
if not version:
return "No updates available"
app.fetch_version(version)
app.install_version(version)
app.cleanup()
except Exception as e:
return e
restarted = []
for service in __opts__['update_restart_services']:
restarted.append(__salt__['service.restart'](service))
return {'comment': "Updated from %s to %s" % (__version__, version),
'restarted': restarted}
def sync_modules(env=None):
'''
Sync the modules from the _modules directory on the salt master file

11
salt/utils/__init__.py
View File

@ -476,7 +476,16 @@ def copyfile(source, dest, backup_mode='', cachedir=''):
if backup_mode == 'master' or backup_mode == 'both' and bkroot:
# TODO, backup to master
pass
shutil.move(tgt, dest)
try:
shutil.move(tgt, dest)
except Exception:
pass
if os.path.isfile(tgt):
# The temp file failed to move
try:
os.remove(tgt)
except Exception:
pass
def path_join(*parts):

4
scripts/salt-minion
View File

@ -4,7 +4,11 @@ This script is used to kick off a salt minion daemon
'''
from salt.scripts import salt_minion
from multiprocessing import freeze_support
if __name__ == '__main__':
# This handles the bootstrapping code that is included with frozen
# scripts. It is a no-op on unfrozen code.
freeze_support()
salt_minion()

42
setup.py
View File

@ -128,8 +128,50 @@ setup_kwargs = {'name': NAME,
('share/man/man7', ['doc/man/salt.7']),
],
'install_requires': requirements,
# The dynamic module loading in salt.modules makes this
# package zip unsafe.
'zip_safe': False
}
# bbfreeze explicit includes
# Sometimes the auto module traversal doesn't find everything, so we
# explicitly add it. The auto dependency tracking especially does not work for
# imports occurring in salt.modules, as they are loaded at salt runtime.
# Specifying includes that don't exist doesn't appear to cause a freezing
# error.
freezer_includes = [
'zmq.core.*',
'zmq.utils.*',
'ast',
]
if sys.platform == 'win32':
freezer_includes.extend([
'win32con',
'win32security',
'ntsecuritycon'
])
elif sys.platform.startswith('linux'):
freezer_includes.extend([
'yum'
])
if 'bdist_esky' in sys.argv:
# Add the esky bdist target if the module is available
# may require additional modules depending on platform
from esky import bdist_esky
# bbfreeze chosen for its tight integration with distutils
import bbfreeze
options = setup_kwargs.get('options', {})
options['bdist_esky'] = {
"freezer_module": "bbfreeze",
"freezer_options": {
"includes": freezer_includes
}
}
setup_kwargs['options'] = options
if with_setuptools:
setup_kwargs['entry_points'] = {
"console_scripts": ["salt-master = salt.scripts:salt_master",