Merge remote-tracking branch 'upstream/2014.7' into merge-forward-2015.2

Conflicts:
    doc/topics/installation/windows.rst
    doc/topics/tutorials/gitfs.rst
This commit is contained in:
Colton Myers 2015-04-20 11:39:41 -06:00
commit 1f9934c634
9 changed files with 406 additions and 140 deletions

View File

@ -20,6 +20,10 @@ minion exe>` should match the contents of the corresponding md5 file.
.. admonition:: Download here
* 2014.7.5
* `Salt-Minion-2014.7.5-x86-Setup.exe <http://docs.saltstack.com/downloads/Salt-Minion-2014.7.5-x86-Setup.exe>`__ | `md5 <http://docs.saltstack.com/downloads/Salt-Minion-2014.7.5-x86-Setup.exe.md5>`__
* `Salt-Minion-2014.7.5-AMD64-Setup.exe <http://docs.saltstack.com/downloads/Salt-Minion-2014.7.5-AMD64-Setup.exe>`__ | `md5 <http://docs.saltstack.com/downloads/Salt-Minion-2014.7.5-AMD64-Setup.exe.md5>`__
* 2014.7.4
* `Salt-Minion-2014.7.4-x86-Setup.exe <http://docs.saltstack.com/downloads/Salt-Minion-2014.7.4-x86-Setup.exe>`__ | `md5 <http://docs.saltstack.com/downloads/Salt-Minion-2014.7.4-x86-Setup.exe.md5>`__
* `Salt-Minion-2014.7.4-AMD64-Setup.exe <http://docs.saltstack.com/downloads/Salt-Minion-2014.7.4-AMD64-Setup.exe>`__ | `md5 <http://docs.saltstack.com/downloads/Salt-Minion-2014.7.4-AMD64-Setup.exe.md5>`__
@ -133,7 +137,7 @@ minion exe>` should match the contents of the corresponding md5 file.
.. note::
The executables above will install all dependencies that the Salt minion
The executables above will install dependencies that the Salt minion
requires.
The 64bit installer has been tested on Windows 7 64bit and Windows Server
@ -161,7 +165,7 @@ line. The options `/master` and `/minion-name` allow for configuring the master
hostname and minion name, respectively. Here's an example of using the silent
installer:
.. code-block:: bash
.. code-block:: bat
Salt-Minion-0.17.0-Setup-amd64.exe /S /master=yoursaltmaster /minion-name=yourminionname
@ -169,136 +173,305 @@ installer:
Setting up a Windows build environment
======================================
1. Install the Microsoft Visual C++ 2008 SP1 Redistributable, `vcredist_x86`_
or `vcredist_x64`_.
This document will explain how to set up a development environment for salt on
Windows. The development environment allows you to work with the source code to
customize or fix bugs. It will also allow you to build your own installation.
2. Install `msysgit`_
The Easy Way
------------
3. Clone the Salt git repository from GitHub
Prerequisite Software
^^^^^^^^^^^^^^^^^^^^^
.. code-block:: bash
To do this the easy way you only need to install `Git for Windows <https://github.com/msysgit/msysgit/releases/download/Git-1.9.5-preview20150319/Git-1.9.5-preview20150319.exe/>`_.
git clone git://github.com/saltstack/salt.git
Create the Build Environment
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
4. Install the latest point release of `Python 2.7`_ for the architecture you
wish to target
1. Clone the `Salt-Windows-Dev <https://github.com/saltstack/salt-windows-dev/>`_
repo from github.
5. Add C:\\Python27 and C:\\Python27\\Scripts to your system path
Open a command line and type:
6. Download and run the Setuptools bootstrap - `ez_setup.py`_
.. code-block:: bat
.. code-block:: bash
git clone https://github.com/saltstack/salt-windows-dev
python ez_setup.py
2. Build the Python Environment
7. Install Pip
Go into the salt-windows-dev directory. Right-click the file named
**dev_env.ps1** and select **Run with PowerShell**
.. code-block:: bash
If you get an error, you may need to change the execution policy.
easy_install pip
Open a powershell window and type the following:
8. Install the latest point release of `OpenSSL for Windows`_
.. code-block:: powershell
#. During setup, choose first option to install in Windows system
directory
Set-ExecutionPolicy RemoteSigned
9. Install the latest point release of `M2Crypto`_
This will download and install Python with all the dependencies needed to
develop and build salt.
#. In general, be sure to download installers targeted at py2.7 for your
chosen architecture
3. Build the Salt Environment
10. Install the latest point release of `pycrypto`_
Right-click on the file named **dev_env_salt.ps1** and select **Run with
Powershell**
11. Install the latest point release of `pywin32`_
This will clone salt into ``C:\Salt-Dev\salt`` and set it to the 2015.2
branch. You could optionally run the command from a powershell window with a
``-Version`` switch to pull a different version. For example:
12. Install the latest point release of `Cython`_
.. code-block:: powershell
13. Install the latest point release of `jinja2`_
dev_env_salt.ps1 -Version '2014.7'
14. Install the latest point release of `msgpack`_
To view a list of available branches and tags, open a command prompt in your
`C:\Salt-Dev\salt` directory and type:
15. Install psutil
.. code-block:: bat
.. code-block:: bash
easy_install psutil
16. Install pyzmq
.. code-block:: bash
easy_install pyzmq
17. Install PyYAML
.. code-block:: bash
easy_install pyyaml
18. Install bbfreeze
.. code-block:: bash
easy_install bbfreeze
19. Install wmi
.. code-block:: bash
pip install wmi
20. Install requests
.. code-block:: bash
pip install requests
21. Install markupsafe
.. code-block:: bash
pip install markupsafe
22. Install six
.. code-block:: bash
pip install six
23. Install esky
.. code-block:: bash
pip install esky
24. Install Salt
.. code-block:: bash
cd salt
python setup.py install
25. Build a frozen binary distribution of Salt
.. code-block:: bash
python setup.py bdist_esky
A zip file has been created in the ``dist/`` folder, containing a frozen copy
of Python and the dependency libraries, along with Windows executables for each
of the Salt scripts.
git branch -a
git tag -n
Building the installer
======================
The Hard Way
------------
The Salt Windows installer is built with the open-source NSIS compiler. The
source for the installer is found in the pkg directory of the Salt repo here:
:blob:`pkg/windows/installer/Salt-Minion-Setup.nsi`. To create the installer,
extract the frozen archive from ``dist/`` into ``pkg/windows/buildenv/`` and
run NSIS.
Prerequisite Software
^^^^^^^^^^^^^^^^^^^^^
The NSIS installer can be found here: http://nsis.sourceforge.net/Main_Page
Install the following software:
1. `Git for Windows <https://github.com/msysgit/msysgit/releases/download/Git-1.9.5-preview20150319/Git-1.9.5-preview20150319.exe/>`_
2. `Nullsoft Installer <http://downloads.sourceforge.net/project/nsis/NSIS%203%20Pre-release/3.0b1/nsis-3.0b1-setup.exe/>`_
Download the Prerequisite zip file for your CPU architecture from the
SaltStack download site:
* `Salt32.zip <http://docs.saltstack.com/downloads/windows-deps/Salt32.zip/>`_
* `Salt64.zip <http://docs.saltstack.com/downloads/windows-deps/Salt64.zip/>`_
These files contain all sofware required to build and develop salt. Unzip the
contents of the file to ``C:\Salt-Dev\temp``.
Create the Build Environment
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1. Build the Python Environment
* Install Python:
Browse to the ``C:\Salt-Dev\temp`` directory and find the Python
installation file for your CPU Architecture under the corresponding
subfolder. Double-click the file to install python.
Make sure the following are in your **PATH** environment variable:
.. code-block:: bat
C:\Python27
C:\Python27\Scripts
* Install Pip
Open a command prompt and navigate to ``C:\Salt-Dev\temp``
Run the following command:
.. code-block:: bat
python get-pip.py
* Easy Install compiled binaries.
M2Crypto, PyCrypto, and PyWin32 need to be installed using Easy Install.
Open a command prompt and navigate to ``C:\Salt-Dev\temp\<cpuarch>``.
Run the following commands:
.. code-block:: bat
easy_install -Z <M2Crypto file name>
easy_install -Z <PyCrypto file name>
easy_install -Z <PyWin32 file name>
.. note::
You can type the first part of the file name and then press the tab key
to auto-complete the name of the file.
* Pip Install Additional Prerequisites
All remaining prerequisites need to be pip installed. These prerequisites
are as follow:
* MarkupSafe
* Jinja
* MsgPack
* PSUtil
* PyYAML
* PyZMQ
* WMI
* Requests
* Certifi
Open a command prompt and navigate to ``C:\Salt-Dev\temp``. Run the following
commands:
.. code-block:: bat
pip install <cpuarch>\<MarkupSafe file name>
pip install <Jinja file name>
pip install <cpuarch>\<MsgPack file name>
pip install <cpuarch>\<psutil file name>
pip install <cpuarch>\<PyYAML file name>
pip install <cpuarch>\<pyzmq file name>
pip install <WMI file name>
pip install <requests file name>
pip install <certifi file name>
2. Build the Salt Environment
* Clone Salt
Open a command prompt and navigate to ``C:\Salt-Dev``. Run the following command
to clone salt:
.. code-block:: bat
git clone https://github.com/saltstack/salt
* Checkout Branch
Checkout the branch or tag of salt you want to work on or build. Open a
command prompt and navigate to ``C:\Salt-Dev\salt``. Get a list of
available tags and branches by running the following commands:
.. code-block:: bat
git fetch --all
To view a list of available branches:
git branch -a
To view a list of availabel tags:
git tag -n
Checkout the branch or tag by typing the following command:
.. code-block:: bat
git checkout <branch/tag name>
* Clean the Environment
When switching between branches residual files can be left behind that
will interfere with the functionality of salt. Therefore, after you check
out the branch you want to work on, type the following commands to clean
the salt environment:
.. code-block: bat
git clean -fxd
git reset --hard HEAD
Developing with Salt
====================
There are two ways to develop with salt. You can run salt's setup.py each time
you make a change to source code or you can use the setup tools develop mode.
Configure the Minion
--------------------
Both methods require that the minion configuration be in the ``C:\salt``
directory. Copy the conf and var directories from ``C:\Salt-Dev\salt\pkg\
windows\buildenv`` to ``C:\salt``. Now go into the ``C:\salt\conf`` directory
and edit the file name ``minion`` (no extension). You need to configure the
master and id parameters in this file. Edit the following lines:
.. code-block:: bat
master: <ip or name of your master>
id: <name of your minion>
Setup.py Method
---------------
Go into the ``C:\Salt-Dev\salt`` directory from a cmd prompt and type:
.. code-block:: bat
python setup.py install --force
This will install python into your python installation at ``C:\Python27``.
Everytime you make an edit to your source code, you'll have to stop the minion,
run the setup, and start the minion.
To start the salt-minion go into ``C:\Python27\Scripts`` from a cmd prompt and
type:
.. code-block:: bat
salt-minion
For debug mode type:
.. code-block:: bat
salt-minion -l debug
To stop the minion press Ctrl+C.
Setup Tools Develop Mode (Preferred Method)
-------------------------------------------
To use the Setup Tools Develop Mode go into ``C:\Salt-Dev\salt`` from a cmd
prompt and type:
.. code-block:: bat
pip install -e .
This will install pointers to your source code that resides at
``C:\Salt-Dev\salt``. When you edit your source code you only have to restart
the minion.
Build the windows installer
===========================
This is the method of building the installer as of version 2014.7.4.
Clean the Environment
---------------------
Make sure you don't have any leftover salt files from previous versions of salt
in your Python directory.
1. Remove all files that start with salt in the ``C:\Python27\Scripts``
directory
2. Remove all files and directorys that start with salt in the
``C:\Python27\Lib\site-packages`` directory
Install Salt
------------
Install salt using salt's setup.py. From the ``C:\Salt-Dev\salt`` directory
type the following command:
.. code-block:: bat
python setup.py install --force
Build the Installer
-------------------
From cmd prompt go into the ``C:\Salt-Dev\salt\pkg\windows`` directory. Type
the following command for the branch or tag of salt you're building:
.. code-block:: bat
BuildSalt.bat <branch or tag>
This will copy python with salt installed to the ``buildenv\bin`` directory,
make it portable, and then create the windows installer . The .exe for the
windows installer will be placed in the ``installer`` directory.
Testing the Salt minion
@ -353,7 +526,7 @@ salt, including all dependencies:
This script is not up to date. Please use the installer found above
.. code-block:: bash
.. code-block:: powershell
# (All in one line.)
@ -373,12 +546,10 @@ Packages management under Windows 2003
On windows Server 2003, you need to install optional component "wmi windows
installer provider" to have full list of installed packages. If you don't have
this, salt-minion can't report some installed packages.
this, salt-minion can't report some installed softwares.
.. _http://csa-net.dk/salt: http://csa-net.dk/salt
.. _vcredist_x86: http://www.microsoft.com/en-us/download/details.aspx?id=5582
.. _vcredist_x64: http://www.microsoft.com/en-us/download/details.aspx?id=2092
.. _msysgit: http://code.google.com/p/msysgit/downloads/list?can=3
.. _Python 2.7: http://www.python.org/downloads
.. _ez_setup.py: https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py
@ -389,3 +560,4 @@ this, salt-minion can't report some installed packages.
.. _Cython: http://www.lfd.uci.edu/~gohlke/pythonlibs/#cython
.. _jinja2: http://www.lfd.uci.edu/~gohlke/pythonlibs/#jinja2
.. _msgpack: http://www.lfd.uci.edu/~gohlke/pythonlibs/#msgpack

View File

@ -0,0 +1,58 @@
===========================
Salt 2014.7.5 Release Notes
===========================
:release: TBA
Version 2014.7.5 is a bugfix release for :doc:`2014.7.0
</topics/releases/2014.7.0>`.
Changes:
- Fixed a key error bug in salt-cloud
- Updated man pages to better match documentation
- Fixed bug concerning high CPU usage with salt-ssh
- Fixed bugs with remounting cvfs and fuse filesystems
- Fixed bug with alowing requisite tracking of entire sls files
- Fixed bug with aptpkg.mod_repo returning OK even if apt-add-repository fails
- Increased frequency of ssh terminal output checking
- Fixed malformed locale string in localmod module
- Fixed checking of available version of package when accept_keywords were changed
- Fixed bug to make git.latest work with empty repositories
- Added **kwargs to service.mod_watch which removes warnings about `enable` and `__reqs__` not being supported by the function
- Improved state comments to not grow so quickly on failed requisites
- Added force argument to service to trigger force_reload
- Fixed bug to andle pkgrepo keyids that have been converted to int
- Fixed module.portage_config bug with appending accept_keywords
- Fixed bug to correctly report disk usage on windows minion
- Added the ability to specify key prefix for S3 ext_pillar
- Fixed issues with batch mode operating on the incorrect number of minions
- Fixed a bug with the proxmox cloud provider stacktracing on disk definition
- Fixed a bug with the changes dictionary in the file state
- Fixed the TCP keep alive settings to work better with SREQ caching
- Fixed many bugs within the iptables state and module
- Fixed bug with states by adding `fun`, `state`, and `unless` to the state runtime internal keywords listing
- Added ability to eAuth against Active Directory
- Fixed some salt-ssh issues when running on Fedora 21
- Fixed grains.get_or_set_hash to work with multiple entries under same key
- Added better explanations and more examples of how the Reactor calls functions to docs
- Fixed bug to not pass `ex_config_drive` to libcloud unless it's explicitly enabled
- Fixed bug with pip.install on windows
- Fixed bug where puppet.run always returns a 0 retcode
- Fixed race condition bug with minion scheduling via pillar
- Made efficiency improvements and bug fixes to the windows installer
- Updated environment variables to fix bug with pygit2 when running salt as non-root user
- Fixed cas behavior on data module -- data.cas was not saving changes
- Fixed GPG rendering error
- Fixed strace error in virt.query
- Fixed stacktrace when running chef-solo command
- Fixed possible bug wherein uncaught exceptions seem to make zmq3 tip over when threading is involved
- Fixed argument passing to the reactor
- Fixed glibc caching to prevent bug where salt-minion getaddrinfo in dns_check() never got updated nameservers
Known issues:
- In multimaster mode, a minion may become temporarily unresponsive
if modules or pillars are refreshed at the same time that one
or more masters are down. This can be worked around by setting
'auth_timeout' and 'auth_tries' down to shorter periods.

View File

@ -59,13 +59,12 @@ be used to install it:
If pygit2_ is not packaged for the platform on which the Master is running, the
pygit2_ website has installation instructions here__. Keep in mind however that
following these instructions will install libgit2 and pygit2_ without system
packages. Also, while this is not explicitly mentioned in the pygit2_
installation instructions, libssh2 development headers must be installed before
building libgit2 in order to enable access to SSH-protected git repositories.
Luckily, these are available in most distros' repositories, usually as either
``libssh2-devel`` or ``libssh2-dev``, depending on platform.
packages. Additionally, keep in mind that :ref:`SSH authentication in pygit2
<pygit2-authentication-ssh>` requires libssh2_ (*not* libssh) development
libraries to be present before libgit2 is built.
.. __: http://www.pygit2.org/install.html
.. _libssh2: http://www.libssh2.org/
GitPython
---------
@ -543,6 +542,8 @@ an ``insecure_auth`` parameter:
- password: mypassword
- insecure_auth: True
.. _pygit2-authentication-ssh:
SSH
~~~

View File

@ -573,6 +573,8 @@ def list_nodes(call=None):
ret[node]['public_ips'] = nodes[node]['primaryIpAddress']
if 'primaryBackendIpAddress' in nodes[node]:
ret[node]['private_ips'] = nodes[node]['primaryBackendIpAddress']
if 'status' in nodes[node]:
ret[node]['state'] = str(nodes[node]['status']['name'])
return ret

View File

@ -573,7 +573,7 @@ def _verify_auth(repo):
'''
Helper function to log errors about missing auth parameters
'''
log.error(
log.critical(
'Incomplete authentication information for remote {0}. Missing '
'parameters: {1}'.format(remote_url, ', '.join(missing))
)
@ -596,9 +596,9 @@ def _verify_auth(repo):
user = address.split('@')[0]
if user == address:
# No '@' sign == no user. This is a problem.
log.error(
'Password / keypair specified for remote {0}, but remote '
'URL is missing a username'.format(repo['url'])
log.critical(
'Keypair specified for remote {0}, but remote URL is missing '
'a username'.format(repo['url'])
)
_failhard()
@ -621,7 +621,7 @@ def _verify_auth(repo):
return True
if password_ok:
if transport == 'http' and not repo['insecure_auth']:
log.error(
log.critical(
'Invalid configuration for gitfs remote {0}. '
'Authentication is disabled by default on http remotes. '
'Either set gitfs_insecure_auth to True in the master '
@ -637,7 +637,7 @@ def _verify_auth(repo):
missing_auth = [x for x in required_params if not bool(repo[x])]
_incomplete_auth(repo['url'], missing_auth)
else:
log.error(
log.critical(
'Invalid configuration for remote {0}. Unsupported transport '
'{1!r}.'.format(repo['url'], transport)
)
@ -701,7 +701,7 @@ def init():
salt.utils.repack_dictlist(remote[repo_url]).items()]
)
if not per_remote_conf:
log.error(
log.critical(
'Invalid per-remote configuration for gitfs remote {0}. '
'If no per-remote parameters are being specified, there '
'may be a trailing colon after the URL, which should be '
@ -812,7 +812,7 @@ def init():
'{0}'.format(exc))
if provider == 'gitpython':
msg += ' Perhaps git is not available.'
log.error(msg, exc_info_on_loglevel=logging.DEBUG)
log.critical(msg, exc_info_on_loglevel=logging.DEBUG)
_failhard()
if new_remote:
@ -1148,7 +1148,21 @@ def update():
except KeyError:
# No credentials configured for this repo
pass
fetch = origin.fetch()
try:
fetch = origin.fetch()
except pygit2.errors.GitError as exc:
# Using exc.__str__() here to avoid deprecation warning
# when referencing exc.message
if 'unsupported url protocol' in exc.__str__().lower() \
and isinstance(repo.get('credentials'),
pygit2.Keypair):
log.error(
'Unable to fetch SSH-based gitfs remote {0}. '
'libgit2 must be compiled with libssh2 to support '
'SSH authentication.'.format(repo['url'])
)
continue
raise
try:
# pygit2.Remote.fetch() returns a dict in pygit2 < 0.21.0
received_objects = fetch['received_objects']
@ -1156,10 +1170,16 @@ def update():
# pygit2.Remote.fetch() returns a class instance in
# pygit2 >= 0.21.0
received_objects = fetch.received_objects
log.debug(
'gitfs received {0} objects for remote {1}'
.format(received_objects, repo['url'])
)
if received_objects != 0:
log.debug(
'gitfs received {0} objects for remote {1}'
.format(received_objects, repo['url'])
)
else:
log.debug(
'gitfs remote {0} is up-to-date'
.format(repo['url'])
)
# Clean up any stale refs
refs_post = repo['repo'].listall_references()
cleaned = _clean_stale(repo['repo'], refs_post)
@ -1176,14 +1196,14 @@ def update():
try:
refs_post = client.fetch(path, repo['repo'])
except dulwich.errors.NotGitRepository:
log.critical(
log.error(
'Dulwich does not recognize remote {0} as a valid '
'remote URL. Perhaps it is missing \'.git\' at the '
'end.'.format(repo['url'])
)
continue
except KeyError:
log.critical(
log.error(
'Local repository cachedir {0!r} (corresponding '
'remote: {1}) has been corrupted. Salt will now '
'attempt to remove the local checkout to allow it to '
@ -1194,7 +1214,7 @@ def update():
try:
salt.utils.rm_rf(repo['cachedir'])
except OSError as exc:
log.critical(
log.error(
'Unable to remove {0!r}: {1}'
.format(repo['cachedir'], exc)
)
@ -1622,7 +1642,7 @@ def _file_lists(load, form):
try:
os.makedirs(list_cachedir)
except os.error:
log.critical('Unable to make cachedir {0}'.format(list_cachedir))
log.error('Unable to make cachedir {0}'.format(list_cachedir))
return []
list_cache = os.path.join(
list_cachedir,

View File

@ -48,7 +48,7 @@ def load():
datastore_path = os.path.join(__opts__['cachedir'], 'datastore')
fn_ = salt.utils.fopen(datastore_path, 'rb')
return serial.load(fn_)
except (IOError, OSError):
except (IOError, OSError, NameError):
return {}
@ -76,7 +76,7 @@ def dump(new_data):
return True
except (IOError, OSError):
except (IOError, OSError, NameError):
return False

View File

@ -538,7 +538,11 @@ def find_cached_job(jid):
buf = fp_.read()
fp_.close()
if buf:
data = serial.loads(buf)
try:
data = serial.loads(buf)
except NameError:
# msgpack error in salt-ssh
return
else:
return
if not isinstance(data, dict):

View File

@ -275,7 +275,8 @@ def _load_accumulators():
with open(path, 'rb') as f:
loaded = serial.load(f)
return loaded if loaded else ret
except IOError:
except (IOError, NameError):
# NameError is a msgpack error from salt-ssh
return ret
loaded = _deserialize(_get_accumulator_filepath())
@ -289,8 +290,12 @@ def _persist_accummulators(accumulators, accumulators_deps):
'accumulators_deps': accumulators_deps}
serial = salt.payload.Serial(__opts__)
with open(_get_accumulator_filepath(), 'w+b') as f:
serial.dump(accumm_data, f)
try:
with open(_get_accumulator_filepath(), 'w+b') as f:
serial.dump(accumm_data, f)
except NameError:
# msgpack error from salt-ssh
pass
def _check_user(user, group):

View File

@ -270,9 +270,13 @@ class ZeroMQChannel(Channel):
def crypted_transfer_decode_dictentry(self, load, dictkey=None, tries=3, timeout=60):
ret = self.sreq.send('aes', self.auth.crypticle.dumps(load), tries, timeout)
key = self.auth.get_keys()
aes = key.private_decrypt(ret['key'], 4)
pcrypt = salt.crypt.Crypticle(self.opts, aes)
return pcrypt.loads(ret[dictkey])
try:
aes = key.private_decrypt(ret['key'], 4)
except (TypeError, KeyError):
return None
else:
pcrypt = salt.crypt.Crypticle(self.opts, aes)
return pcrypt.loads(ret[dictkey])
def _crypted_transfer(self, load, tries=3, timeout=60):
'''