mirror of
https://github.com/valitydev/salt.git
synced 2024-11-07 17:09:03 +00:00
268 lines
8.7 KiB
ReStructuredText
268 lines
8.7 KiB
ReStructuredText
.. _troubleshooting:
|
|
|
|
===============
|
|
Troubleshooting
|
|
===============
|
|
|
|
The intent of the troubleshooting section is to introduce solutions to a
|
|
number of common issues encountered by users and the tools that are available
|
|
to aid in developing States and Salt code.
|
|
|
|
Troubleshooting the Salt Master
|
|
===============================
|
|
|
|
If your Salt master is having issues such as minions not returning data, slow
|
|
execution times, or a variety of other issues, the following links contain
|
|
details on troubleshooting the most common issues encountered:
|
|
|
|
.. toctree::
|
|
:maxdepth: 2
|
|
|
|
master
|
|
|
|
Troubleshooting the Salt Minion
|
|
===============================
|
|
|
|
In the event that your Salt minion is having issues, a variety of solutions
|
|
and suggestions are available. Please refer to the following links for more information:
|
|
|
|
.. toctree::
|
|
:maxdepth: 2
|
|
|
|
minion
|
|
|
|
Running in the Foreground
|
|
=========================
|
|
|
|
A great deal of information is available via the debug logging system, if you
|
|
are having issues with minions connecting or not starting run the minion and/or
|
|
master in the foreground:
|
|
|
|
.. code-block:: bash
|
|
|
|
salt-master -l debug
|
|
salt-minion -l debug
|
|
|
|
Anyone wanting to run Salt daemons via a process supervisor such as `monit`_,
|
|
`runit`_, or `supervisord`_, should omit the ``-d`` argument to the daemons and
|
|
run them in the foreground.
|
|
|
|
.. _`monit`: http://mmonit.com/monit/
|
|
.. _`runit`: http://smarden.org/runit/
|
|
.. _`supervisord`: http://supervisord.org/
|
|
|
|
What Ports do the Master and Minion Need Open?
|
|
==============================================
|
|
|
|
No ports need to be opened up on each minion. For the master, TCP ports 4505
|
|
and 4506 need to be open. If you've put both your Salt master and minion in
|
|
debug mode and don't see an acknowledgment that your minion has connected,
|
|
it could very well be a firewall.
|
|
|
|
You can check port connectivity from the minion with the nc command:
|
|
|
|
.. code-block:: bash
|
|
|
|
nc -v -z salt.master.ip 4505
|
|
nc -v -z salt.master.ip 4506
|
|
|
|
There is also a :ref:`firewall configuration<firewall>`
|
|
document that might help as well.
|
|
|
|
If you've enabled the right TCP ports on your operating system or Linux
|
|
distribution's firewall and still aren't seeing connections, check that no
|
|
additional access control system such as `SELinux`_ or `AppArmor`_ is blocking
|
|
Salt.
|
|
|
|
.. _`SELinux`: https://en.wikipedia.org/wiki/Security-Enhanced_Linux
|
|
.. _`AppArmor`: http://wiki.apparmor.net/index.php/Main_Page
|
|
|
|
|
|
.. _using-salt-call:
|
|
|
|
Using salt-call
|
|
===============
|
|
|
|
The ``salt-call`` command was originally developed for aiding in the development
|
|
of new Salt modules. Since then, many applications have been developed for
|
|
running any Salt module locally on a minion. These range from the original
|
|
intent of salt-call, development assistance, to gathering more verbose output
|
|
from calls like :mod:`state.apply <salt.modules.state.apply_>`.
|
|
|
|
When initially creating your state tree, it is generally recommended to invoke
|
|
:mod:`state.apply <salt.modules.state.apply_>` directly from the minion with
|
|
``salt-call``, rather than remotely from the master. This displays far more
|
|
information about the execution than calling it remotely. For even more
|
|
verbosity, increase the loglevel using the ``-l`` argument:
|
|
|
|
.. code-block:: bash
|
|
|
|
salt-call -l debug state.apply
|
|
|
|
The main difference between using ``salt`` and using ``salt-call`` is that
|
|
``salt-call`` is run from the minion, and it only runs the selected function on
|
|
that minion. By contrast, ``salt`` is run from the master, and requires you to
|
|
specify the minions on which to run the command using salt's :ref:`targeting
|
|
system <targeting>`.
|
|
|
|
Too many open files
|
|
===================
|
|
|
|
The salt-master needs at least 2 sockets per host that connects to it, one for
|
|
the Publisher and one for response port. Thus, large installations may, upon
|
|
scaling up the number of minions accessing a given master, encounter:
|
|
|
|
.. code-block:: bash
|
|
|
|
12:45:29,289 [salt.master ][INFO ] Starting Salt worker process 38
|
|
Too many open files
|
|
sock != -1 (tcp_listener.cpp:335)
|
|
|
|
The solution to this would be to check the number of files allowed to be
|
|
opened by the user running salt-master (root by default):
|
|
|
|
.. code-block:: bash
|
|
|
|
[root@salt-master ~]# ulimit -n
|
|
1024
|
|
|
|
And modify that value to be at least equal to the number of minions x 2.
|
|
This setting can be changed in limits.conf as the nofile value(s),
|
|
and activated upon new a login of the specified user.
|
|
|
|
So, an environment with 1800 minions, would need 1800 x 2 = 3600 as a minimum.
|
|
|
|
|
|
Salt Master Stops Responding
|
|
============================
|
|
|
|
There are known bugs with ZeroMQ versions less than 2.1.11 which can cause the
|
|
Salt master to not respond properly. If you're running a ZeroMQ version greater
|
|
than or equal to 2.1.9, you can work around the bug by setting the sysctls
|
|
``net.core.rmem_max`` and ``net.core.wmem_max`` to 16777216. Next, set the third
|
|
field in ``net.ipv4.tcp_rmem`` and ``net.ipv4.tcp_wmem`` to at least 16777216.
|
|
|
|
You can do it manually with something like:
|
|
|
|
.. code-block:: bash
|
|
|
|
# echo 16777216 > /proc/sys/net/core/rmem_max
|
|
# echo 16777216 > /proc/sys/net/core/wmem_max
|
|
# echo "4096 87380 16777216" > /proc/sys/net/ipv4/tcp_rmem
|
|
# echo "4096 87380 16777216" > /proc/sys/net/ipv4/tcp_wmem
|
|
|
|
Or with the following Salt state:
|
|
|
|
.. code-block:: yaml
|
|
:linenos:
|
|
|
|
net.core.rmem_max:
|
|
sysctl:
|
|
- present
|
|
- value: 16777216
|
|
|
|
net.core.wmem_max:
|
|
sysctl:
|
|
- present
|
|
- value: 16777216
|
|
|
|
net.ipv4.tcp_rmem:
|
|
sysctl:
|
|
- present
|
|
- value: 4096 87380 16777216
|
|
|
|
net.ipv4.tcp_wmem:
|
|
sysctl:
|
|
- present
|
|
- value: 4096 87380 16777216
|
|
|
|
Salt and SELinux
|
|
================
|
|
|
|
Currently there are no SELinux policies for Salt. For the most part Salt runs
|
|
without issue when SELinux is running in Enforcing mode. This is because when
|
|
the minion executes as a daemon the type context is changed to ``initrc_t``.
|
|
The problem with SELinux arises when using salt-call or running the minion in
|
|
the foreground, since the type context stays ``unconfined_t``.
|
|
|
|
This problem is generally manifest in the rpm install scripts when using the
|
|
pkg module. Until a full SELinux Policy is available for Salt the solution
|
|
to this issue is to set the execution context of ``salt-call`` and
|
|
``salt-minion`` to rpm_exec_t:
|
|
|
|
.. code-block:: bash
|
|
|
|
# CentOS 5 and RHEL 5:
|
|
chcon -t system_u:system_r:rpm_exec_t:s0 /usr/bin/salt-minion
|
|
chcon -t system_u:system_r:rpm_exec_t:s0 /usr/bin/salt-call
|
|
|
|
# CentOS 6 and RHEL 6:
|
|
chcon system_u:object_r:rpm_exec_t:s0 /usr/bin/salt-minion
|
|
chcon system_u:object_r:rpm_exec_t:s0 /usr/bin/salt-call
|
|
|
|
This works well, because the ``rpm_exec_t`` context has very broad control over
|
|
other types.
|
|
|
|
Red Hat Enterprise Linux 5
|
|
==========================
|
|
|
|
Salt requires Python 2.6 or 2.7. Red Hat Enterprise Linux 5 and its variants
|
|
come with Python 2.4 installed by default. When installing on RHEL 5 from the
|
|
`EPEL repository`_ this is handled for you. But, if you run Salt from git, be
|
|
advised that its dependencies need to be installed from EPEL and that Salt
|
|
needs to be run with the ``python26`` executable.
|
|
|
|
.. _`EPEL repository`: http://fedoraproject.org/wiki/EPEL
|
|
|
|
Common YAML Gotchas
|
|
===================
|
|
|
|
An extensive list of YAML idiosyncrasies has been compiled:
|
|
|
|
.. toctree::
|
|
:maxdepth: 2
|
|
|
|
yaml_idiosyncrasies
|
|
|
|
Live Python Debug Output
|
|
========================
|
|
|
|
If the minion or master seems to be unresponsive, a SIGUSR1 can be passed to
|
|
the processes to display where in the code they are running. If encountering a
|
|
situation like this, this debug information can be invaluable. First make
|
|
sure the master of minion are running in the foreground:
|
|
|
|
.. code-block:: bash
|
|
|
|
salt-master -l debug
|
|
salt-minion -l debug
|
|
|
|
Then pass the signal to the master or minion when it seems to be unresponsive:
|
|
|
|
.. code-block:: bash
|
|
|
|
killall -SIGUSR1 salt-master
|
|
killall -SIGUSR1 salt-minion
|
|
|
|
Also under BSD and macOS in addition to SIGUSR1 signal, debug subroutine set
|
|
up for SIGINFO which has an advantage of being sent by Ctrl+T shortcut.
|
|
|
|
When filing an issue or sending questions to the mailing list for a problem
|
|
with an unresponsive daemon this information can be invaluable.
|
|
|
|
Salt 0.16.x minions cannot communicate with a 0.17.x master
|
|
===========================================================
|
|
|
|
As of release 0.17.1 you can no longer run different versions of Salt on your
|
|
Master and Minion servers. This is due to a protocol change for security
|
|
purposes. The Salt team will continue to attempt to ensure versions are as
|
|
backwards compatible as possible.
|
|
|
|
|
|
Debugging the Master and Minion
|
|
===============================
|
|
|
|
A list of common :ref:`master<troubleshooting-salt-master>` and
|
|
:ref:`minion<troubleshooting-minion-salt-call>` troubleshooting steps provide a
|
|
starting point for resolving issues you may encounter.
|