salt/doc/topics/jobs/job_cache.rst

73 lines
2.7 KiB
ReStructuredText
Raw Normal View History

[develop] Merge forward from 2016.3 to develop (#33408) * Fix master hanging after a request from minion with removed key. (#33333) * ZMQ monitor for MWorker connections. * Reauth minion if the key was removed on the master side. * Allow concurrency mode in state runs if using sudo (#33325) Closes #30130 * Disambiguate non-exact matches when checking if sysv service is enabled (#33324) Fixes #33323 * remove redundant, incorrect sudo_runas config documentation (#33318) * remove sudo_runas documentation `sudo_runas` was renamed to `sudo_user` and the documentation was not updated accordingly. * conf/minion: update sudo_user description The description from sudo_runas was better. * import ps from psutil_compat in beacons (#33334) * beacons.network_info: import gate psutil * beacons.ps: import gate psutil * Add docs for mine_functions config var (#33326) * Add docs for mine_functions config var * Note that mine_enabled essentially just doesn't add the mine update function to the scheduler. * Bp 28467 calm mine (#33327) * make minion mine update behavior more configurable * Add docs for mine_functions config var * Remove config dup from mine config options Refs #28467 * 2015.8 does not have _DFLT_MULTIPROCESSING_MODE * This won't be in until 2015.8.10. * Fix network.managed for windows (#33312) * Fix some link errors in the test writing tutorial (#33347) * Describes parameters in register_instances function (#33339) * Fix UnboundLocalError in git.latest (#33340) Resolves #32260. * Expanded documentation for boto_elb state and module (#33341) * Describes what happens when the CNAME parameter is given. * Describes what the recognized attributes are for for ELBs. * Properly detect newer Linux Mint distros (#33359) * Properly detect newer Linux Mint distros LMDE 2 and Linux Mint 17.3 changed the DISTRIB_ID in /etc/lsb-release to ``LinuxMint``, breaking OS detection for these distros. This commit fixes that by adding an entry to the OS_NAME_MAP in the core grains. * Remove LinuxMint os_family from aptpkg.py It is no longer necessary as the distro is now detected properly, which will lead to an os_family of Debian. * Update job_cache and keep_jobs docs to be more specific to their behavior (#33328) * Update job_cache and keep_jobs docs to be more specific to their behavior Also fixed a bug discovered when investigating job_cache/keep_jobs functionality where the jid directory and files were removed by the cache cleaner, but not the original jid clash detection directory created in /var/cache/salt/master/jobs/. Fixes #29286 * Add testcase for the changes in the local_cache.clean_old_jobs func * Mark tests as destructive * Put destructive test decorator in correct location * Remove mentions of windows not supporting pkgs param (#33361) Fixes #33313 * Updates docs version to 2015.8.9 Adds note regarding the os grain on Mint Linux Adds an FAQ regarding grains that change due to upstream changes * revved 2015.8 branch to .9 in version selector * Add initscripts, SystemD service units and environment files for Debian (#32857) * Add note to docs about api settings for Hipchat API v2 (#33365) Fixes #27779 * Add win_pkg to list of modules that support "version" in pkg.installed (#33362) Fixes #32913 * Add note about name parameter in git_pillar docs (#33369) Fixes #27737 * Better YAML syntax error handling (#33375) Closes #26574 * Improve doc clarity for disable_modules documentation (#33379) * Improve doc clarity for disable_modules documentation * Additional clarification on blacklisted name * maintain the fallabck because I am totally sick of this crap * blast, put the try/except int he right place * restore whitespace * Fix traceback in logging for config validation (#33386) * 2015.8.10 release notes * Sync pillarstack to latest upstream version (#33391) * Don't lay down all available opts (#33385) * Don't lay down all available opts * We need at least one opt in there * Condense defaults * Put the default hash type back
2016-05-20 20:48:40 +00:00
.. _managing_the_job_cache:
======================
2014-09-03 22:36:00 +00:00
Managing the Job Cache
======================
2014-09-03 21:36:48 +00:00
The Salt Master maintains a job cache of all job executions which can be
queried via the jobs runner. This job cache is called the Default Job Cache.
2014-09-03 21:36:48 +00:00
2015-03-31 19:59:11 +00:00
.. _default_job_cache:
2014-09-03 21:36:48 +00:00
Default Job Cache
=================
A number of options are available when configuring the job cache. The default
caching system uses local storage on the Salt Master and can be found in the
job cache directory (on Linux systems this is typically
``/var/cache/salt/master/jobs``). The default caching system is suitable for most
2014-09-03 22:36:00 +00:00
deployments as it does not typically require any further configuration or
2014-09-03 21:36:48 +00:00
management.
The default job cache is a temporary cache and jobs will be stored for 24
hours. If the default cache needs to store jobs for a different period the
2014-09-03 22:36:00 +00:00
time can be easily adjusted by changing the `keep_jobs` parameter in the
2014-09-03 21:36:48 +00:00
Salt Master configuration file. The value passed in is measured via hours:
.. code-block:: yaml
keep_jobs: 24
[develop] Merge forward from 2016.3 to develop (#33408) * Fix master hanging after a request from minion with removed key. (#33333) * ZMQ monitor for MWorker connections. * Reauth minion if the key was removed on the master side. * Allow concurrency mode in state runs if using sudo (#33325) Closes #30130 * Disambiguate non-exact matches when checking if sysv service is enabled (#33324) Fixes #33323 * remove redundant, incorrect sudo_runas config documentation (#33318) * remove sudo_runas documentation `sudo_runas` was renamed to `sudo_user` and the documentation was not updated accordingly. * conf/minion: update sudo_user description The description from sudo_runas was better. * import ps from psutil_compat in beacons (#33334) * beacons.network_info: import gate psutil * beacons.ps: import gate psutil * Add docs for mine_functions config var (#33326) * Add docs for mine_functions config var * Note that mine_enabled essentially just doesn't add the mine update function to the scheduler. * Bp 28467 calm mine (#33327) * make minion mine update behavior more configurable * Add docs for mine_functions config var * Remove config dup from mine config options Refs #28467 * 2015.8 does not have _DFLT_MULTIPROCESSING_MODE * This won't be in until 2015.8.10. * Fix network.managed for windows (#33312) * Fix some link errors in the test writing tutorial (#33347) * Describes parameters in register_instances function (#33339) * Fix UnboundLocalError in git.latest (#33340) Resolves #32260. * Expanded documentation for boto_elb state and module (#33341) * Describes what happens when the CNAME parameter is given. * Describes what the recognized attributes are for for ELBs. * Properly detect newer Linux Mint distros (#33359) * Properly detect newer Linux Mint distros LMDE 2 and Linux Mint 17.3 changed the DISTRIB_ID in /etc/lsb-release to ``LinuxMint``, breaking OS detection for these distros. This commit fixes that by adding an entry to the OS_NAME_MAP in the core grains. * Remove LinuxMint os_family from aptpkg.py It is no longer necessary as the distro is now detected properly, which will lead to an os_family of Debian. * Update job_cache and keep_jobs docs to be more specific to their behavior (#33328) * Update job_cache and keep_jobs docs to be more specific to their behavior Also fixed a bug discovered when investigating job_cache/keep_jobs functionality where the jid directory and files were removed by the cache cleaner, but not the original jid clash detection directory created in /var/cache/salt/master/jobs/. Fixes #29286 * Add testcase for the changes in the local_cache.clean_old_jobs func * Mark tests as destructive * Put destructive test decorator in correct location * Remove mentions of windows not supporting pkgs param (#33361) Fixes #33313 * Updates docs version to 2015.8.9 Adds note regarding the os grain on Mint Linux Adds an FAQ regarding grains that change due to upstream changes * revved 2015.8 branch to .9 in version selector * Add initscripts, SystemD service units and environment files for Debian (#32857) * Add note to docs about api settings for Hipchat API v2 (#33365) Fixes #27779 * Add win_pkg to list of modules that support "version" in pkg.installed (#33362) Fixes #32913 * Add note about name parameter in git_pillar docs (#33369) Fixes #27737 * Better YAML syntax error handling (#33375) Closes #26574 * Improve doc clarity for disable_modules documentation (#33379) * Improve doc clarity for disable_modules documentation * Additional clarification on blacklisted name * maintain the fallabck because I am totally sick of this crap * blast, put the try/except int he right place * restore whitespace * Fix traceback in logging for config validation (#33386) * 2015.8.10 release notes * Sync pillarstack to latest upstream version (#33391) * Don't lay down all available opts (#33385) * Don't lay down all available opts * We need at least one opt in there * Condense defaults * Put the default hash type back
2016-05-20 20:48:40 +00:00
Reducing the Size of the Default Job Cache
------------------------------------------
The Default Job Cache can sometimes be a burden on larger deployments (over 5000
minions). Disabling the job cache will make previously executed jobs unavailable
to the jobs system and is not generally recommended. Normally it is wise to make
sure the master has access to a faster IO system or a tmpfs is mounted to the
jobs dir.
However, you can disable the :conf_master:`job_cache` by setting it to ``False``
in the Salt Master configuration file. Setting this value to ``False`` means that
the Salt Master will no longer cache minion returns, but a JID directory and ``jid``
file for each job will still be created. This JID directory is necessary for
checking for and preventing JID collisions.
The default location for the job cache is in the ``/var/cache/salt/master/jobs/``
directory.
Setting the :conf_master:`job_cache`` to ``False`` in addition to setting
the :conf_master:`keep_jobs` option to a smaller value, such as ``1``, in the Salt
Master configuration file will reduce the size of the Default Job Cache, and thus
the burden on the Salt Master.
.. note::
Changing the ``keep_jobs`` option sets the number of hours to keep old job
information and defaults to ``24`` hours. Do not set this value to ``0`` when
trying to make the cache cleaner run more frequently, as this means the cache
cleaner will never run.
Additional Job Cache Options
============================
2014-09-03 21:36:48 +00:00
Many deployments may wish to use an external database to maintain a long term
2014-09-03 22:36:00 +00:00
register of executed jobs. Salt comes with two main mechanisms to do this, the
master job cache and the external job cache.
2014-09-03 21:36:48 +00:00
See :ref:`Storing Job Results in an External System <external-master-cache>`.
2015-03-31 19:59:11 +00:00
2014-09-03 21:36:48 +00:00