mirror of
https://github.com/valitydev/salt.git
synced 2024-11-07 17:09:03 +00:00
Merge pull request #9503 from pakdel/minionfs
MinionFS fileserver backend
This commit is contained in:
Thomas S Hatch
commit
aacaf6c24c
@ -14,3 +14,4 @@ Full list of builtin fileserver modules
|
||||
hgfs
|
||||
roots
|
||||
s3fs
|
||||
minionfs
|
||||
|
||||
6
doc/ref/file_server/all/salt.fileserver.minionfs.rst
Normal file
6
doc/ref/file_server/all/salt.fileserver.minionfs.rst
Normal file
@ -0,0 +1,6 @@
|
||||
=====================
|
||||
salt.fileserver.minionfs
|
||||
=====================
|
||||
|
||||
.. automodule:: salt.fileserver.minionfs
|
||||
:members:
|
||||
132
doc/topics/tutorials/minionfs.rst
Normal file
132
doc/topics/tutorials/minionfs.rst
Normal file
@ -0,0 +1,132 @@
|
||||
.. _tutorial-minionfs:
|
||||
|
||||
============================
|
||||
MinionFS Backend Walkthrough
|
||||
============================
|
||||
|
||||
Sometimes, you might need to propagate files that are generated on a minion. Salt already has a feature to send files from a minion to the
|
||||
master:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
salt 'minion-id' cp.push /path/to/the/file
|
||||
|
||||
This command will store the file, including its full path, under :conf_master:`cachedir` ``/master/minions/minion-id/files``. With the default
|
||||
:conf_master:`cachedir` the example file above would be stored as `/var/cache/salt/master/minions/minion-id/files/path/to/the/file`.
|
||||
|
||||
.. note::
|
||||
|
||||
This walkthrough assumes basic knowledge of Salt and :salt.modules.cp:`push`. To get up to speed, check
|
||||
out the :doc:`walkthrough </topics/tutorials/walkthrough>`.
|
||||
|
||||
Since it is not a good idea to expose the whole :conf_master:`cachedir`, MinionFS
|
||||
should be used to send these files to other minions.
|
||||
|
||||
Simple Configuration
|
||||
====================
|
||||
|
||||
To use the minionfs backend only two configuration changes are required on the
|
||||
master. The :conf_master:`fileserver_backend` option needs to contain a value of
|
||||
``minion`` and :conf_master:`file_recv` needs to be set to true:
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
fileserver_backend:
|
||||
- roots
|
||||
- minion
|
||||
|
||||
file_recv: True
|
||||
|
||||
These changes require a restart of the master, then new requests for the ``salt://minion-id/`` protocol will send files that are pushed by ``cp.push``
|
||||
from ``minion-id`` to the master.
|
||||
|
||||
.. note::
|
||||
|
||||
All of the files that are pushed to the master are going to be available to all
|
||||
of the minions. If this is not what you want, please remove ``minion`` from
|
||||
:conf_master:`fileserver_backend`.
|
||||
|
||||
.. note::
|
||||
|
||||
Having directories with the same name as your minions in the root
|
||||
that can be accessed like ``salt://minion-id/`` might cause confusion.
|
||||
|
||||
Commandline Example
|
||||
===================
|
||||
|
||||
Lets assume that we are going to generate SSH keys on a minion called ``minion-source`` and put the public part in ``~/.ssh/authorized_keys`` of root user
|
||||
of a minion called ``minion-destination``.
|
||||
|
||||
First, lets make sure that ``/root/.ssh`` exists and has the right permissions:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
[root@salt-master file]# salt '*' file.mkdir dir_path=/root/.ssh user=root group=root mode=700
|
||||
minion-source:
|
||||
None
|
||||
minion-destination:
|
||||
None
|
||||
|
||||
We create an RSA key pair without a passphrase [*]_:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
[root@salt-master file]# salt 'minion-source' cmd.run 'ssh-keygen -N "" -f /root/.ssh/id_rsa'
|
||||
minion-source:
|
||||
Generating public/private rsa key pair.
|
||||
Your identification has been saved in /root/.ssh/id_rsa.
|
||||
Your public key has been saved in /root/.ssh/id_rsa.pub.
|
||||
The key fingerprint is:
|
||||
9b:cd:1c:b9:c2:93:8e:ad:a3:52:a0:8b:0a:cc:d4:9b root@minion-source
|
||||
The key's randomart image is:
|
||||
+--[ RSA 2048]----+
|
||||
| |
|
||||
| |
|
||||
| |
|
||||
| o . |
|
||||
| o o S o |
|
||||
|= + . B o |
|
||||
|o+ E B = |
|
||||
|+ . .+ o |
|
||||
|o ...ooo |
|
||||
+-----------------+
|
||||
|
||||
and we send the public part to the master to be available to all minions:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
[root@salt-master file]# salt 'minion-source' cp.push /root/.ssh/id_rsa.pub
|
||||
minion-source:
|
||||
True
|
||||
|
||||
now it can be seen by everyone:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
[root@salt-master file]# salt 'minion-destination' cp.list_master_dirs
|
||||
minion-destination:
|
||||
- .
|
||||
- etc
|
||||
- minion-source/root
|
||||
- minion-source/root/.ssh
|
||||
|
||||
Lets copy that as the only authorized key to ``minion-destination``:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
[root@salt-master file]# salt 'minion-destination' cp.get_file salt://minion-source/root/.ssh/id_rsa.pub /root/.ssh/authorized_keys
|
||||
minion-destination:
|
||||
/root/.ssh/authorized_keys
|
||||
|
||||
Or we can use a more elegant and salty way to add an SSH key:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
[root@salt-master file]# salt 'minion-destination' ssh.set_auth_key_from_file user=root source=salt://minion-source/root/.ssh/id_rsa.pub
|
||||
minion-destination:
|
||||
new
|
||||
|
||||
|
||||
|
||||
|
||||
.. [*] Yes, that was the actual key on my server, but the server is already destroyed.
|
||||
251
salt/fileserver/minionfs.py
Normal file
251
salt/fileserver/minionfs.py
Normal file
@ -0,0 +1,251 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
'''
|
||||
The backend for serving files pushed to master by cp.push (file_recv).
|
||||
|
||||
:conf_master:`file_recv` needs to be enabled.
|
||||
'''
|
||||
|
||||
# Import python libs
|
||||
import os
|
||||
import logging
|
||||
|
||||
try:
|
||||
import fcntl
|
||||
HAS_FCNTL = True
|
||||
except ImportError:
|
||||
# fcntl is not available on windows
|
||||
HAS_FCNTL = False
|
||||
|
||||
# Import salt libs
|
||||
import salt.fileserver
|
||||
import salt.utils
|
||||
|
||||
logger = logging.getLogger()
|
||||
|
||||
|
||||
# Define the module's virtual name
|
||||
__virtualname__ = 'minion'
|
||||
|
||||
|
||||
def __virtual__():
|
||||
'''
|
||||
Only load if file_recv is enabled
|
||||
'''
|
||||
if not __virtualname__ in __opts__['fileserver_backend']:
|
||||
return False
|
||||
if not __opts__['file_recv']:
|
||||
return False
|
||||
return __virtualname__
|
||||
|
||||
|
||||
def find_file(path, env='base', **kwargs):
|
||||
'''
|
||||
Search the environment for the relative path
|
||||
'''
|
||||
# AP logger.debug('minionfs is asked for {0}'.format(path))
|
||||
fnd = {'path': '', 'rel': ''}
|
||||
if os.path.isabs(path):
|
||||
return fnd
|
||||
if env not in envs():
|
||||
return fnd
|
||||
if path[-7:] == 'top.sls':
|
||||
logger.debug('minionfs will NOT serve top.sls '
|
||||
'for security reasons: {0}'.format(path))
|
||||
return fnd
|
||||
minion, pushed_file = path.split(os.sep,1)
|
||||
full = os.path.join(__opts__['cachedir'], 'minions',
|
||||
minion, 'files', pushed_file)
|
||||
if os.path.isfile(full) and not salt.fileserver.is_file_ignored(__opts__, full):
|
||||
fnd['path'] = full
|
||||
fnd['rel'] = path
|
||||
return fnd
|
||||
# AP logger.debug('minionfs: full path for {0} is {1}'.format(path, full))
|
||||
return fnd
|
||||
|
||||
|
||||
def envs():
|
||||
'''
|
||||
Return "base" as the file server environment, because there is only one set
|
||||
of minions.
|
||||
'''
|
||||
return ['base']
|
||||
|
||||
|
||||
def serve_file(load, fnd):
|
||||
'''
|
||||
Return a chunk from a file based on the data received
|
||||
|
||||
CLI Example:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ salt 'source-minion' cp.push /path/to/the/file # Push the file to the master
|
||||
$ salt 'destination-minion' cp.get_file salt://source-minion/path/to/the/file /destination/file
|
||||
'''
|
||||
ret = {'data': '', 'dest': ''}
|
||||
if not fnd['path']:
|
||||
return ret
|
||||
ret['dest'] = fnd['rel']
|
||||
gzip = load.get('gzip', None)
|
||||
|
||||
# AP
|
||||
# May I sleep here to slow down serving of big files?
|
||||
# How many threads are serving files?
|
||||
with salt.utils.fopen(fnd['path'], 'rb') as fp_:
|
||||
fp_.seek(load['loc'])
|
||||
data = fp_.read(__opts__['file_buffer_size'])
|
||||
if gzip and data:
|
||||
data = salt.utils.gzip_util.compress(data, gzip)
|
||||
ret['gzip'] = gzip
|
||||
ret['data'] = data
|
||||
return ret
|
||||
|
||||
|
||||
def update():
|
||||
'''
|
||||
When we are asked to update (regular interval) lets reap the cache
|
||||
'''
|
||||
# AP logger.debug("minionfs: updating {0}".format(
|
||||
# AP os.path.join(__opts__['cachedir'], 'minionfs/hash')))
|
||||
try:
|
||||
salt.fileserver.reap_fileserver_cache_dir(
|
||||
os.path.join(__opts__['cachedir'], 'minionfs/hash'),
|
||||
find_file)
|
||||
except os.error:
|
||||
# Hash file won't exist if no files have yet been served up
|
||||
pass
|
||||
|
||||
|
||||
def file_hash(load, fnd):
|
||||
'''
|
||||
Return a file hash, the hash type is set in the master config file
|
||||
'''
|
||||
path = fnd['path']
|
||||
ret = {}
|
||||
|
||||
# if the file doesn't exist, we can't get a hash
|
||||
if not path or not os.path.isfile(path):
|
||||
return ret
|
||||
|
||||
# set the hash_type as it is determined by config-- so mechanism won't change that
|
||||
ret['hash_type'] = __opts__['hash_type']
|
||||
|
||||
# check if the hash is cached
|
||||
# cache file's contents should be "hash:mtime"
|
||||
cache_path = os.path.join(__opts__['cachedir'],
|
||||
'minionfs/hash',
|
||||
load['saltenv'],
|
||||
'{0}.hash.{1}'.format(
|
||||
fnd['rel'], __opts__['hash_type'])
|
||||
)
|
||||
# if we have a cache, serve that if the mtime hasn't changed
|
||||
if os.path.exists(cache_path):
|
||||
try:
|
||||
with salt.utils.fopen(cache_path, 'rb') as fp_:
|
||||
try:
|
||||
hsum, mtime = fp_.read().split(':')
|
||||
except ValueError:
|
||||
log.debug('Fileserver attempted to read incomplete cache file. Retrying.')
|
||||
file_hash(load, fnd)
|
||||
return(ret)
|
||||
if os.path.getmtime(path) == mtime:
|
||||
# check if mtime changed
|
||||
ret['hsum'] = hsum
|
||||
return ret
|
||||
except os.error: # Can't use Python select() because we need Windows support
|
||||
log.debug("Fileserver encountered lock when reading cache file. Retrying.")
|
||||
file_hash(load, fnd)
|
||||
return(ret)
|
||||
|
||||
# if we don't have a cache entry-- lets make one
|
||||
ret['hsum'] = salt.utils.get_hash(path, __opts__['hash_type'])
|
||||
cache_dir = os.path.dirname(cache_path)
|
||||
# make cache directory if it doesn't exist
|
||||
if not os.path.exists(cache_dir):
|
||||
os.makedirs(cache_dir)
|
||||
# save the cache object "hash:mtime"
|
||||
if HAS_FCNTL:
|
||||
with salt.utils.flopen(cache_path, 'w') as fp_:
|
||||
fp_.write('{0}:{1}'.format(ret['hsum'], os.path.getmtime(path)))
|
||||
fcntl.flock(fp_.fileno(), fcntl.LOCK_UN)
|
||||
return ret
|
||||
else:
|
||||
with salt.utils.fopen(cache_path, 'w') as fp_:
|
||||
fp_.write('{0}:{1}'.format(ret['hsum'], os.path.getmtime(path)))
|
||||
return ret
|
||||
|
||||
|
||||
def file_list(load):
|
||||
'''
|
||||
Return a list of all files on the file server in a specified environment
|
||||
'''
|
||||
# AP logger.debug('minionfs is asked for file_list of {0}'.format(os.path.join(__opts__['cachedir'], 'minions')))
|
||||
ret = []
|
||||
prefix = load.get('prefix', '').strip('/')
|
||||
minions_cache_dir = os.path.join(__opts__['cachedir'], 'minions')
|
||||
for minion_dir in os.listdir(minions_cache_dir):
|
||||
minion_files_dir = os.path.join(minions_cache_dir, minion_dir, 'files')
|
||||
if not os.path.isdir(minion_files_dir):
|
||||
logger.debug('minionfs: could not find files directory under {0}!'
|
||||
.format(os.path.join(minions_cache_dir, minion_dir))
|
||||
)
|
||||
continue
|
||||
# Always ignore links for security reasons
|
||||
for root, dirs, files in os.walk(
|
||||
os.path.join(minion_files_dir,
|
||||
prefix
|
||||
), followlinks=False):
|
||||
for fname in files:
|
||||
if os.path.islink(os.path.join(root, fname)):
|
||||
continue
|
||||
rel_fn = os.path.join(minion_dir,
|
||||
os.path.relpath(os.path.join(root, fname),
|
||||
minion_files_dir
|
||||
)
|
||||
)
|
||||
if not salt.fileserver.is_file_ignored(__opts__, rel_fn):
|
||||
ret.append(rel_fn)
|
||||
# AP logger.debug('minionfs: file_list is returning {0}'.format(ret))
|
||||
return ret
|
||||
|
||||
|
||||
# There should be no emptydirs
|
||||
#def file_list_emptydirs(load):
|
||||
|
||||
|
||||
def dir_list(load):
|
||||
'''
|
||||
Return a list of all directories on the master
|
||||
|
||||
CLI Example:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
$ salt 'source-minion' cp.push /absolute/path/file # Push the file to the master
|
||||
$ salt 'destination-minion' cp.list_master_dirs
|
||||
destination-minion:
|
||||
- .
|
||||
- source-minion/absolute
|
||||
- source-minion/absolute/path
|
||||
'''
|
||||
ret = []
|
||||
prefix = load.get('prefix', '').strip('/')
|
||||
minions_cache_dir = os.path.join(__opts__['cachedir'], 'minions')
|
||||
for minion_dir in os.listdir(minions_cache_dir):
|
||||
minion_files_dir = os.path.join(minions_cache_dir, minion_dir, 'files')
|
||||
if not os.path.isdir(minion_files_dir):
|
||||
logger.debug('minionfs: could not find files directory under {0}!'
|
||||
.format(os.path.join(minions_cache_dir, minion_dir))
|
||||
)
|
||||
continue
|
||||
# Always ignore links for security reasons
|
||||
for root, dirs, files in os.walk(
|
||||
os.path.join(minion_files_dir,
|
||||
prefix
|
||||
), followlinks=False):
|
||||
rel_fn = os.path.join(minion_dir,
|
||||
os.path.relpath(root, minion_files_dir)
|
||||
)
|
||||
ret.append(rel_fn)
|
||||
# AP logger.debug('minionfs: dir_list is returning {0}'.format(ret))
|
||||
return ret
|
||||
Loading…
Reference in New Issue
Block a user