diff --git a/doc/topics/releases/2018.3.0.rst b/doc/topics/releases/2018.3.0.rst index 27dfdbd4f1..a1befac851 100644 --- a/doc/topics/releases/2018.3.0.rst +++ b/doc/topics/releases/2018.3.0.rst @@ -45,8 +45,8 @@ Custom subnets can now be configured. Both IPv4 and mixed IPv4/IPv6 networks are supported. See :ref:`here ` for more information. -Network Configuration in :py:func:`docker_container.running` States -******************************************************************* +Network Configuration in :py:func:`docker_container.running ` States +********************************************************************************************************** A long-requested feature has finally been added! It is now possible to configure static IPv4/IPv6 addresses, as well as links and labels. See @@ -54,9 +54,10 @@ configure static IPv4/IPv6 addresses, as well as links and labels. See information. .. note:: - While the ``containers`` argument to :py:func:`docker_network.present` - will continue to be supported, it will no longer be the recommended way of - ensuring that a container is attached to a network. + While the ``containers`` argument to :py:func:`docker_network.present + ` will continue to be supported, it + will no longer be the recommended way of ensuring that a container is + attached to a network. Improved Handling of Images from Custom Registries ************************************************** diff --git a/doc/topics/releases/2018.3.2.rst b/doc/topics/releases/2018.3.2.rst index 1a10c32ad9..b2457d1b22 100644 --- a/doc/topics/releases/2018.3.2.rst +++ b/doc/topics/releases/2018.3.2.rst @@ -5,12 +5,69 @@ In Progress: Salt 2018.3.2 Release Notes Version 2018.3.2 is an **unreleased** bugfix release for :ref:`2018.3.0 `. This release is still in progress and has not been released yet. -Changes to win_timezone -======================= +The ``2018.3.2`` release contains only a small number of fixes, detailed below. -Improves timezone detection by using the pytz module. +Mainly, this release fixes Issue `#48038`_, which is a critical bug that occurs +in a multi-syndic setup where the same job is run multiple times on a minion. -``timezone.get_offset`` and ``timezone.get_zonecode`` now work properly. +Statistics +========== -Adds ``timezone.list`` to list supported timezones in either Windows or Unix -format. +- Total Merges: **3** +- Total Issue References: **1** +- Total PR References: **6** + +- Contributors: **3** (`cro`_, `garethgreenaway`_, `rallytime`_) + + +Changelog for v2018.3.1..v2018.3.2 +================================== + +*Generated at: 2018-06-14 13:24:42 UTC* + +* **PR** `#48100`_: (`rallytime`_) Back-port `#48014`_ to 2018.3.2 + @ *2018-06-14 12:54:52 UTC* + + * **PR** `#48014`_: (`cro`_) Find job pause (refs: `#48100`_) + + * 36b99ae80a Merge pull request `#48100`_ from rallytime/bp-48014 + + * 77feccc5c4 Lint: Add blank line + + * 159b052962 One more case where returner doesn't respond + + * 91b45b4cc4 Catch two cases when a returner is not able to be contacted--these would throw a stacktrace. + +* **PR** `#48099`_: (`rallytime`_) Back-port `#47915`_ to 2018.3.2 + @ *2018-06-14 12:54:23 UTC* + + * **PR** `#47915`_: (`garethgreenaway`_) [2018.3] state runner pause resume kill (refs: `#48099`_) + + * 40c1bfdec9 Merge pull request `#48099`_ from rallytime/bp-47915 + + * 3556850058 fixing typo in alias_function call. + + * 4b0ff496fa Some fixes to the set_pause and rm_pause function in the state runner, renaming to in line with the functions in the state module. Including aliases to previous names for back-ward compatibility. Including a soft_kill function to kill running orchestration states. A new test to test soft_kill functionality. + +* **ISSUE** `#48038`_: (`austinpapp`_) jobs are not dedup'ing minion side (refs: `#48075`_) + +* **PR** `#48097`_: (`rallytime`_) Back-port `#48075`_ to 2018.3.2 + @ *2018-06-14 12:52:44 UTC* + + * **PR** `#48075`_: (`garethgreenaway`_) [2017.7] Ensure that the shared list of jids is passed (refs: `#48097`_) + + * 074a97dcfa Merge pull request `#48097`_ from rallytime/bp-48075 + + * e4c719b55f Ensure that the shared list of jids is passed when creating the Minion. Fixes an issue when minions are pointed at multiple syndics. + +.. _`#47915`: https://github.com/saltstack/salt/pull/47915 +.. _`#48014`: https://github.com/saltstack/salt/pull/48014 +.. _`#48038`: https://github.com/saltstack/salt/issues/48038 +.. _`#48075`: https://github.com/saltstack/salt/pull/48075 +.. _`#48097`: https://github.com/saltstack/salt/pull/48097 +.. _`#48099`: https://github.com/saltstack/salt/pull/48099 +.. _`#48100`: https://github.com/saltstack/salt/pull/48100 +.. _`austinpapp`: https://github.com/austinpapp +.. _`cro`: https://github.com/cro +.. _`garethgreenaway`: https://github.com/garethgreenaway +.. _`rallytime`: https://github.com/rallytime diff --git a/doc/topics/releases/2018.3.3.rst b/doc/topics/releases/2018.3.3.rst new file mode 100644 index 0000000000..3f448947bc --- /dev/null +++ b/doc/topics/releases/2018.3.3.rst @@ -0,0 +1,16 @@ +======================================== +In Progress: Salt 2018.3.3 Release Notes +======================================== + +Version 2018.3.3 is an **unreleased** bugfix release for :ref:`2018.3.0 `. +This release is still in progress and has not been released yet. + +Changes to win_timezone +======================= + +Improves timezone detection by using the pytz module. + +``timezone.get_offset`` and ``timezone.get_zonecode`` now work properly. + +Adds ``timezone.list`` to list supported timezones in either Windows or Unix +format. diff --git a/requirements/dev.txt b/requirements/dev.txt index 57af6f81e7..f362e35b2f 100644 --- a/requirements/dev.txt +++ b/requirements/dev.txt @@ -1,10 +1,6 @@ -r base.txt mock>=2.0.0 -apache-libcloud>=0.14.0 -boto>=2.32.1 -boto3>=1.2.1 -moto>=0.3.6 SaltPyLint>=v2017.3.6 pytest>=3.5.0 git+https://github.com/saltstack/pytest-salt.git@master#egg=pytest-salt diff --git a/requirements/tests.txt b/requirements/tests.txt new file mode 100644 index 0000000000..ffd8207ec8 --- /dev/null +++ b/requirements/tests.txt @@ -0,0 +1,36 @@ +-r zeromq.txt +-r dev.txt +-r pytest.txt +apache-libcloud>=1.0.0 +boto>=2.32.1 +boto3>=1.2.1 +moto>=0.3.6 +docker; sys.platform != 'win32' +docker==2.7.0; sys.platform == 'win32' +virtualenv +setuptools>=30 +six>=1.10.0 +timelib +coverage +keyring==5.7.1 +python-gnupg +python-etcd==0.4.2 +GitPython +supervisor; python_version < '3' +kubernetes<4.0 +psutil +pyvmomi +setproctitle +cherrypy; sys.platform != 'win32' and sys.platform != 'darwin' +pyinotify; sys.platform != 'win32' and sys.platform != 'darwin' +PyMySQL; sys.platform != 'win32' and sys.platform != 'darwin' +jsonschema +strict_rfc3339 +rfc3987 +jinja2 +pyOpenSSL +ioflo +dnspython +SaltTesting==2017.6.1 +junos-eznc +jxmlease diff --git a/salt/beacons/wtmp.py b/salt/beacons/wtmp.py index 7a9f9448e1..45215ba0a3 100644 --- a/salt/beacons/wtmp.py +++ b/salt/beacons/wtmp.py @@ -20,7 +20,7 @@ import salt.utils.stringutils import salt.utils.files # Import 3rd-party libs -import salt.ext.six +from salt.ext import six # pylint: disable=import-error from salt.ext.six.moves import map # pylint: enable=import-error @@ -257,7 +257,7 @@ def beacon(config): event = {} for ind, field in enumerate(FIELDS): event[field] = pack[ind] - if isinstance(event[field], salt.ext.six.string_types): + if isinstance(event[field], six.string_types): if isinstance(event[field], bytes): event[field] = salt.utils.stringutils.to_unicode(event[field]) event[field] = event[field].strip('\x00') diff --git a/salt/modules/mac_sysctl.py b/salt/modules/mac_sysctl.py index 030a0db9f5..f0f4447e9b 100644 --- a/salt/modules/mac_sysctl.py +++ b/salt/modules/mac_sysctl.py @@ -179,13 +179,10 @@ def persist(name, value, config='/etc/sysctl.conf', apply_change=False): rest = rest[len(rest_v):] if rest_v == value: return 'Already set' - new_line = '{0}={1}'.format(name, value) - nlines.append(new_line) - nlines.append('\n') + nlines.append('{0}={1}\n'.format(name, value)) edited = True if not edited: - nlines.append('{0}={1}'.format(name, value)) - nlines.append('\n') + nlines.append('{0}={1}\n'.format(name, value)) nlines = [salt.utils.stringutils.to_str(_l) for _l in nlines] with salt.utils.files.fopen(config, 'w+') as ofile: ofile.writelines(nlines) diff --git a/salt/modules/mount.py b/salt/modules/mount.py index e0a36922f9..2c59a1a236 100644 --- a/salt/modules/mount.py +++ b/salt/modules/mount.py @@ -121,18 +121,30 @@ def _active_mounts_aix(ret): ''' for line in __salt__['cmd.run_stdout']('mount -p').split('\n'): comps = re.sub(r"\s+", " ", line).split() - if comps and comps[0] == 'node' or comps[0] == '--------': - continue - if len(comps) < 8: - ret[comps[1]] = {'device': comps[0], - 'fstype': comps[2], - 'opts': _resolve_user_group_names(comps[6].split(','))} - else: - ret[comps[2]] = {'node': comps[0], - 'device': comps[1], - 'fstype': comps[3], - 'opts': _resolve_user_group_names(comps[7].split(','))} - + if comps: + if comps[0] == 'node' or comps[0] == '--------': + continue + comps_len = len(comps) + if line.startswith((' ', '\t')): + curr_opts = _resolve_user_group_names(comps[6].split(',')) if 7 == comps_len else [] + if curr_opts: + ret[comps[1]] = {'device': comps[0], + 'fstype': comps[2], + 'opts': curr_opts} + else: + ret[comps[1]] = {'device': comps[0], + 'fstype': comps[2]} + else: + curr_opts = _resolve_user_group_names(comps[7].split(',')) if 8 == comps_len else [] + if curr_opts: + ret[comps[2]] = {'node': comps[0], + 'device': comps[1], + 'fstype': comps[3], + 'opts': curr_opts} + else: + ret[comps[2]] = {'node': comps[0], + 'device': comps[1], + 'fstype': comps[3]} return ret @@ -228,7 +240,7 @@ def active(extended=False): ret = {} if __grains__['os'] == 'FreeBSD': _active_mounts_freebsd(ret) - elif __grains__['kernel'] == 'AIX': + elif 'AIX' in __grains__['kernel']: _active_mounts_aix(ret) elif __grains__['kernel'] == 'SunOS': _active_mounts_solaris(ret) @@ -1046,7 +1058,7 @@ def mount(name, device, mkmnt=False, fstype='', opts='defaults', user=None, util return False # Darwin doesn't expect defaults when mounting without other options - if 'defaults' in opts and __grains__['os'] in ['MacOS', 'Darwin']: + if 'defaults' in opts and __grains__['os'] in ['MacOS', 'Darwin', 'AIX']: opts = None if isinstance(opts, six.string_types): @@ -1059,7 +1071,9 @@ def mount(name, device, mkmnt=False, fstype='', opts='defaults', user=None, util if opts is not None: lopts = ','.join(opts) args = '-o {0}'.format(lopts) - if fstype: + + # use of fstype on AIX is with /etc/filesystems + if fstype and 'AIX' not in __grains__['os']: args += ' -t {0}'.format(fstype) cmd = 'mount {0} {1} {2} '.format(args, device, name) out = __salt__['cmd.run_all'](cmd, runas=user, python_shell=False) @@ -1086,6 +1100,10 @@ def remount(name, device, mkmnt=False, fstype='', opts='defaults', user=None): if fstype == 'smbfs': force_mount = True + if 'AIX' in __grains__['os']: + if opts == 'defaults': + opts = '' + if isinstance(opts, six.string_types): opts = opts.split(',') mnts = active() @@ -1098,7 +1116,9 @@ def remount(name, device, mkmnt=False, fstype='', opts='defaults', user=None): umount(name, device, user=user) lopts = ','.join(opts) args = '-o {0}'.format(lopts) - if fstype: + + # use of fstype on AIX is with /etc/filesystems + if fstype and 'AIX' not in __grains__['os']: args += ' -t {0}'.format(fstype) if __grains__['os'] not in ['OpenBSD', 'MacOS', 'Darwin'] or force_mount: cmd = 'mount {0} {1} {2} '.format(args, device, name) @@ -1192,6 +1212,17 @@ def swaps(): 'size': int(comps[3]), 'used': (int(comps[3]) - int(comps[4])), 'priority': '-'} + elif 'AIX' in __grains__['kernel']: + for line in __salt__['cmd.run_stdout']('swap -l').splitlines(): + if line.startswith('device'): + continue + comps = line.split() + + # AIX uses MB for units + ret[comps[0]] = {'type': 'device', + 'size': int(comps[3][:-2]) * 1024, + 'used': (int(comps[3][:-2]) - int(comps[4][:-2])) * 1024, + 'priority': '-'} elif __grains__['os'] != 'OpenBSD': with salt.utils.files.fopen('/proc/swaps') as fp_: for line in fp_: @@ -1244,7 +1275,7 @@ def swapon(name, priority=None): return False else: cmd = 'swapon {0}'.format(name) - if priority: + if priority and 'AIX' not in __grains__['kernel']: cmd += ' -p {0}'.format(priority) __salt__['cmd.run'](cmd, python_shell=False) diff --git a/salt/modules/rpm.py b/salt/modules/rpm.py index 3683234f59..15738b2523 100644 --- a/salt/modules/rpm.py +++ b/salt/modules/rpm.py @@ -9,6 +9,7 @@ import logging import os import re import datetime +from salt.utils.versions import LooseVersion # Import Salt libs import salt.utils.decorators.path @@ -609,7 +610,7 @@ def info(*packages, **kwargs): # pick only latest versions # (in case multiple packages installed, e.g. kernel) ret = dict() - for pkg_data in reversed(sorted(_ret, key=lambda x: x['edition'])): + for pkg_data in reversed(sorted(_ret, key=lambda x: LooseVersion(x['edition']))): pkg_name = pkg_data.pop('name') # Filter out GPG public keys packages if pkg_name.startswith('gpg-pubkey'): diff --git a/salt/modules/rpmbuild.py b/salt/modules/rpmbuild.py index 3fd3c94931..832570e6df 100644 --- a/salt/modules/rpmbuild.py +++ b/salt/modules/rpmbuild.py @@ -213,7 +213,7 @@ def make_src_pkg(dest_dir, spec, sources, env=None, template=None, saltenv='base runas The user to run the build process as - .. versionadded:: 2018.3.2 + .. versionadded:: 2018.3.3 .. note:: diff --git a/salt/modules/win_timezone.py b/salt/modules/win_timezone.py index da04122ea0..8fe0ab9ff0 100644 --- a/salt/modules/win_timezone.py +++ b/salt/modules/win_timezone.py @@ -338,7 +338,7 @@ def list(unix_style=True): Return a list of Timezones that this module supports. These can be in either Unix or Windows format. - .. versionadded:: 2018.3.2 + .. versionadded:: 2018.3.3 Args: unix_style (bool): diff --git a/salt/state.py b/salt/state.py index 6808003b97..d6600aeafb 100644 --- a/salt/state.py +++ b/salt/state.py @@ -1866,6 +1866,9 @@ class State(object): '__lowstate__': immutabletypes.freeze(chunks) if chunks else {} } + if '__env__' in low: + inject_globals['__env__'] = six.text_type(low['__env__']) + if self.inject_globals: inject_globals.update(self.inject_globals) @@ -1899,11 +1902,6 @@ class State(object): # allow setting the OS environ also make use of the "env" # keyword argument, which is not a string inject_globals['__env__'] = six.text_type(cdata['kwargs']['env']) - elif '__env__' in low: - # The user is passing an alternative environment using - # __env__ which is also not the appropriate choice, still, - # handle it - inject_globals['__env__'] = six.text_type(low['__env__']) if '__env__' not in inject_globals: # Let's use the default environment diff --git a/salt/states/archive.py b/salt/states/archive.py index 6838b2202d..7cbbe11325 100644 --- a/salt/states/archive.py +++ b/salt/states/archive.py @@ -66,14 +66,28 @@ def _gen_checksum(path): def _checksum_file_path(path): - relpath = '.'.join((os.path.relpath(path, __opts__['cachedir']), 'hash')) - if re.match(r'..[/\\]', relpath): - # path is a local file - relpath = salt.utils.path.join( - 'local', - os.path.splitdrive(path)[-1].lstrip('/\\'), - ) - return salt.utils.path.join(__opts__['cachedir'], 'archive_hash', relpath) + try: + relpath = '.'.join((os.path.relpath(path, __opts__['cachedir']), 'hash')) + if re.match(r'..[/\\]', relpath): + # path is a local file + relpath = salt.utils.path.join( + 'local', + os.path.splitdrive(path)[-1].lstrip('/\\'), + ) + except ValueError as exc: + # The path is on a different drive (Windows) + if six.text_type(exc).startswith('path is on'): + drive, path = os.path.splitdrive(path) + relpath = salt.utils.path.join( + 'local', + drive.rstrip(':'), + path.lstrip('/\\'), + ) + else: + raise + ret = salt.utils.path.join(__opts__['cachedir'], 'archive_hash', relpath) + log.debug('Using checksum file %s for cached archive file %s', ret, path) + return ret def _update_checksum(path): diff --git a/salt/states/file.py b/salt/states/file.py index 1a53e041c3..ff33e3aa78 100644 --- a/salt/states/file.py +++ b/salt/states/file.py @@ -678,13 +678,13 @@ def _error(ret, err_msg): def _check_directory(name, - user, - group, - recurse, - mode, - clean, - require, - exclude_pat, + user=None, + group=None, + recurse=False, + mode=None, + clean=False, + require=False, + exclude_pat=None, max_depth=None, follow_symlinks=False): ''' @@ -769,7 +769,7 @@ def _check_directory(name, def _check_directory_win(name, - win_owner, + win_owner=None, win_perms=None, win_deny_perms=None, win_inheritance=None, @@ -783,9 +783,10 @@ def _check_directory_win(name, changes = {name: {'directory': 'new'}} else: # Check owner - owner = salt.utils.win_dacl.get_owner(name) - if not owner.lower() == win_owner.lower(): - changes['owner'] = win_owner + if win_owner is not None: + owner = salt.utils.win_dacl.get_owner(name) + if not owner.lower() == win_owner.lower(): + changes['owner'] = win_owner # Check perms perms = salt.utils.win_dacl.get_permissions(name) @@ -961,33 +962,46 @@ def _check_touch(name, atime, mtime): def _get_symlink_ownership(path): - return ( - __salt__['file.get_user'](path, follow_symlinks=False), - __salt__['file.get_group'](path, follow_symlinks=False) - ) + if salt.utils.platform.is_windows(): + owner = salt.utils.win_dacl.get_owner(path) + return owner, owner + else: + return ( + __salt__['file.get_user'](path, follow_symlinks=False), + __salt__['file.get_group'](path, follow_symlinks=False) + ) -def _check_symlink_ownership(path, user, group): +def _check_symlink_ownership(path, user, group, win_owner): ''' Check if the symlink ownership matches the specified user and group ''' cur_user, cur_group = _get_symlink_ownership(path) - return (cur_user == user) and (cur_group == group) + if salt.utils.platform.is_windows(): + return win_owner == cur_user + else: + return (cur_user == user) and (cur_group == group) -def _set_symlink_ownership(path, user, group): +def _set_symlink_ownership(path, user, group, win_owner): ''' Set the ownership of a symlink and return a boolean indicating success/failure ''' - try: - __salt__['file.lchown'](path, user, group) - except OSError: - pass - return _check_symlink_ownership(path, user, group) + if salt.utils.platform.is_windows(): + try: + salt.utils.win_dacl.set_owner(path, win_owner) + except CommandExecutionError: + pass + else: + try: + __salt__['file.lchown'](path, user, group) + except OSError: + pass + return _check_symlink_ownership(path, user, group, win_owner) -def _symlink_check(name, target, force, user, group): +def _symlink_check(name, target, force, user, group, win_owner): ''' Check the symlink function ''' @@ -1006,7 +1020,7 @@ def _symlink_check(name, target, force, user, group): else: result = True msg = 'The symlink {0} is present'.format(name) - if not _check_symlink_ownership(name, user, group): + if not _check_symlink_ownership(name, user, group, win_owner): result = None pchanges['ownership'] = '{0}:{1}'.format(*_get_symlink_ownership(name)) msg += ( @@ -1230,6 +1244,57 @@ def _shortcut_check(name, 'should be. Did you mean to use force?'.format(name)), pchanges +def _makedirs(name, + user=None, + group=None, + dir_mode=None, + win_owner=None, + win_perms=None, + win_deny_perms=None, + win_inheritance=None): + ''' + Helper function for creating directories when the ``makedirs`` option is set + to ``True``. Handles Unix and Windows based systems + + .. versionadded:: 2017.7.7 + + Args: + name (str): The directory path to create + user (str): The linux user to own the directory + group (str): The linux group to own the directory + dir_mode (str): The linux mode to apply to the directory + win_owner (str): The Windows user to own the directory + win_perms (dict): A dictionary of grant permissions for Windows + win_deny_perms (dict): A dictionary of deny permissions for Windows + win_inheritance (bool): True to inherit permissions on Windows + + Returns: + bool: True if successful, otherwise False on Windows + str: Error messages on failure on Linux + None: On successful creation on Linux + + Raises: + CommandExecutionError: If the drive is not mounted on Windows + ''' + if salt.utils.platform.is_windows(): + # Make sure the drive is mapped before trying to create the + # path in windows + drive, path = os.path.splitdrive(name) + if not os.path.isdir(drive): + raise CommandExecutionError(drive) + win_owner = win_owner if win_owner else user + return __salt__['file.makedirs'](path=name, + owner=win_owner, + grant_perms=win_perms, + deny_perms=win_deny_perms, + inheritance=win_inheritance) + else: + return __salt__['file.makedirs'](path=name, + user=user, + group=group, + mode=dir_mode) + + def symlink( name, target, @@ -1239,6 +1304,10 @@ def symlink( user=None, group=None, mode=None, + win_owner=None, + win_perms=None, + win_deny_perms=None, + win_inheritance=None, **kwargs): ''' Create a symbolic link (symlink, soft link) @@ -1290,6 +1359,28 @@ def symlink( The default mode for new files and directories corresponds umask of salt process. For existing files and directories it's not enforced. + + win_owner : None + The owner of the symlink and directories if ``makedirs`` is True. If + this is not passed, ``user`` will be used. If ``user`` is not passed, + the account under which Salt is running will be used. + + .. versionadded:: 2017.7.7 + + win_perms : None + A dictionary containing permissions to grant + + .. versionadded:: 2017.7.7 + + win_deny_perms : None + A dictionary containing permissions to deny + + .. versionadded:: 2017.7.7 + + win_inheritance : None + True to inherit permissions from parent, otherwise False + + .. versionadded:: 2017.7.7 ''' name = os.path.expanduser(name) @@ -1318,10 +1409,16 @@ def symlink( if not user: user = 'SYSTEM' + # If win_owner is not passed, use user + if win_owner is None: + win_owner = user if user else None + + # Group isn't relevant to Windows, use win_perms/win_deny_perms if group is not None: log.warning( 'The group argument for {0} has been ignored as this ' - 'is a Windows system.'.format(name) + 'is a Windows system. Please use the `win_*` parameters to set ' + 'permissions in Windows.'.format(name) ) group = user @@ -1331,14 +1428,31 @@ def symlink( ) preflight_errors = [] - uid = __salt__['file.user_to_uid'](user) - gid = __salt__['file.group_to_gid'](group) + if salt.utils.platform.is_windows(): + # Make sure the passed owner exists + if not salt.utils.win_functions.get_sid_from_name(win_owner): + preflight_errors.append('User {0} does not exist'.format(win_owner)) - if uid == '': - preflight_errors.append('User {0} does not exist'.format(user)) + # Make sure users passed in win_perms exist + if win_perms: + for name_check in win_perms: + if not salt.utils.win_functions.get_sid_from_name(name_check): + preflight_errors.append('User {0} does not exist'.format(name_check)) - if gid == '': - preflight_errors.append('Group {0} does not exist'.format(group)) + # Make sure users passed in win_deny_perms exist + if win_deny_perms: + for name_check in win_deny_perms: + if not salt.utils.win_functions.get_sid_from_name(name_check): + preflight_errors.append('User {0} does not exist'.format(name_check)) + else: + uid = __salt__['file.user_to_uid'](user) + gid = __salt__['file.group_to_gid'](group) + + if uid == '': + preflight_errors.append('User {0} does not exist'.format(user)) + + if gid == '': + preflight_errors.append('Group {0} does not exist'.format(group)) if not os.path.isabs(name): preflight_errors.append( @@ -1355,48 +1469,77 @@ def symlink( target, force, user, - group) + group, + win_owner) + + if not os.path.isdir(os.path.dirname(name)): + if makedirs: + if __opts__['test']: + pcomment += '\n{0} will be created'.format(os.path.dirname(name)) + else: + try: + _makedirs(name=name, + user=user, + group=group, + dir_mode=mode, + win_owner=win_owner, + win_perms=win_perms, + win_deny_perms=win_deny_perms, + win_inheritance=win_inheritance) + except CommandExecutionError as exc: + return _error(ret, 'Drive {0} is not mapped'.format(exc.message)) + else: + if __opts__['test']: + pcomment += '\nDirectory {0} for symlink is not present' \ + ''.format(os.path.dirname(name)) + else: + return _error( + ret, + 'Directory {0} for symlink is not present'.format( + os.path.dirname(name) + ) + ) + if __opts__['test']: ret['result'] = presult ret['comment'] = pcomment return ret - if not os.path.isdir(os.path.dirname(name)): - if makedirs: - __salt__['file.makedirs']( - name, - user=user, - group=group, - mode=mode) - else: - return _error( - ret, - 'Directory {0} for symlink is not present'.format( - os.path.dirname(name) - ) - ) - if __salt__['file.is_link'](name): # The link exists, verify that it matches the target if os.path.normpath(__salt__['file.readlink'](name)) != os.path.normpath(target): # The target is wrong, delete the link os.remove(name) else: - if _check_symlink_ownership(name, user, group): + if _check_symlink_ownership(name, user, group, win_owner): # The link looks good! - ret['comment'] = ('Symlink {0} is present and owned by ' - '{1}:{2}'.format(name, user, group)) - else: - if _set_symlink_ownership(name, user, group): - ret['comment'] = ('Set ownership of symlink {0} to ' + if salt.utils.platform.is_windows(): + ret['comment'] = ('Symlink {0} is present and owned by {1}' + ''.format(name, win_owner)) + else: + ret['comment'] = ('Symlink {0} is present and owned by ' '{1}:{2}'.format(name, user, group)) - ret['changes']['ownership'] = '{0}:{1}'.format(user, group) + else: + if _set_symlink_ownership(name, user, group, win_owner): + if salt.utils.platform.is_windows(): + ret['comment'] = ('Set ownership of symlink {0} to ' + '{1}'.format(name, win_owner)) + ret['changes']['ownership'] = win_owner + else: + ret['comment'] = ('Set ownership of symlink {0} to ' + '{1}:{2}'.format(name, user, group)) + ret['changes']['ownership'] = '{0}:{1}'.format(user, + group) else: ret['result'] = False - ret['comment'] += ( - 'Failed to set ownership of symlink {0} to ' - '{1}:{2}'.format(name, user, group) - ) + if salt.utils.platform.is_windows(): + ret['comment'] += ( + 'Failed to set ownership of symlink ' + '{0} to {1}'.format(name, win_owner)) + else: + ret['comment'] += ( + 'Failed to set ownership of symlink {0} to ' + '{1}:{2}'.format(name, user, group)) return ret elif os.path.isfile(name) or os.path.isdir(name): @@ -1460,8 +1603,8 @@ def symlink( '{1}'.format(name, target)) ret['changes']['new'] = name - if not _check_symlink_ownership(name, user, group): - if not _set_symlink_ownership(name, user, group): + if not _check_symlink_ownership(name, user, group, win_owner): + if not _set_symlink_ownership(name, user, group, win_owner): ret['result'] = False ret['comment'] += (', but was unable to set ownership to ' '{0}:{1}'.format(user, group)) @@ -3020,23 +3163,17 @@ def directory(name, # The parent directory does not exist, create them if makedirs: # Everything's good, create the parent Dirs - if salt.utils.platform.is_windows(): - # Make sure the drive is mapped before trying to create the - # path in windows - drive, path = os.path.splitdrive(name) - if not os.path.isdir(drive): - return _error( - ret, 'Drive {0} is not mapped'.format(drive)) - __salt__['file.makedirs']( - path=name, - owner=win_owner, - grant_perms=win_perms, - deny_perms=win_deny_perms, - inheritance=win_inheritance, - reset=win_perms_reset) - else: - __salt__['file.makedirs'](name, user=user, group=group, - mode=dir_mode) + try: + _makedirs(name=name, + user=user, + group=group, + dir_mode=dir_mode, + win_owner=win_owner, + win_perms=win_perms, + win_deny_perms=win_deny_perms, + win_inheritance=win_inheritance) + except CommandExecutionError as exc: + return _error(ret, 'Drive {0} is not mapped'.format(exc.message)) else: return _error( ret, 'No directory to create {0} in'.format(name)) @@ -3231,6 +3368,10 @@ def recurse(name, maxdepth=None, keep_symlinks=False, force_symlinks=False, + win_owner=None, + win_perms=None, + win_deny_perms=None, + win_inheritance=True, **kwargs): ''' Recurse through a subdirectory on the master and copy said subdirectory @@ -3390,6 +3531,28 @@ def recurse(name, If a file or directory is obstructing symlink creation it will be recursively removed so that symlink creation can proceed. This option is usually not needed except in special circumstances. + + win_owner : None + The owner of the symlink and directories if ``makedirs`` is True. If + this is not passed, ``user`` will be used. If ``user`` is not passed, + the account under which Salt is running will be used. + + .. versionadded:: 2017.7.7 + + win_perms : None + A dictionary containing permissions to grant + + .. versionadded:: 2017.7.7 + + win_deny_perms : None + A dictionary containing permissions to deny + + .. versionadded:: 2017.7.7 + + win_inheritance : None + True to inherit permissions from parent, otherwise False + + .. versionadded:: 2017.7.7 ''' if 'env' in kwargs: # "env" is not supported; Use "saltenv". @@ -3488,7 +3651,18 @@ def recurse(name, return _error( ret, 'The path {0} exists and is not a directory'.format(name)) if not __opts__['test']: - __salt__['file.makedirs_perms'](name, user, group, dir_mode) + if salt.utils.platform.is_windows(): + win_owner = win_owner if win_owner else user + __salt__['file.makedirs_perms'](path=name, + owner=win_owner, + grant_perms=win_perms, + deny_perms=win_deny_perms, + inheritance=win_inheritance) + else: + __salt__['file.makedirs_perms'](name=name, + user=user, + group=group, + mode=dir_mode) def add_comment(path, comment): comments = ret['comment'].setdefault(path, []) @@ -4870,10 +5044,16 @@ def append(name, if makedirs is True: dirname = os.path.dirname(name) if not __salt__['file.directory_exists'](dirname): - __salt__['file.makedirs'](name) - check_res, check_msg, ret['pchanges'] = _check_directory( - dirname, None, None, False, None, False, False, None - ) + try: + _makedirs(name=name) + except CommandExecutionError as exc: + return _error(ret, 'Drive {0} is not mapped'.format(exc.message)) + + if salt.utils.platform.is_windows(): + check_res, check_msg, ret['pchanges'] = _check_directory_win(dirname) + else: + check_res, check_msg, ret['pchanges'] = _check_directory(dirname) + if not check_res: return _error(ret, check_msg) @@ -4984,6 +5164,90 @@ def prepend(name, The text will not be prepended again if it already exists in the file. You may specify a single line of text or a list of lines to append. + name + The location of the file to append to. + + text + The text to be appended, which can be a single string or a list + of strings. + + makedirs + If the file is located in a path without a parent directory, + then the state will fail. If makedirs is set to True, then + the parent directories will be created to facilitate the + creation of the named file. Defaults to False. + + source + A single source file to append. This source file can be hosted on either + the salt master server, or on an HTTP or FTP server. Both HTTPS and + HTTP are supported as well as downloading directly from Amazon S3 + compatible URLs with both pre-configured and automatic IAM credentials + (see s3.get state documentation). File retrieval from Openstack Swift + object storage is supported via swift://container/object_path URLs + (see swift.get documentation). + + For files hosted on the salt file server, if the file is located on + the master in the directory named spam, and is called eggs, the source + string is salt://spam/eggs. + + If the file is hosted on an HTTP or FTP server, the source_hash argument + is also required. + + source_hash + This can be one of the following: + 1. a source hash string + 2. the URI of a file that contains source hash strings + + The function accepts the first encountered long unbroken alphanumeric + string of correct length as a valid hash, in order from most secure to + least secure: + + .. code-block:: text + + Type Length + ====== ====== + sha512 128 + sha384 96 + sha256 64 + sha224 56 + sha1 40 + md5 32 + + See the ``source_hash`` parameter description for :mod:`file.managed + ` function for more details and examples. + + template + The named templating engine will be used to render the appended-to file. + Defaults to ``jinja``. The following templates are supported: + + - :mod:`cheetah` + - :mod:`genshi` + - :mod:`jinja` + - :mod:`mako` + - :mod:`py` + - :mod:`wempy` + + sources + A list of source files to append. If the files are hosted on an HTTP or + FTP server, the source_hashes argument is also required. + + source_hashes + A list of source_hashes corresponding to the sources list specified in + the sources argument. + + defaults + Default context passed to the template. + + context + Overrides default context variables passed to the template. + + ignore_whitespace + .. versionadded:: 2015.8.4 + + Spaces and Tabs in text are ignored by default, when searching for the + appending content, one space or multiple tabs are the same for salt. + Set this option to ``False`` if you want to change this behavior. + Multi-line example: .. code-block:: yaml @@ -5062,10 +5326,15 @@ def prepend(name, if makedirs is True: dirname = os.path.dirname(name) if not __salt__['file.directory_exists'](dirname): - __salt__['file.makedirs'](name) - check_res, check_msg, ret['pchanges'] = _check_directory( - dirname, None, None, False, None, False, False, None - ) + try: + _makedirs(name=name) + except CommandExecutionError as exc: + return _error(ret, 'Drive {0} is not mapped'.format(exc.message)) + + if salt.utils.platform.is_windows(): + check_res, check_msg, ret['pchanges'] = _check_directory_win(dirname) + else: + check_res, check_msg, ret['pchanges'] = _check_directory(dirname) if not check_res: return _error(ret, check_msg) @@ -5657,7 +5926,10 @@ def touch(name, atime=None, mtime=None, makedirs=False): return ret if makedirs: - __salt__['file.makedirs'](name) + try: + _makedirs(name=name) + except CommandExecutionError as exc: + return _error(ret, 'Drive {0} is not mapped'.format(exc.message)) if not os.path.isdir(os.path.dirname(name)): return _error( ret, 'Directory not present to touch file {0}'.format(name) @@ -5853,7 +6125,10 @@ def copy_(name, dname = os.path.dirname(name) if not os.path.isdir(dname): if makedirs: - __salt__['file.makedirs'](name) + try: + _makedirs(name=name) + except CommandExecutionError as exc: + return _error(ret, 'Drive {0} is not mapped'.format(exc.message)) else: return _error( ret, @@ -5950,7 +6225,10 @@ def rename(name, source, force=False, makedirs=False): dname = os.path.dirname(name) if not os.path.isdir(dname): if makedirs: - __salt__['file.makedirs'](name) + try: + _makedirs(name=name) + except CommandExecutionError as exc: + return _error(ret, 'Drive {0} is not mapped'.format(exc.message)) else: return _error( ret, @@ -6871,9 +7149,10 @@ def shortcut( if not os.path.isdir(os.path.dirname(name)): if makedirs: - __salt__['file.makedirs']( - name, - user=user) + try: + _makedirs(name=name, user=user) + except CommandExecutionError as exc: + return _error(ret, 'Drive {0} is not mapped'.format(exc.message)) else: return _error( ret, @@ -6896,7 +7175,10 @@ def shortcut( time.sleep(1) # wait for asynchronous deletion if not os.path.isdir(os.path.dirname(backupname)): if makedirs: - os.makedirs(backupname) + try: + _makedirs(name=backupname) + except CommandExecutionError as exc: + return _error(ret, 'Drive {0} is not mapped'.format(exc.message)) else: return _error(ret, ( 'Directory does not exist for' diff --git a/salt/utils/gitfs.py b/salt/utils/gitfs.py index 6963f40226..caf539080f 100644 --- a/salt/utils/gitfs.py +++ b/salt/utils/gitfs.py @@ -476,6 +476,10 @@ class GitProvider(object): use_tags = 'tag' in self.ref_types ret = set() + if salt.utils.stringutils.is_hex(self.base): + # gitfs_base or per-saltenv 'base' may point to a commit ID, which + # would not show up in the refs. Make sure we include it. + ret.add('base') for ref in salt.utils.data.decode(refs): if ref.startswith('refs/'): ref = ref[5:] diff --git a/salt/utils/network.py b/salt/utils/network.py index 2fc71b99dc..b63a29568c 100644 --- a/salt/utils/network.py +++ b/salt/utils/network.py @@ -148,14 +148,18 @@ def _generate_minion_id(): addr=hosts.first() or 'localhost (N/A)', message=socket.gaierror) ) # Universal method for everywhere (Linux, Slowlaris, Windows etc) - for f_name in ['/etc/hostname', '/etc/nodename', '/etc/hosts', - r'{win}\system32\drivers\etc\hosts'.format(win=os.getenv('WINDIR'))]: - if not os.path.exists(f_name): - continue - with salt.utils.files.fopen(f_name) as f_hdl: - for hst in (line.strip().split('#')[0].strip().split() or None for line in f_hdl.read().split(os.linesep)): - if hst and (hst[0][:4] in ['127.', '::1'] or len(hst) == 1): - hosts.extend(hst) + for f_name in ('/etc/hostname', '/etc/nodename', '/etc/hosts', + r'{win}\system32\drivers\etc\hosts'.format(win=os.getenv('WINDIR'))): + try: + with salt.utils.files.fopen(f_name) as f_hdl: + for line in f_hdl: + line = salt.utils.stringutils.to_unicode(line) + hst = line.strip().split('#')[0].strip().split() + if hst: + if hst[0][:4] in ('127.', '::1') or len(hst) == 1: + hosts.extend(hst) + except IOError: + pass # include public and private ipaddresses return hosts.extend([addr for addr in ip_addrs() diff --git a/salt/utils/parsers.py b/salt/utils/parsers.py index 10c1d7534e..c654f033cd 100644 --- a/salt/utils/parsers.py +++ b/salt/utils/parsers.py @@ -977,9 +977,17 @@ class DaemonMixIn(six.with_metaclass(MixInMeta, object)): # Log error only when running salt-master as a root user. # Otherwise this can be ignored, since salt-master is able to # overwrite the PIDfile on the next start. - if not os.getuid(): - logger.info('PIDfile could not be deleted: %s', six.text_type(self.config['pidfile'])) - logger.debug(six.text_type(err)) + err_msg = ('PIDfile could not be deleted: %s', + six.text_type(self.config['pidfile'])) + if salt.utils.platform.is_windows(): + user = salt.utils.win_functions.get_current_user() + if salt.utils.win_functions.is_admin(user): + logger.info(*err_msg) + logger.debug(six.text_type(err)) + else: + if not os.getuid(): + logger.info(*err_msg) + logger.debug(six.text_type(err)) def set_pidfile(self): from salt.utils.process import set_pidfile diff --git a/salt/utils/saltclass.py b/salt/utils/saltclass.py index e5a0ae0bbd..351f179f37 100644 --- a/salt/utils/saltclass.py +++ b/salt/utils/saltclass.py @@ -209,7 +209,7 @@ def expand_classes_in_order(minion_dict, dict_merge(salt_data['__pillar__'], expanded_classes[klass].get('pillars', {})) # Now replace class element in classes_to_expand by expansion - if 'classes' in expanded_classes[klass]: + if expanded_classes[klass].get('classes'): l_id = classes_to_expand.index(klass) classes_to_expand[l_id:l_id] = expanded_classes[klass]['classes'] expand_classes_in_order(minion_dict, diff --git a/salt/utils/win_update.py b/salt/utils/win_update.py index 963380f0df..6769607f9e 100644 --- a/salt/utils/win_update.py +++ b/salt/utils/win_update.py @@ -239,7 +239,8 @@ class WindowsUpdateAgent(object): # Error codes found at the following site: # https://msdn.microsoft.com/en-us/library/windows/desktop/hh968413(v=vs.85).aspx # https://technet.microsoft.com/en-us/library/cc720442(v=ws.10).aspx - fail_codes = {-2145124300: 'Download failed: 0x80240034', + fail_codes = {-2145107924: 'WinHTTP Send/Receive failed: 0x8024402C', + -2145124300: 'Download failed: 0x80240034', -2145124302: 'Invalid search criteria: 0x80240032', -2145124305: 'Cancelled by policy: 0x8024002F', -2145124307: 'Missing source: 0x8024002D', diff --git a/tests/conftest.py b/tests/conftest.py index 0d7afa5e79..44f8dbc326 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -254,11 +254,13 @@ def pytest_runtest_setup(item): if destructive_tests_marker is not None: if item.config.getoption('--run-destructive') is False: pytest.skip('Destructive tests are disabled') + os.environ['DESTRUCTIVE_TESTS'] = six.text_type(item.config.getoption('--run-destructive')) expensive_tests_marker = item.get_marker('expensive_test') if expensive_tests_marker is not None: if item.config.getoption('--run-expensive') is False: pytest.skip('Expensive tests are disabled') + os.environ['EXPENSIVE_TESTS'] = six.text_type(item.config.getoption('--run-expensive')) skip_if_not_root_marker = item.get_marker('skip_if_not_root') if skip_if_not_root_marker is not None: diff --git a/tests/integration/files/file/base/win/repo-ng/7zip.sls b/tests/integration/files/file/base/win/repo-ng/7zip.sls index c1c5905c37..4b93665c5b 100644 --- a/tests/integration/files/file/base/win/repo-ng/7zip.sls +++ b/tests/integration/files/file/base/win/repo-ng/7zip.sls @@ -1,6 +1,6 @@ {% set versions = {'18':['05', '03', '01'], '16':['04', '03', '02', '00'], '9':['20']} %} -Zzip: +7zip: {% for major, subversions in versions.items() %} {% for minor in subversions %} '{{major}}.{{minor}}.00.0': diff --git a/tests/integration/files/saltclass/examples/classes/default/empty.yml b/tests/integration/files/saltclass/examples/classes/default/empty.yml new file mode 100644 index 0000000000..146d37d231 --- /dev/null +++ b/tests/integration/files/saltclass/examples/classes/default/empty.yml @@ -0,0 +1,4 @@ +# https://github.com/saltstack/salt/issues/48145 +classes: +states: +pillars: diff --git a/tests/integration/files/saltclass/examples/classes/default/init.yml b/tests/integration/files/saltclass/examples/classes/default/init.yml index 8f917b796f..834194cae5 100644 --- a/tests/integration/files/saltclass/examples/classes/default/init.yml +++ b/tests/integration/files/saltclass/examples/classes/default/init.yml @@ -1,6 +1,7 @@ classes: - default.users - default.motd + - default.empty states: - openssh diff --git a/tests/integration/modules/test_service.py b/tests/integration/modules/test_service.py index a27389b0dd..3de0bdd1a2 100644 --- a/tests/integration/modules/test_service.py +++ b/tests/integration/modules/test_service.py @@ -130,7 +130,11 @@ class ServiceModuleTest(ModuleCase): # currently upstart does not have a mechanism to report if disabling a service fails if does not exist self.assertTrue(self.run_function('service.disable', [srv_name])) else: - self.assertFalse(self.run_function('service.disable', [srv_name])) + if salt.utils.platform.is_windows(): + disable = self.run_function('service.disable', [srv_name]) + self.assertTrue('error' in disable.lower()) + else: + self.assertFalse(self.run_function('service.disable', [srv_name])) if salt.utils.platform.is_darwin(): self.assertFalse(self.run_function('service.disabled', [srv_name])) diff --git a/tests/integration/runners/test_state.py b/tests/integration/runners/test_state.py index ae92bc2102..ea822e2657 100644 --- a/tests/integration/runners/test_state.py +++ b/tests/integration/runners/test_state.py @@ -20,7 +20,7 @@ from salt.ext.six.moves import queue from tests.support.case import ShellCase from tests.support.unit import skipIf from tests.support.paths import TMP -from tests.support.helpers import flaky +from tests.support.helpers import flaky, expensiveTest from tests.support.mock import MagicMock, patch # Import Salt Libs @@ -479,6 +479,7 @@ class OrchEventTest(ShellCase): del listener signal.alarm(0) + @expensiveTest def test_orchestration_soft_kill(self): ''' Test to confirm that the parallel state requisite works in orch diff --git a/tests/integration/states/test_pkg.py b/tests/integration/states/test_pkg.py index 38c91f9d27..0182588173 100644 --- a/tests/integration/states/test_pkg.py +++ b/tests/integration/states/test_pkg.py @@ -270,13 +270,15 @@ class PkgTest(ModuleCase, SaltReturnAssertsMixin): except AssertionError: self.assertSaltTrueReturn(self.run_state('pkg.removed', name=None, pkgs=pkg_targets)) - ret = self.run_state('pkg.installed', - name=None, - pkgs=pkg_targets, - refresh=False) - self.assertSaltTrueReturn(ret) - ret = self.run_state('pkg.removed', name=None, pkgs=pkg_targets) - self.assertSaltTrueReturn(ret) + try: + ret = self.run_state('pkg.installed', + name=None, + pkgs=pkg_targets, + refresh=False) + self.assertSaltTrueReturn(ret) + finally: + ret = self.run_state('pkg.removed', name=None, pkgs=pkg_targets) + self.assertSaltTrueReturn(ret) @requires_system_grains def test_pkg_004_installed_multipkg_with_version(self, grains=None): @@ -318,13 +320,15 @@ class PkgTest(ModuleCase, SaltReturnAssertsMixin): pkgs = [{pkg_targets[0]: version}, pkg_targets[1]] - ret = self.run_state('pkg.installed', - name=None, - pkgs=pkgs, - refresh=False) - self.assertSaltTrueReturn(ret) - ret = self.run_state('pkg.removed', name=None, pkgs=pkg_targets) - self.assertSaltTrueReturn(ret) + try: + ret = self.run_state('pkg.installed', + name=None, + pkgs=pkgs, + refresh=False) + self.assertSaltTrueReturn(ret) + finally: + ret = self.run_state('pkg.removed', name=None, pkgs=pkg_targets) + self.assertSaltTrueReturn(ret) @requires_system_grains def test_pkg_005_installed_32bit(self, grains=None): diff --git a/tests/minionswarm.py b/tests/minionswarm.py index 2b09cd6e96..edbf7d41f8 100644 --- a/tests/minionswarm.py +++ b/tests/minionswarm.py @@ -9,7 +9,6 @@ on a single system to test scale capabilities # Import Python Libs from __future__ import absolute_import, print_function import os -import pwd import time import signal import optparse @@ -29,6 +28,7 @@ import salt.utils.yaml # Import third party libs from salt.ext import six from salt.ext.six.moves import range # pylint: disable=import-error,redefined-builtin +import tests.support.helpers OSES = [ @@ -148,7 +148,7 @@ def parse(): '-c', '--config-dir', default='', help=('Pass in a configuration directory containing base configuration.') ) - parser.add_option('-u', '--user', default=pwd.getpwuid(os.getuid()).pw_name) + parser.add_option('-u', '--user', default=tests.support.helpers.this_user()) options, _args = parser.parse_args() diff --git a/tests/support/mock.py b/tests/support/mock.py index 9d3ca0656f..fe5cb60fae 100644 --- a/tests/support/mock.py +++ b/tests/support/mock.py @@ -15,10 +15,13 @@ # pylint: disable=unused-import,function-redefined,blacklisted-module,blacklisted-external-module from __future__ import absolute_import +import errno +import fnmatch import sys # Import salt libs from salt.ext import six +import salt.utils.stringutils try: from mock import ( @@ -98,30 +101,33 @@ file_spec = None def _iterate_read_data(read_data): - # Helper for mock_open: - # Retrieve lines from read_data via a generator so that separate calls to - # readline, read, and readlines are properly interleaved - if six.PY3 and isinstance(read_data, six.binary_type): - data_as_list = ['{0}\n'.format(l.decode(__salt_system_encoding__)) for l in read_data.split(b'\n')] - else: - data_as_list = ['{0}\n'.format(l) for l in read_data.split('\n')] + ''' + Helper for mock_open: + Retrieve lines from read_data via a generator so that separate calls to + readline, read, and readlines are properly interleaved + ''' + # Newline will always be a bytestring on PY2 because mock_open will have + # normalized it to one. + newline = b'\n' if isinstance(read_data, six.binary_type) else '\n' - if data_as_list[-1] == '\n': + read_data = [line + newline for line in read_data.split(newline)] + + if read_data[-1] == newline: # If the last line ended in a newline, the list comprehension will have an - # extra entry that's just a newline. Remove this. - data_as_list = data_as_list[:-1] + # extra entry that's just a newline. Remove this. + read_data = read_data[:-1] else: # If there wasn't an extra newline by itself, then the file being - # emulated doesn't have a newline to end the last line remove the - # newline that our naive format() added - data_as_list[-1] = data_as_list[-1][:-1] + # emulated doesn't have a newline to end the last line, so remove the + # newline that we added in the list comprehension. + read_data[-1] = read_data[-1][:-1] - for line in data_as_list: + for line in read_data: yield line -def mock_open(mock=None, read_data=''): - """ +def mock_open(mock=None, read_data='', match=None): + ''' A helper function to create a mock to replace the use of `open`. It works for `open` called directly or used as a context manager. @@ -131,7 +137,18 @@ def mock_open(mock=None, read_data=''): `read_data` is a string for the `read` methoddline`, and `readlines` of the file handle to return. This is an empty string by default. - """ + + If passed, `match` can be either a string or an iterable containing + patterns to attempt to match using fnmatch.fnmatch(). A side_effect will be + added to the mock object returned, which will cause an IOError(2, 'No such + file or directory') to be raised when the file path is not a match. This + allows you to make your mocked filehandle only work for certain file paths. + ''' + # Normalize read_data, Python 2 filehandles should never produce unicode + # types on read. + if six.PY2: + read_data = salt.utils.stringutils.to_str(read_data) + def _readlines_side_effect(*args, **kwargs): if handle.readlines.return_value is not None: return handle.readlines.return_value @@ -140,7 +157,8 @@ def mock_open(mock=None, read_data=''): def _read_side_effect(*args, **kwargs): if handle.read.return_value is not None: return handle.read.return_value - return ''.join(_data) + joiner = b'' if isinstance(read_data, six.binary_type) else '' + return joiner.join(_data) def _readline_side_effect(): if handle.readline.return_value is not None: @@ -170,10 +188,25 @@ def mock_open(mock=None, read_data=''): handle.readline.return_value = None handle.readlines.return_value = None + # Support iteration via for loop + handle.__iter__ = lambda x: _readline_side_effect() + # This is salt specific and not in the upstream mock handle.read.side_effect = _read_side_effect handle.readline.side_effect = _readline_side_effect() handle.readlines.side_effect = _readlines_side_effect + if match is not None: + if isinstance(match, six.string_types): + match = [match] + + def fopen_side_effect(name, *args, **kwargs): + for pat in match: + if fnmatch.fnmatch(name, pat): + return DEFAULT + raise IOError(errno.ENOENT, 'No such file or directory', name) + + mock.side_effect = fopen_side_effect + mock.return_value = handle return mock diff --git a/tests/unit/beacons/test_btmp_beacon.py b/tests/unit/beacons/test_btmp_beacon.py index 5c8106da8e..4865108e55 100644 --- a/tests/unit/beacons/test_btmp_beacon.py +++ b/tests/unit/beacons/test_btmp_beacon.py @@ -4,7 +4,6 @@ from __future__ import absolute_import import datetime import logging -import sys # Salt testing libs from tests.support.unit import skipIf, TestCase @@ -21,12 +20,9 @@ try: except ImportError: _TIME_SUPPORTED = False -if sys.version_info >= (3,): - raw = bytes('\x06\x00\x00\x00Nt\x00\x00ssh:notty\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00garet\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00::1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xdd\xc7\xc2Y\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', 'utf-8') - pack = (6, 29774, b'ssh:notty\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', b'\x00\x00\x00\x00', b'garet\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', b'::1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', 0, 0, 0, 1505937373, 0, 0, 0, 0, 16777216) -else: - raw = b'\x06\x00\x00\x00Nt\x00\x00ssh:notty\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00garet\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00::1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xdd\xc7\xc2Y\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' - pack = (6, 29774, 'ssh:notty\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', '\x00\x00\x00\x00', 'garet\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', '::1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', 0, 0, 0, 1505937373, 0, 0, 0, 0, 16777216) +raw = b'\x06\x00\x00\x00Nt\x00\x00ssh:notty\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00garet\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00::1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xdd\xc7\xc2Y\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' +pack = (6, 29774, b'ssh:notty\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', b'\x00\x00\x00\x00', b'garet\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', b'::1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', 0, 0, 0, 1505937373, 0, 0, 0, 0, 16777216) + log = logging.getLogger(__name__) diff --git a/tests/unit/beacons/test_wtmp_beacon.py b/tests/unit/beacons/test_wtmp_beacon.py index 5dc816e52f..8be45c81fa 100644 --- a/tests/unit/beacons/test_wtmp_beacon.py +++ b/tests/unit/beacons/test_wtmp_beacon.py @@ -4,7 +4,6 @@ from __future__ import absolute_import import datetime import logging -import sys # Salt testing libs from tests.support.unit import skipIf, TestCase @@ -21,12 +20,8 @@ try: except ImportError: _TIME_SUPPORTED = False -if sys.version_info >= (3,): - raw = bytes('\x07\x00\x00\x00H\x18\x00\x00pts/14\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00s/14gareth\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00::1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x13I\xc5YZf\x05\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', 'utf-8') - pack = (7, 6216, b'pts/14\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', b's/14', b'gareth\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', b'::1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', 0, 0, 0, 1506101523, 353882, 0, 0, 0, 16777216) -else: - raw = b'\x07\x00\x00\x00H\x18\x00\x00pts/14\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00s/14gareth\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00::1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x13I\xc5YZf\x05\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' - pack = (7, 6216, 'pts/14\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', 's/14', 'gareth\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', '::1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', 0, 0, 0, 1506101523, 353882, 0, 0, 0, 16777216) +raw = b'\x07\x00\x00\x00H\x18\x00\x00pts/14\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00s/14gareth\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00::1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x13I\xc5YZf\x05\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' +pack = (7, 6216, b'pts/14\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', b's/14', b'gareth\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', b'::1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', 0, 0, 0, 1506101523, 353882, 0, 0, 0, 16777216) log = logging.getLogger(__name__) diff --git a/tests/unit/daemons/test_masterapi.py b/tests/unit/daemons/test_masterapi.py index 8ba0b04d2a..441bfd44e3 100644 --- a/tests/unit/daemons/test_masterapi.py +++ b/tests/unit/daemons/test_masterapi.py @@ -3,6 +3,7 @@ # Import Python libs from __future__ import absolute_import, print_function, unicode_literals from functools import wraps +import os import io import stat @@ -12,6 +13,7 @@ import salt.daemons.masterapi as masterapi import salt.utils.platform # Import Salt Testing Libs +from tests.support.paths import TMP_CONF_DIR from tests.support.unit import TestCase, skipIf from tests.support.mock import ( patch, @@ -568,7 +570,7 @@ class RemoteFuncsTestCase(TestCase): ''' def setUp(self): - opts = salt.config.master_config(None) + opts = salt.config.master_config(os.path.join(TMP_CONF_DIR, 'master')) self.funcs = masterapi.RemoteFuncs(opts) self.funcs.cache = FakeCache() diff --git a/tests/unit/grains/test_core.py b/tests/unit/grains/test_core.py index e36e6cfbe1..276d08779d 100644 --- a/tests/unit/grains/test_core.py +++ b/tests/unit/grains/test_core.py @@ -8,6 +8,7 @@ from __future__ import absolute_import, print_function, unicode_literals import logging import os import socket +import textwrap # Import Salt Testing Libs try: @@ -69,9 +70,8 @@ class CoreGrainsTestCase(TestCase, LoaderModuleMockMixin): def test_parse_etc_os_release(self, path_isfile_mock): path_isfile_mock.side_effect = lambda x: x == "/usr/lib/os-release" with salt.utils.files.fopen(os.path.join(OS_RELEASE_DIR, "ubuntu-17.10")) as os_release_file: - os_release_content = os_release_file.readlines() - with patch("salt.utils.files.fopen", mock_open()) as os_release_file: - os_release_file.return_value.__iter__.return_value = os_release_content + os_release_content = os_release_file.read() + with patch("salt.utils.files.fopen", mock_open(read_data=os_release_content)): os_release = core._parse_os_release(["/etc/os-release", "/usr/lib/os-release"]) self.assertEqual(os_release, { "NAME": "Ubuntu", @@ -272,34 +272,26 @@ class CoreGrainsTestCase(TestCase, LoaderModuleMockMixin): return orig_import(name, *args) # Skip the first if statement - with patch.object(salt.utils.platform, 'is_proxy', - MagicMock(return_value=False)): - # Skip the selinux/systemd stuff (not pertinent) - with patch.object(core, '_linux_bin_exists', - MagicMock(return_value=False)): - # Skip the init grain compilation (not pertinent) - with patch.object(os.path, 'exists', path_isfile_mock): - # Ensure that lsb_release fails to import - with patch('{0}.__import__'.format(built_in), - side_effect=_import_mock): - # Skip all the /etc/*-release stuff (not pertinent) - with patch.object(os.path, 'isfile', path_isfile_mock): - with patch.object(core, '_parse_os_release', os_release_mock): - # Mock linux_distribution to give us the OS - # name that we want. - distro_mock = MagicMock( - return_value=os_release_map['linux_distribution'] - ) - with patch('salt.utils.files.fopen', mock_open()) as suse_release_file: - suse_release_file.return_value.__iter__.return_value = \ - os_release_map.get('suse_release_file', '').splitlines() - with patch.object(core, 'linux_distribution', distro_mock): - with patch.object(core, '_linux_gpu_data', empty_mock): - with patch.object(core, '_linux_cpudata', empty_mock): - with patch.object(core, '_virtual', empty_mock): - # Mock the osarch - with patch.dict(core.__salt__, {'cmd.run': osarch_mock}): - os_grains = core.os_data() + # Skip the selinux/systemd stuff (not pertinent) + # Skip the init grain compilation (not pertinent) + # Ensure that lsb_release fails to import + # Skip all the /etc/*-release stuff (not pertinent) + # - Mock linux_distribution to give us the OS name that we want. + # Mock the osarch + distro_mock = MagicMock(return_value=os_release_map['linux_distribution']) + with patch.object(salt.utils.platform, 'is_proxy', MagicMock(return_value=False)), \ + patch.object(core, '_linux_bin_exists', MagicMock(return_value=False)), \ + patch.object(os.path, 'exists', path_isfile_mock), \ + patch('{0}.__import__'.format(built_in), side_effect=_import_mock), \ + patch.object(os.path, 'isfile', path_isfile_mock), \ + patch.object(core, '_parse_os_release', os_release_mock), \ + patch('salt.utils.files.fopen', mock_open(read_data=os_release_map.get('suse_release_file', ''))), \ + patch.object(core, 'linux_distribution', distro_mock), \ + patch.object(core, '_linux_gpu_data', empty_mock), \ + patch.object(core, '_linux_cpudata', empty_mock), \ + patch.object(core, '_virtual', empty_mock), \ + patch.dict(core.__salt__, {'cmd.run': osarch_mock}): + os_grains = core.os_data() grains = {k: v for k, v in os_grains.items() if k in set(["os", "os_family", "osfullname", "oscodename", "osfinger", @@ -318,10 +310,11 @@ class CoreGrainsTestCase(TestCase, LoaderModuleMockMixin): Test if OS grains are parsed correctly in SLES 11 SP3 ''' _os_release_map = { - 'suse_release_file': '''SUSE Linux Enterprise Server 11 (x86_64) -VERSION = 11 -PATCHLEVEL = 3 -''', + 'suse_release_file': textwrap.dedent(''' + SUSE Linux Enterprise Server 11 (x86_64) + VERSION = 11 + PATCHLEVEL = 3 + '''), 'files': ["/etc/SuSE-release"], } expectation = { @@ -590,8 +583,9 @@ PATCHLEVEL = 3 ) empty_mock = MagicMock(return_value={}) - _proc_meminfo_file = '''MemTotal: 16277028 kB -SwapTotal: 4789244 kB''' + _proc_meminfo = textwrap.dedent('''\ + MemTotal: 16277028 kB + SwapTotal: 4789244 kB''') orig_import = __import__ if six.PY2: @@ -604,49 +598,28 @@ SwapTotal: 4789244 kB''' raise ImportError('No module named lsb_release') return orig_import(name, *args) - # Skip the first if statement - with patch.object(salt.utils.platform, 'is_proxy', - MagicMock(return_value=False)): - # Skip the selinux/systemd stuff (not pertinent) - with patch.object(core, '_linux_bin_exists', - MagicMock(return_value=False)): - # Skip the init grain compilation (not pertinent) - with patch.object(os.path, 'exists', path_exists_mock): - # Ensure that lsb_release fails to import - with patch('{0}.__import__'.format(built_in), - side_effect=_import_mock): - # Skip all the /etc/*-release stuff (not pertinent) - with patch.object(os.path, 'isfile', path_isfile_mock): - # Make a bunch of functions return empty dicts, - # we don't care about these grains for the - # purposes of this test. - with patch.object( - core, - '_linux_cpudata', - empty_mock): - with patch.object( - core, - '_linux_gpu_data', - empty_mock): - with patch('salt.utils.files.fopen', mock_open()) as _proc_meminfo: - _proc_meminfo.return_value.__iter__.return_value = _proc_meminfo_file.splitlines() - with patch.object( - core, - '_hw_data', - empty_mock): - with patch.object( - core, - '_virtual', - empty_mock): - with patch.object( - core, - '_ps', - empty_mock): - # Mock the osarch - with patch.dict( - core.__salt__, - {'cmd.run': cmd_run_mock}): - os_grains = core.os_data() + # Mock a bunch of stuff so we can isolate the mem stuff: + # - Skip the first if statement + # - Skip the init grain compilation (not pertinent) + # - Ensure that lsb_release fails to import + # - Skip all the /etc/*-release stuff (not pertinent) + # - Make a bunch of functions return empty dicts, we don't care + # about these grains for the purposes of this test. + # - Mock the osarch + # - And most importantly, mock the contents of /proc/meminfo + with patch.object(salt.utils.platform, 'is_proxy', MagicMock(return_value=False)), \ + patch.object(core, '_linux_bin_exists', MagicMock(return_value=False)), \ + patch.object(os.path, 'exists', path_exists_mock), \ + patch('{0}.__import__'.format(built_in), side_effect=_import_mock), \ + patch.object(os.path, 'isfile', path_isfile_mock), \ + patch.object(core, '_linux_cpudata', empty_mock), \ + patch.object(core, '_linux_gpu_data', empty_mock), \ + patch.object(core, '_hw_data', empty_mock), \ + patch.object(core, '_virtual', empty_mock), \ + patch.object(core, '_ps', empty_mock), \ + patch.dict(core.__salt__, {'cmd.run': cmd_run_mock}), \ + patch('salt.utils.files.fopen', mock_open(read_data=_proc_meminfo)): + os_grains = core.os_data() self.assertEqual(os_grains.get('mem_total'), 15895) self.assertEqual(os_grains.get('swap_total'), 4676) diff --git a/tests/unit/grains/test_iscsi.py b/tests/unit/grains/test_iscsi.py index 9ca4ad4809..2f4ccc8b85 100644 --- a/tests/unit/grains/test_iscsi.py +++ b/tests/unit/grains/test_iscsi.py @@ -5,6 +5,7 @@ # Import Python libs from __future__ import absolute_import, print_function, unicode_literals import os +import textwrap # Import Salt Testing Libs from tests.support.unit import TestCase, skipIf @@ -51,15 +52,16 @@ class IscsiGrainsTestCase(TestCase): ['iqn.localhost.hostid.7f000001']) def test_linux_iscsi_iqn_grains(self): - _iscsi_file = '## DO NOT EDIT OR REMOVE THIS FILE!\n' \ - '## If you remove this file, the iSCSI daemon will not start.\n' \ - '## If you change the InitiatorName, existing access control lists\n' \ - '## may reject this initiator. The InitiatorName must be unique\n' \ - '## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.\n' \ - 'InitiatorName=iqn.1993-08.org.debian:01:d12f7aba36\n' + _iscsi_file = textwrap.dedent('''\ + ## DO NOT EDIT OR REMOVE THIS FILE! + ## If you remove this file, the iSCSI daemon will not start. + ## If you change the InitiatorName, existing access control lists + ## may reject this initiator. The InitiatorName must be unique + ## for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames. + InitiatorName=iqn.1993-08.org.debian:01:d12f7aba36 + ''') - with patch('salt.utils.files.fopen', mock_open()) as iscsi_initiator_file: - iscsi_initiator_file.return_value.__iter__.return_value = _iscsi_file.splitlines() + with patch('salt.utils.files.fopen', mock_open(read_data=_iscsi_file)): iqn = iscsi._linux_iqn() assert isinstance(iqn, list) diff --git a/tests/unit/modules/test_cp.py b/tests/unit/modules/test_cp.py index 0d51ab208c..f09664b294 100644 --- a/tests/unit/modules/test_cp.py +++ b/tests/unit/modules/test_cp.py @@ -147,7 +147,7 @@ class CpTestCase(TestCase, LoaderModuleMockMixin): cmd='_file_recv', tok='token', path=['saltines', 'test.file'], - data='', # data is empty here because load['data'] is overwritten + data=b'', # data is empty here because load['data'] is overwritten id='abc' ) ) diff --git a/tests/unit/modules/test_dnsmasq.py b/tests/unit/modules/test_dnsmasq.py index 3d97eee3df..55080c68f3 100644 --- a/tests/unit/modules/test_dnsmasq.py +++ b/tests/unit/modules/test_dnsmasq.py @@ -5,6 +5,7 @@ # Import Python libs from __future__ import absolute_import, print_function, unicode_literals import os +import textwrap # Import Salt Testing Libs from tests.support.mixins import LoaderModuleMockMixin @@ -20,7 +21,6 @@ from tests.support.mock import ( # Import Salt Libs from salt.exceptions import CommandExecutionError import salt.modules.dnsmasq as dnsmasq -import salt.utils.stringutils @skipIf(NO_MOCK, NO_MOCK_REASON) @@ -98,13 +98,14 @@ class DnsmasqTestCase(TestCase, LoaderModuleMockMixin): test for generic function for parsing dnsmasq files including includes. ''' with patch('os.path.isfile', MagicMock(return_value=True)): - text_file_data = salt.utils.stringutils.to_str( - '\n'.join(["line here", "second line", "A=B", "#"])) + text_file_data = textwrap.dedent('''\ + line here + second line + A=B + #''') with patch('salt.utils.files.fopen', - mock_open(read_data=text_file_data), - create=True) as m: - m.return_value.__iter__.return_value = text_file_data.splitlines() + mock_open(read_data=text_file_data)): self.assertDictEqual(dnsmasq._parse_dnamasq('filename'), {'A': 'B', - 'unparsed': ['line here', - 'second line']}) + 'unparsed': ['line here\n', + 'second line\n']}) diff --git a/tests/unit/modules/test_mac_sysctl.py b/tests/unit/modules/test_mac_sysctl.py index ac821b59aa..1ea870726c 100644 --- a/tests/unit/modules/test_mac_sysctl.py +++ b/tests/unit/modules/test_mac_sysctl.py @@ -91,7 +91,12 @@ class DarwinSysctlTestCase(TestCase, LoaderModuleMockMixin): Tests successful write to existing sysctl file ''' to_write = '#\n# Kernel sysctl configuration\n#\n' - m_calls_list = [call.writelines(['net.inet.icmp.icmplim=50', '\n'])] + m_calls_list = [call.writelines([ + '#\n', + '# Kernel sysctl configuration\n', + '#\n', + 'net.inet.icmp.icmplim=50\n', + ])] with patch('salt.utils.files.fopen', mock_open(read_data=to_write)) as m_open, \ patch('os.path.isfile', MagicMock(return_value=True)): mac_sysctl.persist('net.inet.icmp.icmplim', 50, config=to_write) diff --git a/tests/unit/modules/test_win_status.py b/tests/unit/modules/test_win_status.py index 939ecc3d74..314467023e 100644 --- a/tests/unit/modules/test_win_status.py +++ b/tests/unit/modules/test_win_status.py @@ -146,11 +146,7 @@ class TestProcsWMIGetOwnerErrorsAreLogged(TestProcsBase): def test_error_logged_if_process_get_owner_fails(self): with patch('salt.modules.win_status.log') as log: self.call_procs() - log.warning.assert_called_once_with(ANY) - self.assertIn( - str(self.expected_error_code), - log.warning.call_args[0][0] - ) + log.warning.assert_called_once_with(ANY, ANY, self.expected_error_code) class TestEmptyCommandLine(TestProcsBase): diff --git a/tests/unit/pillar/test_saltclass.py b/tests/unit/pillar/test_saltclass.py index ec1b256990..8bd4f050dd 100644 --- a/tests/unit/pillar/test_saltclass.py +++ b/tests/unit/pillar/test_saltclass.py @@ -48,5 +48,5 @@ class SaltclassPillarTestCase(TestCase, LoaderModuleMockMixin): self.assertListEqual(parsed_ret, expected_ret) def test_succeeds(self): - ret = ['default.users', 'default.motd', 'default', 'roles.app'] + ret = ['default.users', 'default.motd', 'default.empty', 'default', 'roles.app'] self._runner(ret) diff --git a/tests/unit/test_doc.py b/tests/unit/test_doc.py index 4cd4caeb96..2e14e68cd1 100644 --- a/tests/unit/test_doc.py +++ b/tests/unit/test_doc.py @@ -59,9 +59,11 @@ class DocTestCase(TestCase): key, val = regex.split(line, 1) # Don't test man pages, this file, - # the page that documents to not use ":doc:", or - # the doc/conf.py file + # the tox virtualenv files, the page + # that documents to not use ":doc:", + # or the doc/conf.py file if 'man' in key \ + or '.tox/' in key \ or key.endswith('test_doc.py') \ or key.endswith(os.sep.join(['doc', 'conf.py'])) \ or key.endswith(os.sep.join(['conventions', 'documentation.rst'])) \ diff --git a/tests/unit/utils/test_network.py b/tests/unit/utils/test_network.py index 0e2c45b17c..487db58d4c 100644 --- a/tests/unit/utils/test_network.py +++ b/tests/unit/utils/test_network.py @@ -3,11 +3,18 @@ from __future__ import absolute_import, unicode_literals, print_function import logging import socket +import textwrap # Import Salt Testing libs from tests.support.unit import skipIf from tests.support.unit import TestCase -from tests.support.mock import NO_MOCK, NO_MOCK_REASON, patch, MagicMock +from tests.support.mock import ( + MagicMock, + mock_open, + patch, + NO_MOCK, + NO_MOCK_REASON, +) # Import salt libs import salt.utils.network as network @@ -147,6 +154,20 @@ class NetworkTestCase(TestCase): def test_generate_minion_id(self): self.assertTrue(network.generate_minion_id()) + def test__generate_minion_id_with_unicode_in_etc_hosts(self): + ''' + Test that unicode in /etc/hosts doesn't raise an error when + _generate_minion_id() helper is called to gather the hosts. + ''' + content = textwrap.dedent('''\ + # 以下为主机名解析 + ## ccc + 127.0.0.1 localhost thisismyhostname # 本机 + ''') + fopen_mock = mock_open(read_data=content, match='/etc/hosts') + with patch('salt.utils.files.fopen', fopen_mock): + assert 'thisismyhostname' in network._generate_minion_id() + def test_is_ip(self): self.assertTrue(network.is_ip('10.10.0.3')) self.assertFalse(network.is_ip('0.9.800.1000')) @@ -320,8 +341,7 @@ class NetworkTestCase(TestCase): patch('socket.gethostname', MagicMock(return_value='hostname')), \ patch('socket.getfqdn', MagicMock(return_value='hostname.domainname.blank')), \ patch('socket.getaddrinfo', MagicMock(return_value=[(2, 3, 0, 'attrname', ('127.0.1.1', 0))])), \ - patch('salt.utils.files.fopen', MagicMock(return_value=False)), \ - patch('os.path.exists', MagicMock(return_value=False)), \ + patch('salt.utils.files.fopen', mock_open()), \ patch('salt.utils.network.ip_addrs', MagicMock(return_value=['1.2.3.4', '5.6.7.8'])): self.assertEqual(network._generate_minion_id(), ['hostname.domainname.blank', 'nodename', 'hostname', '1.2.3.4', '5.6.7.8']) @@ -336,8 +356,7 @@ class NetworkTestCase(TestCase): patch('socket.gethostname', MagicMock(return_value='127')), \ patch('socket.getfqdn', MagicMock(return_value='127.domainname.blank')), \ patch('socket.getaddrinfo', MagicMock(return_value=[(2, 3, 0, 'attrname', ('127.0.1.1', 0))])), \ - patch('salt.utils.files.fopen', MagicMock(return_value=False)), \ - patch('os.path.exists', MagicMock(return_value=False)), \ + patch('salt.utils.files.fopen', mock_open()), \ patch('salt.utils.network.ip_addrs', MagicMock(return_value=['1.2.3.4', '5.6.7.8'])): self.assertEqual(network._generate_minion_id(), ['127.domainname.blank', '127', '1.2.3.4', '5.6.7.8']) @@ -352,8 +371,7 @@ class NetworkTestCase(TestCase): patch('socket.gethostname', MagicMock(return_value='127890')), \ patch('socket.getfqdn', MagicMock(return_value='127890.domainname.blank')), \ patch('socket.getaddrinfo', MagicMock(return_value=[(2, 3, 0, 'attrname', ('127.0.1.1', 0))])), \ - patch('salt.utils.files.fopen', MagicMock(return_value=False)), \ - patch('os.path.exists', MagicMock(return_value=False)), \ + patch('salt.utils.files.fopen', mock_open()), \ patch('salt.utils.network.ip_addrs', MagicMock(return_value=['1.2.3.4', '5.6.7.8'])): self.assertEqual(network._generate_minion_id(), ['127890.domainname.blank', '127890', '1.2.3.4', '5.6.7.8']) @@ -368,8 +386,7 @@ class NetworkTestCase(TestCase): patch('socket.gethostname', MagicMock(return_value='hostname')), \ patch('socket.getfqdn', MagicMock(return_value='hostname')), \ patch('socket.getaddrinfo', MagicMock(return_value=[(2, 3, 0, 'hostname', ('127.0.1.1', 0))])), \ - patch('salt.utils.files.fopen', MagicMock(return_value=False)), \ - patch('os.path.exists', MagicMock(return_value=False)), \ + patch('salt.utils.files.fopen', mock_open()), \ patch('salt.utils.network.ip_addrs', MagicMock(return_value=['1.2.3.4', '1.2.3.4', '1.2.3.4'])): self.assertEqual(network._generate_minion_id(), ['hostname', '1.2.3.4']) @@ -384,8 +401,7 @@ class NetworkTestCase(TestCase): patch('socket.gethostname', MagicMock(return_value='hostname')), \ patch('socket.getfqdn', MagicMock(return_value='')), \ patch('socket.getaddrinfo', MagicMock(return_value=[(2, 3, 0, 'hostname', ('127.0.1.1', 0))])), \ - patch('salt.utils.files.fopen', MagicMock(return_value=False)), \ - patch('os.path.exists', MagicMock(return_value=False)), \ + patch('salt.utils.files.fopen', mock_open()), \ patch('salt.utils.network.ip_addrs', MagicMock(return_value=['1.2.3.4', '1.2.3.4', '1.2.3.4'])): self.assertEqual(network.generate_minion_id(), 'very.long.and.complex.domain.name') @@ -399,8 +415,7 @@ class NetworkTestCase(TestCase): patch('socket.gethostname', MagicMock(return_value='pick.me')), \ patch('socket.getfqdn', MagicMock(return_value='hostname.domainname.blank')), \ patch('socket.getaddrinfo', MagicMock(return_value=[(2, 3, 0, 'hostname', ('127.0.1.1', 0))])), \ - patch('salt.utils.files.fopen', MagicMock(return_value=False)), \ - patch('os.path.exists', MagicMock(return_value=False)), \ + patch('salt.utils.files.fopen', mock_open()), \ patch('salt.utils.network.ip_addrs', MagicMock(return_value=['1.2.3.4', '1.2.3.4', '1.2.3.4'])): self.assertEqual(network.generate_minion_id(), 'hostname.domainname.blank') @@ -414,8 +429,7 @@ class NetworkTestCase(TestCase): patch('socket.gethostname', MagicMock(return_value='ip6-loopback')), \ patch('socket.getfqdn', MagicMock(return_value='ip6-localhost')), \ patch('socket.getaddrinfo', MagicMock(return_value=[(2, 3, 0, 'localhost', ('127.0.1.1', 0))])), \ - patch('salt.utils.files.fopen', MagicMock(return_value=False)), \ - patch('os.path.exists', MagicMock(return_value=False)), \ + patch('salt.utils.files.fopen', mock_open()), \ patch('salt.utils.network.ip_addrs', MagicMock(return_value=['127.0.0.1', '::1', 'fe00::0', 'fe02::1', '1.2.3.4'])): self.assertEqual(network.generate_minion_id(), '1.2.3.4') @@ -429,8 +443,7 @@ class NetworkTestCase(TestCase): patch('socket.gethostname', MagicMock(return_value='ip6-loopback')), \ patch('socket.getfqdn', MagicMock(return_value='ip6-localhost')), \ patch('socket.getaddrinfo', MagicMock(return_value=[(2, 3, 0, 'localhost', ('127.0.1.1', 0))])), \ - patch('salt.utils.files.fopen', MagicMock(return_value=False)), \ - patch('os.path.exists', MagicMock(return_value=False)), \ + patch('salt.utils.files.fopen', mock_open()), \ patch('salt.utils.network.ip_addrs', MagicMock(return_value=['127.0.0.1', '::1', 'fe00::0', 'fe02::1'])): self.assertEqual(network.generate_minion_id(), 'localhost') @@ -444,8 +457,7 @@ class NetworkTestCase(TestCase): patch('socket.gethostname', MagicMock(return_value='ip6-loopback')), \ patch('socket.getfqdn', MagicMock(return_value='pick.me')), \ patch('socket.getaddrinfo', MagicMock(return_value=[(2, 3, 0, 'localhost', ('127.0.1.1', 0))])), \ - patch('salt.utils.files.fopen', MagicMock(return_value=False)), \ - patch('os.path.exists', MagicMock(return_value=False)), \ + patch('salt.utils.files.fopen', mock_open()), \ patch('salt.utils.network.ip_addrs', MagicMock(return_value=['127.0.0.1', '::1', 'fe00::0', 'fe02::1'])): self.assertEqual(network.generate_minion_id(), 'pick.me') @@ -459,8 +471,7 @@ class NetworkTestCase(TestCase): patch('socket.gethostname', MagicMock(return_value='ip6-loopback')), \ patch('socket.getfqdn', MagicMock(return_value='ip6-localhost')), \ patch('socket.getaddrinfo', MagicMock(return_value=[(2, 3, 0, 'pick.me', ('127.0.1.1', 0))])), \ - patch('salt.utils.files.fopen', MagicMock(return_value=False)), \ - patch('os.path.exists', MagicMock(return_value=False)), \ + patch('salt.utils.files.fopen', mock_open()), \ patch('salt.utils.network.ip_addrs', MagicMock(return_value=['127.0.0.1', '::1', 'fe00::0', 'fe02::1'])): self.assertEqual(network.generate_minion_id(), 'pick.me') @@ -474,8 +485,7 @@ class NetworkTestCase(TestCase): patch('socket.gethostname', MagicMock(return_value='ip6-loopback')), \ patch('socket.getfqdn', MagicMock(return_value='ip6-localhost')), \ patch('socket.getaddrinfo', MagicMock(return_value=[(2, 3, 0, 'localhost', ('127.0.1.1', 0))])), \ - patch('salt.utils.files.fopen', MagicMock(return_value=False)), \ - patch('os.path.exists', MagicMock(return_value=False)), \ + patch('salt.utils.files.fopen', mock_open()), \ patch('salt.utils.network.ip_addrs', MagicMock(return_value=['127.0.0.1', '::1', 'fe00::0', 'fe02::1', '1.2.3.4'])): self.assertEqual(network.generate_minion_id(), '1.2.3.4') diff --git a/tests/unit/utils/test_parsers.py b/tests/unit/utils/test_parsers.py index 9836269473..36b14a0072 100644 --- a/tests/unit/utils/test_parsers.py +++ b/tests/unit/utils/test_parsers.py @@ -1036,30 +1036,44 @@ class DaemonMixInTestCase(TestCase): @patch('os.unlink', MagicMock(side_effect=OSError())) @patch('os.path.isfile', MagicMock(return_value=True)) - @patch('os.getuid', MagicMock(return_value=0)) @patch('salt.utils.parsers.logger', MagicMock()) def test_pid_deleted_oserror_as_root(self): ''' PIDfile deletion with exception, running as root. ''' - self.daemon_mixin._mixin_before_exit() - assert salt.utils.parsers.os.unlink.call_count == 1 - salt.utils.parsers.logger.info.assert_called_with('PIDfile could not be deleted: %s', - format(self.daemon_mixin.config['pidfile'])) - salt.utils.parsers.logger.debug.assert_called() + if salt.utils.platform.is_windows(): + patch_args = ('salt.utils.win_functions.is_admin', + MagicMock(return_value=True)) + else: + patch_args = ('os.getuid', MagicMock(return_value=0)) + + with patch(*patch_args): + self.daemon_mixin._mixin_before_exit() + assert salt.utils.parsers.os.unlink.call_count == 1 + salt.utils.parsers.logger.info.assert_called_with( + 'PIDfile could not be deleted: %s', + format(self.daemon_mixin.config['pidfile']) + ) + salt.utils.parsers.logger.debug.assert_called() @patch('os.unlink', MagicMock(side_effect=OSError())) @patch('os.path.isfile', MagicMock(return_value=True)) - @patch('os.getuid', MagicMock(return_value=1000)) @patch('salt.utils.parsers.logger', MagicMock()) def test_pid_deleted_oserror_as_non_root(self): ''' PIDfile deletion with exception, running as non-root. ''' - self.daemon_mixin._mixin_before_exit() - assert salt.utils.parsers.os.unlink.call_count == 1 - salt.utils.parsers.logger.info.assert_not_called() - salt.utils.parsers.logger.debug.assert_not_called() + if salt.utils.platform.is_windows(): + patch_args = ('salt.utils.win_functions.is_admin', + MagicMock(return_value=False)) + else: + patch_args = ('os.getuid', MagicMock(return_value=1000)) + + with patch(*patch_args): + self.daemon_mixin._mixin_before_exit() + assert salt.utils.parsers.os.unlink.call_count == 1 + salt.utils.parsers.logger.info.assert_not_called() + salt.utils.parsers.logger.debug.assert_not_called() # Hide the class from unittest framework when it searches for TestCase classes in the module diff --git a/tests/unit/utils/test_which.py b/tests/unit/utils/test_which.py index df2090bd01..46120b2858 100644 --- a/tests/unit/utils/test_which.py +++ b/tests/unit/utils/test_which.py @@ -51,14 +51,14 @@ class TestWhich(TestCase): True ] # Let's patch os.environ to provide a custom PATH variable - with patch.dict(os.environ, {'PATH': '/bin', + with patch.dict(os.environ, {'PATH': os.sep + 'bin', 'PATHEXT': '.COM;.EXE;.BAT;.CMD'}): # Let's also patch is_windows to return True with patch('salt.utils.platform.is_windows', lambda: True): with patch('os.path.isfile', lambda x: True): self.assertEqual( salt.utils.path.which('this-binary-exists-under-windows'), - os.path.join('/bin', 'this-binary-exists-under-windows.EXE') + os.path.join(os.sep + 'bin', 'this-binary-exists-under-windows.EXE') ) def test_missing_binary_in_windows(self): @@ -73,7 +73,7 @@ class TestWhich(TestCase): False, False, False, False, False ] # Let's patch os.environ to provide a custom PATH variable - with patch.dict(os.environ, {'PATH': '/bin'}): + with patch.dict(os.environ, {'PATH': os.sep + 'bin'}): # Let's also patch is_widows to return True with patch('salt.utils.platform.is_windows', lambda: True): self.assertEqual( @@ -101,7 +101,7 @@ class TestWhich(TestCase): True ] # Let's patch os.environ to provide a custom PATH variable - with patch.dict(os.environ, {'PATH': '/bin', + with patch.dict(os.environ, {'PATH': os.sep + 'bin', 'PATHEXT': '.COM;.EXE;.BAT;.CMD;.VBS;' '.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.PY'}): # Let's also patch is_windows to return True @@ -109,5 +109,5 @@ class TestWhich(TestCase): with patch('os.path.isfile', lambda x: True): self.assertEqual( salt.utils.path.which('this-binary-exists-under-windows'), - os.path.join('/bin', 'this-binary-exists-under-windows.CMD') + os.path.join(os.sep + 'bin', 'this-binary-exists-under-windows.CMD') ) diff --git a/tox.ini b/tox.ini index 52b026a299..c7d64bd1a3 100644 --- a/tox.ini +++ b/tox.ini @@ -2,15 +2,6 @@ envlist = py27,py34,py35,py36 [testenv] -sitepackages = True -deps = - py27,pylint: -r{toxinidir}/requirements/dev_python27.txt - py34,py35,py36: -r{toxinidir}/requirements/dev_python34.txt -commands = - py27: python2 {toxinidir}/tests/runtests.py {posargs:-v --run-destructive} - py34,py35,py36: python3 {toxinidir}/tests/runtests.py {posargs:-v --run-destructive} - -[testenv:pylint] -basepython = python2.7 -commands = pylint --rcfile={toxinidir}/.testing.pylintrc --disable=W1307 {posargs:setup.py salt/} - pylint --rcfile={toxinidir}/.testing.pylintrc --disable=W0232,E1002,W1307 {posargs:tests/} +deps = -r{toxinidir}/requirements/tests.txt +commands = pytest --rootdir {toxinidir} {posargs:--no-print-logs --run-destructive} +passenv = LANG HOME