======================================= Using Salt Cloud with the Event Reactor ======================================= One of the most powerful features of the Salt framework is the Event Reactor. As the Reactor was in development, Salt Cloud was regularly updated to take advantage of the Reactor upon completion. As such, various aspects of both the creation and destruction of instances with Salt Cloud fire events to the Salt Master, which can be used by the Event Reactor. Event Structure =============== As of this writing, all events in Salt Cloud have a tag, which includes the ID of the instance being managed, and a payload which describes the task that is currently being handled. A Salt Cloud tag looks like: .. code-block:: yaml salt/cloud// For instance, the first event fired when creating an instance named ``web1`` would look like: .. code-block:: yaml salt/cloud/web1/creating Assuming this instance is using the ``ec2-centos`` profile, which is in turn using the ``ec2-config`` provider, the payload for this tag would look like: .. code-block:: python {'name': 'web1', 'profile': 'ec2-centos', 'provider': 'ec2-config:ec2'} Available Events ================ When an instance is created in Salt Cloud, whether by map, profile, or directly through an API, a minimum of five events are normally fired. More may be available, depending upon the cloud provider being used. Some of the common events are described below. salt/cloud//creating ------------------------------- This event states simply that the process to create an instance has begun. At this point in time, no actual work has begun. The payload for this event includes: name profile provider salt/cloud//requesting --------------------------------- Salt Cloud is about to make a request to the cloud provider to create an instance. At this point, all of the variables required to make the request have been gathered, and the payload of the event will reflect those variables which do not normally pose a security risk. What is returned here is dependent upon the cloud provider. Some common variables are: name image size location salt/cloud//querying ------------------------------- The instance has been successfully requested, but the necessary information to log into the instance (such as IP address) is not yet available. This event marks the beginning of the process to wait for this information. The payload for this event normally only includes the ``instance_id``. salt/cloud//waiting_for_ssh -------------------------------------- The information required to log into the instance has been retrieved, but the instance is not necessarily ready to be accessed. Following this event, Salt Cloud will wait for the IP address to respond to a ping, then wait for the specified port (usually 22) to respond to a connection, and on Linux systems, for SSH to become available. Salt Cloud will attempt to issue the ``date`` command on the remote system, as a means to check for availability. If no ``ssh_username`` has been specified, a list of usernames (starting with ``root``) will be attempted. If one or more usernames was configured for ``ssh_username``, they will be added to the beginning of the list, in order. The payload for this event normally only includes the ``ip_address``. salt/cloud//deploying -------------------------------- The necessary port has been detected as available, and now Salt Cloud can log into the instance, upload any files used for deployment, and run the deploy script. Once the script has completed, Salt Cloud will log back into the instance and remove any remaining files. A number of variables are used to deploy instances, and the majority of these will be available in the payload. Any keys, passwords or other sensitive data will be scraped from the payload. Most of the variables returned will be related to the profile or provider config, and any default values that could have been changed in the profile or provider, but weren't. salt/cloud//created ------------------------------ The deploy sequence has completed, and the instance is now available, Salted, and ready for use. This event is the final task for Salt Cloud, before returning instance information to the user and exiting. The payload for this event contains little more than the initial ``creating`` event. This event is required in all cloud providers. Filtering Events ================ When creating a VM, it is possible with certain tags to filter how much information is sent to the event bus. The tags that can be filtered on any provider are: * ``salt/cloud//creating`` * ``salt/cloud//requesting`` * ``salt/cloud//created`` Other providers may allow other tags to be filtered; when that is the case, the documentation for that provider will contain more details. To filter information, create a section in your ``/etc/salt/cloud`` file called ``filter_events``. Create a section for each tag that you want to filter, using the last segment of the tag. For instance, use ``creating`` to represent ``salt/cloud//creating``: .. code-block:: yaml filter_events: creating: keys: - name - profile - provider Any keys listed here will be added to the default keys that are already set to be displayed for that provider. If you wish to start with a clean slate and only show the keys specified, add another option called ``use_defaults`` and set it to ``False``. .. code-block:: yaml filter_events: creating: keys: - name - profile - provider use_defaults: False Configuring the Event Reactor ============================= The Event Reactor is built into the Salt Master process, and as such is configured via the master configuration file. Normally this will be a YAML file located at ``/etc/salt/master``. Additionally, master configuration items can be stored, in YAML format, inside the ``/etc/salt/master.d/`` directory. These configuration items may be stored in either location; however, they may only be stored in one location. For organizational and security purposes, it may be best to create a single configuration file, which contains only Event Reactor configuration, at ``/etc/salt/master.d/reactor``. The Event Reactor uses a top-level configuration item called ``reactor``. This block contains a list of tags to be watched for, each of which also includes a list of ``sls`` files. For instance: .. code-block:: yaml reactor: - 'salt/minion/*/start': - '/srv/reactor/custom-reactor.sls' - 'salt/cloud/*/created': - '/srv/reactor/cloud-alert.sls' - 'salt/cloud/*/destroyed': - '/srv/reactor/cloud-destroy-alert.sls' The above configuration configures reactors for three different tags: one which is fired when a minion process has started and is available to receive commands, one which is fired when a cloud instance has been created, and one which is fired when a cloud instance is destroyed. Note that each tag contains a wildcard (``*``) in it. For each of these tags, this will normally refer to a ``minion_id``. This is not required of event tags, but is very common. Reactor SLS Files ================= Reactor ``sls`` files should be placed in the ``/srv/reactor/`` directory for consistency between environments, but this is not currently enforced by Salt. Reactor ``sls`` files follow a similar format to other ``sls`` files in Salt. By default they are written in YAML and can be templated using Jinja, but since they are processed through Salt's rendering system, any available renderer (JSON, Mako, Cheetah, etc.) can be used. As with other ``sls`` files, each stanza will start with a declaration ID, followed by the function to run, and then any arguments for that function. For example: .. code-block:: yaml # /srv/reactor/cloud-alert.sls new_instance_alert: cmd.pagerduty.create_event: - tgt: alertserver - kwarg: description: "New instance: {{ data['name'] }}" details: "New cloud instance created on {{ data['provider'] }}" service_key: 1626dead5ecafe46231e968eb1be29c4 profile: my-pagerduty-account When the Event Reactor receives an event notifying it that a new instance has been created, this ``sls`` will create a new incident in PagerDuty, using the configured PagerDuty account. The declaration ID in this example is ``new_instance_alert``. The function called is ``cmd.pagerduty.create_event``. The ``cmd`` portion of this function specifies that an execution module and function will be called, in this case, the ``pagerduty.create_event`` function. Because an execution module is specified, a target (``tgt``) must be specified on which to call the function. In this case, a minion called ``alertserver`` has been used. Any arguments passed through to the function are declared in the ``kwarg`` block. Example: Reactor-Based Highstate ================================ When Salt Cloud creates an instance, by default it will install the Salt Minion onto the instance, along with any specified minion configuration, and automatically accept that minion's keys on the master. One of the configuration options that can be specified is ``startup_states``, which is commonly set to ``highstate``. This will tell the minion to immediately apply a :ref:`highstate `, as soon as it is able to do so. This can present a problem with some system images on some cloud hosts. For instance, Salt Cloud can be configured to log in as either the ``root`` user, or a user with ``sudo`` access. While some hosts commonly use images that lock out remote ``root`` access and require a user with ``sudo`` privileges to log in (notably EC2, with their ``ec2-user`` login), most cloud hosts fall back to ``root`` as the default login on all images, including for operating systems (such as Ubuntu) which normally disallow remote ``root`` login. For users of these operating systems, it is understandable that a :ref:`highstate ` would include configuration to block remote ``root`` logins again. However, Salt Cloud may not have finished cleaning up its deployment files by the time the minion process has started, and kicked off a :ref:`highstate ` run. Users have reported errors from Salt Cloud getting locked out while trying to clean up after itself. The goal of a startup state may be achieved using the Event Reactor. Because a minion fires an event when it is able to receive commands, this event can effectively be used inside the reactor system instead. The following will point the reactor system to the right ``sls`` file: .. code-block:: yaml reactor: - 'salt/cloud/*/created': - '/srv/reactor/startup_highstate.sls' And the following ``sls`` file will start a :ref:`highstate ` run on the target minion: .. code-block:: yaml # /srv/reactor/startup_highstate.sls reactor_highstate: cmd.state.apply: - tgt: {{ data['name'] }} Because this event will not be fired until Salt Cloud has cleaned up after itself, the :ref:`highstate ` run will not step on salt-cloud's toes. And because every file on the minion is configurable, including ``/etc/salt/minion``, the ``startup_states`` can still be configured for future minion restarts, if desired.