[#2542] Introduce --enable_syslog to explicit enable syslog ingestion (#2543)

docs/wiki/deployment/syslog.md

@@ -1,6 +1,6 @@
-osquery 1.7.3 introduced support for consuming and querying the Mac OSX system log via Apple System Log (ASL). osquery 1.7.4 introduced support for the Linux syslog via rsyslog. This document explains how to configure and use these syslog tables.
+osquery 1.7.3 introduced support for consuming and querying the Mac OSX system log via Apple System Log (ASL). osquery 1.7.4 introduced support for the Linux syslog via **rsyslog**. This document explains how to configure and use these syslog tables.
 
-## OSX Syslog
+## OS X Syslog
 
 On Mac OSX, the `asl` virtual table makes use of Apple's ASL store, querying this structured store using the routines provided in [`asl.h`](https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man3/asl.3.html).
 
@@ -24,17 +24,19 @@ select time, message from asl where facility = 'authpriv' and sender = 'sudo' an
 
 ## Linux Syslog
 
-On linux, the `syslog` table queries logs forwarded over a named pipe from a properly configured `rsyslogd`. This method was chosen to support the widest range of linux flavors (in theory, anything running at least `ryslogd` version 5, and tested with Ubuntu 12/14, centos 7.1, RHEL 7.2), and to ensure that existing syslog routines and configurations are not modified. As syslog is ingested into osquery, it is written into the backing store (RocksDB) and made available for querying.
+On Linux, the `syslog` table queries logs forwarded over a named pipe from a properly configured **rsyslogd**. This method was chosen to support the widest range of Linux flavors (in theory, anything running at least **rsyslogd** version 5, and tested with Ubuntu 12/14, CentOS 7.1, RHEL 7.2), and to ensure that existing syslog routines and configurations are not modified. As syslog is ingested into osquery, it is written into the backing store (RocksDB) and made available for querying.
+
+Note: the Syslog ingestion is NOT recommended for hosts functioning as syslog aggregators. We have not tested ingestion for massive-throughput or lossless setups.
 
 ### Configuration
 
-The `syslog` table requires additional configuration before it can be used.
+The `syslog` table requires additional configuration before it can be used. Append `--enable_syslog` to your command line arguments or `--flagfile` to enable osquery's `syslog` event publisher thread.
 
-When an osquery process that supports the `syslog` table starts up, it will attempt to create (and properly set permissions for) a named pipe for `rsyslogd` to write to. The path for this pipe is determined by the configuration flag `syslog_pipe_path` (defaults to `/var/osquery/syslog_pipe`). If verbose logging is turned on, you should see a status message indicating whether osquery was able to successfully open the pipe for reading.
+When an osquery process that supports the `syslog` table starts up, it will attempt to create (and properly set permissions for) a named pipe for **rsyslogd** to write to. The path for this pipe is determined by the configuration flag `--syslog_pipe_path` (defaults to `/var/osquery/syslog_pipe`). If verbose logging is turned on, you should see a status message indicating whether osquery was able to successfully open the pipe for reading.
 
-Permissions for the pipe must at least allow `rsyslogd` to read/write, and osquery to read. For security, it is advised that the least possible privileges are enabled to allow this.
+Permissions for the pipe must at least allow **rsyslogd** to read/write, and osquery to read. For security, it is advised that the least possible privileges are enabled to allow this.
 
-Once the named pipe is created, `rsyslogd` must be configured to write logs to the pipe. Add the following to your `rsyslog` configuration files (usually located in `/etc/rsyslog.conf` or `/etc/rsyslog.d/`):
+Once the named pipe is created, **rsyslogd** must be configured to write logs to the pipe. Add the following to your **rsyslog** configuration files (usually located in `/etc/rsyslog.conf` or `/etc/rsyslog.d/`):
 
 #### rsyslog versions < 7
 
@@ -45,7 +47,7 @@ $template OsqueryCsvFormat, "%timestamp:::date-rfc3339,csv%,%hostname:::csv%,%sy
 
 #### rsyslog versions >= 7
 
-Note: the above configuration should also work, but `rsyslog` strongly recommends using the new style configuration syntax.
+Note: the above configuration should also work, but **rsyslog** strongly recommends using the new style configuration syntax.
 
 ```
 template(
@@ -58,13 +60,13 @@ template(
 
 #### All versions
 
-`rsyslogd` must be restarted for the changes to take effect. On many systems, this can be achieved by `sudo service rsyslog restart`.
+**rsyslogd** must be restarted for the changes to take effect. On many systems, this can be achieved by `sudo service rsyslog restart`.
 
-Note: `rsyslogd` will only check once, at startup, whether it can write to the pipe. If `rsyslogd` cannot write to the pipe, it will not retry until restart.
+Note: **rsyslogd** will only check once, at startup, whether it can write to the pipe. If **rsyslogd** cannot write to the pipe, it will not retry until restart.
 
 #### Other configuration
 
-Configuration flags control the retention of syslog logs. `syslog_events_expiry` (default 30 days) defines how long (in seconds) to keep logs. `syslog_events_max` (default 100,000) sets a maximum number of logs to retain (oldest logs are deleted first if this number is surpassed).
+Configuration flags control the retention of syslog logs. `--syslog_events_expiry` (default 30 days) defines how long (in seconds) to keep logs. `--syslog_events_max` (default 100,000) sets a maximum number of logs to retain (oldest logs are deleted first if this number is surpassed).
 
 ### Usage
 

docs/wiki/installation/cli-flags.md

@@ -336,13 +336,17 @@ In seconds, the amount of time that osqueryd will wait between periodically chec
 
 There is a `syslog` virtual table that uses Events and a **rsyslog** configuration to capture results *from* syslog. Please see the [Syslog Consumption](../deployment/syslog.md) deployment page for more information.
 
+`--enable_syslog=false`
+
+Turn on the syslog ingestion event publisher. This is an 'explicit'-enable because it requires external configuration of **rsyslog**.
+
 `--syslog_pipe_path=/var/osquery/syslog_pipe`
 
 Path to the named pipe used for forwarding **rsyslog** events.
 
 `--syslog_rate_limit=100`
 
-Maximum number of logs to ingest per run (~100ms between runs). Use this as a fail-safe to prevent osquery from becoming overloaded when syslog is spammed.
+Maximum number of logs to ingest per run (~200ms between runs). Use this as a fail-safe to prevent osquery from becoming overloaded when syslog is spammed.
 
 ## Shell-only flags
 

osquery/events/events.cpp

@@ -582,7 +582,7 @@ Status EventFactory::run(EventPublisherID& type_id) {
     // This is a 'default' cool-off implemented in InterruptableRunnable.
     // If a publisher fails to perform some sort of interruption point, this
     // prevents the thread from thrashing through exiting checks.
-    publisher->pause();
+    publisher->pauseMilli(200);
   }
   if (!status.ok()) {
     // The runloop status is not reflective of the event type's.

osquery/events/linux/syslog.cpp

@@ -35,6 +35,8 @@ namespace pt = boost::property_tree;
 
 namespace osquery {
 
+FLAG(bool, enable_syslog, false, "Enable the syslog ingestion event publisher");
+
 FLAG(string,
      syslog_pipe_path,
      "/var/osquery/syslog_pipe",
@@ -43,7 +45,7 @@ FLAG(string,
 FLAG(uint64,
      syslog_rate_limit,
      100,
-     "Maximum number of logs to ingest per run (~100ms between runs)");
+     "Maximum number of logs to ingest per run (~200ms between runs)");
 
 REGISTER(SyslogEventPublisher, "event_publisher", "syslog");
 
@@ -56,6 +58,10 @@ const std::vector<std::string> kCsvFields = {
 const size_t kErrorThreshold = 10;
 
 Status SyslogEventPublisher::setUp() {
+  if (!FLAGS_enable_syslog) {
+    return Status(1, "Publisher disabled via configuration");
+  }
+
   Status s;
   if (!pathExists(FLAGS_syslog_pipe_path)) {
     VLOG(1) << "Pipe does not exist: creating pipe " << FLAGS_syslog_pipe_path;