events: Add hidden EID to all events tables (#3159)

docs/wiki/deployment/debugging.md

@@ -10,18 +10,10 @@ This is pretty simple! Just append `--verbose` as a switch.
 
 ```
 $ osqueryi --verbose
-I0119 16:38:03.113173 1965629440 init.cpp:278] osquery initialized [version=1.6.3]
-I0119 16:38:03.113536 1965629440 extensions.cpp:177] Could not autoload modules: Failed reading: /etc/osquery/modules.load
-I0119 16:38:03.132020 1064960 interface.cpp:246] Extension manager service starting: /Users/reed/.osquery/shell.em
-I0119 16:38:03.132203 1965629440 db_handle.cpp:165] Opening RocksDB handle: /Users/reed/.osquery/shell.db
-I0119 16:38:03.141836 1965629440 events.cpp:555] Event publisher failed setup: kernel: Cannot access /dev/osquery
-W0119 16:38:03.142004 1965629440 events.cpp:757] Error registering subscriber: process_file_events: No kernel event publisher
-I0119 16:38:03.143363 5844992 events.cpp:498] Starting event publisher run loop: diskarbitration
-I0119 16:38:03.143702 6381568 events.cpp:498] Starting event publisher run loop: fsevents
-I0119 16:38:03.145011 6918144 events.cpp:498] Starting event publisher run loop: iokit
-I0119 16:38:03.149258 7454720 events.cpp:498] Starting event publisher run loop: scnetwork
-osquery - being built, with love, at Facebook
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+I0412 08:04:56.012428 3056837568 init.cpp:380] osquery initialized [version=2.4.0]
+I0412 08:04:56.013499 3056837568 extensions.cpp:308] Could not autoload modules: Failed reading: /var/osquery/modules.load
+I0412 08:04:56.014837 168243200 interface.cpp:317] Extension manager service starting: /Users/$USER/.osquery/shell.em
+I0412 08:04:56.015383 3056837568 init.cpp:615] Error reading config: config file does not exist: /var/osquery/osquery.conf
 Using a virtual database. Need help, type '.help'
 osquery>
 ```
@@ -33,33 +25,23 @@ To see the daemon's verbose messages you'll need to run it in the foreground, se
 The daemon has some restrictions that make verbose debugging difficult, let's walk through how to run it in the foreground.
 
 ```
-$ osqueryd --pidfile /tmp/osquery.pid --database_path /tmp/osquery.db
+$ osqueryd --ephemeral --database_path /tmp/osquery.db
 ```
 
-The `pidfile` and `database_path` must be overridden as the defaults are not writable/readable by a non-privileged user. Now we can append `--verbose`:
+The `ephemeral` flag tells the daemon that it may co-exist with other persistent daemons. The `database_path` must be overridden as the defaults are not writable/readable by a non-privileged user. Now we can append `--verbose`:
 
 ```
-$ osqueryd --pidfile /tmp/osquery.pid --database_path /tmp/osquery.db --verbose
-I0119 16:45:17.785065 1965629440 init.cpp:278] osquery initialized [version=1.6.4-13]
-I0119 16:45:17.816946 1965629440 system.cpp:183] Found stale process for osqueryd (22391) removing pidfile
-I0119 16:45:17.818084 1965629440 system.cpp:218] Writing osqueryd pid (22406) to /tmp/osquery.pid
-I0119 16:45:17.820576 1965629440 extensions.cpp:170] Could not autoload extensions: Failed reading: /etc/osquery/extensions.load
-I0119 16:45:17.823276 528384 watcher.cpp:371] osqueryd watcher (22406) executing worker (22407)
-I0119 16:45:17.840364 1965629440 init.cpp:276] osquery worker initialized [watcher=22406]
-I0119 16:45:17.841305 1965629440 extensions.cpp:177] Could not autoload modules: Failed reading: /etc/osquery/modules.load
-I0119 16:45:17.847304 1965629440 db_handle.cpp:165] Opening RocksDB handle: /tmp/osquery.db
-Could not create log file: Permission denied
-COULD NOT CREATE LOGFILE '20160119'!
-I0119 16:45:17.857830 1965629440 events.cpp:555] Event publisher failed setup: kernel: Cannot access /dev/osquery
-W0119 16:45:17.857889 1965629440 events.cpp:757] Error registering subscriber: process_file_events: No kernel event publisher
-I0119 16:45:17.857990 1965629440 daemon.cpp:39] Not starting the distributed query service: Distributed query service not enabled.
-I0119 16:45:17.858032 3211264 events.cpp:498] Starting event publisher run loop: diskarbitration
-I0119 16:45:17.858038 3747840 events.cpp:498] Starting event publisher run loop: fsevents
-I0119 16:45:17.858070 4284416 events.cpp:498] Starting event publisher run loop: iokit
-I0119 16:45:17.858481 4820992 events.cpp:498] Starting event publisher run loop: scnetwork
+$ osqueryd --ephemeral --database_path /tmp/osquery.db --verbose
+I0412 08:03:59.664191 3056837568 init.cpp:380] osquery initialized [version=2.4.0]
+I0412 08:03:59.666533 196194304 watcher.cpp:465] osqueryd watcher (35549) executing worker (35550)
+I0412 08:03:59.688765 3056837568 init.cpp:377] osquery worker initialized [watcher=35549]
+I0412 08:03:59.689954 3056837568 extensions.cpp:308] Could not autoload modules: Failed reading: /var/osquery/modules.load
+I0412 08:03:59.690062 3056837568 rocksdb.cpp:205] Opening RocksDB handle: /tmp/osquery.db
 ```
 
-There are errors from Glog about logging permissions, to silence them make a directory and override `--logger_path`. Also note the the daemon wants you to execute it as the user who owns the binary if you attempt to run as a superuser. It also resists running in a tmpfs or sticky-bit directory.
+There may be errors from Glog about logging permissions, to silence them make a directory and override `--logger_path`, or use `--disable_logger`.
+
+Also note the daemon expects to be owned by the superuser if executed as the superuser. It also resists running in a tmpfs or sticky-bit directory. For special testing and debugging cases use `--allow_unsafe`.
 
 If you are using a `--flagfile` to define additional command line switches then it should be readable by your user. In cases where the Remote API is used, an enroll secret or TLS client private key is needed. If these are read-restricted to the superuser you may need to also debug as the superuser.
 
@@ -150,3 +132,14 @@ Error registering subscriber: process_file_events: No kernel event publisher
 ```
 
 This is an informational message with mis-categorized severity. The message indicates that a requested companion kernel extension does not exist and the associated `process_file_events` subscriber on OS X cannot start. It is safe to ignore.
+
+### Testing event subscribers
+
+Each event subscriber, tables that end with `_events`, includes a `HIDDEN` column called `eid`. This is an internal incrementing ID assigned by osquery to every event row added to a subscriber table. Each table maintains its own counter. The `eid` can be used to check for drops and duplicates occurring via an optimization or indexing bug.
+
+Consider the query:
+```
+SELECT *, eid FROM file_events;
+```
+
+If this query is in your schedule then the first `eid` should be `000000001` or similar. Each time the query runs the following should hold: `count(0) == max(eid) - min(eid)` and `min(eid) + 1 == max(eid from last run)`.

docs/wiki/deployment/file-integrity-monitoring.md

@@ -1,6 +1,6 @@
-As of osquery version 1.4.2, file integrity monitoring support was introduced for Linux (using inotify) and Darwin (using FSEvents) platforms.  This module reads a list of files/directories to monitor for changes from the osquery config and details changes and hashes to those selected files in the [`file_events`](https://osquery.io/docs/tables/#file_events) table.
+File integrity monitoring (FIM) is available for Linux and Darwin using inotify and FSEvents. The daemon reads a list of files/directories from the osquery configuration. The actions (and hashes when appropriate) to those selected files populate the [`file_events`](https://osquery.io/docs/tables/#file_events) table.
 
-To get started with FIM (file integrity monitoring), you must first identify which files and directories you wish to monitor. Then use *fnmatch*-style, or filesystem globbing, patterns to represent the target paths. You may use standard wildcards "*\**" or SQL-style wildcards "*%*":
+To get started with FIM, you must first identify which files and directories you wish to monitor. Then use *fnmatch*-style, or filesystem globbing, patterns to represent the target paths. You may use standard wildcards "*\**" or SQL-style wildcards "*%*":
 
 **Matching wildcard rules**
 
@@ -19,7 +19,7 @@ To get started with FIM (file integrity monitoring), you must first identify whi
 
 For example, you may want to monitor `/etc` along with other files on a Linux system. After you identify your target files and directories you wish to monitor, add them to a new section in the config *file_paths*.
 
-The two areas below that are relevant to FIM are the `file_events` and `file_paths` sections. The `file_events` query is scheduled to collect all of the FIM events that have occurred on any files within the paths specified within `file_paths` on a five minute interval.
+The two areas below that are relevant to FIM are the scheduled query against `file_events` and the added `file_paths` section. The `file_events` query is scheduled to collect all of the FIM events that have occurred on any files within the paths specified within `file_paths` on a five minute interval. At a high level this means events are buffered within osquery and sent to the configured _logger_ every five minutes.
 
 ## Example FIM Config
 
@@ -53,7 +53,7 @@ The two areas below that are relevant to FIM are the `file_events` and `file_pat
 
 ## Sample Event Output
 
-As file changes happen, events will appear in the [**file_events**](https://osquery.io/docs/tables/#file_events) table.  During a file change event, the md5, sha1, and sha256 for the file will be calculated if possible.  A sample event looks like this:
+As file changes happen, events will appear in the [**file_events**](https://osquery.io/docs/tables/#file_events) table.  During a file change event, the md5, sha1, and sha256 for the file will be calculated if possible. A sample event looks like this:
 
 ```json
 {

osquery/events/events.cpp

@@ -568,6 +568,7 @@ Status EventSubscriberPlugin::add(Row& r, EventTime event_time) {
   }
 
   r["time"] = std::to_string(event_time);
+  r["eid"] = eid;
   // Serialize and store the row data, for query-time retrieval.
   std::string data;
   auto status = serializeRowJSON(r, data);

specs/darwin/disk_events.table

@@ -16,6 +16,7 @@ schema([
     Column("filesystem", TEXT, "Filesystem if available"),
     Column("checksum", TEXT, "UDIF Master checksum if available (CRC32)"),
     Column("time", BIGINT, "Time of appearance/disappearance in UNIX time"),
+    Column("eid", TEXT, "Event ID", hidden=True),
 ])
 attributes(event_subscriber=True)
 implementation("events/darwin/disk_events@disk_events::genTable")

specs/darwin/process_file_events.table

@@ -19,6 +19,7 @@ schema([
     Column("ctime", BIGINT, "Time of last status change"),
     Column("time", BIGINT, "Time of event in UNIX epoch time"),
     Column("uptime", BIGINT, "Time of event in system uptime"),
+    Column("eid", TEXT, "Event ID", hidden=True),
 ])
 attributes(event_subscriber=True)
 implementation("process_file_events@process_file_events::genTable")

specs/linux/socket_events.table

@@ -16,6 +16,7 @@ schema([
     Column("socket", TEXT, "The local path (UNIX domain socket only)"),
     Column("time", BIGINT, "Time of execution in UNIX time"),
     Column("uptime", BIGINT, "Time of execution in system uptime"),
+    Column("eid", TEXT, "Event ID", hidden=True),
 ])
 attributes(event_subscriber=True)
 implementation("socket_events@socket_events::genTable")

specs/linux/syslog_events.table

@@ -7,6 +7,7 @@ schema([
     Column("facility", TEXT, "Syslog facility"),
     Column("tag", TEXT, "The syslog tag"),
     Column("message", TEXT, "The syslog message"),
+    Column("eid", TEXT, "Event ID", hidden=True),
 ])
 attributes(event_subscriber=True)
 implementation("syslog_events@SyslogEventSubscriber::genTable")

specs/linux/user_events.table

@@ -11,6 +11,7 @@ schema([
     Column("terminal", TEXT, "The network protocol ID"),
     Column("time", BIGINT, "Time of execution in UNIX time"),
     Column("uptime", BIGINT, "Time of execution in system uptime"),
+    Column("eid", TEXT, "Event ID", hidden=True),
 ])
 attributes(event_subscriber=True)
 implementation("user_events@user_events::genTable")

specs/posix/file_events.table

@@ -19,6 +19,7 @@ schema([
     Column("hashed", INTEGER,
       "1 if the file was hashed, 0 if not, -1 if hashing failed"),
     Column("time", BIGINT, "Time of file event"),
+    Column("eid", TEXT, "Event ID", hidden=True),
 ])
 attributes(event_subscriber=True)
 implementation("file_events@file_events::genTable")

specs/posix/hardware_events.table

@@ -12,6 +12,7 @@ schema([
     Column("serial", TEXT, "Device serial (optional)"),
     Column("revision", TEXT, "Device revision (optional)"),
     Column("time", BIGINT, "Time of hardware event"),
+    Column("eid", TEXT, "Event ID", hidden=True),
 ])
 attributes(event_subscriber=True)
 implementation("events/hardware_events@hardware_events::genTable")

specs/posix/process_events.table

@@ -31,6 +31,7 @@ schema([
     Column("parent", BIGINT, "Process parent's PID"),
     Column("time", BIGINT, "Time of execution in UNIX time"),
     Column("uptime", BIGINT, "Time of execution in system uptime"),
+    Column("eid", TEXT, "Event ID", hidden=True),
 ])
 attributes(event_subscriber=True)
 implementation("process_events@process_events::genTable")

specs/posix/yara_events.table

@@ -7,9 +7,10 @@ schema([
     Column("transaction_id", BIGINT, "ID used during bulk update"),
     Column("matches", TEXT, "List of YARA matches"),
     Column("count", INTEGER, "Number of YARA matches"),
-    Column("time", BIGINT, "Time of the scan"),
     Column("strings", TEXT, "Matching strings"),
     Column("tags", TEXT, "Matching tags"),
+    Column("time", BIGINT, "Time of the scan"),
+    Column("eid", TEXT, "Event ID", hidden=True),
 ])
 attributes(event_subscriber=True)
 implementation("yara@yara_events::genTable")

specs/windows/windows_events.table

@@ -11,6 +11,7 @@ schema([
     Column("level", INTEGER, "The severity level associated with the event"),
     Column("keywords", BIGINT, "A bitmask of the keywords defined in the event"),
     Column("data", TEXT, "Data associated with the event"),
+    Column("eid", TEXT, "Event ID", hidden=True),
 ])
 attributes(event_subscriber=True)
 implementation("windows_events@WindowsEventSubscriber::genTable")

tools/codegen/gentable.py

@@ -72,6 +72,7 @@ COLUMN_OPTIONS = {
     "additional": "ADDITIONAL",
     "required": "REQUIRED",
     "optimized": "OPTIMIZED",
+    "hidden": "HIDDEN",
 }
 
 # Column options that render tables uncacheable.