1. Table implementations (spec files) can mark the table as 'cachable'.
2. Cached results depend on the shortest/quickest interval of scheduled
queries that act on results of the table.
3. The table API generator blocks caching on index/additional/required
table column options.
The refactor of config/packs was initiated because event subscribers needed
a method for toggling `::init` based on some configurable option. In the case
of auditd, turning on the support with `--disable_audit=false` used to start
auditing the EXECVE syscall. It was understandable that this would cause
latency based on the number of processes executing per measure of time.
A new `socket_events` table will do the same but for `bind` and `connect`. These
are less-obvious and for now, require a scan of /proc for socket tuples. In the
future this file descriptor to socket tuple will be faster.
There was a bug in the `osquery::Schedule` container object such that,
when the iteration through the schedule occured, pack objects were being
passed by value (copied) instead of passed by reference. Thus, the
discovery query would be executed, the object's cache would be updated,
and then the object would go out of scope and be destructed, thus
leaving the original object without ever having ran the discovery query.
This caused discovery queries to thrash. Bad times.
I added a new test so that we don't regress here as well as const'd a
few functions that should have been const in `osquery::Pack`.
Distributed tests were failing every now and then because the test
plugin didn't implement retry's and the test server wasn't always
starting up fast enough. I fixed this by refactoring the tests to use
the real TLS plugin, which has retry logic. This required some mangling
of the configuration options, which should serve as a good reference as
well.
This commit contains the features specified in #1390 as well as a
refactoring of the general osquery configuration code.
The API for the config plugins hasn't changed, although now there's a
`genPack` method that config plugins can implement. If a plugin doesn't
implement `genPack`, then the map<string, string> format cannot be used.
The default config plugin, the filesystem plugin, now implements
`genPack`, so existing query packs code will continue to work as it
always has.
Now many other config plugins can implement custom pack handling for
what makes sense in their context. `genPacks` is not a pure virtual, so
it doesn't have to be implemented in your plugin if you don't want to
use it. Also, more importantly, all config plugins can use the standard
inline pack format if they want to use query packs. Which is awesome.
For more information, refer to #1390, the documentation and the doxygen
comments included with this pull requests, as well as the following
example config which is now supported, regardless of what config plugin
you're using:
```json
{
"options": {
"enable_monitor": "true"
},
"packs": {
"core_os_monitoring": {
"version": "1.4.5",
"discovery": [
"select pid from processes where name like '%osqueryd%';"
],
"queries": {
"kernel_modules": {
"query": "SELECT name, size FROM kernel_modules;",
"interval": 600
},
"system_controls": {
"query": "SELECT * FROM system_controls;",
"interval": 600,
"snapshot": true,
},
"usb_devices": {
"query": "SELECT * FROM usb_devices;",
"interval": 600
}
}
},
"osquery_internal_info": {
"version": "1.4.5",
"discovery": [
"select pid from processes where name like '%osqueryd%';"
],
"queries": {
"info": {
"query": "select i.*, p.resident_size, p.user_time, p.system_time, time.minutes as counter from osquery_info i, processes p, time where p.pid = i.pid;",
"interval": 60,
"snapshot": true
},
"registry": {
"query": "SELECT * FROM osquery_registry;",
"interval": 600,
"snapshot": true
},
"schedule": {
"query": "select name, interval, executions, output_size, wall_time, (user_time/executions) as avg_user_time, (system_time/executions) as avg_system_time, average_memory from osquery_schedule;",
"interval": 60,
"snapshot": true
}
}
}
}
}
```
The `osquery_packs` table was modified to remove the superfluous
columns which could already have been found in `osquery_schedule`. Two
more columns were added in their place, representing stats about pack's
discovery query execution history.
Notably, the internal API for the `osquery::Config` class has changed
rather dramatically as apart of the refactoring. We think this is an
improvement. While strictly adhering to the osquery config plugin
interface will have avoided any compatibility errors, advanced users may
notice compilation errors if they access config data directly. All
internal users of the config have obviously been updated. Yet another
reason to merge your code into mainline; we update it for you when we
refactor!
Previously, event publishers used a canonicalized 'type' name for async callbacks.
This type was used to lookup the publisher plugin in the registry as well as for backing store namespacing.
The type is still used but subscribers, which made heavy used of the lookup, store the value locally.
This prevents unneeded publisher plugin allocation when adding events.
Previously, `ConfigParserPlugin`s could only maintain an internal derived object called `data_`.
Then parts of the code that knew to use the plugin's data would call `getParsedData` and provide the name of the plugin.
Parser plugins can now request a mutable version of the `ConfigData` using `::mutableConfigData`.
This requires a lock on the `ConfigDataInstance` and must be provided to their mutable accessor.
Acess to a mutable config enables parsers to make modifications to internal config structures like options and the query schedule.
There are 3 new options that control how files are read:
--read_max: controls the maximum size, in bytes, for file reads. If a file is larger than `read_max` the read will fail.
--read_user_max: similar to `read_max` but applies additional limitations to user-controlled files.
--read_user_links: a boolean control to enable/disable following symlinks for user-controlled files.
Important highlights:
If files exceed the configured max, those reads will fail.
The `read_max` will override `read_user_max` if it is set lower.
A default integer value of `0` will disable the limitations.
The default `read_max` is set to 50M and the default `read_user_max` is 10M.
POSIX-globbing will allow event publishers/subscribers to post-check
results against glob-syntax, fnpath matching, and POSIX C-regex.
These checks are anecdotally speedy.
