The refactor of config/packs was initiated because event subscribers needed
a method for toggling `::init` based on some configurable option. In the case
of auditd, turning on the support with `--disable_audit=false` used to start
auditing the EXECVE syscall. It was understandable that this would cause
latency based on the number of processes executing per measure of time.
A new `socket_events` table will do the same but for `bind` and `connect`. These
are less-obvious and for now, require a scan of /proc for socket tuples. In the
future this file descriptor to socket tuple will be faster.
This commit contains the features specified in #1390 as well as a
refactoring of the general osquery configuration code.
The API for the config plugins hasn't changed, although now there's a
`genPack` method that config plugins can implement. If a plugin doesn't
implement `genPack`, then the map<string, string> format cannot be used.
The default config plugin, the filesystem plugin, now implements
`genPack`, so existing query packs code will continue to work as it
always has.
Now many other config plugins can implement custom pack handling for
what makes sense in their context. `genPacks` is not a pure virtual, so
it doesn't have to be implemented in your plugin if you don't want to
use it. Also, more importantly, all config plugins can use the standard
inline pack format if they want to use query packs. Which is awesome.
For more information, refer to #1390, the documentation and the doxygen
comments included with this pull requests, as well as the following
example config which is now supported, regardless of what config plugin
you're using:
```json
{
"options": {
"enable_monitor": "true"
},
"packs": {
"core_os_monitoring": {
"version": "1.4.5",
"discovery": [
"select pid from processes where name like '%osqueryd%';"
],
"queries": {
"kernel_modules": {
"query": "SELECT name, size FROM kernel_modules;",
"interval": 600
},
"system_controls": {
"query": "SELECT * FROM system_controls;",
"interval": 600,
"snapshot": true,
},
"usb_devices": {
"query": "SELECT * FROM usb_devices;",
"interval": 600
}
}
},
"osquery_internal_info": {
"version": "1.4.5",
"discovery": [
"select pid from processes where name like '%osqueryd%';"
],
"queries": {
"info": {
"query": "select i.*, p.resident_size, p.user_time, p.system_time, time.minutes as counter from osquery_info i, processes p, time where p.pid = i.pid;",
"interval": 60,
"snapshot": true
},
"registry": {
"query": "SELECT * FROM osquery_registry;",
"interval": 600,
"snapshot": true
},
"schedule": {
"query": "select name, interval, executions, output_size, wall_time, (user_time/executions) as avg_user_time, (system_time/executions) as avg_system_time, average_memory from osquery_schedule;",
"interval": 60,
"snapshot": true
}
}
}
}
}
```
The `osquery_packs` table was modified to remove the superfluous
columns which could already have been found in `osquery_schedule`. Two
more columns were added in their place, representing stats about pack's
discovery query execution history.
Notably, the internal API for the `osquery::Config` class has changed
rather dramatically as apart of the refactoring. We think this is an
improvement. While strictly adhering to the osquery config plugin
interface will have avoided any compatibility errors, advanced users may
notice compilation errors if they access config data directly. All
internal users of the config have obviously been updated. Yet another
reason to merge your code into mainline; we update it for you when we
refactor!
When strings match they will be populated into the "strings" column of
the table. The format is identifier:offset.
When a matching rule has tags defined the tags will be put into the
"tags" column of the table in a comma separated list.
We now have a Publisher to report on disk events and its metadata,
using the DiskArbitration framework on OS X. Currently disk appearance
and disappearance events are published for both physical and
virtual disks (DMG files). On an event trigger, disk properties are
parsed and that metadata is reported along with the action.
The Subscriber subscribes to virtual disk events currently.
This closes#1103.
This implements the following tables on FreeBSD:
process_envs
process_memory_map
process_open_files
process_open_sockets
processes
All the heavy lifting is done with libprocstat(3). All the tables follow
the same general principle. Use the common function, getProcesses() in
procstat.cpp, to get the processes and then generate the rows for each
process returned. There is also a procstatCleanup() function commonly
used across all the tables.
The one thing I am not able to test is the process_open_sockets table on
an IPv6 machine.
This was causing a crash when executing a query using the yara table
from the command line, because YARA was never initialized properly, so
the thread index was whatever was left on the stack. Eventually YARA
would attempt to set a rule that matches using this thread index and
would explode in flames.
Fix it by moving the initialization to a place that is always called.
1. Rename yara_matches to yara_events.
2. Add support for Config::getParser().
- This returns a ConfigPluginRef, which is the ConfigParser for the
given key.
- Being able to get the parser is useful because the
YARAConfigParserPlugin uses it to store the compiled rules as an
attribute.
3. Finish rename and use ConfigParserPlugin.
- Finish the table rename to yara_events.
- Use the new ConfigParserPlugin interface to parse the YARA
configuration. The file_paths and signatures are stored in the
ConfigParserPlugin named "yara" under the key "yara". The rules are
compiled and stored as a private attribute of the same
ConfigParserPlugin object.
Here is an example config using this new structure:
{
// Description of the YARA feature.
"yara": {
"signatures": {
// Each key is an arbitrary group name to give the signatures listed
"sig_group_1": [ "/Users/wxs/foo.sig", "/Users/wxs//bar.sig" ],
"sig_group_2": [ "/Users/wxs/baz.sig" ]
},
"file_paths": {
// Each key is a key from file_paths
// The value is a list of signature groups to run when an event fires
// These will be watched for and scanned when the event framework
// fire off an event to yara_events table
"system_binaries": [ "sig_group_1" ],
"tmp": [ "sig_group_1", "sig_group_2" ]
}
},
// Paths to watch for filesystem events
"file_paths": {
"system_binaries": [ "/usr/bin/%", "/usr/sbin/%" ],
"tmp": [ "/Users/wxs/tmp/%%" ]
}
}
- Currently the signature file must be an absolute path.
3. Move common YARA code to yara_utils.
- In preparation for the yara table (different from yara_events) I'm
moving the common YARA code into a separate place which is shared
between the two tables.
4. Add yara table.
- This allows you to do things like:
```sql
select * from yara where path="/bin/ls" and sigfile="/tmp/foo.sig";
select * from yara where path="/bin/ls" and sig_group="sig_group_1";
```
- The latter will use the signature grouping from the config.
5. Check for keys not existing.
After the merge you need to include yara.h after fsevents.h on OS X.
Apaprently DECLARE_SUBSCRIBER() was dropped during the merge, which
meant that attempts to find the subscriber would fail, throwing an
unhandled exception.
Fix the table spec to use the new table name.
Must use recursive on Linux.
In order to watch a directory you must set recursive to true in
SubscriptionContext under Linux.
