This will remove the use of current time for syslog.time and introduce
a new column called 'datetime'.
Events now uses an "optimize_id" alongside "optimize" to prevent returning
colliding events added within the same second as the previous genTable call.
This change causes Doxygen to error if there are unrecognized commands, or
undocumented/incorrect parameters in documented functions. It does not require
that every function be fully documented, just that those that are do not have
errors.
Old documentation with problems was fixed to comply.
This changes several initialization steps:
- The daemon (and shell, though not needed) have a new --ephemeral flag.
- Events are now disabled in the shell by default, use --nodisable_events to
re-enable.
- RocksDB-based backing storage is now disabled in the shell by default.
The --ephemeral flag for the daemon is disabled by default and will allow
skipping configuration and database path sanity, and skipping pidfile checks.
This is intended to be used when debugging or monitoring the daemon process.
To make the RocksDB backing storage feature usage very clear we introduce a new
flag: --disable_database. The shell sets this to true unless overridden in
a flagfile or via command line arguments.
Integrated process abstraction code into more locations
Defined new macros for abstracting across various platforms
Added GLOG_NO_ABBREVIATED_SEVERITIES for glog to support Windows
Fixed some minor CMake issues involving thrift
Updated gflags package; reflecting change in provision script
Preparing CMake config files for WIN32 support
This commit adds an event-based virtual table implementation for
querying the linux syslog. It introduces an event publisher that
attaches to a named pipe to ingest CSV formatted syslog forwarded from
rsyslogd. An event subscriber/virtual table makes these log lines
available for queries. Currently, no additional processing is done on
the input data besides parsing.
Using this table requires a properly configured rsyslogd. Documentation
for this configuration is forthcoming in the wiki.
The refactor of config/packs was initiated because event subscribers needed
a method for toggling `::init` based on some configurable option. In the case
of auditd, turning on the support with `--disable_audit=false` used to start
auditing the EXECVE syscall. It was understandable that this would cause
latency based on the number of processes executing per measure of time.
A new `socket_events` table will do the same but for `bind` and `connect`. These
are less-obvious and for now, require a scan of /proc for socket tuples. In the
future this file descriptor to socket tuple will be faster.
Previously, event publishers used a canonicalized 'type' name for async callbacks.
This type was used to lookup the publisher plugin in the registry as well as for backing store namespacing.
The type is still used but subscribers, which made heavy used of the lookup, store the value locally.
This prevents unneeded publisher plugin allocation when adding events.
Currently only for OS X, will port to others soon.
Also need to add tests.
Remove old comment and add loading message.
Implement YARA table for Linux.
Use mask properly.
Use the various masks to specify the kinds of events we are interested
in. This removes the need to do the dirty "DELETED" check when the event
fires.
Make getYARAFiles return a const map.
Switch to LOG(WARNING) and emit error number.
Add vim .swp files to .gitignore.
Add yara_utils.(c|h).
Start to condense common code between the Linux and Darwin YARA tables
into a yara_utils.h. Right now it includes a function to compile rules
and store the results back in the map, indexed by category. It also has
the callback used by YARA when a rule is processed. I can not move much
more than that for the row creation code because the structures used in
the event callback are slightly different.
Include a better error message.
The errors are still printed by the compiler callback, but this will
allow my future work to return a Status from the event initialization to
print a useful message in summary.
Make Subscriber init() return Status.
Each EventSubscriber::init() now returns a Status. If the init() fails
for any reason the EventSubscriber is still stored but the failure is
tracked.
EventSubscribers now have a state member, which represents the current
state of the subscriber. The current supported states are:
uninitialized, running, paused, failed. Currently the only meaningful
ones are running and failed, but I put paused in there as a
forward-looking feature.
Subscriptions now have a subscriber_name member. This is used in
EventPublisherPlugin::fire() as a lookup to get the EventSubscriber and
check the state. If the EventSubscriber is not running the event will
not fire.
Only the EventSubscribers on OS X are using this. I'll do the Linux
implementation next.
Chase the init() changes to Linux.
This brings the Linux YARA table in line with the OS X one.
Require a EventSubscriberID when creating a subscription.
Now that Subscriptions are "tied" to EventSubscribers you must create a
Subscription with the name of the Subscriber it is for. This is because
when the event fires the list of Subscriptions is walked and the name is
used to lookup the EventSubscriber and make sure it is in the running
state.
Fix various tests.
Some tests would fire an event with only a Subscription, which is no
longer a valid thing to do. For these tests an EventSubscription is
created and registered in the EventFactory.
When Subscriptions are created pass the name of the EventSubscriber to
them. In some cases where no event is ever fired it is fine to pass a
bogus name.
Fix inotify tests.
Move a test down so the class is defined and make sure to create an
EventSubscriber and use it properly.
Add support for yara to provision.sh.
Right now this grabs yara 3.3.0 and applies the patch to fix min() and max(),
which is commit fc4696c8b725be1ac099d340359c8d550d116041 in the yara repo.
This has been tested under Ubuntu 14.04 only.
Remove NOMINMAX.
This is no longer necessary after the patch was backported to 3.3.0.
Revert "Add support for yara to provision.sh."
This reverts commit a8bd371498c0979f070adeff23d05571882ac3f1.
Use vendored YARA code in third-party.
This switches to using the YARA code contained in third-party, including
the patch to fix min/max macros.
Fix mismerge.
Remove unused function after merge.
Well, soon to be unused as soon as I fix up the Linux YARA table. ;)
Chase config changes.
Make the Linux YARA table use ConfigDataInstance along with files() and
yaraFiles().
