This is fairly important, it changes the default mode for what status
events are logged to INFO. It had been set to WARNING and INFO was relatively
unused.
This also removes expected support for RocksDB 'in-memory' databases.
If a shell-user requests a database via CLI flags it will now work.
This commit fixes two issues with `path` in the linux processes table:
(1) Fixes a bug in which `on_disk` is set to `NULL` instead of `0` when the
binary is not on disk.
(2) Fixes a bug in which a filename ending in ` (deleted)` could cause osquery
to return an incorrect value for `on_disk`. See
https://github.com/facebook/osquery/issues/1607
This includes a minor SDK refactor as it move quite a few specialized
functions and facilities from core.h into system.h. There was a breaking point
for needing to frequently update core includes.
The new logger systemLog function allows a call site to bypass logging config
and write a line to the OS logger (aka syslog).
Include table changes necessary for a Windows processes table and changes to other tables needed for daemon and shell to run. The Windows processes table uses WMI as a backend to gather information. This commit does not yet build these tables.
This commit bumps the third-party SQLite to the 3.14.0 pre-release (18:59).
With 3.14.0 the LIKE and EQUALS constraint operators may be mixed within a
query. Previously these would fail to produce a valid set.
As part of the support, each virtual table should choose to bypass rowid-based
deduplication using the new "WITHOUT ROWID" create table epilog. This will
be appended to the schema if the table defines a PRIMARY KEY using index=True.
This changes several initialization steps:
- The daemon (and shell, though not needed) have a new --ephemeral flag.
- Events are now disabled in the shell by default, use --nodisable_events to
re-enable.
- RocksDB-based backing storage is now disabled in the shell by default.
The --ephemeral flag for the daemon is disabled by default and will allow
skipping configuration and database path sanity, and skipping pidfile checks.
This is intended to be used when debugging or monitoring the daemon process.
To make the RocksDB backing storage feature usage very clear we introduce a new
flag: --disable_database. The shell sets this to true unless overridden in
a flagfile or via command line arguments.
Integrated process abstraction code into more locations
Defined new macros for abstracting across various platforms
Added GLOG_NO_ABBREVIATED_SEVERITIES for glog to support Windows
Fixed some minor CMake issues involving thrift
Updated gflags package; reflecting change in provision script
Preparing CMake config files for WIN32 support
The OS X kernel subscribers have not been starting because they expect the
publisher thread to run before they begin configuration. Due to some recent
refactors the publisher thread creation now occurs after configuration.
The subscriber logic to check for a valid kernel connection is still valid.
This commit has two additional side-effects:
- The RocksDB plugin is modified to use 3 background merge threads.
- The OS X kernel publisher syncing thread is now non-blocking.
The mobile 'LowBattery' crash logs provide little if any data of value,
and seem to be nothing more than notification events. We're filtering
these out of the crashes table.
This adds a new optimization feature that allows expensive tables to cache
their results between JOINs. Consider JOINing a list of open sockets, for each
process, then requesting to hash each process path. This query may hash
the same path multiple times.
Within-query caching allows the hash table to respond with the previous
result of the hash request as long as the requested computation was the result
of a single query. Subsequent queries will perform subsequent hashing.
This commit adds mobile device crashes to the list of crash logs parsed by the Crashes table as well as adding a lamdba to improve code reuse. The commit also adds a 'type' column to the table to indicate what kind of log this crash log was.
Renamed the crash_log table to crashes for future abstraction to other
operating systems. Also fixed how the table was parsing the most recent
stack trace and the registers. Register values are now all parsed into
one column 'registers', which will be a space delimited string of the
form:
register:value register:value ... register:value
in order to best allow for OS abstraction.
It is often helpful to know the local timezone of the machine. For this use
local_timezone, as the base timezone will use local or UTC depending on the
--utc flag. This will be default=UTC in osquery 1.8.0.
The datetime field is added to mimic ISO 8601, along with iso_8601.
The timestamp field remains as the time stamp used for logging (within osquery)
and commonly outside of osquery. The goal for adding multiple representations
is to allow joining/augmenting of other tables.
Beginning in version 1.8.0 all time uses will converge on an osquery-provided
getUnixTime() API call that returns, by default, UNIX time integers converted
to UTC/GMT. The 'time' table will respond with the parsed time for the
configuration. If the timezone is not UTC then osquery is using localtime.
This configuration option will affect the 'unix_time' response in the 'time'
table. Because of this configurable-effect the table is extended to include
'local_time' which is always the system local UNIX time.
Added a table that parses out some of the informaiton in the OS X logs
stored in /Library/Logs/DiagnosticReports as well as
/Users/<user>/Library/Logs/DiagnosticReports
This commit adds an event-based virtual table implementation for
querying the linux syslog. It introduces an event publisher that
attaches to a named pipe to ingest CSV formatted syslog forwarded from
rsyslogd. An event subscriber/virtual table makes these log lines
available for queries. Currently, no additional processing is done on
the input data besides parsing.
Using this table requires a properly configured rsyslogd. Documentation
for this configuration is forthcoming in the wiki.
Previously, this test relied on there being console messages available
to query for. Now, it actually writes a message to ASL, then verifies
that the message can be found. It also exercises a more complicated
query than previously. Also remove potentially high latency queries.
Fixes issue #1975
1. Update boost to 1.60 from 1.55 on Linux platforms
2. Add asio (1.11.0) to the deps set
3. Update snappy to 1.1.3 on Linux platforms
4. Update cpp-netlib to 0.12.0-rc1 from 0.11 on Linux platforms
- OS X and brew also include 0.12.0-rc1 as a devel option
5. Update libapt to 1.2.6 from 0.8.6 on Ubuntu/Debian
- This adds lzma as a dependent link
Process states in `processes` table now use mnemonics instead of the
integer code. They follow closely to the states defined sys/proc.h,
which means processes with state as:
state value meaning mnemonic
-------- ------- --------------------------------- ----------
SIDL 1 process being created by fork 'I'
SRUN 2 currently runnable 'R'
SSLEEP 3 sleeping on an address 'S'
SSTOP 4 process debugging or suspension 'T'
SZOMB 5 awaiting collection by parent 'Z'
Note: The mnemonics here will not match up with what `ps` or `top`
reports.
`ps` and `top` queries run state of all Mach Threads (by grabbing a Mach Task)
of a process, and then coalesces it to represent the process state.
This is no longer possible to do by non-Apple signed binaries,
as the only way to a Mach Task (and therefore list of Mach Threads)
is by calling `task_for_pid()`, which no longer works under
System Integrity Protection (SIP) introduced in OS X 10.11
2. Introduce a SQLite-based database plugin
3. Refactor database usage to include local 'fast-calls'
4. Introduce an 'ephemeral' database plugin for testing (like a mock)
Lumping parsing of smc fpe2 values in with the else case, which is also
used for parsing sp78 values, seems to have it show correct values that
match those outputted by `./smc -l | grep -i fpe2` via the smc-fuzzer
tool.
