This will remove the use of current time for syslog.time and introduce
a new column called 'datetime'.
Events now uses an "optimize_id" alongside "optimize" to prevent returning
colliding events added within the same second as the previous genTable call.
This is fairly important, it changes the default mode for what status
events are logged to INFO. It had been set to WARNING and INFO was relatively
unused.
This also removes expected support for RocksDB 'in-memory' databases.
If a shell-user requests a database via CLI flags it will now work.
This improves dispatcher tests by allowing units to act like component
tests and use embedded std::thread-based osquery APIs. A unit may force
a 'service' to run by joining the Dispatcher before deconstructing.
Along with the platform defines and platform string defines provided by
CMake to the build, add a PLATFORM_MASK define.
Use this define as a platform-type mask with the PlatformType enum.
Missing initialization of the BufferedLogForwarder was causing an underflow in
the count of buffered logs, and error messages as described in #2324. This
commit brings the initialization of the forwarder for TLS in line with
aws_kinesis and aws_firehose, removing that error.
Prior to this change, both --disable_database=false and --database_path had to
be specified together. Now, if the user specifies --database_path the database
is enabled automatically.
This commit fixes two issues with `path` in the linux processes table:
(1) Fixes a bug in which `on_disk` is set to `NULL` instead of `0` when the
binary is not on disk.
(2) Fixes a bug in which a filename ending in ` (deleted)` could cause osquery
to return an incorrect value for `on_disk`. See
https://github.com/facebook/osquery/issues/1607
This includes a minor SDK refactor as it move quite a few specialized
functions and facilities from core.h into system.h. There was a breaking point
for needing to frequently update core includes.
The new logger systemLog function allows a call site to bypass logging config
and write a line to the OS logger (aka syslog).
RocksDB is the default "database" plugin. Writes are normally kept in an
in-memory memtable. Writes that are not part of the event pubsub system can
be forced to sync to disk.
This test was intermittently failing because it relies on the actual thread
scheduling. Our discussion in issue #2218 decided that it was worth keeping the
test around, while trying to mitigate the flakiness. The longer sleeps in this
test ran successfully hundreds of times in local testing.
This alters the osquery.thrift spec to add a ::shutdown method to the
Extension class. The ExtensionManager inherits from this but includes a
no-op shutdown method.
When an ExtensionManager (osquery core) stops, it optionally requests all
Extensions to shutdown immediately. This helps quit extensions processes
faster.
Include table changes necessary for a Windows processes table and changes to other tables needed for daemon and shell to run. The Windows processes table uses WMI as a backend to gather information. This commit does not yet build these tables.
* Integrated filesystem operation abstraction code into filesystem.cpp
* Modified filesystem unit tests to be more platform agnostic
* Added append mode for PlatformFile
* Minor bug fixes in filesystem operations
This commit bumps the third-party SQLite to the 3.14.0 pre-release (18:59).
With 3.14.0 the LIKE and EQUALS constraint operators may be mixed within a
query. Previously these would fail to produce a valid set.
As part of the support, each virtual table should choose to bypass rowid-based
deduplication using the new "WITHOUT ROWID" create table epilog. This will
be appended to the schema if the table defines a PRIMARY KEY using index=True.
Some of the types in fileops tests were causing warnings to be thrown
during build, due to type mismatch. I've added a few local variables to
quiet these warnings.
* Implemented filesystem operations abstraction code
* Added filesystem operations abstraction unit tests
* Modified CMake configurations to support the building of the abstraction code and unit tests
The three new SQLite functions:
- split: Splits a column using a set of tokens and a selected index.
- regex_split: Similar to split but with a regex instead of tokens.
- inet_aton: Returns the IPv4 decimal value for a string-formatted address.
With a removeService method, combined with the abstracted thread start in
the Dispatcher API, services auto-remove when finished.
This will un-break the kernel communication tests. These tests only stop
when all their producer threads/services have ended.
This also promotes the OS X kernel build to 10.11.
This changes several initialization steps:
- The daemon (and shell, though not needed) have a new --ephemeral flag.
- Events are now disabled in the shell by default, use --nodisable_events to
re-enable.
- RocksDB-based backing storage is now disabled in the shell by default.
The --ephemeral flag for the daemon is disabled by default and will allow
skipping configuration and database path sanity, and skipping pidfile checks.
This is intended to be used when debugging or monitoring the daemon process.
To make the RocksDB backing storage feature usage very clear we introduce a new
flag: --disable_database. The shell sets this to true unless overridden in
a flagfile or via command line arguments.
Table options includes a change to the Registry::call API for TablePlugins.
When requesting route information or the 'columns' action, a new 'op' key is included.
Integrated process abstraction code into more locations
Defined new macros for abstracting across various platforms
Added GLOG_NO_ABBREVIATED_SEVERITIES for glog to support Windows
Fixed some minor CMake issues involving thrift
Updated gflags package; reflecting change in provision script
Preparing CMake config files for WIN32 support
This introduces a new testing-related flag to the osqueryi shell.
The --pack flag takes a single argument, the name of a configured pack.
To list the configured packs try:
$ osqueryi -A osquery_packs
To run all queries, immediately and serially, try:
$ osqueryi --pack NAME_OF_PACK
When selecting UUID as the host identifier the client follows:
- If host_identifier is set to hostname, return the hostname
- If UUID check a static UUID container
- If empty request the UUID from the backing storage, RocksDB
- If empty use the Apple hardware UUID or generate a UUID
- If the Apple UUID request fails generate a UUID
When no UUID is available in the backing store save the new UUID.
The Thrift library should be discovered simiarly to all others.
When using the build-macros for library discovery the caller can choose
how to discover static or dynamic links.
Added Windows support for process operations.
Added unit tests for process abstraction code for POSIX and Windows.
Modified CMake config files to support building the new code and unit tests.
The initialization of a logger plugin was confusing. The 'init' step was
introduced to allow a daemon to buffer status events before a logger plugin
is determined by external/remote configuration. The buffered statuses could
then be transferred via a medium other than Glog (the default). To determine
if Glog should continue to write statuses to the filesystem the 'init' method
returned a Status.
Logger plugins should now use a feature method override to select how status
logs should be handled.
The OS X kernel subscribers have not been starting because they expect the
publisher thread to run before they begin configuration. Due to some recent
refactors the publisher thread creation now occurs after configuration.
The subscriber logic to check for a valid kernel connection is still valid.
This commit has two additional side-effects:
- The RocksDB plugin is modified to use 3 background merge threads.
- The OS X kernel publisher syncing thread is now non-blocking.
The mobile 'LowBattery' crash logs provide little if any data of value,
and seem to be nothing more than notification events. We're filtering
these out of the crashes table.
This adds a new optimization feature that allows expensive tables to cache
their results between JOINs. Consider JOINing a list of open sockets, for each
process, then requesting to hash each process path. This query may hash
the same path multiple times.
Within-query caching allows the hash table to respond with the previous
result of the hash request as long as the requested computation was the result
of a single query. Subsequent queries will perform subsequent hashing.
This commit adds mobile device crashes to the list of crash logs parsed by the Crashes table as well as adding a lamdba to improve code reuse. The commit also adds a 'type' column to the table to indicate what kind of log this crash log was.
Renamed the crash_log table to crashes for future abstraction to other
operating systems. Also fixed how the table was parsing the most recent
stack trace and the registers. Register values are now all parsed into
one column 'registers', which will be a space delimited string of the
form:
register:value register:value ... register:value
in order to best allow for OS abstraction.
It is often helpful to know the local timezone of the machine. For this use
local_timezone, as the base timezone will use local or UTC depending on the
--utc flag. This will be default=UTC in osquery 1.8.0.
The datetime field is added to mimic ISO 8601, along with iso_8601.
The timestamp field remains as the time stamp used for logging (within osquery)
and commonly outside of osquery. The goal for adding multiple representations
is to allow joining/augmenting of other tables.
Beginning in version 1.8.0 all time uses will converge on an osquery-provided
getUnixTime() API call that returns, by default, UNIX time integers converted
to UTC/GMT. The 'time' table will respond with the parsed time for the
configuration. If the timezone is not UTC then osquery is using localtime.
This configuration option will affect the 'unix_time' response in the 'time'
table. Because of this configurable-effect the table is extended to include
'local_time' which is always the system local UNIX time.
This commit adds logger plugin implementations for the Amazon
Kinesis (https://aws.amazon.com/kinesis/) and Kinesis
Firehose (https://aws.amazon.com/kinesis/firehose/) services. To support
these plugins there are a number of utility classes and functions for
AWS authentication, configuration and API integration. The logger plugin
implementations take advantage of the BufferedLogForwarder base class
for reliable buffering and batch sending of logs. In their current
implementations, the logger plugins only support sending of result logs
to these AWS services.
* Committing changes related to our experimentation with a "pure" Win64 build
* Placates CMake such that a Visual Studio 14 x64 solution is generated!
* Updated changes to fix the issue of GFlags not being found.
* Added cases to handle Win64 specific CMake options such as include/link
directories and compiler options
* Comment change in CMakeList.txt
* Changed wording of get_platform error message. Adding Powershell
provisioning script.
* Finalized provision powershell script
* Added a deployment XML file for insuring C++ support exists for VS2015
* Added admin check and resolved some potential PATH issues.
* Fixed some potential bugs in VS2015 automated install
* Adding a wrapper for provision.ps1 so people don't need to know obscure
powershell syntax
* Fixing a bug with third-party archive extraction
* Ignoring the updating of pip for now...
* Fix invocation of choco.exe
* Resolved pip install issues
* Removed some debugging residue
* Changing get_platform.py from tabs to spaces
* Added distro detection for ubuntu and darwin
* Using 4 spaces
* Added a newline after powershell invocation
* Added OS detection for freebsd and fedora
* Fixed bug with freebsd
* Adding arguments parsing to prepare for modularizing platform detection
* Removing bash platform detection logic and forwarding the task to
get_platform.py
* Removing get_platform support in provision.sh since it doesn't appear to
be used anywhere now
* Fixed some comment/spacing issues. Made a few efficiency changes
* A few bug fixes, revereted back to WIN32 variable for now
* Added Facebook copyright information.
* Fixed boost and rocksdb library paths
* Added support for installing our custom chocolatey packages
* Fixed path to ignore the symlink
* Force environment variable propagation
* Forgot to add new line after make-win64-dev-env.bat
* Added error handling for choco install failures
* Handles download and python/pip errors
* When chocolatey is not detected, refreshenv.bat is not found in the PATH.
Hardcode the PATH as per chocolatey install instructions
* Takes care of updating git submodules in third-party\
* Fixes a bug in which Linux provisioning fails because of unset FAMILY
env var.
BufferedLogForwarder is a base class for external log forwarding schemes
that require buffering and retries. It generalizes the logic from the
existing TLSLogForwarderRunner.
The existing TLSLoggerPlugin and TLSLogForwarderRunner are refactored to
demonstrate the use of this new base class.
Added a table that parses out some of the informaiton in the OS X logs
stored in /Library/Logs/DiagnosticReports as well as
/Users/<user>/Library/Logs/DiagnosticReports
These APIs using std::chrono::duration allow us to take advantage of
automatic time conversion and type safety among different units of time.
No changes were made to existing call sites.
This commit adds an event-based virtual table implementation for
querying the linux syslog. It introduces an event publisher that
attaches to a named pipe to ingest CSV formatted syslog forwarded from
rsyslogd. An event subscriber/virtual table makes these log lines
available for queries. Currently, no additional processing is done on
the input data besides parsing.
Using this table requires a properly configured rsyslogd. Documentation
for this configuration is forthcoming in the wiki.
If the distributed plugin's `writeResults` method can be successfully
called, we must clear the local vector of results so that we're not
constantly growing it over time.
Previously, this test relied on there being console messages available
to query for. Now, it actually writes a message to ASL, then verifies
that the message can be found. It also exercises a more complicated
query than previously. Also remove potentially high latency queries.
Fixes issue #1975
As with all other appearances of IOKitLib.h in the osquery sources, use
the capitalization "IOKitLib.h" not "IOKitlib.h" to avoid build failure
on case-sensitive file systems.
1. Update boost to 1.60 from 1.55 on Linux platforms
2. Add asio (1.11.0) to the deps set
3. Update snappy to 1.1.3 on Linux platforms
4. Update cpp-netlib to 0.12.0-rc1 from 0.11 on Linux platforms
- OS X and brew also include 0.12.0-rc1 as a devel option
5. Update libapt to 1.2.6 from 0.8.6 on Ubuntu/Debian
- This adds lzma as a dependent link
