This commit bumps the third-party SQLite to the 3.14.0 pre-release (18:59).
With 3.14.0 the LIKE and EQUALS constraint operators may be mixed within a
query. Previously these would fail to produce a valid set.
As part of the support, each virtual table should choose to bypass rowid-based
deduplication using the new "WITHOUT ROWID" create table epilog. This will
be appended to the schema if the table defines a PRIMARY KEY using index=True.
Some of the types in fileops tests were causing warnings to be thrown
during build, due to type mismatch. I've added a few local variables to
quiet these warnings.
* Implemented filesystem operations abstraction code
* Added filesystem operations abstraction unit tests
* Modified CMake configurations to support the building of the abstraction code and unit tests
The three new SQLite functions:
- split: Splits a column using a set of tokens and a selected index.
- regex_split: Similar to split but with a regex instead of tokens.
- inet_aton: Returns the IPv4 decimal value for a string-formatted address.
With a removeService method, combined with the abstracted thread start in
the Dispatcher API, services auto-remove when finished.
This will un-break the kernel communication tests. These tests only stop
when all their producer threads/services have ended.
This also promotes the OS X kernel build to 10.11.
This changes several initialization steps:
- The daemon (and shell, though not needed) have a new --ephemeral flag.
- Events are now disabled in the shell by default, use --nodisable_events to
re-enable.
- RocksDB-based backing storage is now disabled in the shell by default.
The --ephemeral flag for the daemon is disabled by default and will allow
skipping configuration and database path sanity, and skipping pidfile checks.
This is intended to be used when debugging or monitoring the daemon process.
To make the RocksDB backing storage feature usage very clear we introduce a new
flag: --disable_database. The shell sets this to true unless overridden in
a flagfile or via command line arguments.
Table options includes a change to the Registry::call API for TablePlugins.
When requesting route information or the 'columns' action, a new 'op' key is included.
Integrated process abstraction code into more locations
Defined new macros for abstracting across various platforms
Added GLOG_NO_ABBREVIATED_SEVERITIES for glog to support Windows
Fixed some minor CMake issues involving thrift
Updated gflags package; reflecting change in provision script
Preparing CMake config files for WIN32 support
This introduces a new testing-related flag to the osqueryi shell.
The --pack flag takes a single argument, the name of a configured pack.
To list the configured packs try:
$ osqueryi -A osquery_packs
To run all queries, immediately and serially, try:
$ osqueryi --pack NAME_OF_PACK
When selecting UUID as the host identifier the client follows:
- If host_identifier is set to hostname, return the hostname
- If UUID check a static UUID container
- If empty request the UUID from the backing storage, RocksDB
- If empty use the Apple hardware UUID or generate a UUID
- If the Apple UUID request fails generate a UUID
When no UUID is available in the backing store save the new UUID.
The Thrift library should be discovered simiarly to all others.
When using the build-macros for library discovery the caller can choose
how to discover static or dynamic links.
Added Windows support for process operations.
Added unit tests for process abstraction code for POSIX and Windows.
Modified CMake config files to support building the new code and unit tests.
The initialization of a logger plugin was confusing. The 'init' step was
introduced to allow a daemon to buffer status events before a logger plugin
is determined by external/remote configuration. The buffered statuses could
then be transferred via a medium other than Glog (the default). To determine
if Glog should continue to write statuses to the filesystem the 'init' method
returned a Status.
Logger plugins should now use a feature method override to select how status
logs should be handled.
The OS X kernel subscribers have not been starting because they expect the
publisher thread to run before they begin configuration. Due to some recent
refactors the publisher thread creation now occurs after configuration.
The subscriber logic to check for a valid kernel connection is still valid.
This commit has two additional side-effects:
- The RocksDB plugin is modified to use 3 background merge threads.
- The OS X kernel publisher syncing thread is now non-blocking.
The mobile 'LowBattery' crash logs provide little if any data of value,
and seem to be nothing more than notification events. We're filtering
these out of the crashes table.
This adds a new optimization feature that allows expensive tables to cache
their results between JOINs. Consider JOINing a list of open sockets, for each
process, then requesting to hash each process path. This query may hash
the same path multiple times.
Within-query caching allows the hash table to respond with the previous
result of the hash request as long as the requested computation was the result
of a single query. Subsequent queries will perform subsequent hashing.
This commit adds mobile device crashes to the list of crash logs parsed by the Crashes table as well as adding a lamdba to improve code reuse. The commit also adds a 'type' column to the table to indicate what kind of log this crash log was.
Renamed the crash_log table to crashes for future abstraction to other
operating systems. Also fixed how the table was parsing the most recent
stack trace and the registers. Register values are now all parsed into
one column 'registers', which will be a space delimited string of the
form:
register:value register:value ... register:value
in order to best allow for OS abstraction.
It is often helpful to know the local timezone of the machine. For this use
local_timezone, as the base timezone will use local or UTC depending on the
--utc flag. This will be default=UTC in osquery 1.8.0.
The datetime field is added to mimic ISO 8601, along with iso_8601.
The timestamp field remains as the time stamp used for logging (within osquery)
and commonly outside of osquery. The goal for adding multiple representations
is to allow joining/augmenting of other tables.
Beginning in version 1.8.0 all time uses will converge on an osquery-provided
getUnixTime() API call that returns, by default, UNIX time integers converted
to UTC/GMT. The 'time' table will respond with the parsed time for the
configuration. If the timezone is not UTC then osquery is using localtime.
This configuration option will affect the 'unix_time' response in the 'time'
table. Because of this configurable-effect the table is extended to include
'local_time' which is always the system local UNIX time.
This commit adds logger plugin implementations for the Amazon
Kinesis (https://aws.amazon.com/kinesis/) and Kinesis
Firehose (https://aws.amazon.com/kinesis/firehose/) services. To support
these plugins there are a number of utility classes and functions for
AWS authentication, configuration and API integration. The logger plugin
implementations take advantage of the BufferedLogForwarder base class
for reliable buffering and batch sending of logs. In their current
implementations, the logger plugins only support sending of result logs
to these AWS services.
* Committing changes related to our experimentation with a "pure" Win64 build
* Placates CMake such that a Visual Studio 14 x64 solution is generated!
* Updated changes to fix the issue of GFlags not being found.
* Added cases to handle Win64 specific CMake options such as include/link
directories and compiler options
* Comment change in CMakeList.txt
* Changed wording of get_platform error message. Adding Powershell
provisioning script.
* Finalized provision powershell script
* Added a deployment XML file for insuring C++ support exists for VS2015
* Added admin check and resolved some potential PATH issues.
* Fixed some potential bugs in VS2015 automated install
* Adding a wrapper for provision.ps1 so people don't need to know obscure
powershell syntax
* Fixing a bug with third-party archive extraction
* Ignoring the updating of pip for now...
* Fix invocation of choco.exe
* Resolved pip install issues
* Removed some debugging residue
* Changing get_platform.py from tabs to spaces
* Added distro detection for ubuntu and darwin
* Using 4 spaces
* Added a newline after powershell invocation
* Added OS detection for freebsd and fedora
* Fixed bug with freebsd
* Adding arguments parsing to prepare for modularizing platform detection
* Removing bash platform detection logic and forwarding the task to
get_platform.py
* Removing get_platform support in provision.sh since it doesn't appear to
be used anywhere now
* Fixed some comment/spacing issues. Made a few efficiency changes
* A few bug fixes, revereted back to WIN32 variable for now
* Added Facebook copyright information.
* Fixed boost and rocksdb library paths
* Added support for installing our custom chocolatey packages
* Fixed path to ignore the symlink
* Force environment variable propagation
* Forgot to add new line after make-win64-dev-env.bat
* Added error handling for choco install failures
* Handles download and python/pip errors
* When chocolatey is not detected, refreshenv.bat is not found in the PATH.
Hardcode the PATH as per chocolatey install instructions
* Takes care of updating git submodules in third-party\
* Fixes a bug in which Linux provisioning fails because of unset FAMILY
env var.
BufferedLogForwarder is a base class for external log forwarding schemes
that require buffering and retries. It generalizes the logic from the
existing TLSLogForwarderRunner.
The existing TLSLoggerPlugin and TLSLogForwarderRunner are refactored to
demonstrate the use of this new base class.
Added a table that parses out some of the informaiton in the OS X logs
stored in /Library/Logs/DiagnosticReports as well as
/Users/<user>/Library/Logs/DiagnosticReports
