This commit bumps the third-party SQLite to the 3.14.0 pre-release (18:59).
With 3.14.0 the LIKE and EQUALS constraint operators may be mixed within a
query. Previously these would fail to produce a valid set.
As part of the support, each virtual table should choose to bypass rowid-based
deduplication using the new "WITHOUT ROWID" create table epilog. This will
be appended to the schema if the table defines a PRIMARY KEY using index=True.
Thus begins our need to include local (modified) brew formulas.
This commit adds a new provision library method: local_brew. Use this function
within provision scripts to install packages that are not appropriate for
homebrew-core.
With a removeService method, combined with the abstracted thread start in
the Dispatcher API, services auto-remove when finished.
This will un-break the kernel communication tests. These tests only stop
when all their producer threads/services have ended.
This also promotes the OS X kernel build to 10.11.
Table options includes a change to the Registry::call API for TablePlugins.
When requesting route information or the 'columns' action, a new 'op' key is included.
Integrated process abstraction code into more locations
Defined new macros for abstracting across various platforms
Added GLOG_NO_ABBREVIATED_SEVERITIES for glog to support Windows
Fixed some minor CMake issues involving thrift
Updated gflags package; reflecting change in provision script
Preparing CMake config files for WIN32 support
This adds a new optimization feature that allows expensive tables to cache
their results between JOINs. Consider JOINing a list of open sockets, for each
process, then requesting to hash each process path. This query may hash
the same path multiple times.
Within-query caching allows the hash table to respond with the previous
result of the hash request as long as the requested computation was the result
of a single query. Subsequent queries will perform subsequent hashing.
This commit adds logger plugin implementations for the Amazon
Kinesis (https://aws.amazon.com/kinesis/) and Kinesis
Firehose (https://aws.amazon.com/kinesis/firehose/) services. To support
these plugins there are a number of utility classes and functions for
AWS authentication, configuration and API integration. The logger plugin
implementations take advantage of the BufferedLogForwarder base class
for reliable buffering and batch sending of logs. In their current
implementations, the logger plugins only support sending of result logs
to these AWS services.
There seems to have been a regression in package building.
The ./tools/lib.sh script now overloads the SCRIPT_DIR variable,
which is also used in the package build scripts.
This changes the file-local variable in ./tools/lib.sh.
* Committing changes related to our experimentation with a "pure" Win64 build
* Placates CMake such that a Visual Studio 14 x64 solution is generated!
* Updated changes to fix the issue of GFlags not being found.
* Added cases to handle Win64 specific CMake options such as include/link
directories and compiler options
* Comment change in CMakeList.txt
* Changed wording of get_platform error message. Adding Powershell
provisioning script.
* Finalized provision powershell script
* Added a deployment XML file for insuring C++ support exists for VS2015
* Added admin check and resolved some potential PATH issues.
* Fixed some potential bugs in VS2015 automated install
* Adding a wrapper for provision.ps1 so people don't need to know obscure
powershell syntax
* Fixing a bug with third-party archive extraction
* Ignoring the updating of pip for now...
* Fix invocation of choco.exe
* Resolved pip install issues
* Removed some debugging residue
* Changing get_platform.py from tabs to spaces
* Added distro detection for ubuntu and darwin
* Using 4 spaces
* Added a newline after powershell invocation
* Added OS detection for freebsd and fedora
* Fixed bug with freebsd
* Adding arguments parsing to prepare for modularizing platform detection
* Removing bash platform detection logic and forwarding the task to
get_platform.py
* Removing get_platform support in provision.sh since it doesn't appear to
be used anywhere now
* Fixed some comment/spacing issues. Made a few efficiency changes
* A few bug fixes, revereted back to WIN32 variable for now
* Added Facebook copyright information.
* Fixed boost and rocksdb library paths
* Added support for installing our custom chocolatey packages
* Fixed path to ignore the symlink
* Force environment variable propagation
* Forgot to add new line after make-win64-dev-env.bat
* Added error handling for choco install failures
* Handles download and python/pip errors
* When chocolatey is not detected, refreshenv.bat is not found in the PATH.
Hardcode the PATH as per chocolatey install instructions
* Takes care of updating git submodules in third-party\
* Fixes a bug in which Linux provisioning fails because of unset FAMILY
env var.
When I added the flag file switch it was aimed at `/var/osquery`, but the package is built such that everything exists in `/private/var/osquery`. This simply makes this more consistent.
1. Update boost to 1.60 from 1.55 on Linux platforms
2. Add asio (1.11.0) to the deps set
3. Update snappy to 1.1.3 on Linux platforms
4. Update cpp-netlib to 0.12.0-rc1 from 0.11 on Linux platforms
- OS X and brew also include 0.12.0-rc1 as a devel option
5. Update libapt to 1.2.6 from 0.8.6 on Ubuntu/Debian
- This adds lzma as a dependent link
2. Introduce a SQLite-based database plugin
3. Refactor database usage to include local 'fast-calls'
4. Introduce an 'ephemeral' database plugin for testing (like a mock)
[Added] python packages which are not installable via pip on lucid
[Moved] installing latest bison and openssl right before thrift building for lucid
[Fixed] package bison installation for not lucid
[Added] OpenSSL dependency for lucid
[Changed] prefix to /usr/local
[Fixed] wrong file for checking if dependency is intalled
1. Table implementations (spec files) can mark the table as 'cachable'.
2. Cached results depend on the shortest/quickest interval of scheduled
queries that act on results of the table.
3. The table API generator blocks caching on index/additional/required
table column options.
If the option of remembering known Wi-Fi networks is enabled on a system,
they are persisted to disk as a preferences property list file.
This table is populated by parsing that file.
There was a bug in the `osquery::Schedule` container object such that,
when the iteration through the schedule occured, pack objects were being
passed by value (copied) instead of passed by reference. Thus, the
discovery query would be executed, the object's cache would be updated,
and then the object would go out of scope and be destructed, thus
leaving the original object without ever having ran the discovery query.
This caused discovery queries to thrash. Bad times.
I added a new test so that we don't regress here as well as const'd a
few functions that should have been const in `osquery::Pack`.
This commit contains the features specified in #1390 as well as a
refactoring of the general osquery configuration code.
The API for the config plugins hasn't changed, although now there's a
`genPack` method that config plugins can implement. If a plugin doesn't
implement `genPack`, then the map<string, string> format cannot be used.
The default config plugin, the filesystem plugin, now implements
`genPack`, so existing query packs code will continue to work as it
always has.
Now many other config plugins can implement custom pack handling for
what makes sense in their context. `genPacks` is not a pure virtual, so
it doesn't have to be implemented in your plugin if you don't want to
use it. Also, more importantly, all config plugins can use the standard
inline pack format if they want to use query packs. Which is awesome.
For more information, refer to #1390, the documentation and the doxygen
comments included with this pull requests, as well as the following
example config which is now supported, regardless of what config plugin
you're using:
```json
{
"options": {
"enable_monitor": "true"
},
"packs": {
"core_os_monitoring": {
"version": "1.4.5",
"discovery": [
"select pid from processes where name like '%osqueryd%';"
],
"queries": {
"kernel_modules": {
"query": "SELECT name, size FROM kernel_modules;",
"interval": 600
},
"system_controls": {
"query": "SELECT * FROM system_controls;",
"interval": 600,
"snapshot": true,
},
"usb_devices": {
"query": "SELECT * FROM usb_devices;",
"interval": 600
}
}
},
"osquery_internal_info": {
"version": "1.4.5",
"discovery": [
"select pid from processes where name like '%osqueryd%';"
],
"queries": {
"info": {
"query": "select i.*, p.resident_size, p.user_time, p.system_time, time.minutes as counter from osquery_info i, processes p, time where p.pid = i.pid;",
"interval": 60,
"snapshot": true
},
"registry": {
"query": "SELECT * FROM osquery_registry;",
"interval": 600,
"snapshot": true
},
"schedule": {
"query": "select name, interval, executions, output_size, wall_time, (user_time/executions) as avg_user_time, (system_time/executions) as avg_system_time, average_memory from osquery_schedule;",
"interval": 60,
"snapshot": true
}
}
}
}
}
```
The `osquery_packs` table was modified to remove the superfluous
columns which could already have been found in `osquery_schedule`. Two
more columns were added in their place, representing stats about pack's
discovery query execution history.
Notably, the internal API for the `osquery::Config` class has changed
rather dramatically as apart of the refactoring. We think this is an
improvement. While strictly adhering to the osquery config plugin
interface will have avoided any compatibility errors, advanced users may
notice compilation errors if they access config data directly. All
internal users of the config have obviously been updated. Yet another
reason to merge your code into mainline; we update it for you when we
refactor!
* It wasn't straightforward to get OpenSSL building
without avx/vxoprs optimizations on 10.10
* libressl is essentially a modern/lean-ish drop-in replacement for OpenSSL
and can build without avx optimizations to support older Macs
This change:
* Installs libressl (builds a bottle) using homebrew
* And statically links `libcrypto.a` and `libssl.a` unless
`BUILD_LINK_SHARED` is requested.
Fixes#1329
OS X 10.9 should not build/test a kernel extension yet. The MAC policy framework is slightly different and the APIs/version dependencies need to be tested.
Having `-mtune=i386` is causing compilation failure for gflags on ubuntu.
This change removes the `mtune` compile flag.
`-march` flag is already set to `x86-64` and according to gcc doc,
Specifying `-march=cpu-type` implies `-mtune=cpu-type.`
Fixes#1428
Boost property trees are level delimited using '.' characters.
An Apple property list may contain keys with '.' characters, so the plist conversion must use iterators and raw node appends.
POSIX-globbing will allow event publishers/subscribers to post-check
results against glob-syntax, fnpath matching, and POSIX C-regex.
These checks are anecdotally speedy.
We found that not installing the headers for snappy caused RocksDB's
snappy detection to not find that snappy was installed:
https://goo.gl/YOWJl0
The snippet there requires that the headers are installed, not just the
library. By installing the headers, we can ensure that snappy is linked.
OR, alternatively, we could just leave it and not link snappy. It's
uncertain what the specific benefits of including snappy are for our
use-case. (CC @igorcanadi)
