Commit Graph

521 Commits

Author SHA1 Message Date
Teddy Reed
944e3de206 Merge pull request #1496 from theopolis/events_table
[#1487] Add osquery_events table to track pubsub stats
2015-09-14 15:27:35 -04:00
Mike Arpaia
aaa03a1058 Distributed queries client-side 2015-09-08 13:33:48 -07:00
Teddy Reed
b57040db60 Add osquery_events table to track pubsub stats 2015-09-03 15:10:53 -07:00
Teddy Reed
2813d3ab87 Add a Linux audit event publisher 2015-09-03 08:45:02 -07:00
Mike Arpaia
a140333441 [fix #1390] query pack re-org
This commit contains the features specified in #1390 as well as a
refactoring of the general osquery configuration code.

The API for the config plugins hasn't changed, although now there's a
`genPack` method that config plugins can implement. If a plugin doesn't
implement `genPack`, then the map<string, string> format cannot be used.
The default config plugin, the filesystem plugin, now implements
`genPack`, so existing query packs code will continue to work as it
always has.

Now many other config plugins can implement custom pack handling for
what makes sense in their context. `genPacks` is not a pure virtual, so
it doesn't have to be implemented in your plugin if you don't want to
use it. Also, more importantly, all config plugins can use the standard
inline pack format if they want to use query packs. Which is awesome.

For more information, refer to #1390, the documentation and the doxygen
comments included with this pull requests, as well as the following
example config which is now supported, regardless of what config plugin
you're using:

```json
{
  "options": {
    "enable_monitor": "true"
  },
  "packs": {
    "core_os_monitoring": {
        "version": "1.4.5",
        "discovery": [
          "select pid from processes where name like '%osqueryd%';"
        ],
        "queries": {
          "kernel_modules": {
              "query": "SELECT name, size FROM kernel_modules;",
              "interval": 600
          },
          "system_controls": {
              "query": "SELECT * FROM system_controls;",
              "interval": 600,
              "snapshot": true,
          },
          "usb_devices": {
              "query": "SELECT * FROM usb_devices;",
              "interval": 600
          }
        }
    },
    "osquery_internal_info": {
        "version": "1.4.5",
        "discovery": [
          "select pid from processes where name like '%osqueryd%';"
        ],
        "queries": {
          "info": {
              "query": "select i.*, p.resident_size, p.user_time, p.system_time, time.minutes as counter from osquery_info i, processes p, time where p.pid = i.pid;",
              "interval": 60,
              "snapshot": true
          },
          "registry": {
              "query": "SELECT * FROM osquery_registry;",
              "interval": 600,
              "snapshot": true
          },
          "schedule": {
              "query": "select name, interval, executions, output_size, wall_time, (user_time/executions) as avg_user_time, (system_time/executions) as avg_system_time, average_memory from osquery_schedule;",
              "interval": 60,
              "snapshot": true
          }
        }
    }
  }
}
```

The `osquery_packs` table was modified to remove the superfluous
columns which could already have been found in `osquery_schedule`. Two
more columns were added in their place, representing stats about pack's
discovery query execution history.

Notably, the internal API for the `osquery::Config` class has changed
rather dramatically as apart of the refactoring. We think this is an
improvement. While strictly adhering to the osquery config plugin
interface will have avoided any compatibility errors, advanced users may
notice compilation errors if they access config data directly. All
internal users of the config have obviously been updated. Yet another
reason to merge your code into mainline; we update it for you when we
refactor!
2015-09-02 10:56:26 -07:00
Teddy Reed
d63510f8bd Remove benchmark from third-party, prefer deps-build 2015-08-31 15:01:13 -07:00
Teddy Reed
4dd77a43a7 Remove cpp-netlib from third-party, prefer deps-build 2015-08-31 09:27:01 -07:00
Javier Marcos
1a50977a23 Adding magic table to check for libmagic data 2015-08-28 12:49:46 -07:00
Dave Gosselin
6899127e1e Add support for building on Fedora 21 2015-08-25 11:10:56 -04:00
Mathieu Kooiman
b151ecedc2 Refs https://github.com/facebook/osquery/issues/320
Add provisioning scripts to build osquery on Debian Wheezy and Debian Jessie.
2015-08-20 20:57:22 +02:00
Teddy Reed
73da2f1448 Merge pull request #1443 from sharvilshah/libressl
[#1329] Statically link against brew bottled libressl on OS X
2015-08-18 01:09:45 -07:00
Teddy Reed
fb93b73253 Merge pull request #1450 from eastebry/fix-package-dependencies
Fix lucid package building
2015-08-17 16:04:00 -07:00
Teddy Reed
5bf30a779d RocksDB usage speedups 2015-08-15 20:43:53 -07:00
Bryan Eastes
1ac7c5d21a Installing specific FPM version 2015-08-15 15:11:29 -07:00
Sharvil Shah
ccc086f050 Statically link against brew bottled libressl on OS X
* It wasn't straightforward to get OpenSSL building
 without avx/vxoprs optimizations on 10.10
* libressl is essentially a modern/lean-ish drop-in replacement for OpenSSL
and can build without avx optimizations to support older Macs

This change:

* Installs libressl (builds a bottle) using homebrew
* And statically links `libcrypto.a` and `libssl.a` unless
`BUILD_LINK_SHARED` is requested.

Fixes #1329
2015-08-13 15:31:26 -07:00
Javier Marcos
ee98bbde67 Fix for the version variable 2015-08-13 07:02:24 +02:00
Teddy Reed
251aded11c Merge pull request #1442 from javuto/darwin_package_10.9
Updating the script to create packages for 10.9 building
2015-08-12 18:16:36 -07:00
Teddy Reed
fd1b4b06e8 Merge pull request #1436 from theopolis/pack_place
[#1402] Add notes around pack paths in example.conf
2015-08-12 18:15:50 -07:00
Teddy Reed
b9ded9e7af [#1402] Add notes around pack paths in example.conf 2015-08-12 17:15:42 -07:00
Javier Marcos
861e0e4273 Using the code in lib.sh 2015-08-13 00:53:44 +02:00
Javier Marcos
54cb08c193 Updating the script to create packages for 10.9 building 2015-08-12 20:34:50 +02:00
Teddy Reed
ea0ca195f6 Update build/test for 10.9 (do not test extension)
OS X 10.9 should not build/test a kernel extension yet. The MAC policy framework is slightly different and the APIs/version dependencies need to be tested.
2015-08-12 10:40:53 -07:00
Sharvil Shah
e4f52589f5 Remove -mtune compile flag
Having `-mtune=i386` is causing compilation failure for gflags on ubuntu.
This change removes the `mtune` compile flag.
`-march` flag is already set to `x86-64` and according to gcc doc,
Specifying `-march=cpu-type` implies `-mtune=cpu-type.`

Fixes #1428
2015-08-06 17:14:33 -07:00
Teddy Reed
67b0f51ab5 Several small optimizations around internal SQL queries 2015-08-03 07:56:55 -07:00
Michael O'Farrell
dda11ce74a Executable size benchmark change. 2015-07-30 15:44:25 -07:00
Michael O'Farrell
346743e87f Benchmark using mean across 5 runs. 2015-07-29 16:50:19 -07:00
Michael O'Farrell
5956e685e9 Report bytes using word count 2015-07-29 15:10:20 -07:00
Michael O'Farrell
46ee4b491c Benchmark stripped binary size. 2015-07-29 11:35:04 -07:00
Michael O'Farrell
0cb5730d55 Added benchmark for executable size. 2015-07-28 22:34:22 -07:00
Teddy Reed
ff9cb71628 Various additional tests and benchmarks 2015-07-28 12:26:17 -07:00
Elan Ruusamäe
bb40956844 don't qualify any system with lsb-release as ubuntu 2015-07-25 14:52:59 +03:00
Michael O'Farrell
66b075a685 Merge pull request #1377 from mofarrell/benchmark
Added benchmarking targets.
2015-07-23 17:37:56 -07:00
Michael O'Farrell
a65f8dd93c Added benchmarking targets. 2015-07-23 17:07:42 -07:00
Teddy Reed
f03ec9ddac [Fix #1368] Restore autostart post-install scripts 2015-07-21 19:09:24 -07:00
Teddy Reed
fc24682816 Fix profile platform bug in leaks checking 2015-07-20 02:06:52 -07:00
Teddy Reed
95775be1d9 [Fix #1355] Allow plist keys with '.'
Boost property trees are level delimited using '.' characters.
An Apple property list may contain keys with '.' characters, so the plist conversion must use iterators and raw node appends.
2015-07-19 16:24:43 -07:00
Teddy Reed
a713d09f0e Install additional configs for HB/packages 2015-07-17 16:07:22 -07:00
Teddy Reed
270b4da540 [Fix #1339] Add kernel-build to packages when used 2015-07-16 15:23:29 -07:00
Javier Marcos
7241becda1 Fix tables JSON file name 2015-07-16 13:38:31 -07:00
Javier Marcos
84e0c77a98 Generation of table docs with packages and docs targets 2015-07-16 12:23:44 -07:00
Javier Marcos
01fabf910d Merge pull request #1343 from javuto/generate_tables_output
Adding support to generate documentation to external files
2015-07-15 17:10:27 -07:00
Javier Marcos
ba69bf8efa Adding support to generate documentation to external files 2015-07-15 13:18:41 -07:00
Teddy Reed
341245f751 Build/install gflags' static library on build hosts.
As of [homebrew #41151](https://github.com/Homebrew/homebrew/pull/41151) gflags is not installed with a static library.
Our build hosts must have static versions of gflags.
2015-07-14 17:20:55 -07:00
Teddy Reed
c269bbeaf3 Rollup of build changes 2015-07-14 13:45:53 -07:00
Teddy Reed
19d7a9e735 Merge pull request #1328 from mofarrell/kernel
Fixed cleanup in testing script in failure cases.
2015-07-13 16:41:44 -07:00
Michael O'Farrell
b2b1f0483d Fixed cleanup in testing script in failure cases. 2015-07-13 16:11:45 -07:00
Teddy Reed
0e49a3a9a1 Build separate OS X packages 2015-07-13 15:44:16 -07:00
Michael O'Farrell
dd1f0af0ff Build system changes for kernel extension testing and deployment. 2015-07-09 11:50:23 -07:00
Michael O'Farrell
0284b9e60d Merge branch 'master' into kernel
Conflicts:
	mkdocs.yml
2015-07-08 10:26:32 -07:00
Teddy Reed
f48619ed28 [#1285, #1276] Faster, optimized subscriber results 2015-07-07 00:59:28 -07:00
Matthew White
11f447a959 Minor fixes to support building on Ubuntu 10.04 2015-07-06 15:18:11 -07:00
Teddy Reed
7aac5fd358 Replace custom wildcarding with POSIX-glob
POSIX-globbing will allow event publishers/subscribers to post-check
results against glob-syntax, fnpath matching, and POSIX C-regex.
These checks are anecdotally speedy.
2015-07-02 13:53:16 -07:00
Teddy Reed
64e4afa136 Merge pull request #1294 from theopolis/relax_test_timesouts
Relax extensions and shell timeouts
2015-07-02 13:50:07 -07:00
Teddy Reed
89e5b6c729 Relax extensions and shell timeouts 2015-07-02 12:14:44 -07:00
Mike Arpaia
ba89b67cc5 Install snappy headers instead of just the library
We found that not installing the headers for snappy caused RocksDB's
snappy detection to not find that snappy was installed:
https://goo.gl/YOWJl0

The snippet there requires that the headers are installed, not just the
library. By installing the headers, we can ensure that snappy is linked.

OR, alternatively, we could just leave it and not link snappy. It's
uncertain what the specific benefits of including snappy are for our
use-case. (CC @igorcanadi)
2015-07-01 16:14:06 -07:00
Michael O'Farrell
a7bd4bd3db Merge pull request #1278 from facebook/master
Merge branch 'master' into kernel
2015-06-30 13:12:16 -07:00
Teddy Reed
757940fe6f Towards CMake-powered kernel extension building 2015-06-30 00:49:16 -07:00
Mike Arpaia
06793f9d00 Merge pull request #1267 from marpaia/osquery-latest-no-more
Remove "latest" from the osquery package names
2015-06-29 15:09:31 -07:00
Teddy Reed
0d6ab16281 Yara events was not building 2015-06-29 14:45:31 -07:00
Mike Arpaia
d6719f9ef7 Remove "latest" from the osquery package names 2015-06-29 11:18:49 -07:00
Teddy Reed
8db6ca4a3f [Fix #1198] Add a small retry to ext watcher 2015-06-28 02:12:50 -07:00
Teddy Reed
5566d8cd96 Merge pull request #1194 from theopolis/lucid-build
Loose support for building on Ubuntu 10.04
2015-06-27 20:47:53 -07:00
Teddy Reed
e7ed68e187 [Fix #1198] Faster death/timeout checks in extensions tests 2015-06-25 02:53:53 -07:00
Sharvil Shah
368517c6a6 Use psutil's Process.children() instead of Process.get_children() as the latter has been deprecated.
Process.get_children() had been deprecated in psutil 2.x and is compeletely removed in 3.x versions
in favor of Process.children().

This fixes #1220.
2015-06-23 16:44:05 -07:00
Mike Arpaia
2b9bbb6bd4 Merge pull request #1223 from marpaia/yara-3.4.0
updating yara to 3.4.0
2015-06-22 09:33:25 -07:00
Teddy Reed
37188f788b Fixups in tables, add DOUBLE, shell extensions 2015-06-22 04:17:23 -04:00
Mike Arpaia
6f85f2f617 updating yara to 3.4.0 2015-06-21 11:40:51 -04:00
Teddy Reed
46ceb7aa6d Merge pull request #1213 from theopolis/certs2
Update testing x509 certs
2015-06-13 02:24:29 -07:00
Teddy Reed
2fb774218a Update testing x509 certs 2015-06-13 02:13:31 -07:00
Teddy Reed
e7ab2fc47b Limit scope of git/tag version defines.
Harden plist parsing against internal fuzzing tests.
Improve file/stream read speeds.
2015-06-12 10:10:20 -07:00
Teddy Reed
727f5b091f Various table perf improvements and TLS docs 2015-06-05 22:03:15 -07:00
Teddy Reed
ccb1c2cd69 Loose support for building on Ubuntu 10.04 2015-06-04 18:25:49 -07:00
Teddy Reed
4e59bcf4c1 Merge pull request #1191 from theopolis/feature-backoffs
[#1190] Schedule queries without logging removed results
2015-06-04 14:58:19 -07:00
Teddy Reed
a678f8f46a Merge pull request #1192 from theopolis/rocksdb-from-homebrew
[Fix #1185, #1183] Use RocksDB from Homebrew on OS X
2015-06-04 14:34:52 -07:00
Teddy Reed
650a43d053 [Fix #1185, #1183] Use RocksDB from Homebrew on OS X 2015-06-04 13:56:58 -07:00
Teddy Reed
e244883ea4 [#1190] Schedule queries without logging removed results 2015-06-04 13:53:55 -07:00
Teddy Reed
a70828c2a4 Merge pull request #1187 from sharvilshah/xattr_update
Extended Attributes: Use LaunchServices API for quarantine data
2015-06-03 22:38:17 -07:00
Sharvil Shah
065fe6412d Use LaunchServices (part of CoreServices) to grab quarantine properties instead of manually parsing the colon separated attribute data.
Fall back to deprecated LaunchService API for OS X 10.9 Mavericks.

Added tests for extended_attributes

Better error handling and cleanup
2015-06-03 22:18:45 -07:00
Teddy Reed
c934ad0df3 Update tooling/profiling paths 2015-06-03 21:22:12 -07:00
Teddy Reed
8aacaca7eb Query pack platform binds should match any/all 2015-06-03 13:56:39 -07:00
Teddy Reed
a105924804 Move specs to a top-level path, add query examples
1. Example queries will run with an (optional) integration test.
2. Fix bad accesses with OS X package BOMs
3. Move spec files from ./osquery/tables/specs to ./specs
4. Remove server parsers (netlib) from client builds.
2015-06-03 10:39:05 -07:00
Teddy Reed
5899bbb8f5 Merge pull request #1182 from theopolis/osx_rocksdb_portable
Build RocksDB from source on Darwin
2015-06-02 15:50:15 -07:00
Teddy Reed
eeab588d8f Build RocksDB from source on Darwin 2015-06-02 15:25:16 -07:00
Teddy Reed
f41fb6b107 Remove package-manager installed autoconf tools for older distros 2015-06-02 03:05:47 -07:00
Teddy Reed
0669d8205e Merge pull request #1174 from theopolis/remote_logger
TLS/HTTPS-based logger plugin
2015-06-02 02:59:34 -07:00
Teddy Reed
33f53809ad Fix DBHandle checking with concurrent processes.
`make tests` fails with another osquery process running.
The backing-store check happens after a config plugin is setUp and
the initial load occures. This may involve calls to cached keys, the
check should occur pre-config initialize.
2015-06-02 02:50:04 -07:00
Teddy Reed
da9bd5801b Migrate HTTP remote logger to TLS logger 2015-06-01 10:12:31 -07:00
Teddy Reed
4064fa6eb5 Pack and testing fixups 2015-05-28 12:17:27 -07:00
Blake Frantz
28d9237b50 Amazon EC2-based vagrant targets for RHEL/Amazon Linux
1. added docs for vagrant-aws support in Vagrantfile
2. removed aws target that have local vagrant support. inline-string'd aws.user_data
3. support building rhel6/7 in aws
4. correct aws-rhel6.6 name. it should be rhel6.5
2015-05-26 21:03:10 -07:00
Teddy Reed
8b3686a58a TLS plugin workflow tests 2015-05-26 19:55:00 -07:00
Teddy Reed
b90b21bc2d [Fix #1154] Clean up CMake messages and check TP 2015-05-23 17:15:28 -07:00
Teddy Reed
5969ae4fbf Clean up TLS-version from OpenSSL detection 2015-05-23 13:04:36 -07:00
Javier Marcos
9a4f611baf Merge pull request #1155 from javuto/osquery_packs_table
Osquery packs table
2015-05-21 20:32:45 -07:00
Javier Marcos
c6855fab43 Table for osquery packs 2015-05-19 18:44:28 -07:00
Teddy Reed
b3338dc5d2 Merge pull request #1146 from theopolis/tls
Towards TLS config/logging
2015-05-19 17:17:04 -07:00
Teddy Reed
2a1f496cc5 Towards TLS config/logging 2015-05-19 17:05:55 -07:00
Javier Marcos
65e6e38e0f Merge pull request #1143 from javuto/pack_config_changes
Support to load query packs as scheduled queries
2015-05-16 15:37:27 -07:00
Javier Marcos
47e680e825 Adding tests and implementing version checker 2015-05-15 22:25:19 -07:00
Teddy Reed
f5945f98b4 Oracle 5.11 2015-05-14 22:44:01 -07:00
Teddy Reed
525c584a0b Merge pull request #1141 from theopolis/static_cryptsetup
Build libcryptsetup statically
2015-05-14 22:33:56 -07:00