Commit Graph

573 Commits

Author SHA1 Message Date
Teddy Reed
2fe800d9b8 Add ASAN blacklists for GFlags and RocksDB 2015-12-14 15:09:46 -08:00
Teddy Reed
9d394065e3 [#1636] Add simple sharding to packs and pack queries 2015-12-10 10:01:53 -08:00
Teddy Reed
b88d6816f3 Additional TSK tables 2015-12-07 08:36:22 -08:00
Teddy Reed
c020bb87b4 Merge pull request #1705 from theopolis/dump
[#1702] Add config and database dumping to stdout
2015-12-06 21:41:31 -08:00
pathcl
6c8cc20117 PEP8 Compliance && Python 3.X compatibility
Signed-off-by: Teddy Reed
Merge-conflicts-by: Teddy Reed

Closes: #1586
2015-12-06 20:57:30 -08:00
Teddy Reed
9ebd292eb6 [#1648] Support multiple loggers 2015-12-06 11:10:10 -08:00
Teddy Reed
fef53fa0d0 Add config and database dumping to stdout 2015-12-06 11:01:26 -08:00
Teddy Reed
1acba4dfa6 Merge pull request #1700 from theopolis/tsk2
TSK integration and example tables
2015-12-04 11:26:03 -08:00
Teddy Reed
f687a84840 [Fix #1689] Remove C-style comments from config examples 2015-12-04 11:08:54 -08:00
Teddy Reed
373ce339dc TSK integration and example tables 2015-12-04 11:08:51 -08:00
Teddy Reed
e5bc6410ba Merge pull request #1697 from theopolis/fix_1660
[Fix #1660] Prevent spurious NETLINK recv retries
2015-12-02 23:56:39 -08:00
Teddy Reed
4dc6b9f0a3 [Fix #1660] Prevent spurious NETLINK recv retries 2015-12-02 23:33:20 -08:00
Teddy Reed
ffb5b7020e [Fix #1693, #1527] Add osquery-specific query planner output 2015-12-02 19:57:24 -08:00
Mykola Kokalko
652e5a24f3 [Fixed] build for lucid
[Added] python packages which are not installable via pip on lucid

[Moved] installing latest bison and openssl right before thrift building for lucid

[Fixed] package bison installation for not lucid

[Added] OpenSSL dependency for lucid

[Changed] prefix to /usr/local

[Fixed] wrong file for checking if dependency is intalled
2015-11-30 17:49:31 +01:00
Teddy Reed
44286eb611 Add hardware/internal (monitoring) packs and reduce FPs, duplicate queries 2015-11-26 16:00:53 -08:00
Teddy Reed
204b16a946 Merge pull request #1675 from theopolis/planner_or
Fix constraints stacking
2015-11-24 12:25:15 -08:00
Teddy Reed
3221fbd9b3 Fix constraints stacking 2015-11-22 22:53:23 -08:00
Teddy Reed
a3a05e7e1e [#1527] Add a --profile option to the shell, replace 'run' 2015-11-21 22:45:40 -08:00
Teddy Reed
98f212e7a9 Add a SQLite query planner for type detection 2015-11-15 13:56:16 -08:00
Teddy Reed
cef8f59054 Merge pull request #1639 from theopolis/cache
Table results caching
2015-11-14 16:22:24 -08:00
Teddy Reed
e1d7511600 Remove column type string representations 2015-11-14 15:57:30 -08:00
Teddy Reed
c2be670806 Table results caching
1. Table implementations (spec files) can mark the table as 'cachable'.
2. Cached results depend on the shortest/quickest interval of scheduled
queries that act on results of the table.
3. The table API generator blocks caching on index/additional/required
table column options.
2015-11-14 15:57:23 -08:00
Andrew Dunham
4ccdcc7864 Allow setting the mode of log files
This also sets the appropriate flags in glog
2015-11-11 11:37:55 -08:00
Teddy Reed
28bdcecc19 Custom flaky test marker 2015-11-08 02:29:49 -08:00
Teddy Reed
b29168a7b7 Use a null DB for the run test binary 2015-11-04 10:39:40 -08:00
Teddy Reed
5aa225d4c3 Merge pull request #1619 from sharvilshah/wifi
Implement wifi_networks tables for OS X
2015-11-02 16:11:21 -08:00
Teddy Reed
15215cdbc0 Add persistent splays 2015-11-02 14:10:04 -08:00
Teddy Reed
6aae4c9aa0 Fix tests and shell escape errors (faults) 2015-11-02 10:57:01 -08:00
Teddy Reed
50550e607a Build and provision edits for FreeBSD CI 2015-11-02 01:47:09 -08:00
Sharvil Shah
9a6d6d1293 Implement wifi_networks tables for OS X
If the option of remembering known Wi-Fi networks is enabled on a system,
they are persisted to disk as a preferences property list file.
This table is populated by parsing that file.
2015-11-01 16:53:51 -08:00
Teddy Reed
6a07135648 Passing clang Address/Leak Sanitize version 3.7 2015-11-01 04:00:21 -08:00
Teddy Reed
cd8f42844a Mark flaky integration tests 2015-10-28 09:40:17 -07:00
Teddy Reed
8ca2925ef0 [Fix #1583] Require osqueryd to have R/W access to RocksDB 2015-10-27 16:09:24 -07:00
Teddy Reed
654830cf11 Merge pull request #1594 from rcseacord/additional-sign-fixes
eliminated some warnings from Clang 3.7 analyze mode
2015-10-23 13:03:54 -03:00
Robert C. Seacord
1d9695ac31 eliminated some warnings from Clang 3.7 analyze mode 2015-10-21 06:02:58 +00:00
Teddy Reed
7ba87a88bb Merge pull request #1585 from rcseacord/additional-sign-fixes
Additional sign fixes
2015-10-19 11:25:18 -07:00
Scott J Roberts
1258800279 basic flag added
wrong thing on the clipboard

updated to ProgramArguments vs Program per theopolis
2015-10-19 11:56:12 -04:00
Robert C. Seacord
acb2f6f628 eliminating diagnostics, mostly for comparisons between signed and unsigned operations 2015-10-16 16:10:37 +00:00
Teddy Reed
c26f7bf8da Merge pull request #1565 from PickmanSec/master
added comment parsing for profile.py
2015-10-15 12:19:59 -07:00
Teddy Reed
2e7415f871 Convert wiki to UNIX format and refresh most content 2015-10-15 12:18:16 -07:00
Michael George
ed00e54aa7 added comment parsing
added packs to profiling

added comment scrubing.
2015-10-14 17:04:38 -07:00
Luis San Martin
0f0d873a96 sudo added as requeriment 2015-10-14 20:05:12 -03:00
Teddy Reed
1c12d274f4 Minor fix, return an empty query data 2015-10-13 09:25:10 -07:00
Teddy Reed
0440bb970f Fix test_5_daemon_sigint, and hardware_events tests 2015-10-13 08:40:23 -07:00
Javier Marcos
7442392c7f Fixing default configuration syntax, trailing comma 2015-10-09 14:50:00 +02:00
Mike Arpaia
5789d889f4 Merge pull request #1538 from marpaia/discovery_queries
[fix #1536] Schedule iteration pass-by-reference
2015-09-30 15:50:05 -07:00
Mike Arpaia
65df593d33 [fix #1536] Schedule iteration pass-by-reference
There was a bug in the `osquery::Schedule` container object such that,
when the iteration through the schedule occured, pack objects were being
passed by value (copied) instead of passed by reference. Thus, the
discovery query would be executed, the object's cache would be updated,
and then the object would go out of scope and be destructed, thus
leaving the original object without ever having ran the discovery query.
This caused discovery queries to thrash. Bad times.

I added a new test so that we don't regress here as well as const'd a
few functions that should have been const in `osquery::Pack`.
2015-09-30 15:41:43 -07:00
Teddy Reed
ad4b41cb84 Merge pull request #1524 from mathieuk/build_on_debian_wheezy_with_benchmark_change
Build on debian wheezy with recent changes
2015-09-30 13:32:55 -07:00
Mathieu Kooiman
d26d12a63c Make osquery build on debian again 2015-09-30 19:37:56 +02:00
Teddy Reed
bb65ec49ac [#1488] Shutdown Linux event publishers responsibly 2015-09-22 23:06:23 -07:00
Teddy Reed
7852c356ec Merge pull request #1494 from theopolis/signals
[#1488] Use signal handlers for teardown and reloading
2015-09-15 16:14:40 -07:00
Teddy Reed
7c2a625ef2 Use signal handlers for teardown and reloading 2015-09-14 16:57:00 -07:00
Teddy Reed
944e3de206 Merge pull request #1496 from theopolis/events_table
[#1487] Add osquery_events table to track pubsub stats
2015-09-14 15:27:35 -04:00
Mike Arpaia
aaa03a1058 Distributed queries client-side 2015-09-08 13:33:48 -07:00
Teddy Reed
b57040db60 Add osquery_events table to track pubsub stats 2015-09-03 15:10:53 -07:00
Teddy Reed
2813d3ab87 Add a Linux audit event publisher 2015-09-03 08:45:02 -07:00
Mike Arpaia
a140333441 [fix #1390] query pack re-org
This commit contains the features specified in #1390 as well as a
refactoring of the general osquery configuration code.

The API for the config plugins hasn't changed, although now there's a
`genPack` method that config plugins can implement. If a plugin doesn't
implement `genPack`, then the map<string, string> format cannot be used.
The default config plugin, the filesystem plugin, now implements
`genPack`, so existing query packs code will continue to work as it
always has.

Now many other config plugins can implement custom pack handling for
what makes sense in their context. `genPacks` is not a pure virtual, so
it doesn't have to be implemented in your plugin if you don't want to
use it. Also, more importantly, all config plugins can use the standard
inline pack format if they want to use query packs. Which is awesome.

For more information, refer to #1390, the documentation and the doxygen
comments included with this pull requests, as well as the following
example config which is now supported, regardless of what config plugin
you're using:

```json
{
  "options": {
    "enable_monitor": "true"
  },
  "packs": {
    "core_os_monitoring": {
        "version": "1.4.5",
        "discovery": [
          "select pid from processes where name like '%osqueryd%';"
        ],
        "queries": {
          "kernel_modules": {
              "query": "SELECT name, size FROM kernel_modules;",
              "interval": 600
          },
          "system_controls": {
              "query": "SELECT * FROM system_controls;",
              "interval": 600,
              "snapshot": true,
          },
          "usb_devices": {
              "query": "SELECT * FROM usb_devices;",
              "interval": 600
          }
        }
    },
    "osquery_internal_info": {
        "version": "1.4.5",
        "discovery": [
          "select pid from processes where name like '%osqueryd%';"
        ],
        "queries": {
          "info": {
              "query": "select i.*, p.resident_size, p.user_time, p.system_time, time.minutes as counter from osquery_info i, processes p, time where p.pid = i.pid;",
              "interval": 60,
              "snapshot": true
          },
          "registry": {
              "query": "SELECT * FROM osquery_registry;",
              "interval": 600,
              "snapshot": true
          },
          "schedule": {
              "query": "select name, interval, executions, output_size, wall_time, (user_time/executions) as avg_user_time, (system_time/executions) as avg_system_time, average_memory from osquery_schedule;",
              "interval": 60,
              "snapshot": true
          }
        }
    }
  }
}
```

The `osquery_packs` table was modified to remove the superfluous
columns which could already have been found in `osquery_schedule`. Two
more columns were added in their place, representing stats about pack's
discovery query execution history.

Notably, the internal API for the `osquery::Config` class has changed
rather dramatically as apart of the refactoring. We think this is an
improvement. While strictly adhering to the osquery config plugin
interface will have avoided any compatibility errors, advanced users may
notice compilation errors if they access config data directly. All
internal users of the config have obviously been updated. Yet another
reason to merge your code into mainline; we update it for you when we
refactor!
2015-09-02 10:56:26 -07:00
Teddy Reed
d63510f8bd Remove benchmark from third-party, prefer deps-build 2015-08-31 15:01:13 -07:00
Teddy Reed
4dd77a43a7 Remove cpp-netlib from third-party, prefer deps-build 2015-08-31 09:27:01 -07:00
Javier Marcos
1a50977a23 Adding magic table to check for libmagic data 2015-08-28 12:49:46 -07:00
Dave Gosselin
6899127e1e Add support for building on Fedora 21 2015-08-25 11:10:56 -04:00
Mathieu Kooiman
b151ecedc2 Refs https://github.com/facebook/osquery/issues/320
Add provisioning scripts to build osquery on Debian Wheezy and Debian Jessie.
2015-08-20 20:57:22 +02:00
Teddy Reed
73da2f1448 Merge pull request #1443 from sharvilshah/libressl
[#1329] Statically link against brew bottled libressl on OS X
2015-08-18 01:09:45 -07:00
Teddy Reed
fb93b73253 Merge pull request #1450 from eastebry/fix-package-dependencies
Fix lucid package building
2015-08-17 16:04:00 -07:00
Teddy Reed
5bf30a779d RocksDB usage speedups 2015-08-15 20:43:53 -07:00
Bryan Eastes
1ac7c5d21a Installing specific FPM version 2015-08-15 15:11:29 -07:00
Sharvil Shah
ccc086f050 Statically link against brew bottled libressl on OS X
* It wasn't straightforward to get OpenSSL building
 without avx/vxoprs optimizations on 10.10
* libressl is essentially a modern/lean-ish drop-in replacement for OpenSSL
and can build without avx optimizations to support older Macs

This change:

* Installs libressl (builds a bottle) using homebrew
* And statically links `libcrypto.a` and `libssl.a` unless
`BUILD_LINK_SHARED` is requested.

Fixes #1329
2015-08-13 15:31:26 -07:00
Javier Marcos
ee98bbde67 Fix for the version variable 2015-08-13 07:02:24 +02:00
Teddy Reed
251aded11c Merge pull request #1442 from javuto/darwin_package_10.9
Updating the script to create packages for 10.9 building
2015-08-12 18:16:36 -07:00
Teddy Reed
fd1b4b06e8 Merge pull request #1436 from theopolis/pack_place
[#1402] Add notes around pack paths in example.conf
2015-08-12 18:15:50 -07:00
Teddy Reed
b9ded9e7af [#1402] Add notes around pack paths in example.conf 2015-08-12 17:15:42 -07:00
Javier Marcos
861e0e4273 Using the code in lib.sh 2015-08-13 00:53:44 +02:00
Javier Marcos
54cb08c193 Updating the script to create packages for 10.9 building 2015-08-12 20:34:50 +02:00
Teddy Reed
ea0ca195f6 Update build/test for 10.9 (do not test extension)
OS X 10.9 should not build/test a kernel extension yet. The MAC policy framework is slightly different and the APIs/version dependencies need to be tested.
2015-08-12 10:40:53 -07:00
Sharvil Shah
e4f52589f5 Remove -mtune compile flag
Having `-mtune=i386` is causing compilation failure for gflags on ubuntu.
This change removes the `mtune` compile flag.
`-march` flag is already set to `x86-64` and according to gcc doc,
Specifying `-march=cpu-type` implies `-mtune=cpu-type.`

Fixes #1428
2015-08-06 17:14:33 -07:00
Teddy Reed
67b0f51ab5 Several small optimizations around internal SQL queries 2015-08-03 07:56:55 -07:00
Michael O'Farrell
dda11ce74a Executable size benchmark change. 2015-07-30 15:44:25 -07:00
Michael O'Farrell
346743e87f Benchmark using mean across 5 runs. 2015-07-29 16:50:19 -07:00
Michael O'Farrell
5956e685e9 Report bytes using word count 2015-07-29 15:10:20 -07:00
Michael O'Farrell
46ee4b491c Benchmark stripped binary size. 2015-07-29 11:35:04 -07:00
Michael O'Farrell
0cb5730d55 Added benchmark for executable size. 2015-07-28 22:34:22 -07:00
Teddy Reed
ff9cb71628 Various additional tests and benchmarks 2015-07-28 12:26:17 -07:00
Elan Ruusamäe
bb40956844 don't qualify any system with lsb-release as ubuntu 2015-07-25 14:52:59 +03:00
Michael O'Farrell
66b075a685 Merge pull request #1377 from mofarrell/benchmark
Added benchmarking targets.
2015-07-23 17:37:56 -07:00
Michael O'Farrell
a65f8dd93c Added benchmarking targets. 2015-07-23 17:07:42 -07:00
Teddy Reed
f03ec9ddac [Fix #1368] Restore autostart post-install scripts 2015-07-21 19:09:24 -07:00
Teddy Reed
fc24682816 Fix profile platform bug in leaks checking 2015-07-20 02:06:52 -07:00
Teddy Reed
95775be1d9 [Fix #1355] Allow plist keys with '.'
Boost property trees are level delimited using '.' characters.
An Apple property list may contain keys with '.' characters, so the plist conversion must use iterators and raw node appends.
2015-07-19 16:24:43 -07:00
Teddy Reed
a713d09f0e Install additional configs for HB/packages 2015-07-17 16:07:22 -07:00
Teddy Reed
270b4da540 [Fix #1339] Add kernel-build to packages when used 2015-07-16 15:23:29 -07:00
Javier Marcos
7241becda1 Fix tables JSON file name 2015-07-16 13:38:31 -07:00
Javier Marcos
84e0c77a98 Generation of table docs with packages and docs targets 2015-07-16 12:23:44 -07:00
Javier Marcos
01fabf910d Merge pull request #1343 from javuto/generate_tables_output
Adding support to generate documentation to external files
2015-07-15 17:10:27 -07:00
Javier Marcos
ba69bf8efa Adding support to generate documentation to external files 2015-07-15 13:18:41 -07:00
Teddy Reed
341245f751 Build/install gflags' static library on build hosts.
As of [homebrew #41151](https://github.com/Homebrew/homebrew/pull/41151) gflags is not installed with a static library.
Our build hosts must have static versions of gflags.
2015-07-14 17:20:55 -07:00
Teddy Reed
c269bbeaf3 Rollup of build changes 2015-07-14 13:45:53 -07:00
Teddy Reed
19d7a9e735 Merge pull request #1328 from mofarrell/kernel
Fixed cleanup in testing script in failure cases.
2015-07-13 16:41:44 -07:00
Michael O'Farrell
b2b1f0483d Fixed cleanup in testing script in failure cases. 2015-07-13 16:11:45 -07:00
Teddy Reed
0e49a3a9a1 Build separate OS X packages 2015-07-13 15:44:16 -07:00
Michael O'Farrell
dd1f0af0ff Build system changes for kernel extension testing and deployment. 2015-07-09 11:50:23 -07:00