Commit Graph

3191 Commits

Author SHA1 Message Date
Teddy Reed
9c01d4a6e3 Add quicklook_cache to Darwin (#2099) 2016-05-13 23:49:10 -07:00
Teddy Reed
77273f6500 Add logEvent API to logger plugins (#2088) 2016-05-13 19:48:40 -07:00
Teddy Reed
314e3467bc Check for systemd service in osqueryctl (#2090) 2016-05-13 19:48:22 -07:00
yying
15d1455957 Process Operations - osquery/core Integration (#2087)
This integrates the process abstraction operations within osquery core.
2016-05-13 19:47:45 -07:00
Marcin Wielgoszewski
484cf9c919 Add blurb about doorman (#2094) 2016-05-12 15:37:22 -07:00
Teddy Reed
0ffe0c9d16 Add --pack flag to the shell (#2093)
This introduces a new testing-related flag to the osqueryi shell.
The --pack flag takes a single argument, the name of a configured pack.

To list the configured packs try:
$ osqueryi -A osquery_packs

To run all queries, immediately and serially, try:
$ osqueryi --pack NAME_OF_PACK
2016-05-12 09:22:05 -07:00
Teddy Reed
e61116f5db Add Ubuntu Xenial 16.04 build support (#2089)
The current Ubuntu-provided Xenial does not mount a shared folder.
2016-05-12 08:47:05 -07:00
Christian McHugh
1f4d4f6965 - Install CMake on all Debian versions- Only install Ruby 1.8's development package on Wheeze
- Install CMake on all Debian versions
- Only install Ruby 1.8's development package on Wheezy
2016-05-12 08:45:33 -07:00
Teddy Reed
3813f09abb Clean up host identifier UUID access (#2092)
When selecting UUID as the host identifier the client follows:
- If host_identifier is set to hostname, return the hostname
- If UUID check a static UUID container
- If empty request the UUID from the backing storage, RocksDB
- If empty use the Apple hardware UUID or generate a UUID
- If the Apple UUID request fails generate a UUID

When no UUID is available in the backing store save the new UUID.
2016-05-12 08:40:37 -07:00
Teddy Reed
fcd00aff2c [Fix #2032] Remove manual Thrift library discovery (#2085)
The Thrift library should be discovered simiarly to all others.
When using the build-macros for library discovery the caller can choose
how to discover static or dynamic links.
2016-05-11 14:17:12 -07:00
yying
63c0258aec Abstracted platform specific process operations into a common interface. (#2069)
Added Windows support for process operations.
Added unit tests for process abstraction code for POSIX and Windows.
Modified CMake config files to support building the new code and unit tests.
2016-05-11 14:16:32 -07:00
Teddy Reed
ce2ba6d9c8 Refactor logStatus and logger initialization (#2081)
The initialization of a logger plugin was confusing. The 'init' step was
introduced to allow a daemon to buffer status events before a logger plugin
is determined by external/remote configuration. The buffered statuses could
then be transferred via a medium other than Glog (the default). To determine
if Glog should continue to write statuses to the filesystem the 'init' method
returned a Status.

Logger plugins should now use a feature method override to select how status
logs should be handled.
2016-05-11 12:05:09 -07:00
Teddy Reed
57c6b2a521 Revive the OS X kernel-based publishers (#2083)
The OS X kernel subscribers have not been starting because they expect the
publisher thread to run before they begin configuration. Due to some recent
refactors the publisher thread creation now occurs after configuration.

The subscriber logic to check for a valid kernel connection is still valid.

This commit has two additional side-effects:
- The RocksDB plugin is modified to use 3 background merge threads.
- The OS X kernel publisher syncing thread is now non-blocking.
2016-05-11 11:47:42 -07:00
Teddy Reed
d49c401324 Add decorators to example config (#2082) 2016-05-10 10:07:25 -07:00
Teddy Reed
0bb79c83e0 Remove the logHealth method from logger plugins (#2079) 2016-05-10 08:34:55 -07:00
Nick Anderson
2ac2eef3c8 Filtering out LowBattery crash logs (#2080)
The mobile 'LowBattery' crash logs provide little if any data of value,
and seem to be nothing more than notification events.  We're filtering
these out of the crashes table.
2016-05-10 08:01:28 -07:00
Teddy Reed
947799d28e Introduce within-query caching (#2077)
This adds a new optimization feature that allows expensive tables to cache
their results between JOINs. Consider JOINing a list of open sockets, for each
process, then requesting to hash each process path. This query may hash
the same path multiple times.

Within-query caching allows the hash table to respond with the previous
result of the hash request as long as the requested computation was the result
of a single query. Subsequent queries will perform subsequent hashing.
2016-05-09 10:32:33 -07:00
Teddy Reed
83b16fb579 SQLite 3.12.2 (#2078)
This updates the submodule commit which includes the SQLite 3.12.2 amalgamation.
2016-05-08 23:59:36 -07:00
Nick Anderson
209900d5a8 Adding mobile device crash parsing and 'type' column to Crashes table (#2076)
This commit adds mobile device crashes to the list of crash logs parsed by the Crashes table as well as adding a lamdba to improve code reuse.  The commit also adds a 'type' column to the table to indicate what kind of log this crash log was.
2016-05-06 13:14:06 -07:00
Nick Anderson
27fa7001c9 Renamed crash_log table. Small fixes to parsing behavior (#2074)
Renamed the crash_log table to crashes for future abstraction to other
operating systems. Also fixed how the table was parsing the most recent
stack trace and the registers.  Register values are now all parsed into
one column 'registers', which will be a space delimited string of the
form:

register:value register:value ... register:value

in order to best allow for OS abstraction.
2016-05-06 09:18:06 -07:00
Uma Reddy
e86a773a2e Update install-osx.md (#2071)
Add post-install notes for brew-based installs.
2016-05-06 08:31:22 -07:00
Teddy Reed
9011123f2d Add local_timezone and datetime (ISO) to time table (#2073)
It is often helpful to know the local timezone of the machine. For this use
local_timezone, as the base timezone will use local or UTC depending on the
--utc flag. This will be default=UTC in osquery 1.8.0.

The datetime field is added to mimic ISO 8601, along with iso_8601.
The timestamp field remains as the time stamp used for logging (within osquery)
and commonly outside of osquery. The goal for adding multiple representations
is to allow joining/augmenting of other tables.
2016-05-05 23:05:51 -07:00
Teddy Reed
921221ed46 Add osquery version to status logs (#2067) 2016-05-05 13:19:25 -07:00
Teddy Reed
9053a1f125 Include decorations within results events (#2068)
Currently, the decorations are applied to statuses and results in batch format.
This seems like an oversight, decorations should be on all results.
2016-05-05 13:19:10 -07:00
Zachary Wasserman
9fd807df54 Add --build-bottle argument to AWS SDK install (#2066) 2016-05-03 14:25:29 -07:00
Teddy Reed
5a56805970 Introduce --utc flag to convert all calendar UNIX times to UTC (#2064)
Beginning in version 1.8.0 all time uses will converge on an osquery-provided
getUnixTime() API call that returns, by default, UNIX time integers converted
to UTC/GMT. The 'time' table will respond with the parsed time for the
configuration. If the timezone is not UTC then osquery is using localtime.

This configuration option will affect the 'unix_time' response in the 'time'
table. Because of this configurable-effect the table is extended to include
'local_time' which is always the system local UNIX time.
2016-05-03 13:00:31 -07:00
Teddy Reed
b6f09a7ecb Add action for snapshot query results (#2061) 2016-05-03 11:16:22 -07:00
Zachary Wasserman
4bb95a3a29 Fix AWS credentials test failure when env vars set (#2062) 2016-05-02 19:17:09 -07:00
Teddy Reed
9c541ee1ce [Fix #2058] Send Content-Encoding when using TLS compression (#2060)
When the configuration flag: --logger_tls_compress is used the client should
send a "Content-Encoding: gzip".
2016-05-02 18:03:04 -07:00
Zachary Wasserman
5f778a201b Prevent error when TLS response body is empty (#2053) 2016-04-25 18:10:12 -07:00
Zachary Wasserman
10781e896e Add wiki docs for AWS logger plugins (#2052) 2016-04-25 16:20:01 -07:00
Zachary Wasserman
262833c86a Add AWS Kinesis and Firehose logger plugins (#2045)
This commit adds logger plugin implementations for the Amazon
Kinesis (https://aws.amazon.com/kinesis/) and Kinesis
Firehose (https://aws.amazon.com/kinesis/firehose/) services. To support
these plugins there are a number of utility classes and functions for
AWS authentication, configuration and API integration. The logger plugin
implementations take advantage of the BufferedLogForwarder base class
for reliable buffering and batch sending of logs. In their current
implementations, the logger plugins only support sending of result logs
to these AWS services.
2016-04-25 16:19:51 -07:00
Zachary Wasserman
31748d249b Add AWS C++ SDK to provision scripts (#2046) 2016-04-25 09:01:29 -07:00
Zachary Wasserman
9be5c465ef Improve error messages for TLS JSON (de)serialization (#2050) 2016-04-22 11:29:56 -07:00
Teddy Reed
87f7b32d0a Fix make_package build for OS X/Linux (#2049)
There seems to have been a regression in package building.
The ./tools/lib.sh script now overloads the SCRIPT_DIR variable,
which is also used in the package build scripts.

This changes the file-local variable in ./tools/lib.sh.
2016-04-22 11:15:26 -07:00
Mike Arpaia
77e8253c4d Changing the string 'true' to the number 1 in the README example 2016-04-22 09:57:50 -07:00
Mike Arpaia
8e73cf82d2 formatting changes for windows documentation 2016-04-19 10:25:28 -07:00
yying
0c15e1e732 Adding README explaining Windows provisioning and moved Windows provisioning README to development/, added it as link in navbar. Also added a mini-HOWTO section for running the provisioning script.
Adding README explaining Windows provisioning and moved Windows provisioning README to development/, added it as link in navbar.  Also added a mini-HOWTO section for running the provisioning script.
2016-04-19 09:38:18 -07:00
yying
18013da24b Initial support for building on Windows (#2015)
* Committing changes related to our experimentation with a "pure" Win64 build

* Placates CMake such that a Visual Studio 14 x64 solution is generated!

* Updated changes to fix the issue of GFlags not being found.

* Added cases to handle Win64 specific CMake options such as include/link
directories and compiler options

* Comment change in CMakeList.txt

* Changed wording of get_platform error message. Adding Powershell
provisioning script.

* Finalized provision powershell script

* Added a deployment XML file for insuring C++ support exists for VS2015

* Added admin check and resolved some potential PATH issues.

* Fixed some potential bugs in VS2015 automated install

* Adding a wrapper for provision.ps1 so people don't need to know obscure
powershell syntax

* Fixing a bug with third-party archive extraction

* Ignoring the updating of pip for now...

* Fix invocation of choco.exe

* Resolved pip install issues

* Removed some debugging residue

* Changing get_platform.py from tabs to spaces

* Added distro detection for ubuntu and darwin

* Using 4 spaces

* Added a newline after powershell invocation

* Added OS detection for freebsd and fedora

* Fixed bug with freebsd

* Adding arguments parsing to prepare for modularizing platform detection

* Removing bash platform detection logic and forwarding the task to
get_platform.py

* Removing get_platform support in provision.sh since it doesn't appear to
be used anywhere now

* Fixed some comment/spacing issues. Made a few efficiency changes

* A few bug fixes, revereted back to WIN32 variable for now

* Added Facebook copyright information.

* Fixed boost and rocksdb library paths

* Added support for installing our custom chocolatey packages

* Fixed path to ignore the symlink

* Force environment variable propagation

* Forgot to add new line after make-win64-dev-env.bat

* Added error handling for choco install failures

* Handles download and python/pip errors

* When chocolatey is not detected, refreshenv.bat is not found in the PATH.
Hardcode the PATH as per chocolatey install instructions

* Takes care of updating git submodules in third-party\

* Fixes a bug in which Linux provisioning fails because of unset FAMILY
env var.
2016-04-15 14:01:16 -07:00
Zachary Wasserman
9370d9e1f5 Fix pessimizing move warnings (#2040)
Fixes warnings generated by Clang when `std::move` is used in a `return`
statement. Verified by `make clean`, saw no warnings, `make test` passes.
2016-04-15 13:15:23 -07:00
Zachary Wasserman
2376fedb73 Introduce BufferedLogForwarder (#2036)
BufferedLogForwarder is a base class for external log forwarding schemes
that require buffering and retries. It generalizes the logic from the
existing TLSLogForwarderRunner.

The existing TLSLoggerPlugin and TLSLogForwarderRunner are refactored to
demonstrate the use of this new base class.
2016-04-15 09:49:35 -07:00
Serey Ty
a8c8e9e9cb fix the cpu id displayed in core column (#2038) 2016-04-15 08:38:32 -07:00
Nick Anderson
134c2750c2 Adding Crash Logs table for OS X (#2027)
Added a table that parses out some of the informaiton in the OS X logs
stored in /Library/Logs/DiagnosticReports as well as
/Users/<user>/Library/Logs/DiagnosticReports
2016-04-13 16:25:40 -07:00
Teddy Reed
a98e483ef5 Reset configuration state in relative tests 2016-04-12 13:06:02 -07:00
Teddy Reed
192953bcd3 [Fix #2030] Add generate_foreign and --enable_foreign (#2034) 2016-04-11 09:33:45 -07:00
Teddy Reed
fd53f7ce6c Remove devel switch for brew's cpp-netlib package 2016-04-10 19:34:33 -07:00
Chris Long
9ccbd08330 Adding Elite Keylogger Detection to osx-attacks (#2031) 2016-04-09 13:54:15 -07:00
Serey Ty
198c8ff978 Add detection for OSX Pirrit (#2029)
See: https://threatpost.com/mac-adware-osx-pirrit-unleashes-ad-overload-for-now/117273/

Someone also wrote a removal for it:
https://github.com/aserper/osx.pirrit_removal/blob/master/remove_pirrit.sh
2016-04-08 11:29:44 -07:00
Zachary Wasserman
8cb862e327 Allow TLS to proceed without node key if enrollment disabled 2016-04-07 08:47:03 -07:00
Justin Gerace
8207feecd1 Add time column to shell_history table
* Add time column to shell_history table

* Add support for zsh timestamp format in shell_history table
2016-04-05 11:02:14 -07:00