valitydev/osquery-1
14
0
Fork 0
mirror of https://github.com/valitydev/osquery-1.git synced 2024-11-06 09:35:20 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

docs: Quick pass through documentation (#3391)

Browse Source
This commit is contained in:
Teddy Reed 2017-06-09 01:47:16 +01:00 committed by GitHub
parent 8ad086098c
commit a66192903c
31 changed files with 198 additions and 280 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
CMakeLists.txt
View File

@ -610,7 +610,7 @@ endif()
# make format-all
find_program(CLANGFORMAT_EXECUTABLE "clang-format" ENV PATH)
add_custom_target(
format-all
format_all
find osquery include tools \( -name "*.h" -o -name "*.cpp" -o -name "*.mm" \)
-exec "${CLANGFORMAT_EXECUTABLE}" -i {} +
WORKING_DIRECTORY "${CMAKE_SOURCE_DIR}"

71
CONTRIBUTING.md
View File

@ -2,8 +2,7 @@ All osquery development occurs in feature branches and all contributions occur v
## Contributor License Agreement ("CLA")
In order to accept your pull request, we need you to submit a CLA. You only need
to do this once to work on any of Facebook's open source projects.
In order to accept your pull request, we need you to submit a CLA. You only need to do this once to work on any of Facebook's open source projects.
Complete your CLA here: <https://code.facebook.com/cla>
@ -27,7 +26,7 @@ origin git@gitHub.com:facebook/osquery.git (push)
Now, use the GitHub UI to fork osquery to your personal GitHub organization. Then, add the remote URL of your fork to git's local remotes:
```
$ git remote add marpaia git@github.com:marpaia/osquery.git
$ git remote add $USER git@github.com:$USER/osquery.git
```
Now, your "remote" should be set up as follows:
@ -51,7 +50,7 @@ Write your code and when you're ready to put up a Pull Request, push your local
```
$ git add .
$ git commit -m "my awesome feature!"
$ git push -u marpaia my-feature
$ git push -u $USER my-feature
```
Visit https://github.com/facebook/osquery and use the web UI to create a Pull Request. Once your pull request has gone through sufficient review and iteration, please squash all of your commits into one commit.
@ -64,71 +63,11 @@ In most cases your PR should represent a single body of work. It is fine to chan
Pull requests will often need revision, most likely after the required code review from the friendly core development team. :D
Our preference is to minimize the number of commits in a pull request and represent each body of change as a single, concise, commit. To do this we ask you to [squash](https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History) your git commits before we merge changes. There are two basic workflows for squashing, let's run through examples of each.
**You create a pull request with several commits**
If you open a pull request from a branch with 'stacked commits' the request will ask us to merge the lot of them into our master. That is no fun, and we will promptly ask you to squash! If you have 5 commits in the pull request you can squash these into 1 using:
```
$ git rebase -i HEAD~5
```
This tells git to perform an interactive rebase onto the `HEAD-5` commit. The interactive part means your favorite editor will prompt you for actions. To turn 5 commits into 1 we'll `pick` the first, then `squash` the remaining, in this case 4. The 'squashed' 4 will be squashed into the commit we 'pick'. Within the interactive editor, change the last 4 'picks' to an `s`, shorthand for squash.
For example here are my 5 commits while editing this guide:
```
pick dc849a9 Update contributing with squash instructions
pick 8b1fa6b Minor change to contributing
pick 45baf1a Delete whitespace in contributing
pick bded8d7 Delete more whitespace
pick ab49a55 Fix small mistake
```
I can squash these into a single commit by updating and saving:
```
pick dc849a9 Update contributing with squash instructions
s 8b1fa6b Minor change to contributing
s 45baf1a Delete whitespace in contributing
s bded8d7 Delete more whitespace
s ab49a55 Fix small mistake
```
The next prompt allows us to amend the commit message:
```
# This is a combination of 5 commits.
# The first commit's message is:
Update contributing with squash instructions
# This is the 2nd commit message:
Minor change to contributing
# This is the 3rd commit message:
Delete whitespace in contributing
# This is the 4th commit message:
Delete more whitespace
# This is the 5th commit message:
Fix small mistake
```
I will remove everything except for the first line, as that is the thesis for all 5 commits, and save:
```
# This is a combination of 5 commits.
# The first commit's message is:
Update contributing with squash instructions
```
When you save you can verify your 5 commits are now 1 by inspecting the `git log`. To update your pull request you'll need to force-push since you just rewrote your local history:
```
$ git push -f
```
Please feel free to add several commits to your Pull Request. When it comes time to merge into **master** all commits in a Pull Request will be squashed using GitHub's tooling into a single commit. The development team will usually choose to remove the commit body and keep the GitHub-appended `(#PR)` number in the commit title.
**You make updates to your pull request**
If the pull request needs changes, or you decide to update the content, please 'amend' your previous commit:
If the pull request needs changes, or you decide to update the content, consider 'amending' your previous commit:
```
$ git commit --amend

3
Makefile
View File

@ -111,6 +111,9 @@ check:
@$(PATH_SET) cppcheck --quiet --enable=warning --error-exitcode=1 \
-I ./include ./osquery
audit:
@tools/audit.sh
debug_build:
cd build/debug_$(BUILD_DIR) && \
$(DEFINES) $(MAKE) --no-print-directory $(MAKEFLAGS)

5
docs/wiki/deployment/anomaly-detection.md
View File

@ -1,11 +1,10 @@
An osquery deployment can help you establish an infrastructural baseline, allowing you to detect malicious activity using scheduled queries.
This approach will help you catch known malware ([WireLurker](http://bits.blogs.nytimes.com/2014/11/05/malicious-software-campaign-targets-apple-users-in-china/), IceFog, Imuler, etc.), and more importantly, unknown malware. Let's look at Mac OS X startup items for a given laptop using [osqueryi](../introduction/using-osqueryi.md):
This approach will help you catch known malware ([WireLurker](http://bits.blogs.nytimes.com/2014/11/05/malicious-software-campaign-targets-apple-users-in-china/), IceFog, Imuler, etc.), and more importantly, unknown malware. Let's look at MacOS startup items for a given laptop using [osqueryi](../introduction/using-osqueryi.md):
```sh
$ osqueryi
osqueryi> SELECT * FROM startup_items;
+--------------+----------------------------------------------------------+
| name | path |
+--------------+----------------------------------------------------------+
@ -43,7 +42,7 @@ It's clear that a suspicious application called "Phone" was added to this host's
### Case-study: WireLurker
In November, Palo Alto Networks [discovered](http://researchcenter.paloaltonetworks.com/2014/11/wirelurker-new-era-os-x-ios-malware/) a new piece of OS X malware called Wirelurker.
In November 2015, Palo Alto Networks [discovered](http://researchcenter.paloaltonetworks.com/2014/11/wirelurker-new-era-os-x-ios-malware/) a new piece of macOS malware called Wirelurker.
If you have osquery deployed, you can search for their static IOCs (indicators of compromise):

2
docs/wiki/deployment/aws-logging.md
View File

@ -60,4 +60,4 @@ Similarly for Kinesis Firehose delivery streams, the stream name must be specifi
}
```
**Note**: Kinesis services have a maximum 1MB record size. Result logs bigger than this will not be forwarded by `osqueryd` as they will be rejected by the Kinesis services.
**Note**: Kinesis services have a maximum 1MB record size. Result logs bigger than this will not be forwarded by **osqueryd** as they will be rejected by the Kinesis services.

63
docs/wiki/deployment/configuration.md
View File

@ -1,6 +1,6 @@
An osquery deployment consists of:
* Installing the tools for [Windows](../installation/install-windows.md), [OS X](../installation/install-osx.md), or [Linux](../installation/install-linux.md)
* Installing the tools for [Windows](../installation/install-windows.md), [macOS](../installation/install-osx.md), or [Linux](../installation/install-linux.md)
* Reviewing the [osqueryd](../introduction/using-osqueryd.md) introduction
* Configuring and starting the **osqueryd** service (this page)
* Managing and [collecting](log-aggregation.md) the query results
@ -26,12 +26,12 @@ flags](../installation/cli-flags.md) overview for a complete list of these
parameters.
The default config plugin, **filesystem**, reads from a file and optional
directory ".d" based on the filename. The included init scripts set the default
directory ".d" based on the filename. The included initscripts set the default
config path as follows:
* Windows: **C:\ProgramData\osquery\osquery.conf**
* Linux: **/etc/osquery/osquery.conf** and **/etc/osquery/osquery.conf.d/**
* Mac OS X: **/var/osquery/osquery.conf** and **/var/osquery/osquery.conf.d/**
* MacOS: **/var/osquery/osquery.conf** and **/var/osquery/osquery.conf.d/**
You may override the **filesystem** plugin's path using
`--config_path=/path/to/osquery.conf`. You may also use the ".d/" directory
@ -46,7 +46,7 @@ Here is an example config that includes options and the query schedule:
"schedule_splay_percent": 10
},
"schedule": {
"macosx_kextstat": {
"macos_kextstat": {
"query": "SELECT * FROM kernel_extensions;",
"interval": 10
},
@ -58,13 +58,13 @@ Here is an example config that includes options and the query schedule:
}
```
This config tells osqueryd to schedule two queries, **macosx_kextstat** and
This config tells osqueryd to schedule two queries, **macos_kextstat** and
**foobar**:
* the schedule keys must be unique
* the `interval` specifies query frequency (in seconds)
The first query will log changes to the OS X host's kernel extensions,
The first query will log changes to the macOS host's kernel extensions,
with a query interval of 10 seconds. Consider using osquery's [performance
tooling](performance-safety.md) to understand the performance impact for each
query.
@ -76,7 +76,7 @@ stored in RocksDB. On subsequent runs, only result-set-difference (changes) are
Scheduled queries can also set: `"removed":false` and `"snapshot":true`. See
the next section on [logging](../deployment/logging.md), and the below configuration specification to learn how query options affect the output.
**Note** that the `interval` time in seconds is how many seconds the _daemon_
> NOTICE: that the `interval` time in seconds is how many seconds the _daemon_
itself has been running before the scheduled query will be executed. If the
system is suspended or put to sleep the progression of time "freezes" and
resumes when the system comes back online. For example a scheduled query with
@ -102,13 +102,13 @@ consist of pack name to pack content JSON data structures.
"packs": {
"internal_stuff": {
"discovery": [
"select pid from processes where name = 'ldap';"
"SELECT pid FROM processes WHERE name = 'ldap';"
],
"platform": "linux",
"version": "1.5.2",
"queries": {
"active_directory": {
"query": "select * from ad_config;",
"query": "SELECT * FROM ad_config;",
"interval": "1200",
"description": "Check each user's active directory cached settings."
}
@ -118,7 +118,7 @@ consist of pack name to pack content JSON data structures.
"shard": "10",
"queries": {
"suid_bins": {
"query": "select * from suid_bins;",
"query": "SELECT * FROM suid_bins;",
"interval": "3600"
}
}
@ -188,8 +188,8 @@ Discovery queries look like:
```json
{
"discovery": [
"select pid from processes where name = 'foobar';",
"select count(*) from users where username like 'www%';"
"SELECT pid FROM processes WHERE name = 'foobar';",
"SELECT count(*) FROM users WHERE username like 'www%';"
],
"queries": {}
}
@ -288,7 +288,7 @@ Example:
{
"schedule": {
"users_browser_plugins": {
"query": "SELECT * FROM users JOIN browser_plugins USING (uid)",
"query": "SELECT * FROM users JOIN browser_plugins USING (uid);",
"interval": 60
},
"hashes_of_bin": {
@ -315,9 +315,10 @@ The basic scheduled query specification includes:
* `shard`: restrict this query to a percentage (1-100) of target hosts
The `platform` key can be:
* `darwin` for OS X hosts
* `darwin` for MacOS hosts
* `freebsd` for FreeBSD hosts
* `linux` for any RedHat or Debian-based hosts
* `posix` for `linux`, `freebsd`, and `linux` hosts
* `windows` for any Windows desktop or server hosts
* `any` or `all` for all, alternatively no platform key selects all
@ -342,7 +343,7 @@ The above section on packs almost covers all you need to know about query packs.
"version": "1.7.0",
"platform": "linux",
"discovery": [
"SELECT * FROM processes WHERE name = 'osqueryi'"
"SELECT * FROM processes WHERE name = 'osqueryi';"
]
}
}
@ -428,13 +429,13 @@ Example:
```json
{
"views": {
"kernel_hashses" : "select hash.path as kernel_binary, version, hash.sha256 as sha256, hash.sha1 as sha1, hash.md5 as md5 from (select path || '/Contents/MacOS/' as directory, name, version from kernel_extensions) join hash using (directory)"
"kernel_hashses" : "SELECT hash.path AS kernel_binary, version, hash.sha256 AS sha256, hash.sha1 AS sha1, hash.md5 AS md5 FROM (SELECT path || '/Contents/MacOS/' AS directory, name, version FROM kernel_extensions) JOIN hash USING (directory);"
}
}
```
```SQL
select * from kernel_hashes where kernel_binary not like "%apple%"
SELECT * FROM kernel_hashes WHERE kernel_binary NOT LIKE "%apple%";
```
### Decorator queries
@ -445,8 +446,8 @@ Decorator queries exist in osquery versions 1.7.3+ and are used to add additiona
{
"decorators": {
"load": [
"SELECT version FROM osquery_info",
"SELECT uuid AS host_uuid FROM system_info"
"SELECT version FROM osquery_info;",
"SELECT uuid AS host_uuid FROM system_info;"
],
"always": [
"SELECT user AS username FROM logged_in_users WHERE user <> '' ORDER BY time LIMIT 1;"
@ -483,12 +484,12 @@ Example configuration:
{
"decorators": {
"load": [
"SELECT 'collision' as name",
"SELECT 'collision' as hostIdentifier",
"SELECT 'collision' as calendarTime",
"SELECT 'collision' as unixTime",
"SELECT 'collision' as columns",
"SELECT 'collision' as action"
"SELECT 'collision' AS name;",
"SELECT 'collision' AS hostIdentifier;",
"SELECT 'collision' AS calendarTime;",
"SELECT 'collision' AS unixTime;",
"SELECT 'collision' AS columns;",
"SELECT 'collision' AS action;"
]
}
}
@ -515,18 +516,18 @@ The `interval` type uses a map of interval 'periods' as keys, and the set of dec
## Chef Configuration
Here are example chef cookbook recipes and files for OS X and Linux
Here are example chef cookbook recipes and files for macOS and Linux
deployments. Consider improving the recipes using node attributes to further
control what nodes and clients enable osquery. It helps to create a canary or a
testing set that implements a separate "testing" configuration. These recipes
assume you are deploying the OS X package or the Linux package separately.
assume you are deploying the macOS package or the Linux package separately.
### Chef OS X
### Chef OS X / macOS
Consider the default recipe:
```ruby
# Domain used by the OS X LaunchDaemon.
# Domain used by the macOS LaunchDaemon.
domain = 'com.facebook.osquery.osqueryd'
config_path = '/var/osquery/osquery.conf'
pid_path = '/var/osquery/osquery.pid'
@ -647,8 +648,8 @@ service service_name do
end
```
And the same configuration file from the OS X example is appropriate.
And the same configuration file from the macOS example is appropriate.
## osqueryctl helper
To test a deploy or configuration we include a short helper script called **osqueryctl**. There are several actions including "start", "stop", and "config-check" that apply to both OS X and Linux.
To test a deploy or configuration we include a short helper script called **osqueryctl**. There are several actions including "start", "stop", and "config-check" that apply to both macOS and Linux.

4
docs/wiki/deployment/debugging.md
View File

@ -2,7 +2,7 @@ Here are some debugging tips and tricks related to the daemon and shell from a d
Almost every situation requiring debugging should ultimately be solved with bug fixes or better documentation. In these cases the documentation usually surfaces in the form of verbose messages in the tools.
Please feel encouraged to add additional messages in the code, or create Github issues documenting your experience and suggestions for documentation or code improvements.
Please feel encouraged to add additional messages in the code, or create GitHub issues documenting your experience and suggestions for documentation or code improvements.
### Running the shell or daemon in verbose mode
@ -129,7 +129,7 @@ If you see:
Error registering subscriber: process_file_events: No kernel event publisher
```
This is an informational message with mis-categorized severity. The message indicates that a requested companion kernel extension does not exist and the associated `process_file_events` subscriber on OS X cannot start. It is safe to ignore.
This is an informational message with mis-categorized severity. The message indicates that a requested companion kernel extension does not exist and the associated `process_file_events` subscriber on macOS cannot start. It is safe to ignore.
### Testing event subscribers

12
docs/wiki/deployment/file-integrity-monitoring.md
View File

@ -29,11 +29,11 @@ The two areas below that are relevant to FIM are the scheduled query against `fi
{
"schedule": {
"crontab": {
"query": "select * from crontab;",
"query": "SELECT * FROM crontab;",
"interval": 300
},
"file_events": {
"query": "select * from file_events;",
"query": "SELECT * FROM file_events;",
"removed": false,
"interval": 300
}
@ -113,10 +113,10 @@ File accesses on Linux using inotify may induce unexpected and unwanted performa
The above configuration snippet will enable file integrity monitoring for 'homes', 'etc', and 'tmp' but only enable access monitoring for the 'homes' and 'etc' directories.
**WARNING:** The hashes of files will not be calculated to avoid generating additional access events.
> NOTICE: The hashes of files will not be calculated to avoid generating additional access events.
### Process File Accesses on OS X
### Process File Accesses on macOS
It is possible to monitor for file accesses by process using the osquery OS X kernel module. File accesses induce a LOT of stress on the system and are more or less useless giving the context from userland monitoring systems (aka, not having the process that caused the modification).
It is possible to monitor for file accesses by process using the osquery macOS kernel module. File accesses induce a LOT of stress on the system and are more or less useless giving the context from userland monitoring systems (aka, not having the process that caused the modification).
If the OS X kernel extension is running, the `process_file_events` table will be populated using the same **file_paths** key in the osquery config. This implementation of access monitoring includes process PIDs and should not cause CPU or memory latency outside of the normal kernel extension/module guarantees. See [../development/kernel.md](Kernel) for more information.
If the macOS kernel extension is running, the `process_file_events` table will be populated using the same **file_paths** key in the osquery config. This implementation of access monitoring includes process PIDs and should not cause CPU or memory latency outside of the normal kernel extension/module guarantees. See [../development/kernel.md](Kernel) for more information.

6
docs/wiki/deployment/logging.md
View File

@ -36,7 +36,7 @@ As the above directory listing reveals, *osqueryd.INFO* is a symlink to the most
The results of your scheduled queries are logged to the "results log". These are differential changes between the last (most recent) query execution and the current execution. Each log line is a JSON string that indicates what data has been added/removed by which query. There are two format options, *single*, or event, and *batched*. Some queries do not make sense to log "removed" events like:
```sql
SELECT i.*, p.resident_size, p.user_time, p.system_time, t.minutes as c
SELECT i.*, p.resident_size, p.user_time, p.system_time, t.minutes AS c
FROM osquery_info i, processes p, time t
WHERE p.pid = i.pid;
```
@ -47,7 +47,7 @@ By adding an outer join of `time` and using `time.minutes` as a counter this que
{
"schedule": {
"osquery_monitor": {
"query": "SELECT ... t.minutes as c FROM time t WHERE ...",
"query": "SELECT ... t.minutes AS c FROM time t WHERE ...",
"interval": 60,
"removed": false
}
@ -66,7 +66,7 @@ To schedule a snapshot query, use:
{
"schedule": {
"mounts": {
"query": "select * from mounts",
"query": "SELECT * FROM mounts;",
"interval": 3600,
"snapshot": true
}

24
docs/wiki/deployment/performance-safety.md
View File

@ -12,27 +12,27 @@ Consider the following `osquery.conf`:
{
"schedule": {
"alf_services": {
"query": "select service, process from alf_services where state != 0;",
"query": "SELECT service, process FROM alf_services WHERE state != 0;",
"interval": 60
},
"installed_applications": {
"query": "select name, path, bundle_version, minimum_system_version, applescript_enabled, bundle_executable from apps;",
"query": "SELECT name, path, bundle_version, minimum_system_version, applescript_enabled, bundle_executable FROM apps;",
"interval": 60
},
"all_kexts": {
"query": "select name, version from kernel_extensions;",
"query": "SELECT name, version FROM kernel_extensions;",
"interval": 60
},
"non_apple_kexts": {
"query": "select * from kernel_extensions where name not like 'com.apple.%' and name != '__kernel__';",
"query": "SELECT * FROM kernel_extensions WHERE name NOT LIKE 'com.apple.%' AND name != '__kernel__';",
"interval": 60
},
"processes_binding_to_ports": {
"query": "select distinct process.name, listening.port, listening.protocol, listening.family, listening.address, process.pid, process.path, process.on_disk, process.parent, process.start_time from processes as process join listening_ports as listening on process.pid = listening.pid;",
"query": "SELECT DISTINCT process.name, listening.port, listening.protocol, listening.family, listening.address, process.pid, process.path, process.on_disk, process.parent, process.start_time FROM processes AS process JOIN listening_ports AS listening ON process.pid = listening.pid;",
"interval": 60
},
"processes_not_on_disk": {
"query": "select * from processes where on_disk != 1;",
"query": "SELECT * FROM processes WHERE on_disk != 1;",
"interval": 60
}
}
@ -45,17 +45,17 @@ For this we can use `./tools/analysis/profile.py` to profile the queries by runn
```
$ sudo -E python ./tools/analysis/profile.py --config osquery.conf
Profiling query: select * from kernel_extensions where name not like 'com.apple.%' and name != '__kernel__';
Profiling query: SELECT * FROM kernel_extensions WHERE name NOT LIKE 'com.apple.%' AND name != '__kernel__';
D:0 C:0 M:0 F:0 U:1 non_apple_kexts (1/1): duration: 0.519426107407 cpu_time: 0.096729864 memory: 6447104 fds: 5 utilization: 9.5
Profiling query: select name, path, bundle_version, minimum_system_version, applescript_enabled, bundle_executable from apps;
Profiling query: SELECT name, path, bundle_version, minimum_system_version, applescript_enabled, bundle_executable FROM apps;
D:0 C:0 M:0 F:0 U:1 installed_applications (1/1): duration: 0.507317066193 cpu_time: 0.113432314 memory: 7639040 fds: 6 utilization: 11.15
Profiling query: select service, process from alf_services where state != 0;
Profiling query: SELECT service, process FROM alf_services WHERE state != 0;
D:0 C:0 M:0 F:0 U:0 alf_services (1/1): duration: 0.525090932846 cpu_time: 0.021108868 memory: 5406720 fds: 5 utilization: 1.9
Profiling query: select * from processes where on_disk != 1;
Profiling query: SELECT * FROM processes WHERE on_disk != 1;
D:0 C:0 M:0 F:0 U:0 processes_not_on_disk (1/1): duration: 0.521270990372 cpu_time: 0.030440911 memory: 6148096 fds: 5 utilization: 2.8
Profiling query: select name, version from kernel_extensions;
Profiling query: SELECT name, version FROM kernel_extensions;
D:0 C:0 M:0 F:0 U:1 all_kexts (1/1): duration: 0.522475004196 cpu_time: 0.089579066 memory: 6500352 fds: 5 utilization: 8.65
Profiling query: select distinct process.name, listening.port, listening.protocol, listening.family, listening.address, process.pid, process.path, process.on_disk, process.parent, process.start_time from processes as process join listening_ports as listening on process.pid = listening.pid;
Profiling query: SELECT DISTINCT process.name, listening.port, listening.protocol, listening.family, listening.address, process.pid, process.path, process.on_disk, process.parent, process.start_time FROM processes AS process JOIN listening_ports AS listening ON process.pid = listening.pid;
D:2 C:1 M:0 F:0 U:2 processes_binding_to_ports (1/1): duration: 1.02116107941 cpu_time: 0.668809664 memory: 6340608 fds: 5 utilization: 44.3
```

4
docs/wiki/deployment/process-auditing.md
View File

@ -26,13 +26,13 @@ Another audit-based table is provided on Linux: `socket_events`. This table repo
Use `--audit_allow_sockets` to enable the associated event subscriber.
## OS X process auditing
## macOS process auditing
osquery does not (yet?) support audit on Darwin platforms. It is possible to enable process auditing using a kernel extension. The extension can be downloaded and installed from the [http://osquery.io/downloads](http://osquery.io/downloads) page. It must be kept up to date alongside the osquery daemon and shell since there are automatic API restrictions applied. If you are running a 1.7.5 daemon, a 1.7.5 extension is needed otherwise the extension will not be used. If you are interested in the extension's design and development please check out the [kernel](../development/kernel.md) development guide.
At the heart of the extension is a replica of the userland Table Pubsub Framework. All osquery-developed kernel code is 100% OS public API compatible and designed to introduce as little stability risk as possible. Running the kernel extension without the osquery daemon should not impact performance.
There are no configuration or additional options required to enable process auditing on OS X if the kernel extension is installed. The osquery daemon will auto-detect the extension and attempt to load and connect when the process starts.
There are no configuration or additional options required to enable process auditing on macOS if the kernel extension is installed. The osquery daemon will auto-detect the extension and attempt to load and connect when the process starts.
## osquery events optimization

22
docs/wiki/deployment/remote.md
View File

@ -16,8 +16,6 @@ The initial step is called an "enroll step" and in the case of **tls** plugins,
The validity of a **node_key** is determined and implemented in the TLS server. The node only manages to ask for the content during enroll, and posts the content during subsequent requests.
With osquery version 1.7.0, OS X clients **MUST** use a `--tls_server_certs` bundle since osquery is built using LibreSSL and no default certificate bundle is available on OS X.
### Simple shared secret enrollment
A deployment key, called an enrollment shared secret, is the simplest **tls** plugin enrollment authentication method. A protected shared secret is written to disk and osquery reads then posts the content to `--enroll_tls_endpoint` once during enrollment. The TLS server may implement an enrollment request approval process that requires manual intervention/approval for each new enrollment request.
@ -112,9 +110,9 @@ The read request sends the enrollment **node_key** for identification. The distr
```json
{
"queries": {
"id1": "select * from osquery_info;",
"id2": "select * from osquery_schedule;",
"id3": "select * from does_not_exist;"
"id1": "SELECT * FROM osquery_info;",
"id2": "SELECT * FROM osquery_schedule;",
"id3": "SELECT * FROM does_not_exist;"
},
"node_invalid": false // Optional, return true to indicate re-enrollment.
}
@ -172,9 +170,9 @@ The TLS client does not handle HTTP errors, if the service returns a bad request
We include a very basic example python TLS/HTTPS server: [./tools/tests/test_http_server.py](https://github.com/facebook/osquery/blob/master/tools/tests/test_http_server.py). And a set of unit/integration tests: [./osquery/remote/transports/tests/tls_transports_tests.cpp](https://github.com/facebook/osquery/blob/master/osquery/remote/transports/tests/tls_transports_tests.cpp) for a reference server implementation.
The TLS clients built into osquery use the system-provided OpenSSL libraries. The clients use boost's ASIO header-libraries through the [cpp-netlib](http://cpp-netlib.org/) HTTPS library. OpenSSL is very outdated on OS X (deprecated since OS X 10.7), so a Homebrew-provided Libressl is statically linked into the osquery tools.
The TLS clients built into osquery use the system-provided OpenSSL libraries. The clients use boost's ASIO header-libraries through the [cpp-netlib](http://cpp-netlib.org/) HTTPS library.
On OS X, Linux, and FreeBSD the TLS client prefers the TLS 1.2 protocol, but includes TLS 1.1/1.0 as well as the following cipher suites:
On macOS, Linux, and FreeBSD the TLS client prefers the TLS 1.2 protocol, but includes TLS 1.1/1.0 as well as the following cipher suites:
```
ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:\
@ -230,9 +228,7 @@ This starts a HTTPS server bound to port 8080 using some fake CA/server cert and
We will use an **osqueryd** client and set the required TLS settings. When enforcing TLS server authentication, note that the example server is using a toy certificate with the subject: `C=US, ST=California, O=osquery-testing, CN=localhost`:
```
$ osqueryd --verbose \
--pidfile /tmp/osqueryd.pid \
--database_path /tmp/osquery.db/ \
$ osqueryd --verbose --ephemeral --disable_database \
--tls_hostname localhost:8080 \
--tls_server_certs ./tools/tests/test_server_ca.pem \
--config_plugin tls \
@ -243,16 +239,14 @@ $ osqueryd --verbose \
--enroll_secret_path ./tools/tests/test_enroll_secret.txt
```
There is a LOT of command line switches here! The basics notes are (1) set a temporary pidfile and database for this **osqueryd**; (2) set the TLS hostname and port, note that no <i>https://</i> is used, as well as the explicit set of certificates to expect; (3) set the plugin options for the config and logger; (4) set the plugin options for enrollment. Turning <i>verbose</i> mode on helps describe the expected behavior.
There are a LOT of command line switches here! The basics notes are (1) set the TLS hostname and port, note that no <i>https://</i> is used, as well as the explicit set of certificates to expect; (2) set the plugin options for the config and logger; (3) set the plugin options for enrollment. Turning <i>verbose</i> mode on helps describe the expected behavior.
```
I1015 10:36:06.894544 2032685056 init.cpp:263] osquery initialized [version=1.5.3]
I1015 10:36:06.924180 2032685056 system.cpp:207] Writing osqueryd pid (19651) to /tmp/osqueryd.pid
I1015 10:36:06.925974 2032685056 db_handle.cpp:124] Opening RocksDB handle: /tmp/osquery.db/
I1015 10:36:06.935755 2032685056 tls.cpp:68] TLSEnrollPlugin requesting a node enroll key from: https://localhost:8080/enroll
I1015 10:36:06.936123 2032685056 tls.cpp:196] TLS/HTTPS POST request to URI: https://localhost:8080/enroll
I1015 10:36:06.947465 2032685056 tls.cpp:196] TLS/HTTPS POST request to URI: https://localhost:8080/config
I1015 10:36:10.288635 3825664 scheduler.cpp:56] Executing query: select * from processes
I1015 10:36:10.288635 3825664 scheduler.cpp:56] Executing query: SELECT * FROM processes;
I1015 10:36:10.366140 3825664 scheduler.cpp:101] Found results for query (tls_proc) for host: YOURHOSTNAME.local
I1015 10:36:11.019227 528384 tls.cpp:196] TLS/HTTPS POST request to URI: https://localhost:8080/logger
[...]

15
docs/wiki/deployment/syslog.md
View File

@ -1,6 +1,6 @@
osquery 1.7.3 introduced support for consuming and querying the Mac OSX system log via Apple System Log (ASL). osquery 1.7.4 introduced support for the Linux syslog via **rsyslog**. This document explains how to configure and use these syslog tables.
## OS X Syslog
## Apple macOS Syslog
On Mac OSX, the `asl` virtual table makes use of Apple's ASL store, querying this structured store using the routines provided in [`asl.h`](https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man3/asl.3.html).
@ -10,7 +10,7 @@ No configuration is required to begin using the `asl` table. Note, however, that
If your target logs are not already being sent to the ASL store by your current configuration, take a look at the [man page](https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man5/asl.conf.5.html) for `asl.conf`, and use the `store` action to ensure your logs of interest are available in the store. `asl.conf` is also responsible for the rotation and retention settings of the ASL store.
Note: the configuration for `/var/log/install.log` and `/var/log/commerce.log` is hardcoded into the Apple provided syslog binaries, and we are not aware of a way to configure ASL to send these logs to the store.
The configuration for `/var/log/install.log` and `/var/log/commerce.log` is hardcoded into the Apple provided syslog binaries, and we are not aware of a way to configure ASL to send these logs to the store.
### Usage
@ -19,7 +19,7 @@ The `asl` table can be queried like any other osquery table. It exposes many of
Basic query predicates (`<`, `<=`, `=`, `>=`, `>`) are able to be efficiently queried in the store. The `LIKE` predicate is also supported, however it must be tested after applying all othe predicates and reading logs from the store. For performance reasons, it is suggested to use at least one basic predicate in a query against the `asl` table. For example,
```
select time, message from asl where facility = 'authpriv' and sender = 'sudo' and message like '%python%';
SELECT time, message FROM asl WHERE facility = 'authpriv' AND sender = 'sudo' AND message LIKE '%python%';
```
## Linux Syslog
@ -28,7 +28,7 @@ On Linux, the `syslog` table queries logs forwarded over a named pipe from a pro
Alternatively you can also use **syslog-ng** to forward log messages to osquery.
Note: the Syslog ingestion is NOT recommended for hosts functioning as syslog aggregators. We have not tested ingestion for massive-throughput or lossless setups.
> NOTICE: the Syslog ingestion is NOT recommended for hosts functioning as syslog aggregators. We have not tested ingestion for massive-throughput or lossless setups.
### Configuration
@ -49,7 +49,7 @@ $template OsqueryCsvFormat, "%timestamp:::date-rfc3339,csv%,%hostname:::csv%,%sy
#### rsyslog versions >= 7
Note: the above configuration should also work, but **rsyslog** strongly recommends using the new style configuration syntax.
The above configuration should also work, but **rsyslog** strongly recommends using the new style configuration syntax.
```
template(
@ -64,7 +64,7 @@ template(
**rsyslogd** must be restarted for the changes to take effect. On many systems, this can be achieved by `sudo service rsyslog restart`.
Note: **rsyslogd** will only check once, at startup, whether it can write to the pipe. If **rsyslogd** cannot write to the pipe, it will not retry until restart.
> NOTICE: **rsyslogd** will only check once, at startup, whether it can write to the pipe. If **rsyslogd** cannot write to the pipe, it will not retry until restart.
#### Other configuration
@ -103,12 +103,13 @@ destination d_osquery {
# destination(d_osquery_copy);
};
```
The rewrite is needed to make sure that quotation marks are escaped. The template re-formats the messages as expected by osquery. Binaries provided by the osquery project expect syslog messages in this pipe: you might need to change the location if you compiled osquery yourself. If you want to see what messages are sent to osquery you can uncomment the “d_osquery_copy” destination in the log path. The “s_sys” source refers to your local log messages and might be different on your system (this example is from CentOS).
### Usage
Once configuration is complete, the `syslog` table can be queried like any other osquery table. It's schema can be viewed with `.schema syslog`.
Note: only logs produced after this table was properly configured (and while osquery is running) will be available for querying.
> NOTICE: only logs produced after this table was properly configured (and while osquery is running) will be available for querying.
If no logs are available to query, try turning on verbose logging, and see [issue #1964](https://github.com/facebook/osquery/issues/1964) for debugging suggestions.

14
docs/wiki/deployment/yara.md
View File

@ -44,13 +44,13 @@ For example, when a file in */usr/bin/* and */usr/sbin/* is changed it will be s
Using the configuration above you can see it in action. While osquery was running I executed `touch /Users/wxs/tmp/foo` in another terminal. Here is the relevant queries to show what happened:
```bash
osquery> select * from file_events;
osquery> SELECT * FROM file_events;
+--------------------+----------+------------+---------+----------------+----------------------------------+------------------------------------------+------------------------------------------------------------------+
| target_path | category | time | action | transaction_id | md5 | sha1 | sha256 |
+--------------------+----------+------------+---------+----------------+----------------------------------+------------------------------------------+------------------------------------------------------------------+
| /Users/wxs/tmp/foo | tmp | 1430078285 | CREATED | 33859499 | d41d8cd98f00b204e9800998ecf8427e | da39a3ee5e6b4b0d3255bfef95601890afd80709 | e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 |
+--------------------+----------+------------+---------+----------------+----------------------------------+------------------------------------------+------------------------------------------------------------------+
osquery> select * from yara_events;
osquery> SELECT * FROM yara_events;
+--------------------+----------+------------+---------+----------------+-------------+-------+
| target_path | category | time | action | transaction_id | matches | count |
+--------------------+----------+------------+---------+----------------+-------------+-------+
@ -64,7 +64,7 @@ The [**file_events**](https://osquery.io/docs/tables/#file_events) table recorde
The [**yara_events**](https://osquery.io/docs/tables/#yara_events) table recorded that 1 matching rule (*always_true*) was found when the file was created. In this example every file will always have at least one match because I am using a rule which always evaluates to true. In the next example I'll issue the same command to create a file in a monitored directory but have removed the *always_true* rule from my signature files.
```bash
osquery> select * from yara_events;
osquery> SELECT * FROM yara_events;
+--------------------+----------+------------+---------+----------------+-------------+-------+
| target_path | category | time | action | transaction_id | matches | count |
+--------------------+----------+------------+---------+----------------+-------------+-------+
@ -87,13 +87,13 @@ Once the "where" is out of the way, you must specify the "what" part. This is do
Here are some examples of the **yara** table in action:
```bash
osquery> select * from yara where path="/bin/ls" and sig_group="sig_group_1";
osquery> SE:ECT * FROM yara WHERE path="/bin/ls" AND sig_group="sig_group_1";
+---------+-------------+-------+-------------+---------+---------+
| path | matches | count | sig_group | sigfile | pattern |
+---------+-------------+-------+-------------+---------+---------+
| /bin/ls | always_true | 1 | sig_group_1 | | |
+---------+-------------+-------+-------------+---------+---------+
osquery> select * from yara where path="/bin/ls" and sig_group="sig_group_2";
osquery> SELECT * FROM yara WHERE path="/bin/ls" AND sig_group="sig_group_2";
+---------+---------+-------+-------------+---------+---------+
| path | matches | count | sig_group | sigfile | pattern |
+---------+---------+-------+-------------+---------+---------+
@ -105,7 +105,7 @@ osquery>
As you can see in these examples, I'm scanning the same file with two different signature groups and getting different results.
```bash
osquery> select * from yara where pattern="/bin/%sh" and sig_group="sig_group_1";
osquery> SELECT * FROM yara WHERE pattern="/bin/%sh" AND sig_group="sig_group_1";
+-----------+-------------+-------+-------------+---------+----------+
| path | matches | count | sig_group | sigfile | pattern |
+-----------+-------------+-------+-------------+---------+----------+
@ -122,7 +122,7 @@ osquery>
The above illustrates using the *pattern* constraint to scan */bin/%sh* with a signature group.
```
osquery> select * from yara where pattern="/bin/%sh" and sigfile="/Users/wxs/sigs/baz.sig";
osquery> SELECT * FROM yara WHERE pattern="/bin/%sh" AND sigfile="/Users/wxs/sigs/baz.sig";
+-----------+---------+-------+-----------+-------------------------+----------+
| path | matches | count | sig_group | sigfile | pattern |
+-----------+---------+-------+-----------+-------------------------+----------+

33
docs/wiki/development/building.md
View File

@ -1,6 +1,6 @@
## Dependencies
We include a `make deps` command to make it easier for developers to get started with the osquery project. `make deps` uses Homebrew for OS X and Linuxbrew for Linux. The following basic dependencies are need before running `make deps`:
We include a `make deps` command to make it easier for developers to get started with the osquery project. `make deps` uses Homebrew for macOS and Linuxbrew for Linux. The following basic dependencies are need before running `make deps`:
- `sudo`
- `make` (that is, GNU make)
@ -9,7 +9,7 @@ We include a `make deps` command to make it easier for developers to get started
- `git`
- `bash`
**WARNING:** This will install or build various dependencies on the build host that are not required to "use" osquery, only build osquery binaries and packages.
> NOTICE: This will install or build various dependencies on the build host that are not required to "use" osquery, only build osquery binaries and packages.
For our build hosts (CentOS, Ubuntu 12, 14, 16, macOS 10.12, Windows 2016) we use a `sysprep` target to update the host and install these basic dependencies.
@ -17,7 +17,7 @@ For our build hosts (CentOS, Ubuntu 12, 14, 16, macOS 10.12, Windows 2016) we us
make sysprep
```
## Building on OS X
## Building on macOS
`make deps` will take care of installing the appropriate library dependencies, but it's recommended to take a look at the Makefile, just in case something conflicts with your environment.
@ -45,9 +45,9 @@ $ ls -la ./build/darwin/osquery/
## Building on Linux
osquery supports several distributions of Linux.
osquery supports almost all distributions of Linux.
For each supported distro, we supply vagrant infrastructure for creating native operating system packages. To create a package (e.g. a deb on Ubuntu or an rpm on CentOS), simply spin up a vagrant instance.
For some distros, we supply vagrant infrastructure for creating native operating system packages. To create a package (e.g. a deb on Ubuntu or an rpm on CentOS), simply spin up a vagrant instance.
For example:
@ -60,7 +60,7 @@ By default vagrant will allocate 2 virtual CPUs to the virtual machine instance.
```sh
OSQUERY_BUILD_CPUS=`nproc` # for Linux
OSQUERY_BUILD_CPUS=`sysctl -n hw.ncpu` # for OS X
OSQUERY_BUILD_CPUS=`sysctl -n hw.ncpu` # for MacOS
```
Once you have logged into the vagrant box, run the following to create a package:
@ -99,13 +99,13 @@ In that case the final push becomes `git push USERNAME`.
Our Jenkins CI will test your changes in three steps.
1. A code audit is run using `./tools/audit.sh`.
2. The code is rebuilt, built again for release, then a package is generated using `./tools/build.sh` on various Linux and OS X versions.
1. A code audit is run using `make audit`.
2. The code is rebuilt, built again for release, then a package is generated using `./tools/build.sh` on various Linux and macOS versions.
3. The same step is run on Windows 10.
The audit step attempts to build the documentation, run code formatting checks, and a brief static code analysis. The formatting check is performed with `clang-format` (installed with `make deps` to your osquery dependencies directory). Your changes are compared against the local `master` branch. Within the build host this is always the TIP of `facebook/osquery`, but locally the branch may be behind.
To speed up the format auditing process please configure your code editor to run `clang-format` on files changed. Or periodically during your development run `make format`. Running `make check` is also helpful, but it will use `cppcheck` which is not installed by default.
To speed up the format auditing process please configure your code editor to run `clang-format` on files changed. Or periodically during your development run `make format_master`. Running `make check` is also helpful, but it will use `cppcheck` which is not installed by default.
## Dependencies and build internals
@ -117,7 +117,7 @@ When using `make deps` the environment the resultant binaries will have a minimu
- `libgcc_s`
- `libz`
All other dependencies are built, compiled, and linked statically. This makes for a rather large set of output binaries (15M on Linux and 9M on OS X) but the trade-off for deployment simplicity is very worthwhile.
All other dependencies are built, compiled, and linked statically. This makes for a rather large set of output binaries (15M on Linux and 9M on macOS) but the trade-off for deployment simplicity is very worthwhile.
Under the hood the `make deps` script is calling `./tools/provision.sh`, which performs the simplified set of steps:
@ -156,7 +156,7 @@ This looks A LOT like normal *brew formulas. For a new dependency do not add a `
If you want to make build changes see the Cookbook for `revision` edits. Note that committing new or edited formulas will invalidate package caches this will cause the package to be built from source on the test/build hosts.
**If this is a new dependency** then you need to add a line to `./tools/provision.sh` for Linux and or OS X at the order/time it should be installed.
**If this is a new dependency** then you need to add a line to `./tools/provision.sh` for Linux and or macOS at the order/time it should be installed.
When a dependency is updated by a maintainer or contributor the flow should follow:
* Update the target formula in `./tools/provision/formula/`.
@ -240,12 +240,14 @@ make docs # Build the Doxygen and mkdocs wiki
There are several additional code testing and formatting macros:
```sh
make format # Apply clang-format using osquery's format spec*
make format-all # Not recommended but formats the entire code base
make format_master # Format everything changed from the local master branch
make format_all # Not recommended but formats the entire code base
make format # Apply clang-format using osquery's format spec*
make analyze # Run clean first, then rebuild with the LLVM static analyzer
make sanitize # Run clean first, then rebuild with sanitations
make fuzz # Run basic fuzz tests, as defined in each table spec
make audit # The clang-format and other PR-blocking checks
make check # Run cpp-check and output style/performance/error warnings
```
Generating the osquery SDK or sync:
@ -268,11 +270,12 @@ OSQUERY_BUILD_BOTTLES=True # Create Homebrew bottles from installed dependencies
OSQUERY_BUILD_VERSION=9.9.9 # Set a wacky version string
OSQUERY_PLATFORM=custom_linux;1.0 # Set a wacky platform/distro name
SDK_VERSION=9.9.9 # Set a wacky SDK-version string
OSX_VERSION_MIN=10.11 # Override the native minimum OS X version ABI
OSX_VERSION_MIN=10.11 # Override the native minimum macOS version ABI
OSQUERY_DEPS=/path/to/dependencies # Use or create a custom dependency environment
FAST=True # Build and link as quick as possible
SANITIZE_THREAD=True # Add -fsanitize=thread when using "make sanitize"
SANITIZE_UNDEFINED=True # Add -fsanitize=undefined when using "make sanitize"
OPTIMIZED=True # Enable specific CPU optimizations (not recommended)
SKIP_TESTS=True # Skip unit test building (very very not recommended!)
SKIP_INTEGRATION_TESTS=True # Skip python tests when using "make test"
@ -284,7 +287,7 @@ SQLITE_DEBUG=True # Enable SQLite query debugging (very verbose!)
## Custom Packages
Building osquery on OS X or Linux requires a significant number of dependencies, which are not needed when deploying. It does not make sense to install osquery on your build hosts. See the [Custom Packages](../installation/custom-packages.md) guide for generating pkgs, debs or rpms.
Building osquery on macOS or Linux requires a significant number of dependencies, which are not needed when deploying. It does not make sense to install osquery on your build hosts. See the [Custom Packages](../installation/custom-packages.md) guide for generating PKGs, debs or RPMs.
Debug and from-source package building is not recommended but supported. You may generate several packages including devel, debuginfo, and additional sets of tools:

8
docs/wiki/development/contributing-code.md
View File

@ -2,7 +2,7 @@ For documentation on contributing to osquery, please read the [CONTRIBUTING.md](
## Using `clang-format`
C++ can be very difficult to read, maintain and write if it's not written cleanly. Following uniform patterns throughout the entire codebase improves the entire codebase. With that being said, the last thing that you want to think about when you're trying to iterate quickly is how many of your lines are greater than 80 characters long. For this reason, osquery uses `clang-format` to automatically format code. **After you've written some code, stage your modified files, then execute `make format` from the root of the repository.** This automatically runs `clang-format` on all staged C/C++/Objective-C source code.
C++ can be very difficult to read, maintain and write if it's not written cleanly. Following uniform patterns throughout the entire codebase improves the entire codebase. With that being said, the last thing that you want to think about when you're trying to iterate quickly is how many of your lines are greater than 80 characters long. For this reason, osquery uses `clang-format` to automatically format code. **After you've written some code, stage your modified files, then execute `make format_master` from the root of the repository.** This automatically runs `clang-format` on all staged C/C++/Objective-C source code.
## Format style
@ -12,14 +12,16 @@ The format style used by osquery is defined by the configurations of osquery's [
If you would really like a style guide to follow or refer to, please use the [LLVM Coding Standards](http://llvm.org/docs/CodingStandards.html).
Keep in mind, osquery's style configurations are slightly different than vanilla LLVM coding standards, so be sure to still run `make format` before submitting any code.
Keep in mind, osquery's style configurations are slightly different than vanilla LLVM coding standards, so be sure to still run `make format_master` before submitting any code.
## Linting
osquery has some basic linting for documentation that gets rendered on the [osquery tables](https://osquery.io/docs/tables/) page. Breaking these linting rules will cause table generation to fail and all builds will break.
**Active Linting Rules**
- Table descriptions must end in a period: Try to make table descriptions full sentences.
**Upcoming Linting Rules**
- Table specs must have examples: Give one or two examples of how your table could be used. Generally one simple and a complex one but if your table just needs a `select * from [table]`, that's also fine.
- Table specs must have examples: Give one or two examples of how your table could be used. Generally one simple and a complex one but if your table just needs a `SELECT * FROM [table]`, that's also fine.

10
docs/wiki/development/creating-tables.md
View File

@ -49,12 +49,14 @@ You might wonder "this syntax looks similar to Python?". Well, it is! The build
You may be wondering how osquery handles cross-platform support while still allowing operating-system specific tables. The osquery build process takes care of this by only generating the relevant code based on a directory structure convention.
- Cross-platform: [specs/](https://github.com/facebook/osquery/tree/master/specs/)
- Mac OS X: [specs/darwin/](https://github.com/facebook/osquery/tree/master/specs/darwin)
- MacOS: [specs/darwin/](https://github.com/facebook/osquery/tree/master/specs/darwin)
- General Linux: [specs/linux/](https://github.com/facebook/osquery/tree/master/specs/linux)
- Ubuntu: [specs/ubuntu/](https://github.com/facebook/osquery/tree/master/specs/ubuntu)
- CentOS: [specs/centos/](https://github.com/facebook/osquery/tree/master/specs/centos)
- Windows: [specs/windows/](https://github.com/facebook/osquery/tree/master/specs/windows)
- POSIX: [specs/posix/](https://github.com/facebook/osquery/tree/master/specs/posix)
- FreeBSD: [specs/freebsd/](https://github.com/facebook/osquery/tree/master/specs/freebsd)
- You get the picture ;)
Note: the CMake build provides custom defines for each platform and platform version.
> NOTICE: the CMake build provides custom defines for each platform and platform version.
**Creating your implementation**

18
docs/wiki/development/kernel.md
View File

@ -1,8 +1,8 @@
osquery may optionally use a kernel extension (on OS X)/module (Linux) to introspect into process and socket events. This extends osquery by enabling more tables.
osquery may optionally use a kernel extension (on macOS)/module (Linux) to introspect into process and socket events. This extends osquery by enabling more tables.
Currently an kernel extension is under development for OS X. Once the OS X implementation is finished a Linux module will be considered. There are several options for retrieving real time process and socket events. User-land implementation are preferred to having kernel presence.
Currently an kernel extension is under development for macOS. Once the macOS implementation is finished a Linux module will be considered. There are several options for retrieving real time process and socket events. User-land implementation are preferred to having kernel presence.
**WARNING**: This is currently under heavy development.
> NOTICE: This is currently under heavy development.
The osquery kernel introspection architecture is designed with simplicity and portability. It uses a small ring buffer, backed by shared memory, filled by kernel callback registrations to maintain simple structures. A process creation structure may contain a path to the program image, assigned pid, and ownership information. When the ring buffer fills, osquery drops information. The user-land process, osqueryd, will periodically request a minimum and maximum read into the ring buffer and pass structures to an event subscriber.
@ -10,17 +10,17 @@ This code is mostly shared between BSD-based kernels and Linux. The ring buffer
The kernel applies calling-process ownership limitations to super users. Only 1 process should issues IOCTL commands, if another process (pid) uses the device node the queue and buffer are considered invalid and all pointers are reset. Clean tear down assures deregistration of callback functions and will result in maximum performance. Improper tear down may trigger timeouts and in the worst scenario continue to track callbacks.
# Apple OS X Kernel Extensions
# Apple Kernel Extensions
Kernel extension loading requires extension bundles signed by valid Apple developer certificates. During development it is NOT recommended to sign inline with building and running unit/integration tests. Changes to the kernel-mode code may result in kernel panics. Some of the development-only-enabled unit test code is designed to stress the limits of kernel memory management and event synchronization. It is very highly recommended to restrict these tests to running inside a virtual environment.
The high-level kernel development workflow involves:
- Install a copy of OS X in VMware and share a development/source/build folder.
- Install a copy of macOS in VMware and share a development/source/build folder.
- Compile and build the user-land and kernel-land code on your host.
- Load the kernel extension in the VM and run the kernel-specific unit tests.
- You may optionally configure the VM to break on boot to debug with `lldb`.
## Building on OS X for local loading (not recommended)
## Building on macOS for local loading (not recommended)
First disable signature verification on the machine by running `make kernel-deps` a reboot is required for this to take effect. **CAUTION:** this lowers the security of your system. It is better to have your kernel extensions signed on your host and develop/load in a virtual machine.
@ -32,13 +32,13 @@ make kernel-test-load
make kernel-test-unload
```
Note: do not deploy a kernel built with `make kernel-build` as it is designed for testings. Always deploy artifacts generated through `make packages`.
> NOTICE: do not deploy a kernel built with `make kernel-build` as it is designed for testings. Always deploy artifacts generated through `make packages`.
## Building on OS X for a debugging target machine (recommended)
## Building on macOS for a debugging target machine (recommended)
You may configure your host and virtual machine for kernel and extension debugging. This involves a few extra steps beyond building, loading, and running tests.
- Get a [kernel debug kit](https://developer.apple.com/downloads/) from Apple's OS X Developer portal.
- Get a [kernel debug kit](https://developer.apple.com/downloads/) from Apple's macOS Developer portal.
- Install it on both the target machine VM and the host debugging machine.
- Run make targets to configure the target machine and the debugger machine.
- `make kernel-configure-target` (on VM, requires a reboot)

2
docs/wiki/development/options-arguments.md
View File

@ -22,4 +22,4 @@ If you are declaring a flag before defining it, no change is needed. Use `DECLAR
This will allow osquery callers to show pretty displays when `-h, --help` is used.
Note: restrict your default values to code literals. It does not help to abstract the default variable into a constant then use it singularly in the macro.
> NOTICE: restrict your default values to code literals. It does not help to abstract the default variable into a constant then use it singularly in the macro.

6
docs/wiki/development/osquery-sdk.md
View File

@ -72,7 +72,7 @@ The **osqueryi** shell also allows a quick and easy command-line autoload using
To load the example extension in the shell try:
```
$ ./build/darwin/osquery/osqueryi
osquery> select path from osquery_extensions;
osquery> SELECT path FROM osquery_extensions;
+-------------------------------------+
| path |
+-------------------------------------+
@ -87,7 +87,7 @@ Here we have started a shell process, inspected the UNIX domain socket path used
```
$ ./build/darwin/osquery/example_extension.ext --help
osquery 1.7.0, your OS as a high-performance relational database
Usage: osqueryi [OPTION]...
Usage: example_extension.ext [OPTION]...
osquery extension command line flags:
@ -105,7 +105,7 @@ Before executing the extension we've inspected the potential CLI flags, which ar
[2] 98795
$ fg
[1] - 98777 continued ./build/darwin/osquery/osqueryi
osquery> select * from example;
osquery> SELECT * FROM example;
+--------------+-----------------+
| example_text | example_integer |
+--------------+-----------------+

4
docs/wiki/development/pubsub-framework.md
View File

@ -1,10 +1,10 @@
Most of osquery's virtual tables are generated when an SQL statement requests data. For example, the [time](https://github.com/facebook/osquery/blob/master/osquery/tables/utility/time.cpp) gets the current time and returns it as a single row. So whenever a call selects data from time, e.g., `SELECT * FROM time;` the current time of the call will return.
From an operating systems perspective, query-time synchronous data retrieval is lossy. Consider the [processes](https://github.com/facebook/osquery/blob/master/osquery/tables/system/linux/processes.cpp) table: if a process like `ps` runs for a fraction of a moment there's no way `SELECT * from processes;` will ever include the details.
From an operating systems perspective, query-time synchronous data retrieval is lossy. Consider the [processes](https://github.com/facebook/osquery/blob/master/osquery/tables/system/linux/processes.cpp) table: if a process like `ps` runs for a fraction of a moment there's no way `SELECT * FROM processes;` will ever include the details.
To solve for this osquery exposes a [pubsub framework](https://github.com/facebook/osquery/tree/master/osquery/events) for aggregating operating system information asynchronously at event time, storing related event details in the osquery backing store, and performing a lookup to report stored rows query time. This reporting pipeline is much more complicated than typical query-time virtual table generation. The time of event, storage history, and applicable (final) virtual table data information must be carefully considered. As events occur, the rows returned by a query will compound, as such selecting from an event-based virtual table generator should always include a time range.
If no time range is provided, as in: `SELECT * FROM process_events`, it is assumed you want to scan from `t=[0, now)`. Otherwise, all of the `*_events` tables must have a `time` column, this is used to optimize searching: `SELECT * FROM process_events WHERE time > NOW() - 300`.
If no time range is provided, as in: `SELECT * FROM process_events;`, it is assumed you want to scan from `t=[0, now)`. Otherwise, all of the `*_events` tables must have a `time` column, this is used to optimize searching: `SELECT * FROM process_events WHERE time > NOW() - 300;`.
## Query and table usage

6
docs/wiki/index.md
View File

@ -4,11 +4,11 @@ osquery exposes an operating system as a high-performance relational database. T
## Getting Started
If you're interested in **installing osquery** check out the install guide for [Windows](installation/install-windows.md), [OS X](installation/install-osx.md), [Linux](installation/install-linux.md), and [FreeBSD](installation/install-freebsd.md).
If you're interested in **installing osquery** check out the install guide for [Windows](installation/install-windows.md), [macOS](installation/install-osx.md), [Linux](installation/install-linux.md), and [FreeBSD](installation/install-freebsd.md).
If you're interested in **developing queries** and **exploring** tables, check out [using osqueryi](introduction/using-osqueryi.md).
If you're interested in **deploying osquery** to provide your organization with deeper insight into your Linux, FreeBSD, OS X, and Windows hosts check out the [using osqueryd guide](introduction/using-osqueryd.md).
If you're interested in **deploying osquery** to provide your organization with deeper insight into your Linux, FreeBSD, macOS, and Windows hosts check out the [using osqueryd guide](introduction/using-osqueryd.md).
If you're interested in **extending one of the existing osquery tools** or improving core libraries, read the developer documentation pages. You should start with "[building the code](development/building.md)" and "[contributing code](development/contributing-code.md)".
@ -20,7 +20,7 @@ The high-performance and low-footprint distributed host monitoring daemon, **osq
The interactive query console, **osqueryi**, gives you a SQL interface to try out new queries and explore your operating system. With the power of a complete SQL language and dozens of useful tables built-in, **osqueryi** is an invaluable tool when performing incident response, diagnosing a systems operations problem, troubleshooting a performance issue, etc.
osquery is cross platform. Even though osquery takes advantage of very low-level operating system APIs, you can build and use osquery on Windows, Mac OS X, Ubuntu, CentOS and other popular enterprise Linux distributions. This has the distinct advantage of allowing you to be able to use one platform for monitoring complex operating system state across your entire infrastructure. Monitor your corporate Windows or Mac OS X clients the same way you monitor your production Linux servers.
osquery is cross platform. Even though osquery takes advantage of very low-level operating system APIs, you can build and use osquery on Windows, MacOS, Ubuntu, CentOS and other popular enterprise Linux distributions. This has the distinct advantage of allowing you to be able to use one platform for monitoring complex operating system state across your entire infrastructure. Monitor your corporate Windows or MacOS clients the same way you monitor your production Linux servers.
To make deploying osquery in your infrastructure as easy as possible, osquery comes with native packages for all supported operating systems. There is extensive tooling and documentation around creating packages so packaging and deploying your custom osquery tools can be just as easy too.

30
docs/wiki/installation/cli-flags.md
View File

@ -1,7 +1,7 @@
The osquery shell and daemon use optional command line (CLI) flags to control
initialization, disable/enable features, and select plugins. These flags are powered by Google Flags and are somewhat complicated. Understanding how flags work in osquery will help with stability and greatly reduce issue debugging time.
Most flags apply to both tools, `osqueryi` and `osqueryd`. The shell contains a few more to help with printing and other helpful one-off modes of operation. Expect Linux / OS X / and Windows to include platform specific flags too. Most platform specific flags will control the OS API and library integrations used by osquery. Warning, this list is still not the 'complete set' of flags. Refer to the techniques below for obtaining ground truth and check other components of this Wiki.
Most flags apply to both tools, `osqueryi` and `osqueryd`. The shell contains a few more to help with printing and other helpful one-off modes of operation. Expect Linux / macOS / and Windows to include platform specific flags too. Most platform specific flags will control the OS API and library integrations used by osquery. Warning, this list is still not the 'complete set' of flags. Refer to the techniques below for obtaining ground truth and check other components of this Wiki.
Flags that do not control startup settings may be included as "options" within [configuration](../deployment/configuration.md). Essentially, any flag needed to help osquery determine and discovery a configuration must be supplied via command line arguments. Google Flags enhances this to allow flags to be set within environment variables or via a "master" flag file.
@ -9,20 +9,20 @@ To see a full list of flags for your osquery version use `--help` or select from
```
$ osqueryi
osquery> select * from osquery_flags;
osquery> SELECT * FROM osquery_flags;
```
To see the flags that have been updated by your configuration, a flag file, or by the shell try:
```
osquery> select * from osquery_flags where default_value <> value;
osquery> SELECT * FROM osquery_flags WHERE default_value <> value;
```
## Command line only flags
### Flagfile
A special flag, part of Google Flags, can be used to read additional flags from a line-delimited file. On OS X and Linux this `--flagfile` is the recommended way to add/remove the following CLI-only initialization flags.
A special flag, part of Google Flags, can be used to read additional flags from a line-delimited file. On macOS and Linux this `--flagfile` is the recommended way to add/remove the following CLI-only initialization flags.
`--flagfile="/etc/osquery/osquery.flags"`
`--flagfile /etc/osquery/osquery.flags`
Include line-delimited switches to be interpreted and used as CLI-flags:
@ -35,7 +35,7 @@ Include line-delimited switches to be interpreted and used as CLI-flags:
If no `--flagfile` is provided, osquery will try to find and use a "default" flagfile at `/etc/osquery/osquery.flags.default`. Both the shell and daemon will discover and use the defaults.
**Note:** Flags in a `flagfile` should not be wrapped in quotes, shell-macro/variable expansion is not applied!
> NOTICE: Flags in a `flagfile` should not be wrapped in quotes, shell-macro/variable expansion is not applied!
### Configuration control flags
@ -48,7 +48,7 @@ Built-in options include: **filesystem**, **tls**
`--config_path="/etc/osquery/osquery.conf"`
The **filesystem** config plugin's path to a JSON file.
On OS X the default path is **/var/osquery/osquery.conf**.
On macOS the default path is **/var/osquery/osquery.conf**.
If you want to read from multiple configuration paths create a directory: **/etc/osquery/osquery.conf.d/**. All files within that optional directory will be read and merged in lexical order.
`--config_refresh=0`
@ -67,7 +67,7 @@ Check the format of an osquery config and exit. Arbitrary config plugins may be
Request that the configuration JSON be printed to standard out before it is updated. In this case "updated" means applied to the active config. When osquery starts it performs an initial update from the config plugin. To quickly debug the content retrieved by custom config plugins use this in tandem with `--config_check`.
### osquery daemon control flags
### Daemon control flags
`--force=false`
@ -157,7 +157,7 @@ Extensions are loaded as processes. They are expected to start a thrift service
Optional comma-delimited set of extension names to require before **osqueryi** or **osqueryd** will start. The tool will fail if the extension has not started according to the interval and timeout.
### Remote settings (optional for config/logger/distributed) flags
### Remote settings flags (optional)
When using non-default [remote](../deployment/remote.md) plugins such as the **tls** config, logger and distributed plugins, there are process-wide settings applied to every plugin.
@ -231,7 +231,7 @@ The URI path which will be used, in conjunction with `--tls_hostname`, to create
The total number of attempts that will be made to the remote distributed query server if a request fails when using the **tls** distributed plugin.
### osquery daemon runtime control flags
### Daemon runtime control flags
`--schedule_splay_percent=10`
@ -272,7 +272,7 @@ Comma-delimited list of table names to be disabled. This allows osquery to be la
Maximum file read size. The daemon or shell will first 'stat' each file before reading. If the reported size is greater than `read_max` a "file too large" error will be returned.
### osquery events control flags
### Events control flags
`--disable_events=false`
@ -363,7 +363,7 @@ This is default `true` and will also send log messages in GLog format to the pro
This controls the types of logs sent to the process's `stderr`. It does NOT limit or control the types sent to the logger plugin. The default value 2 is `ERROR`, set this to `0` for all non-verbose types. If the `--verbose` flag is set this value is overridden to `0`.
## Distributed query service flags
### Distributed query service flags
`--distributed_plugin=tls`
@ -377,7 +377,7 @@ Disable distributed queries functionality. By default, this is set to `true` (th
In seconds, the amount of time that osqueryd will wait between periodically checking in with a distributed query server to see if there are any queries to execute.
## Syslog consumption
### Syslog consumption
There is a `syslog` virtual table that uses Events and a **rsyslog** configuration to capture results *from* syslog. Please see the [Syslog Consumption](../deployment/syslog.md) deployment page for more information.
@ -393,7 +393,7 @@ Path to the named pipe used for forwarding **rsyslog** events.
Maximum number of logs to ingest per run (~200ms between runs). Use this as a fail-safe to prevent osquery from becoming overloaded when syslog is spammed.
## Shell-only flags
### Shell-only flags
Most of the shell flags are self-explanatory and are adapted from the SQLite shell. Refer to the shell's ".help" command for details and explanations.

7
docs/wiki/installation/custom-packages.md
View File

@ -16,7 +16,7 @@ $ make packages
This will use CMake and *fpm*, installed as an osquery build dependency, to generate an `osquery-VERSION.{rpm/deb}` and optionally debug and devel packages.
## OS X
## OS X / macOS
In your cloned osquery repository, once you have [built the code](../development/building.md) (hopefully a tagged release):
@ -24,14 +24,13 @@ In your cloned osquery repository, once you have [built the code](../development
$ make packages
```
OS X deployment is a bit more complicated and customizable compared to Linux.
We include some help and guidance for a more esoteric script `make_osx_package.sh`:
The macOS deployment is a bit more complicated and customizable compared to Linux. We include some help and guidance for a more esoteric script `make_osx_package.sh`:
```sh
$ ./tools/deployment/make_osx_package.sh -h
```
This tool will build an OS X package with:
This tool will build an OS X/macOS package with:
- the **osqueryi** and **osqueryd** binaries
- the [LaunchDaemon](https://github.com/facebook/osquery/blob/master/tools/deployment/com.facebook.osqueryd.plist) that is responsible for osqueryd

35
docs/wiki/installation/install-linux.md
View File

@ -1,12 +1,6 @@
Distro-specific packages are built for each supported operating system.
These packages contain the osquery daemon, shell, example configuration and startup scripts. Note that the `/etc/init.d/osqueryd` script does not automatically start the daemon until a configuration file is created*.
A 'universal' Linux package can be created for each package distribution system. These packages contain the osquery daemon, shell, example configuration and startup scripts. Note that the `/etc/init.d/osqueryd` script does not automatically start the daemon until a configuration file is created*.
Supported distributions for package installs are:
- Ubuntu Xenial 16.04 LTS, Trusty 14.04 LTS, Precise 12.04 LTS
- CentOS 6.6, 7.1
Each osquery tag (release) is published to **yum** and **apt** repositories for our supported operating systems: [https://osquery.io/downloads](http://osquery.io/downloads/).
Each osquery tag (stable release) is published to **yum** and **apt** repositories for our supported operating systems: [https://osquery.io/downloads](http://osquery.io/downloads/).
The default packages create the following structure:
@ -23,38 +17,25 @@ The default packages create the following structure:
## yum-based Distros
We publish two packages, osquery and osquery-latest**, in a yum repository for CentOS/RHEL 6.3-6.6 and 7.0-7.1 built from our Jenkins build hosts. You may install the "auto-repo-add" RPM or add the repository target:
**CentOS/RHEL 7.0**
You may install the "auto-repo-add" RPM or add the repository target. These RPMs should work on any x86-64 Linux with a base install from 2011 forward:
```sh
$ sudo rpm -ivh https://osquery-packages.s3.amazonaws.com/centos7/noarch/osquery-s3-centos7-repo-1-0.0.noarch.rpm
$ sudo yum install osquery
```
**CentOS/RHEL 6.6**
```sh
$ sudo rpm -ivh https://osquery-packages.s3.amazonaws.com/centos6/noarch/osquery-s3-centos6-repo-1-0.0.noarch.rpm
$ sudo yum install osquery
```
## dpkg-based Distros
We publish the same two packages, osquery and osquery-latest, in an apt repository for Ubuntu 16.04 (xenial), 14.04 (trusty), 12.04 (precise):
Replace `DISTRO` with `xenial`, `trusty`, or `precise`
Similar to the **yum-based** distributions, the **dpkg-based** DEB packages should work on any x86-64 Linux since 2011:
```sh
$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 1484120AC4E9F8A1A577AEEE97A80C63C9D8B80B
$ sudo add-apt-repository "deb [arch=amd64] https://osquery-packages.s3.amazonaws.com/DISTRO DISTRO main"
$ sudo add-apt-repository "deb [arch=amd64] https://osquery-packages.s3.amazonaws.com/xenial xenial main"
$ sudo apt-get update
$ sudo apt-get install osquery
```
\* You may also set a different config plugin using a [**flagfile**](../installation/cli-flags.md).<br />
\** We do not recommend using the latest/unstable package as it is built
from our master branch and does not guarantee safety.
## Running osquery
@ -64,8 +45,8 @@ After exploring the rest of the documentation you should understand the basics o
```
sudo cp /usr/share/osquery/osquery.example.conf /etc/osquery/osquery.conf
sudo service osqueryd start
sudo service osqueryd status
# sudo service osqueryd start
sudo systemctl start osqueryd
```
Note: The interactive shell and daemon do NOT communicate!
> NOTICE: The interactive shell and daemon do NOT communicate!

8
docs/wiki/installation/install-osx.md
View File

@ -1,10 +1,10 @@
Continuous integration currently tests stable release versions of osquery against macOS 10.12 (as listed under the _Build_status_ column on the project [README](https://github.com/facebook/osquery/blob/master/README.md)). There are no reported issues which block expected core functionality on 10.11, however 10.10 and previous OS X versions do not work.
Continuous integration currently tests stable release versions of osquery against macOS 10.12 (as listed under the _Build_status_ column on the project [README](https://github.com/facebook/osquery/blob/master/README.md)). There are no reported issues which block expected core functionality on 10.11, however 10.9 and previous macOS versions do not work.
## Package Installation
If you plan to manage an enterprise osquery deployment, the easiest installation method is an OS X package installer. You will have to manage and deploy updates.
If you plan to manage an enterprise osquery deployment, the easiest installation method is a macOS package installer. You will have to manage and deploy updates.
Each osquery tag (release) builds an OS X package:
Each osquery tag (release) builds a macOS package:
[osquery.io/downloads](https://osquery.io/downloads/). There are no package or library dependencies.
The default package creates the following structure:
@ -47,4 +47,4 @@ $ sudo cp /var/osquery/com.facebook.osqueryd.plist /Library/LaunchDaemons/
$ sudo launchctl load /Library/LaunchDaemons/com.facebook.osqueryd.plist
```
Note: The interactive shell and daemon do NOT communicate!
> NOTICE: The interactive shell and daemon do NOT communicate!

6
docs/wiki/installation/install-windows.md
View File

@ -1,12 +1,12 @@
As of osquery 1.8.2+ the Windows builds are feature-complete but provide a limited set of tables compared to OS X and Linux.
As of osquery 1.8.2+ the Windows builds are feature-complete but provide a limited set of tables compared to macOS and Linux.
## Chocolatey
Each osquery tag (release) is published to **chocolatey** for our supported versions: [https://chocolatey.org/packages/osquery/](https://chocolatey.org/packages/osquery/)
Each osquery tag (stable release) is published to **chocolatey** for our supported versions: [https://chocolatey.org/packages/osquery/](https://chocolatey.org/packages/osquery/)
## Installing osquery
By default Chocolatey will install the binaries, example packs, example configuration, and an openssl certs bundle to `C:\ProgramData\osquery` and nothing more. You can pass Chocolatey the `--params='/InstallService'` flag or make use of osquery's `--install` flag with `C:\ProgramData\osquery\osqueryd\osqueryd.exe --install` to install a Windows system service for the **osqueryd** daemon.
By default Chocolatey will install the binaries, example packs, example configuration, and an OpenSSL certificate bundle to `C:\ProgramData\osquery` and nothing more. You can pass Chocolatey the `--params='/InstallService'` flag or make use of osquery's `--install` flag with `C:\ProgramData\osquery\osqueryd\osqueryd.exe --install` to install a Windows system service for the **osqueryd** daemon.
## Running osquery

36
docs/wiki/introduction/sql.md
View File

@ -14,14 +14,13 @@ The osquery SQL language is a superset of SQLite's, please read [SQL as understo
> NOTICE: Several tables, `file` for example, require a predicate for one of the columns, and **will not work without it**. See [Tables with arguments](#tables-with-arguments) for more information.
Before diving into the osquery SQL customizations, please familiarize yourself with the osquery [development shell](../introduction/using-osqueryi.md). This shell is designed for ad-hoc exploration of your OS and SQL query prototyping. Then fire up `osqueryi` as your user or as a superuser and try some of the concepts below.
Before diving into the osquery SQL customizations, please familiarize yourself with the osquery [development shell](../introduction/using-osqueryi.md). This shell is designed for ad-hoc exploration of your OS and SQL query prototyping. Then fire up `osqueryi` as your user or as a superuser and try some of the concepts below. Know that this 'shell' does not connect to anything, it is completely standalone.
### Shell help
Within the shell, try: `.help`
```
$ osqueryi
osquery - being built, with love, at Facebook
Using a virtual database. Need help, type '.help'
osquery> .help
Welcome to the osquery shell. Please explore your OS!
@ -49,7 +48,7 @@ This [complete schema](https://osquery.io/docs/tables/) for all supported platfo
### Your first query
On OS X (or Linux), select 1 process's pid, name, and path. Then change the display mode and issue the same query:
On macOS (or Linux), select 1 process's pid, name, and path. Then change the display mode and issue the same query:
```
osquery> SELECT pid, name, path FROM processes LIMIT 1;
+-----+---------+---------------+
@ -73,14 +72,17 @@ Then let's look at a "meta" table that provides details to osquery about osquery
```
osquery> .mode line
osquery> SELECT * FROM osquery_info;
pid = 3811
version = 1.7.4
config_hash =
pid = 15982
uuid = 4892E1C6-F800-5F8E-92B1-BC2216C29D4F
instance_id = 94c004b0-49e5-4ece-93e6-96c1939c0f83
version = 2.4.6
config_hash =
config_valid = 0
extensions = active
build_platform = darwin
build_distro = 10.11
start_time = 1464730373
build_distro = 10.12
start_time = 1496552549
watcher = -1
```
This will always show the current PID of the running osquery process, shell or otherwise.
@ -88,7 +90,7 @@ This will always show the current PID of the running osquery process, shell or o
Let's use this to demonstrate `JOIN`ing:
```
osquery> SELECT pid, name, path FROM osquery_info JOIN processes USING (pid);
pid = 3811
pid = 15982
name = osqueryi
path = /usr/local/bin/osqueryi
```
@ -100,22 +102,22 @@ osquery> SELECT p.pid, name, p.path as process_path, pf.path as open_path
...> JOIN processes p ON p.pid = i.pid
...> JOIN process_open_files pf ON pf.pid = p.pid
...> WHERE pf.path LIKE '/dev/%';
pid = 3811
pid = 15982
name = osqueryi
process_path = /usr/local/bin/osqueryi
open_path = /dev/ttys000
pid = 3811
pid = 15982
name = osqueryi
process_path = /usr/local/bin/osqueryi
open_path = /dev/ttys000
pid = 3811
pid = 15982
name = osqueryi
process_path = /usr/local/bin/osqueryi
open_path = /dev/ttys000
pid = 3811
pid = 15982
name = osqueryi
process_path = /usr/local/bin/osqueryi
open_path = /dev/null
@ -202,12 +204,12 @@ We have added `sha1`, `sha256`, and `md5` functions that take a single argument
**Encoding functions**
There are also encoding functions available to you to process query results.
- base64: base64 encode a string.
- unbase64: Decode a base64 encoded string. If the string is not valid base64 an empty string is returned.
- conditional_base64: Encode a string if and only if the string contains non-ascii characters.
- `base64`: base64 encode a string.
- `unbase64`: Decode a base64 encoded string. If the string is not valid base64 an empty string is returned.
- `conditional_base64`: Encode a string if and only if the string contains non-ASCII characters.
### Table and column name deprecations
Over time it may makes sense to rename tables and columns. osquery tries to apply plurals to table names and achieve the easiest foreign key JOIN syntax. This often means slightly skewing concept attributes or biasing towards diction used by POSIX.
The tools makes an effort to mark deprecated tables and create 'clone' `VIEW`s so previously scheduled queries continue to work. Similarly for old column names, the column will be marked `HIDDEN` and only returned if explicitly selected. This does not make queries using `*` future-proof, as they will begin using the new column names when the client is updated. All of these changes are considered osquery API changes and marked as such in [release notes](https://github.com/facebook/osquery/releases) on Github.
The tools makes an effort to mark deprecated tables and create 'clone' `VIEW`s so previously scheduled queries continue to work. Similarly for old column names, the column will be marked `HIDDEN` and only returned if explicitly selected. This does not make queries using `*` future-proof, as they will begin using the new column names when the client is updated. All of these changes are considered osquery API changes and marked as such in [release notes](https://github.com/facebook/osquery/releases) on GitHub.

6
docs/wiki/introduction/using-osqueryd.md
View File

@ -1,6 +1,6 @@
`osqueryd` is the host monitoring daemon that allows you to **schedule** queries and record OS state changes. The daemon aggregates query results over time and generates logs, which indicate state change according to each query. The daemon also uses OS eventing APIs to record monitored file and directory changes, hardware events, network events, and more.
The installation and deployment guides are mostly focused on the osquery daemon life cycle. On Linux the daemon starts as an SystemV initscript; on OS X as a launch daemon. The service is highly configurable and extendable.
The installation and deployment guides are mostly focused on the osquery daemon life cycle. On Linux the daemon starts as an SystemV initscript; on macOS as a launch daemon. The service is highly configurable and extendable.
## Configuration and query schedule
@ -9,7 +9,7 @@ The primary daemon feature is executing a query schedule. This schedule is defin
```json
{
"usb_devices": {
"query": "select vendor, model from usb_devices",
"query": "SELECT vendor, model FROM usb_devices;",
"interval": 60
}
}
@ -19,7 +19,7 @@ This simple **usb_devices** query will run approximately every 60 seconds on the
## Logging and reporting
Each query represents a monitored view of your operating system. The first time a scheduled query runs it logs every row in the resulting table with the "added" action. In this example, on an OS X laptop, after the first 60 seconds it would log:
Each query represents a monitored view of your operating system. The first time a scheduled query runs it logs every row in the resulting table with the "added" action. In this example, on a macOS laptop, after the first 60 seconds it would log:
```json
[

7
docs/wiki/introduction/using-osqueryi.md
View File

@ -29,7 +29,7 @@ osquery>
The shell accepts a single positional argument and one of the several output modes. If you want to output JSON or CSV values, try:
```
$ osqueryi --json "select * from routes where destination = '::1'"
$ osqueryi --json "SELECT * FROM routes WHERE destination = '::1'"
[
{"destination":"::1","flags":"2098181","gateway":"::1","interface":"","metric":"0","mtu":"16384","netmask":"128","source":"","type":"local"}
]
@ -38,7 +38,7 @@ $ osqueryi --json "select * from routes where destination = '::1'"
You may also pipe a query as *stdin*. The input will be executed on the **osqueryi** shell and must be well-formed SQL or **osqueryi** meta-commands. Note the added ';' to the query when using *stdin*:
```
$ echo "select * from routes where destination = '::1';" | osqueryi --json
$ echo "SELECT * FROM routes WHERE destination = '::1';" | osqueryi --json
```
## Getting help
@ -83,7 +83,6 @@ CREATE VIRTUAL TABLE routes USING routes(
);
osquery> PRAGMA table_info(routes);
+-----+-------------+---------+---------+------------+----+
| cid | name | type | notnull | dflt_value | pk |
+-----+-------------+---------+---------+------------+----+
@ -105,6 +104,6 @@ $
The shell does not keep much state or connect to the **osqueryd** daemon.
If you would like to run queries and log changes to the output or log operating system events, consider deploying a query **schedule** using [osqueryd](using-osqueryd.md).
**Note:** Event publishers are not started by default. To enable event-based tables, use the flag `--disable_events=false`.
> Note: Event publishers are not started by default. To enable event-based tables, use the flag `--disable_events=false`.
**osqueryi** uses an in-memory database by default. To connect to an existing events database, use the flag `--database_path=/var/osquery/osquery.db` (only one process may attach to the database; see [Checking the database sanity](../deployment/debugging.md#checking-the-database-sanity)).

9
tools/deployment/osquery.example.conf
View File

@ -21,12 +21,6 @@
// large numbers of queries that run a smaller or similar intervals.
//"schedule_splay_percent": "10",
// Write the pid of the osqueryd process to a pidfile/mutex.
//"pidfile": "/var/osquery/osquery.pidfile",
// Clear events from the osquery backing store after a number of seconds.
"events_expiry": "3600",
// A filesystem path for disk-based backing storage used for events and
// query results differentials. See also 'use_in_memory_database'.
//"database_path": "/var/osquery/osquery.db",
@ -35,8 +29,7 @@
// This allows osquery to be launched without certain tables.
//"disable_tables": "foo_bar,time",
// Enable debug or verbose debug output when logging.
"verbose": "false"
"utc": "true"
},
// Define a schedule of queries: