valitydev/osquery-1
14
0
Fork 0
mirror of https://github.com/valitydev/osquery-1.git synced 2024-11-07 01:55:20 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Copy-edit and Markdown lint the remaining deployment docs

Browse Source
This commit is contained in:
Mike Myers 2020-09-22 19:08:51 -07:00 committed by Teddy Reed
parent a68c88bc99
commit a9845b65f2
2 changed files with 36 additions and 33 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

35
docs/wiki/deployment/syslog.md
View File

@ -1,10 +1,12 @@
# Reading syslogs with osquery
osquery 1.7.3 introduced support for consuming and querying the macOS system log via Apple System Log (ASL). osquery 1.7.4 introduced support for the Linux syslog via **rsyslog**. This document explains how to configure and use these syslog tables.
## Apple macOS Syslog
## macOS Syslog
On macOS, the `asl` virtual table makes use of Apple's ASL store, querying this structured store using the routines provided in [`asl.h`](https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man3/asl.3.html).
### Configuration
### macOS Configuration
No configuration is required to begin using the `asl` table. Note, however, that the table is only able to query logs that are available in the ASL store.
@ -12,37 +14,37 @@ If your target logs are not already being sent to the ASL store by your current
The configuration for `/var/log/install.log` and `/var/log/commerce.log` is hardcoded into the Apple provided syslog binaries, and we are not aware of a way to configure ASL to send these logs to the store.
### Usage
### macOS Usage
The `asl` table can be queried like any other osquery table. It exposes many of the columns of structured data from the ASL store, and other additional columns are made available as a JSON dictionary in the `extra` column. Use `.schema asl` in the `osqueryi` shell to see the schema.
Basic query predicates (`<`, `<=`, `=`, `>=`, `>`) are able to be efficiently queried in the store. The `LIKE` predicate is also supported, however it must be tested after applying all other predicates and reading logs from the store. For performance reasons, it is suggested to use at least one basic predicate in a query against the `asl` table. For example,
```
```sql
SELECT time, message FROM asl WHERE facility = 'authpriv' AND sender = 'sudo' AND message LIKE '%python%';
```
## Linux Syslog
On Linux, the `syslog` table queries logs forwarded over a named pipe from a properly configured **rsyslogd**. This method was chosen to support the widest range of Linux flavors (in theory, anything running at least **rsyslogd** version 5, and tested with Ubuntu 12/14, CentOS 7.1, RHEL 7.2), and to ensure that existing syslog routines and configurations are not modified. As syslog is ingested into osquery, it is written into the backing store (RocksDB) and made available for querying.
On Linux, the `syslog` table queries logs forwarded over a named pipe from a properly configured `rsyslogd`. This method was chosen to support the widest range of Linux flavors (in theory, anything running at least `rsyslogd` version 5, and tested with Ubuntu 12+, CentOS 7.1+, RHEL 7.2+), and to ensure that existing syslog routines and configurations are not modified. As syslog is ingested into osquery, it is written into the backing store (RocksDB) and made available for querying.
Alternatively you can also use **syslog-ng** to forward log messages to osquery.
Alternatively you can also use **syslog-ng** to forward log messages to osquery.
> NOTICE: the Syslog ingestion is NOT recommended for hosts functioning as syslog aggregators. We have not tested ingestion for massive-throughput or lossless setups.
### Configuration
### Linux Configuration
The `syslog` table requires additional configuration before it can be used. Append `--enable_syslog` to your command line arguments or `--flagfile` to enable osquery's `syslog` event publisher thread.
When an osquery process that supports the `syslog` table starts up, it will attempt to create (and properly set permissions for) a named pipe for **rsyslogd** to write to. The path for this pipe is determined by the configuration flag `--syslog_pipe_path` (defaults to `/var/osquery/syslog_pipe`). If verbose logging is turned on, you should see a status message indicating whether osquery was able to successfully open the pipe for reading.
When an osquery process that supports the `syslog` table starts up, it will attempt to create (and properly set permissions for) a named pipe for `rsyslogd` to write to. The path for this pipe is determined by the configuration flag `--syslog_pipe_path` (defaults to `/var/osquery/syslog_pipe`). If verbose logging is turned on, you should see a status message indicating whether osquery was able to successfully open the pipe for reading.
Permissions for the pipe must at least allow **rsyslogd** to read/write, and osquery to read. For security, it is advised that the least possible privileges are enabled to allow this.
Permissions for the pipe must at least allow `rsyslogd` to read/write, and osquery to read. For security, it is advised that the least possible privileges are enabled to allow this.
Once the named pipe is created, **rsyslogd** must be configured to write logs to the pipe. Add the following to your **rsyslog** configuration files (usually located in `/etc/rsyslog.conf` or `/etc/rsyslog.d/`):
Once the named pipe is created, `rsyslogd` must be configured to write logs to the pipe. Add the following to your **rsyslog** configuration files (usually located in `/etc/rsyslog.conf` or `/etc/rsyslog.d/`):
#### rsyslog versions < 7
```
```t
$template OsqueryCsvFormat, "%timestamp:::date-rfc3339,csv%,%hostname:::csv%,%syslogseverity:::csv%,%syslogfacility-text:::csv%,%syslogtag:::csv%,%msg:::csv%\n"
*.* |/var/osquery/syslog_pipe;OsqueryCsvFormat
```
@ -51,7 +53,7 @@ $template OsqueryCsvFormat, "%timestamp:::date-rfc3339,csv%,%hostname:::csv%,%sy
The above configuration should also work, but **rsyslog** strongly recommends using the new style configuration syntax.
```
```json
template(
name="OsqueryCsvFormat"
type="string"
@ -62,9 +64,9 @@ template(
#### All versions
**rsyslogd** must be restarted for the changes to take effect. On many systems, this can be achieved by `sudo service rsyslog restart`.
`rsyslogd` must be restarted for the changes to take effect. On many systems, this can be achieved by `sudo service rsyslog restart`.
> NOTICE: **rsyslogd** will only check once, at startup, whether it can write to the pipe. If **rsyslogd** cannot write to the pipe, it will not retry until restart.
> NOTICE: `rsyslogd` will only check once, at startup, whether it can write to the pipe. If `rsyslogd` cannot write to the pipe, it will not retry until restart.
#### Other configuration
@ -73,7 +75,8 @@ Configuration flags control the retention of syslog logs. `--syslog_events_expir
#### Configuring syslog-ng
Configuring osquery to receive logs from syslog-ng is no different from rsyslog, so here only the syslog-ng part is shown. Add the following to your **syslog-ng** configuration files (usually located in `/etc/syslog-ng/syslog-ng.conf` or `/etc/syslog-ng/conf.d/`):
```
```t
# Reformat log messages in a format that osquery accepts
rewrite r_csv_message {
set("$MESSAGE", value("CSVMESSAGE") );
@ -104,7 +107,7 @@ destination d_osquery {
};
```
The rewrite is needed to make sure that quotation marks are escaped. The template re-formats the messages as expected by osquery. Binaries provided by the osquery project expect syslog messages in this pipe: you might need to change the location if you compiled osquery yourself. If you want to see what messages are sent to osquery you can uncomment the “d_osquery_copy” destination in the log path. The “s_sys” source refers to your local log messages and might be different on your system (this example is from CentOS).
The rewrite is needed to make sure that quotation marks are escaped. The template re-formats the messages as expected by osquery. Binaries provided by the osquery project expect syslog messages in this pipe: you might need to change the location if you compiled osquery yourself. If you want to see what messages are sent to osquery you can uncomment the `d_osquery_copy` destination in the log path. The `s_sys` source refers to your local log messages and might be different on your system (this example is from CentOS).
### Usage

34
docs/wiki/deployment/yara.md
View File

@ -1,6 +1,6 @@
There are two YARA related tables in osquery, which serve very different purposes. The first table, called `yara_events`, uses osquery's [Events framework](../development/pubsub-framework.md) to monitor for filesystem changes and will execute YARA when a file change event fires. The second table, called **yara**, is an on-demand YARA scanning table.
# YARA-based scanning with osquery
Both of these tables are considered a beta feature right now.
There are two YARA-related tables in osquery, which serve very different purposes. The first table, called `yara_events`, uses osquery's [Events framework](../development/pubsub-framework.md) to monitor for filesystem changes and will execute YARA when a file change event fires. The second table, just called `yara`, is an on-demand YARA scanning table.
## YARA Configuration
@ -33,15 +33,15 @@ The configuration for osquery is simple. Here is an example config:
}
```
The first thing to notice is the **file_paths** section, which is used to describe which paths to monitor for changes. Each key is an arbitrary category name and the value is a list of paths. The syntax used is documented on the osquery wildcarding rules described on the [FIM](../deployment/file-integrity-monitoring.md) page. The paths, when expanded out by osquery, are monitored for changes and processed by the [**file_events**](https://osquery.io/schema/current/#file_events) table.
The first thing to notice is the `file_paths` section, which is used to describe which paths to monitor for changes. Each key is an arbitrary category name and the value is a list of paths. The syntax used is documented on the osquery wildcarding rules described on the [FIM](../deployment/file-integrity-monitoring.md) page. The paths, when expanded out by osquery, are monitored for changes and processed by the [`file_events`](https://osquery.io/schema/current/#file_events) table.
The second thing to notice is the **yara** section, which contains the configuration to use for YARA within osquery. The **yara** section contains two keys: **signatures** and **file_paths**. The **signatures** key contains a set of arbitrary key names, called "signature groups". The value for each of these groups are the paths to the signature files that will be compiled and stored within osquery. The paths to the signature files can be absolute or relative to ```/etc/osquery/yara/```. The **file_paths** key maps the category name for an event described in the global **file_paths** section to a signature grouping to use when scanning.
The second thing to notice is the `yara` section, which contains the configuration to use for YARA within osquery. The `yara` section contains two keys: `signatures` and `file_paths`. The `signatures` key contains a set of arbitrary key names, called "signature groups." The value for each of these groups are the paths to the signature files that will be compiled and stored within osquery. The paths to the signature files can be absolute or relative to ```/etc/osquery/yara/```. The `file_paths` key maps the category name for an event described in the global `file_paths` section to a signature grouping to use when scanning.
For example, when a file in */usr/bin/* and */usr/sbin/* is changed it will be scanned with *sig_group_1*, which consists of *foo.sig* and *bar.sig*. When a file in */Users/%/tmp/* (recursively) is changed it will be scanned with *sig_group_1* and *sig_group_2*, which consists of all three signature files.
For example, when a file in `/usr/bin/` and `/usr/sbin/` is changed it will be scanned with `sig_group_1`, which consists of `foo.sig` and `bar.sig`. When a file in `/Users/%/tmp/` (recursively) is changed it will be scanned with `sig_group_1` and `sig_group_2`, which consists of all three signature files.
# yara_events table
## Continuous monitoring using the yara_events table
Using the configuration above you can see it in action. While osquery was running I executed `touch /Users/wxs/tmp/foo` in another terminal. Here is the relevant queries to show what happened:
Using the configuration above you can see it in action. While osquery is running, we execute `touch /Users/wxs/tmp/foo` in another terminal. Here is the relevant queries to show what happened:
```sql
osquery> SELECT * FROM file_events;
@ -59,9 +59,9 @@ osquery> SELECT * FROM yara_events;
osquery>
```
The [**file_events**](https://osquery.io/schema/current/#file_events) table recorded that a file named */Users/wxs/tmp/foo* was created with the corresponding hashes and a timestamp.
The [`file_events`](https://osquery.io/schema/current/#file_events) table recorded that a file named `/Users/wxs/tmp/foo` was created with the corresponding hashes and a timestamp.
The [**yara_events**](https://osquery.io/schema/current/#yara_events) table recorded that 1 matching rule (*always_true*) was found when the file was created. In this example every file will always have at least one match because I am using a rule which always evaluates to true. In the next example I'll issue the same command to create a file in a monitored directory but have removed the *always_true* rule from my signature files.
The [`yara_events`](https://osquery.io/schema/current/#yara_events) table recorded that 1 matching rule (`always_true`) was found when the file was created. In this example every file will always have at least one match because we are using a rule which always evaluates to true. In the next example we'll issue the same command to create a file in a monitored directory but have removed the `always_true` rule from our signature files.
```sql
osquery> SELECT * FROM yara_events;
@ -74,17 +74,17 @@ osquery> SELECT * FROM yara_events;
osquery>
```
As you can see, even though no matches were found a row is still created and stored.
As you can see, even though no matches were found, a row is still created and stored.
## On-demand YARA scanning
The [**yara**](https://osquery.io/schema/current/#yara) table is used for on-demand scanning. With this table you can arbitrarily YARA scan any available file on the filesystem with any available signature files or signature group from the configuration. In order to scan, the table must be given a constraint which says where to scan and what to scan with.
The [`yara`](https://osquery.io/schema/current/#yara) table is used for on-demand scanning. With this table you can arbitrarily YARA scan any available file on the filesystem with any available signature files or signature group from the configuration. In order to scan, the table must be given a constraint which says where to scan and what to scan with.
In order to determine where to scan, the table accepts either a *path* or a *pattern* constraint. The *path* constraint must be a full path to a single file. There is no expansion or recursion with this constraint. The *pattern* constraint follows the same wildcard rules mentioned before.
In order to determine where to scan, the table accepts either a `path` or a `pattern` constraint. The `path` constraint must be a full path to a single file. There is no expansion or recursion with this constraint. The `pattern` constraint follows the same wildcard rules mentioned before.
Once the "where" is out of the way, you must specify the "what" part. This is done through either the *sigfile* or *sig_group* constraints. The *sigfile* constraint can be either an absolute path to a signature file on disk or a path relative to */var/osquery/*. The signature file will be compiled only for the execution of this one query and removed afterwards. The *sig_group* constraint must consist of a named signature grouping from your configuration file.
Once the `where` is out of the way, you must specify the "what" part. This is done through either the `sigfile` or `sig_group` constraints. The `sigfile` constraint can be either an absolute path to a signature file on disk or a path relative to `/var/osquery/`. The signature file will be compiled only for the execution of this one query and removed afterwards. The `sig_group` constraint must consist of a named signature grouping from your configuration file.
Here are some examples of the **yara** table in action:
Here are some examples of the `yara` table in action:
```sql
osquery> SELECT * FROM yara WHERE path="/bin/ls" AND sig_group="sig_group_1";
@ -102,7 +102,7 @@ osquery> SELECT * FROM yara WHERE path="/bin/ls" AND sig_group="sig_group_2";
osquery>
```
As you can see in these examples, I'm scanning the same file with two different signature groups and getting different results.
As you can see in these examples, we scan the same file with two different signature groups and get different results.
```sql
osquery> SELECT * FROM yara WHERE pattern="/bin/%sh" AND sig_group="sig_group_1";
@ -119,7 +119,7 @@ osquery> SELECT * FROM yara WHERE pattern="/bin/%sh" AND sig_group="sig_group_1"
osquery>
```
The above illustrates using the *pattern* constraint to scan */bin/%sh* with a signature group.
The above illustrates using the `pattern` constraint to scan `/bin/%sh` with a signature group.
```sql
osquery> SELECT * FROM yara WHERE pattern="/bin/%sh" AND sigfile="/Users/wxs/sigs/baz.sig";
@ -136,4 +136,4 @@ osquery> SELECT * FROM yara WHERE pattern="/bin/%sh" AND sigfile="/Users/wxs/sig
osquery>
```
The above is an example of using an absolute path for *sigfile* combined with *pattern*.
The above is an example of using an absolute path for `sigfile` combined with `pattern`.