Commit Graph

1766 Commits

Author SHA1 Message Date
Wesley Shields
5de1f484df First cut at YARA docs. 2015-04-27 23:59:55 -04:00
Teddy Reed
467ecc20ae Merge pull request #1059 from theopolis/shell_improv
Various shell fixups
2015-04-27 17:29:02 -07:00
Teddy Reed
d5b9c0216b Merge pull request #1058 from theopolis/catching_registry
Apply vegas-style rules to call
2015-04-27 17:28:18 -07:00
Teddy Reed
2c5cbdee63 Various shell fixups 2015-04-27 16:40:05 -07:00
Teddy Reed
2b5b9683a4 Apply vegas-style rules to call 2015-04-27 15:08:03 -07:00
Teddy Reed
04f80f1ef3 Merge pull request #1057 from wxsBSD/yara_fix2
Move YARA initialization to setUp().
2015-04-27 14:42:32 -07:00
Teddy Reed
df5ee4aca8 Merge pull request #1054 from theopolis/chrome_profiles
[Fix #1017] Use a fs glob in chrome-extensions searching
2015-04-27 11:45:30 -07:00
Teddy Reed
cd7d68c994 Use a fs glob in chrome-extensions searching
Chrome (non-Opera) may use multiple profiles before the
extensions dir. Use a glob before searching for extensions/versions.
2015-04-27 11:36:44 -07:00
Teddy Reed
beb28a3f96 Merge pull request #1051 from theopolis/slim_shell
Remove unpopular features from shell
2015-04-27 10:53:55 -07:00
Teddy Reed
3f36241448 Merge pull request #1056 from theopolis/fast_tests
Fast tests
2015-04-27 10:49:51 -07:00
Wesley Shields
bb392c42f6 Move YARA initialization to setUp().
This was causing a crash when executing a query using the yara table
from the command line, because YARA was never initialized properly, so
the thread index was whatever was left on the stack. Eventually YARA
would attempt to set a rule that matches using this thread index and
would explode in flames.

Fix it by moving the initialization to a place that is always called.
2015-04-27 13:45:20 -04:00
Teddy Reed
be65922569 Fast tests 2015-04-27 09:40:31 -07:00
Teddy Reed
149cc8594b Remove unpopular features from shell.
Remove modes: HTML, Tabs, Explain.
Remove stats (meaningless for virtual tables).

Use the osquery SQLite DB manager within meta and shell SQL
execution to allow registry/extensions non-locking access.

This allows existing runtime DB manipulators to prefer a locking
modifier. Currently these manipulators will fallback to a more
transient db instance. The effect was, no shell-accessible runtime
created virtual tables.
2015-04-26 18:54:27 -07:00
Teddy Reed
ed69536c06 Update ubuntu.sh 2015-04-26 17:41:08 -07:00
Teddy Reed
337a20cc75 Install cmake 3.2.1 on 14.04
The repo-provided CMake is at 2.x, which will build extra cpp-netlib tests.
2015-04-26 17:41:08 -07:00
Mike Arpaia
b3540034f8 Merge pull request #1038 from marpaia/file_wildcard
[Fix #1013] wildcard support in file table
2015-04-26 16:10:50 -07:00
Teddy Reed
16447e67d6 [Fix #1040] Check for disabled event subs 2015-04-26 12:15:49 -07:00
Teddy Reed
87ebb93bf7 Merge pull request #1045 from theopolis/yara_wip
YARA Set of commits from wxsBSD squashed
2015-04-26 03:14:04 -07:00
Wesley Shields
67bf099207 YARA tests, SQL matching, sigfile loading
1. Minor refactoring.

- Generate one row per sigfile or sig_group.
- While here, when a signature file fails to compile, VLOG() it.

2. Bring in a couple of YARA tests.
Write a couple of tests for YARA functionality. Right now the only tests
make sure rules are compiled properly and that rules match where they
should and don't match where they shouldn't.

3. Allow sigfiles to be relative to /var/osquery.
- Also, only create a row if scanning happened.

4. Add pattern support to yara table.
- Also, optimize things so that rules are only compiled once.
2015-04-26 03:03:48 -07:00
Teddy Reed
fcde6c4bfc Move yara out of core/SDK into additional 2015-04-26 03:01:28 -07:00
Wesley Shields
a9f66fa38b Major YARA refactor and enhancements
1. Rename yara_matches to yara_events.

2. Add support for Config::getParser().
- This returns a ConfigPluginRef, which is the ConfigParser for the
  given key.
- Being able to get the parser is useful because the
  YARAConfigParserPlugin uses it to store the compiled rules as an
  attribute.

3. Finish rename and use ConfigParserPlugin.
- Finish the table rename to yara_events.
- Use the new ConfigParserPlugin interface to parse the YARA
  configuration. The file_paths and signatures are stored in the
  ConfigParserPlugin named "yara" under the key "yara". The rules are
  compiled and stored as a private attribute of the same
  ConfigParserPlugin object.

Here is an example config using this new structure:

{
  // Description of the YARA feature.
  "yara": {
    "signatures": {
      // Each key is an arbitrary group name to give the signatures listed
      "sig_group_1": [ "/Users/wxs/foo.sig", "/Users/wxs//bar.sig" ],
      "sig_group_2": [ "/Users/wxs/baz.sig" ]
    },
    "file_paths": {
      // Each key is a key from file_paths
      // The value is a list of signature groups to run when an event fires
      // These will be watched for and scanned when the event framework
      // fire off an event to yara_events table
      "system_binaries": [ "sig_group_1" ],
      "tmp": [ "sig_group_1", "sig_group_2" ]
    }
  },

  // Paths to watch for filesystem events
  "file_paths": {
    "system_binaries": [ "/usr/bin/%", "/usr/sbin/%" ],
    "tmp": [ "/Users/wxs/tmp/%%" ]
  }
}

- Currently the signature file must be an absolute path.

3. Move common YARA code to yara_utils.
- In preparation for the yara table (different from yara_events) I'm
  moving the common YARA code into a separate place which is shared
  between the two tables.

4. Add yara table.
- This allows you to do things like:

```sql
select * from yara where path="/bin/ls" and sigfile="/tmp/foo.sig";
select * from yara where path="/bin/ls" and sig_group="sig_group_1";
```

- The latter will use the signature grouping from the config.

5. Check for keys not existing.
2015-04-26 03:01:28 -07:00
Mike Arpaia
a3bdc71c83 Merge pull request #1044 from shawndavenport/patch-1
Fix a broken link to the OS X install docs.
2015-04-25 20:18:52 -07:00
Shawn Davenport
abdf273f0a Fix install-osx link. 2015-04-25 19:57:07 -07:00
Teddy Reed
a972b1b0b0 Merge pull request #1027 from sharvilshah/fde_linux
[Implement #933] Add LUKS/dm-crypt disk_encryption support for Linux
2015-04-25 12:43:05 -07:00
Mike Arpaia
f0302cedb6 clang-format in http config plugin 2015-04-25 12:06:48 -07:00
Teddy Reed
5e2ce5c2e9 Merge pull request #1039 from theopolis/fix_flags
Fix dameon flags loading from options
2015-04-25 01:28:30 -07:00
Teddy Reed
b1eee9503a Merge pull request #1021 from theopolis/rpm_files
Add rpm_package_files table
2015-04-25 01:27:55 -07:00
Teddy Reed
13d1ff031b Add rpm_package_files table 2015-04-25 01:18:55 -07:00
Teddy Reed
c2a8187db5 Merge pull request #1042 from theopolis/rhel_easier
Simpler RHEL6 provision
2015-04-24 17:05:36 -07:00
Sharvil Shah
f72dcb5d96 add libcrypysetup-dev library
moved disk_ecryption table spec to crossplatform

link libcryptsetup

implemented get cipher type and cipher_mode:

more idiomatic c++11

no need to explicitly call std::string constructor to convert char * to std::string

update cryptsetup sources for centos

add function prototype for older libcryptsetup which is in centos6

ifdef check for centos6 which uses older libcryptsetup

remove forward declared functions defined in libcryptsetup, stylistic changes
2015-04-24 17:01:14 -07:00
Teddy Reed
5e08b8bf60 Simpler RHEL6 provision 2015-04-24 14:25:59 -07:00
Teddy Reed
b90aeab2fe Fix dameon flags loading from options 2015-04-24 11:37:51 -07:00
mike@arpaia.co
7eb4402a78 VLOG instead of WARNING 2015-04-24 08:55:36 -07:00
mike@arpaia.co
dedee94441 Changing the column wildcard to pattern 2015-04-24 08:54:18 -07:00
Mike Arpaia
3db60378f4 [Fix #1013] wildcard support in file table
Now you can run a query like:

```
[localhost] linux (file_wildcard) * ./osquery/osqueryi
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
osquery - being built, with love, at Facebook
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Connected to a transient in-memory database.
osquery> select path from file where wildcard = "/home/%/git/osquery/%";
+--------------------------------------------+
| path                                       |
+--------------------------------------------+
| /home/marpaia/git/osquery/.clang-format    |
| /home/marpaia/git/osquery/osquery.thrift   |
| /home/marpaia/git/osquery/PATENTS          |
| /home/marpaia/git/osquery/README.md        |
| /home/marpaia/git/osquery/Vagrantfile      |
| /home/marpaia/git/osquery/CONTRIBUTING.md  |
| /home/marpaia/git/osquery/mkdocs.yml       |
| /home/marpaia/git/osquery/Doxyfile         |
| /home/marpaia/git/osquery/.gitmodules      |
| /home/marpaia/git/osquery/requirements.txt |
| /home/marpaia/git/osquery/Makefile         |
| /home/marpaia/git/osquery/LICENSE          |
| /home/marpaia/git/osquery/.gitignore       |
| /home/marpaia/git/osquery/CMakeLists.txt   |
+--------------------------------------------+
```
2015-04-23 23:39:32 -07:00
Teddy Reed
85b6d979d7 Merge pull request #1035 from steven-hilder/fixCsvFlag
Fix --csv flag in osqueryi shell
2015-04-23 16:57:14 -07:00
Teddy Reed
bfb8f64d37 Merge pull request #1024 from sharvilshah/faster_vagrant
Use all available CPUs when provisioning a Vagrant box
2015-04-23 16:56:03 -07:00
Steven Hilder
0937a92cd1 Fix --csv flag in osqueryi shell 2015-04-23 21:32:14 +00:00
Sharvil Shah
6383a9917f Support setting v.cpus via ENV variable, default fallback to 2
Update docs: mention V_CPUS ENV variable for vagrant

Use OSQUERY_BUILD_CPUS as the ENV var instead of V_CPUS
2015-04-23 14:15:16 -07:00
Javier Marcos
6f447ffedb Merge pull request #1031 from javuto/etc_protocols_table
Adding new table for /etc/protocols
2015-04-22 18:18:03 -07:00
Teddy Reed
328bc062aa Merge pull request #1034 from theopolis/fix_cmake_centos7
Build cmake with gcc to avoid gnu++1y
2015-04-22 18:14:54 -07:00
Teddy Reed
b2dc8b7264 Build cmake with gcc to avoid gnu++1y 2015-04-22 17:58:08 -07:00
Javier Marcos
ddb41ae84a Adding tests to the prototocols table 2015-04-22 17:49:27 -07:00
Javier Marcos
93d2b58b60 Adding new table for /etc/protocols 2015-04-22 15:46:29 -07:00
Mike Arpaia
e1c5b5b596 Merge pull request #1028 from arirubinstein/master
Add a missing Page attribute to fix broken mkdocs build
2015-04-22 11:00:41 -07:00
Ari Rubinstein
7f50484f1d Add a missing Page attribute to fix broken mkdocs build 2015-04-22 10:01:42 -07:00
Teddy Reed
b8a54dbcc9 Merge pull request #1026 from sharvilshah/doc_updates
Minor doc updates
2015-04-22 08:36:55 -07:00
Sharvil Shah
b14874aacd update links to osquery.rtfd and better formatting 2015-04-21 20:36:35 -07:00
Teddy Reed
ab4e78f875 Merge pull request #1005 from theopolis/osx_memmap
Towards OS X memory maps
2015-04-20 22:18:36 -07:00
Mike Arpaia
b409049050 Merge pull request #1010 from marpaia/readme
Changing read the docs URL to be https
2015-04-19 21:59:52 -07:00