POSIX-globbing will allow event publishers/subscribers to post-check
results against glob-syntax, fnpath matching, and POSIX C-regex.
These checks are anecdotally speedy.
We found that not installing the headers for snappy caused RocksDB's
snappy detection to not find that snappy was installed:
https://goo.gl/YOWJl0
The snippet there requires that the headers are installed, not just the
library. By installing the headers, we can ensure that snappy is linked.
OR, alternatively, we could just leave it and not link snappy. It's
uncertain what the specific benefits of including snappy are for our
use-case. (CC @igorcanadi)
This test does not evaluate the functionality of the kernel
communication unless the KERNEL_TEST flag was set during the build.
The test will not succeed unless the tests are being run as root.
We now have a Publisher to report on disk events and its metadata,
using the DiskArbitration framework on OS X. Currently disk appearance
and disappearance events are published for both physical and
virtual disks (DMG files). On an event trigger, disk properties are
parsed and that metadata is reported along with the action.
The Subscriber subscribes to virtual disk events currently.
This closes#1103.
`int osquery::filter(const struct dirent*)` seemed like a pretty generic
symbol to have in our symbol table, so I changed it to
`int msrScandirFilter(const struct dirent*)`
This infomation is primarily related to the performance of processor
cores. The information given constitutes only a small portion of
the information in the model specific register, but this table
has been designed so that more information may easily be added.
The table requires osquery be run as the root, and that the msr
kernel module is loaded. The table reads the msr data from /dev
1. Example queries will run with an (optional) integration test.
2. Fix bad accesses with OS X package BOMs
3. Move spec files from ./osquery/tables/specs to ./specs
4. Remove server parsers (netlib) from client builds.
`make tests` fails with another osquery process running.
The backing-store check happens after a config plugin is setUp and
the initial load occures. This may involve calls to cached keys, the
check should occur pre-config initialize.
This change allows QueryContext constraints to be checked for based on
operator type. This makes checks for the existence of an equality
operator allow enumeration.
Example:
if (context.constraints["pid"].exists(EQUALS)) {
pids = context.constraints["pid"].getAll(EQUALS);
} else {
osquery::procProcesses(pids);
}
This implements the following tables on FreeBSD:
process_envs
process_memory_map
process_open_files
process_open_sockets
processes
All the heavy lifting is done with libprocstat(3). All the tables follow
the same general principle. Use the common function, getProcesses() in
procstat.cpp, to get the processes and then generate the rows for each
process returned. There is also a procstatCleanup() function commonly
used across all the tables.
The one thing I am not able to test is the process_open_sockets table on
an IPv6 machine.
The first draft of the remote logger for osquery. This should give a rough idea
of how the code will be structured and function. RFC please.
At the advice of @theopolis, I removed the category type and added the
http_logger key. We figure this should be more efficient and doesn't have to
be known at compile time.
The user_groups table represents the association between user ids and group ids.
Darwin Issue:
Issues arise in darwin systems with users that are members of many groups due
to a bug in Apple's implementation of getgrouplist. If the number of groups a
user is a member of is greater than 64 a truncated association table may
be returned.
Previously this only existed in the yara table, but it now exists in the
yara config parser land, which will compile signature groups upon
update. Now your signature groups can reference signature files using
paths relative to /var/osquery.
This was causing a crash when executing a query using the yara table
from the command line, because YARA was never initialized properly, so
the thread index was whatever was left on the stack. Eventually YARA
would attempt to set a rule that matches using this thread index and
would explode in flames.
Fix it by moving the initialization to a place that is always called.
Remove modes: HTML, Tabs, Explain.
Remove stats (meaningless for virtual tables).
Use the osquery SQLite DB manager within meta and shell SQL
execution to allow registry/extensions non-locking access.
This allows existing runtime DB manipulators to prefer a locking
modifier. Currently these manipulators will fallback to a more
transient db instance. The effect was, no shell-accessible runtime
created virtual tables.
1. Minor refactoring.
- Generate one row per sigfile or sig_group.
- While here, when a signature file fails to compile, VLOG() it.
2. Bring in a couple of YARA tests.
Write a couple of tests for YARA functionality. Right now the only tests
make sure rules are compiled properly and that rules match where they
should and don't match where they shouldn't.
3. Allow sigfiles to be relative to /var/osquery.
- Also, only create a row if scanning happened.
4. Add pattern support to yara table.
- Also, optimize things so that rules are only compiled once.
1. Rename yara_matches to yara_events.
2. Add support for Config::getParser().
- This returns a ConfigPluginRef, which is the ConfigParser for the
given key.
- Being able to get the parser is useful because the
YARAConfigParserPlugin uses it to store the compiled rules as an
attribute.
3. Finish rename and use ConfigParserPlugin.
- Finish the table rename to yara_events.
- Use the new ConfigParserPlugin interface to parse the YARA
configuration. The file_paths and signatures are stored in the
ConfigParserPlugin named "yara" under the key "yara". The rules are
compiled and stored as a private attribute of the same
ConfigParserPlugin object.
Here is an example config using this new structure:
{
// Description of the YARA feature.
"yara": {
"signatures": {
// Each key is an arbitrary group name to give the signatures listed
"sig_group_1": [ "/Users/wxs/foo.sig", "/Users/wxs//bar.sig" ],
"sig_group_2": [ "/Users/wxs/baz.sig" ]
},
"file_paths": {
// Each key is a key from file_paths
// The value is a list of signature groups to run when an event fires
// These will be watched for and scanned when the event framework
// fire off an event to yara_events table
"system_binaries": [ "sig_group_1" ],
"tmp": [ "sig_group_1", "sig_group_2" ]
}
},
// Paths to watch for filesystem events
"file_paths": {
"system_binaries": [ "/usr/bin/%", "/usr/sbin/%" ],
"tmp": [ "/Users/wxs/tmp/%%" ]
}
}
- Currently the signature file must be an absolute path.
3. Move common YARA code to yara_utils.
- In preparation for the yara table (different from yara_events) I'm
moving the common YARA code into a separate place which is shared
between the two tables.
4. Add yara table.
- This allows you to do things like:
```sql
select * from yara where path="/bin/ls" and sigfile="/tmp/foo.sig";
select * from yara where path="/bin/ls" and sig_group="sig_group_1";
```
- The latter will use the signature grouping from the config.
5. Check for keys not existing.
moved disk_ecryption table spec to crossplatform
link libcryptsetup
implemented get cipher type and cipher_mode:
more idiomatic c++11
no need to explicitly call std::string constructor to convert char * to std::string
update cryptsetup sources for centos
add function prototype for older libcryptsetup which is in centos6
ifdef check for centos6 which uses older libcryptsetup
remove forward declared functions defined in libcryptsetup, stylistic changes
First draft of the enrollment plugin and part of the config plugin.
Please comment on potential structure and functionality.
They way it's designed to work is this:
Both the config and logger plugins will call the enroll getKey function. getKey
is in charge of maintaining the state of having a key and doing the enrollment.
If enroll has a key, then it will instantly return doing no HTTP requests, if it
doesn't, then it will try a few times to get a key, if it's not successful, it
fails and returns an empty string, if it succeeds, then it will return the
client enrollment key for the requester to use.
For the second time in a couple of weeks, I've rewritten the xattr table into
a new extended_attributes table.
If we find an attribute that we don't have a parser for, we will check if it
contains non printable characters. If it does, we'll base64 it. If it doesn't,
we will just output the unencoded string.
There are a couple places where this was an issue.
The first place was in the filesystem plugin where it was only checked that it
existed, and not that it was an actual file.
The second was a lack of try and catch on the parse call in config.cpp.
Both of those issues are addressed in this diff.
After the merge you need to include yara.h after fsevents.h on OS X.
Apaprently DECLARE_SUBSCRIBER() was dropped during the merge, which
meant that attempts to find the subscriber would fail, throwing an
unhandled exception.
Fix the table spec to use the new table name.
Must use recursive on Linux.
In order to watch a directory you must set recursive to true in
SubscriptionContext under Linux.
Currently only for OS X, will port to others soon.
Also need to add tests.
Remove old comment and add loading message.
Implement YARA table for Linux.
Use mask properly.
Use the various masks to specify the kinds of events we are interested
in. This removes the need to do the dirty "DELETED" check when the event
fires.
Make getYARAFiles return a const map.
Switch to LOG(WARNING) and emit error number.
Add vim .swp files to .gitignore.
Add yara_utils.(c|h).
Start to condense common code between the Linux and Darwin YARA tables
into a yara_utils.h. Right now it includes a function to compile rules
and store the results back in the map, indexed by category. It also has
the callback used by YARA when a rule is processed. I can not move much
more than that for the row creation code because the structures used in
the event callback are slightly different.
Include a better error message.
The errors are still printed by the compiler callback, but this will
allow my future work to return a Status from the event initialization to
print a useful message in summary.
Make Subscriber init() return Status.
Each EventSubscriber::init() now returns a Status. If the init() fails
for any reason the EventSubscriber is still stored but the failure is
tracked.
EventSubscribers now have a state member, which represents the current
state of the subscriber. The current supported states are:
uninitialized, running, paused, failed. Currently the only meaningful
ones are running and failed, but I put paused in there as a
forward-looking feature.
Subscriptions now have a subscriber_name member. This is used in
EventPublisherPlugin::fire() as a lookup to get the EventSubscriber and
check the state. If the EventSubscriber is not running the event will
not fire.
Only the EventSubscribers on OS X are using this. I'll do the Linux
implementation next.
Chase the init() changes to Linux.
This brings the Linux YARA table in line with the OS X one.
Require a EventSubscriberID when creating a subscription.
Now that Subscriptions are "tied" to EventSubscribers you must create a
Subscription with the name of the Subscriber it is for. This is because
when the event fires the list of Subscriptions is walked and the name is
used to lookup the EventSubscriber and make sure it is in the running
state.
Fix various tests.
Some tests would fire an event with only a Subscription, which is no
longer a valid thing to do. For these tests an EventSubscription is
created and registered in the EventFactory.
When Subscriptions are created pass the name of the EventSubscriber to
them. In some cases where no event is ever fired it is fine to pass a
bogus name.
Fix inotify tests.
Move a test down so the class is defined and make sure to create an
EventSubscriber and use it properly.
Add support for yara to provision.sh.
Right now this grabs yara 3.3.0 and applies the patch to fix min() and max(),
which is commit fc4696c8b725be1ac099d340359c8d550d116041 in the yara repo.
This has been tested under Ubuntu 14.04 only.
Remove NOMINMAX.
This is no longer necessary after the patch was backported to 3.3.0.
Revert "Add support for yara to provision.sh."
This reverts commit a8bd371498c0979f070adeff23d05571882ac3f1.
Use vendored YARA code in third-party.
This switches to using the YARA code contained in third-party, including
the patch to fix min/max macros.
Fix mismerge.
Remove unused function after merge.
Well, soon to be unused as soon as I fix up the Linux YARA table. ;)
Chase config changes.
Make the Linux YARA table use ConfigDataInstance along with files() and
yaraFiles().
properly query IORegistry
remove fde_status implementation from block_devices
scaffolding for disk_encryption table
add disk_encryption table schema
implement disk_enryption table for OS X
clang-format the source
add newline at the end of disk_encryption.table
add device prefix to the bsd_name
provide link to apple's open source
renamed fde_status to disk_encryption, more readable code
preserve alphabetical ordering
tiny formatting fix
change header comment back to original
Apparently if it ain't broke, don't fix it. Opera support seems to be exactly
the same as Chrome except changing the path. The file is basically just copied
with Chrome replaced with Opera + path change.
Added cross platform functionality to chrome and opera. Plus abstracted it in
more general functions that can be used for other chrome based browsers.
The extended attributes table used to have its own parsing algorithm
and functions. These are unnecessary because osquery has built in PLIST parsing
provided by the operating system. Thus, I've moved the code to using that and
removed the xattr tests because they only tested the now non-existant parsing.
Further, the files have been renamed so they should now play nice with the
included profiler.
ConfigPlugin items can use action='update' along with a single
source, data to add to the config structure. If multiple external
sources update at the same time, multiple 'update' actions are needed.
I put the original escape back in but redirected the call to a new function
that will escape characters in the form of \xNN when:
`byte < 0x20 || byte >= 0x80`
This leaves slashes alone and should fix this issue.
UPDATE: Tests have also been added. Added an English test to test for NOP.
I believe the cause of the problem was that an extraneous escape was happening
in the `addNewResults` function in query.cpp.
I believe this can be safely removed because it's purpose is only to make things
JSON safe. However, I don't think this function is ever called with out a JSON
serialization later, making this unnecessary.
Move /osquery/python_tests/* to /tools/tests
Move test_extensions process controls to test_base module
Use test_base.Testing to implement each module's main()
- This applies a default argparse with --build
- test_base.ARGS is the argparse-parsed namespace
- Use test_base.ARGS.build for the platform-specific dir
Move WatchdogTests to /tools/tests/test_watchdog.py
Use osquery-C flags for every object compile.
Add CXX flags without conditional logic.
Move the `python-thrift` target into the CPP generation command.
Remove verbose option for extensions python unittest.
Add thrift as a pip install requirement (for unittests).
Here we create a new CMake macro for adding python integration tests,
as well as a wrapper for easy testing of osqueryi. There is a PoC test
of the time table.
Use --socket for extensions, limit help
Add an 'active' concept to registries, support a blank item call
Add osquery_registry to list the internal/external plugin details
