valitydev/osquery-1
14
0
Fork 0
mirror of https://github.com/valitydev/osquery-1.git synced 2024-11-07 09:58:54 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
3df60e6c7c
osquery-1/specs/yara/yara_events.table

17 lines
739 B
Plaintext
Raw Normal View History

Major YARA refactor and enhancements 1. Rename yara_matches to yara_events. 2. Add support for Config::getParser(). - This returns a ConfigPluginRef, which is the ConfigParser for the given key. - Being able to get the parser is useful because the YARAConfigParserPlugin uses it to store the compiled rules as an attribute. 3. Finish rename and use ConfigParserPlugin. - Finish the table rename to yara_events. - Use the new ConfigParserPlugin interface to parse the YARA configuration. The file_paths and signatures are stored in the ConfigParserPlugin named "yara" under the key "yara". The rules are compiled and stored as a private attribute of the same ConfigParserPlugin object. Here is an example config using this new structure: { // Description of the YARA feature. "yara": { "signatures": { // Each key is an arbitrary group name to give the signatures listed "sig_group_1": [ "/Users/wxs/foo.sig", "/Users/wxs//bar.sig" ], "sig_group_2": [ "/Users/wxs/baz.sig" ] }, "file_paths": { // Each key is a key from file_paths // The value is a list of signature groups to run when an event fires // These will be watched for and scanned when the event framework // fire off an event to yara_events table "system_binaries": [ "sig_group_1" ], "tmp": [ "sig_group_1", "sig_group_2" ] } }, // Paths to watch for filesystem events "file_paths": { "system_binaries": [ "/usr/bin/%", "/usr/sbin/%" ], "tmp": [ "/Users/wxs/tmp/%%" ] } } - Currently the signature file must be an absolute path. 3. Move common YARA code to yara_utils. - In preparation for the yara table (different from yara_events) I'm moving the common YARA code into a separate place which is shared between the two tables. 4. Add yara table. - This allows you to do things like: ```sql select * from yara where path="/bin/ls" and sigfile="/tmp/foo.sig"; select * from yara where path="/bin/ls" and sig_group="sig_group_1"; ``` - The latter will use the signature grouping from the config. 5. Check for keys not existing.
2015-04-17 20:03:43 +00:00
table_name("yara_events")
Yara events was not building
2015-06-29 21:45:31 +00:00
description("Track YARA matches for files specified in configuration data.")
Implement YARA table. Currently only for OS X, will port to others soon. Also need to add tests. Remove old comment and add loading message. Implement YARA table for Linux. Use mask properly. Use the various masks to specify the kinds of events we are interested in. This removes the need to do the dirty "DELETED" check when the event fires. Make getYARAFiles return a const map. Switch to LOG(WARNING) and emit error number. Add vim .swp files to .gitignore. Add yara_utils.(c|h). Start to condense common code between the Linux and Darwin YARA tables into a yara_utils.h. Right now it includes a function to compile rules and store the results back in the map, indexed by category. It also has the callback used by YARA when a rule is processed. I can not move much more than that for the row creation code because the structures used in the event callback are slightly different. Include a better error message. The errors are still printed by the compiler callback, but this will allow my future work to return a Status from the event initialization to print a useful message in summary. Make Subscriber init() return Status. Each EventSubscriber::init() now returns a Status. If the init() fails for any reason the EventSubscriber is still stored but the failure is tracked. EventSubscribers now have a state member, which represents the current state of the subscriber. The current supported states are: uninitialized, running, paused, failed. Currently the only meaningful ones are running and failed, but I put paused in there as a forward-looking feature. Subscriptions now have a subscriber_name member. This is used in EventPublisherPlugin::fire() as a lookup to get the EventSubscriber and check the state. If the EventSubscriber is not running the event will not fire. Only the EventSubscribers on OS X are using this. I'll do the Linux implementation next. Chase the init() changes to Linux. This brings the Linux YARA table in line with the OS X one. Require a EventSubscriberID when creating a subscription. Now that Subscriptions are "tied" to EventSubscribers you must create a Subscription with the name of the Subscriber it is for. This is because when the event fires the list of Subscriptions is walked and the name is used to lookup the EventSubscriber and make sure it is in the running state. Fix various tests. Some tests would fire an event with only a Subscription, which is no longer a valid thing to do. For these tests an EventSubscription is created and registered in the EventFactory. When Subscriptions are created pass the name of the EventSubscriber to them. In some cases where no event is ever fired it is fine to pass a bogus name. Fix inotify tests. Move a test down so the class is defined and make sure to create an EventSubscriber and use it properly. Add support for yara to provision.sh. Right now this grabs yara 3.3.0 and applies the patch to fix min() and max(), which is commit fc4696c8b725be1ac099d340359c8d550d116041 in the yara repo. This has been tested under Ubuntu 14.04 only. Remove NOMINMAX. This is no longer necessary after the patch was backported to 3.3.0. Revert "Add support for yara to provision.sh." This reverts commit a8bd371498c0979f070adeff23d05571882ac3f1. Use vendored YARA code in third-party. This switches to using the YARA code contained in third-party, including the patch to fix min/max macros. Fix mismerge. Remove unused function after merge. Well, soon to be unused as soon as I fix up the Linux YARA table. ;) Chase config changes. Make the Linux YARA table use ConfigDataInstance along with files() and yaraFiles().
2015-03-10 13:22:16 +00:00
schema([
Column("target_path", TEXT, "The path scanned"),
Column("category", TEXT, "The category of the file"),
Column("action", TEXT, "Change action (UPDATE, REMOVE, etc)"),
Column("transaction_id", BIGINT, "ID used during bulk update"),
Column("matches", TEXT, "List of YARA matches"),
Column("count", INTEGER, "Number of YARA matches"),
Add tags and strings columns to YARA tables. When strings match they will be populated into the "strings" column of the table. The format is identifier:offset. When a matching rule has tags defined the tags will be put into the "tags" column of the table in a comma separated list.
2015-07-26 19:38:44 +00:00
Column("strings", TEXT, "Matching strings"),
Column("tags", TEXT, "Matching tags"),
events: Add hidden EID to all events tables (#3159)
2017-04-14 15:20:20 +00:00
Column("time", BIGINT, "Time of the scan"),
Column("eid", TEXT, "Event ID", hidden=True),
Implement YARA table. Currently only for OS X, will port to others soon. Also need to add tests. Remove old comment and add loading message. Implement YARA table for Linux. Use mask properly. Use the various masks to specify the kinds of events we are interested in. This removes the need to do the dirty "DELETED" check when the event fires. Make getYARAFiles return a const map. Switch to LOG(WARNING) and emit error number. Add vim .swp files to .gitignore. Add yara_utils.(c|h). Start to condense common code between the Linux and Darwin YARA tables into a yara_utils.h. Right now it includes a function to compile rules and store the results back in the map, indexed by category. It also has the callback used by YARA when a rule is processed. I can not move much more than that for the row creation code because the structures used in the event callback are slightly different. Include a better error message. The errors are still printed by the compiler callback, but this will allow my future work to return a Status from the event initialization to print a useful message in summary. Make Subscriber init() return Status. Each EventSubscriber::init() now returns a Status. If the init() fails for any reason the EventSubscriber is still stored but the failure is tracked. EventSubscribers now have a state member, which represents the current state of the subscriber. The current supported states are: uninitialized, running, paused, failed. Currently the only meaningful ones are running and failed, but I put paused in there as a forward-looking feature. Subscriptions now have a subscriber_name member. This is used in EventPublisherPlugin::fire() as a lookup to get the EventSubscriber and check the state. If the EventSubscriber is not running the event will not fire. Only the EventSubscribers on OS X are using this. I'll do the Linux implementation next. Chase the init() changes to Linux. This brings the Linux YARA table in line with the OS X one. Require a EventSubscriberID when creating a subscription. Now that Subscriptions are "tied" to EventSubscribers you must create a Subscription with the name of the Subscriber it is for. This is because when the event fires the list of Subscriptions is walked and the name is used to lookup the EventSubscriber and make sure it is in the running state. Fix various tests. Some tests would fire an event with only a Subscription, which is no longer a valid thing to do. For these tests an EventSubscription is created and registered in the EventFactory. When Subscriptions are created pass the name of the EventSubscriber to them. In some cases where no event is ever fired it is fine to pass a bogus name. Fix inotify tests. Move a test down so the class is defined and make sure to create an EventSubscriber and use it properly. Add support for yara to provision.sh. Right now this grabs yara 3.3.0 and applies the patch to fix min() and max(), which is commit fc4696c8b725be1ac099d340359c8d550d116041 in the yara repo. This has been tested under Ubuntu 14.04 only. Remove NOMINMAX. This is no longer necessary after the patch was backported to 3.3.0. Revert "Add support for yara to provision.sh." This reverts commit a8bd371498c0979f070adeff23d05571882ac3f1. Use vendored YARA code in third-party. This switches to using the YARA code contained in third-party, including the patch to fix min/max macros. Fix mismerge. Remove unused function after merge. Well, soon to be unused as soon as I fix up the Linux YARA table. ;) Chase config changes. Make the Linux YARA table use ConfigDataInstance along with files() and yaraFiles().
2015-03-10 13:22:16 +00:00
])
attributes(event_subscriber=True)
Major YARA refactor and enhancements 1. Rename yara_matches to yara_events. 2. Add support for Config::getParser(). - This returns a ConfigPluginRef, which is the ConfigParser for the given key. - Being able to get the parser is useful because the YARAConfigParserPlugin uses it to store the compiled rules as an attribute. 3. Finish rename and use ConfigParserPlugin. - Finish the table rename to yara_events. - Use the new ConfigParserPlugin interface to parse the YARA configuration. The file_paths and signatures are stored in the ConfigParserPlugin named "yara" under the key "yara". The rules are compiled and stored as a private attribute of the same ConfigParserPlugin object. Here is an example config using this new structure: { // Description of the YARA feature. "yara": { "signatures": { // Each key is an arbitrary group name to give the signatures listed "sig_group_1": [ "/Users/wxs/foo.sig", "/Users/wxs//bar.sig" ], "sig_group_2": [ "/Users/wxs/baz.sig" ] }, "file_paths": { // Each key is a key from file_paths // The value is a list of signature groups to run when an event fires // These will be watched for and scanned when the event framework // fire off an event to yara_events table "system_binaries": [ "sig_group_1" ], "tmp": [ "sig_group_1", "sig_group_2" ] } }, // Paths to watch for filesystem events "file_paths": { "system_binaries": [ "/usr/bin/%", "/usr/sbin/%" ], "tmp": [ "/Users/wxs/tmp/%%" ] } } - Currently the signature file must be an absolute path. 3. Move common YARA code to yara_utils. - In preparation for the yara table (different from yara_events) I'm moving the common YARA code into a separate place which is shared between the two tables. 4. Add yara table. - This allows you to do things like: ```sql select * from yara where path="/bin/ls" and sigfile="/tmp/foo.sig"; select * from yara where path="/bin/ls" and sig_group="sig_group_1"; ``` - The latter will use the signature grouping from the config. 5. Check for keys not existing.
2015-04-17 20:03:43 +00:00
implementation("yara@yara_events::genTable")
Reference in New Issue Copy Permalink