osquery-1/osquery/tables/other/yara_utils.h

65 lines
1.8 KiB
C
Raw Normal View History

Major YARA refactor and enhancements 1. Rename yara_matches to yara_events. 2. Add support for Config::getParser(). - This returns a ConfigPluginRef, which is the ConfigParser for the given key. - Being able to get the parser is useful because the YARAConfigParserPlugin uses it to store the compiled rules as an attribute. 3. Finish rename and use ConfigParserPlugin. - Finish the table rename to yara_events. - Use the new ConfigParserPlugin interface to parse the YARA configuration. The file_paths and signatures are stored in the ConfigParserPlugin named "yara" under the key "yara". The rules are compiled and stored as a private attribute of the same ConfigParserPlugin object. Here is an example config using this new structure: { // Description of the YARA feature. "yara": { "signatures": { // Each key is an arbitrary group name to give the signatures listed "sig_group_1": [ "/Users/wxs/foo.sig", "/Users/wxs//bar.sig" ], "sig_group_2": [ "/Users/wxs/baz.sig" ] }, "file_paths": { // Each key is a key from file_paths // The value is a list of signature groups to run when an event fires // These will be watched for and scanned when the event framework // fire off an event to yara_events table "system_binaries": [ "sig_group_1" ], "tmp": [ "sig_group_1", "sig_group_2" ] } }, // Paths to watch for filesystem events "file_paths": { "system_binaries": [ "/usr/bin/%", "/usr/sbin/%" ], "tmp": [ "/Users/wxs/tmp/%%" ] } } - Currently the signature file must be an absolute path. 3. Move common YARA code to yara_utils. - In preparation for the yara table (different from yara_events) I'm moving the common YARA code into a separate place which is shared between the two tables. 4. Add yara table. - This allows you to do things like: ```sql select * from yara where path="/bin/ls" and sigfile="/tmp/foo.sig"; select * from yara where path="/bin/ls" and sig_group="sig_group_1"; ``` - The latter will use the signature grouping from the config. 5. Check for keys not existing.
2015-04-17 20:03:43 +00:00
/*
* Copyright (c) 2015, Welsey Shields
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
*
*/
#include <osquery/config.h>
#include <osquery/tables.h>
#ifdef CONCAT
#undef CONCAT
Major YARA refactor and enhancements 1. Rename yara_matches to yara_events. 2. Add support for Config::getParser(). - This returns a ConfigPluginRef, which is the ConfigParser for the given key. - Being able to get the parser is useful because the YARAConfigParserPlugin uses it to store the compiled rules as an attribute. 3. Finish rename and use ConfigParserPlugin. - Finish the table rename to yara_events. - Use the new ConfigParserPlugin interface to parse the YARA configuration. The file_paths and signatures are stored in the ConfigParserPlugin named "yara" under the key "yara". The rules are compiled and stored as a private attribute of the same ConfigParserPlugin object. Here is an example config using this new structure: { // Description of the YARA feature. "yara": { "signatures": { // Each key is an arbitrary group name to give the signatures listed "sig_group_1": [ "/Users/wxs/foo.sig", "/Users/wxs//bar.sig" ], "sig_group_2": [ "/Users/wxs/baz.sig" ] }, "file_paths": { // Each key is a key from file_paths // The value is a list of signature groups to run when an event fires // These will be watched for and scanned when the event framework // fire off an event to yara_events table "system_binaries": [ "sig_group_1" ], "tmp": [ "sig_group_1", "sig_group_2" ] } }, // Paths to watch for filesystem events "file_paths": { "system_binaries": [ "/usr/bin/%", "/usr/sbin/%" ], "tmp": [ "/Users/wxs/tmp/%%" ] } } - Currently the signature file must be an absolute path. 3. Move common YARA code to yara_utils. - In preparation for the yara table (different from yara_events) I'm moving the common YARA code into a separate place which is shared between the two tables. 4. Add yara table. - This allows you to do things like: ```sql select * from yara where path="/bin/ls" and sigfile="/tmp/foo.sig"; select * from yara where path="/bin/ls" and sig_group="sig_group_1"; ``` - The latter will use the signature grouping from the config. 5. Check for keys not existing.
2015-04-17 20:03:43 +00:00
#endif
#include <yara.h>
[fix #1390] query pack re-org This commit contains the features specified in #1390 as well as a refactoring of the general osquery configuration code. The API for the config plugins hasn't changed, although now there's a `genPack` method that config plugins can implement. If a plugin doesn't implement `genPack`, then the map<string, string> format cannot be used. The default config plugin, the filesystem plugin, now implements `genPack`, so existing query packs code will continue to work as it always has. Now many other config plugins can implement custom pack handling for what makes sense in their context. `genPacks` is not a pure virtual, so it doesn't have to be implemented in your plugin if you don't want to use it. Also, more importantly, all config plugins can use the standard inline pack format if they want to use query packs. Which is awesome. For more information, refer to #1390, the documentation and the doxygen comments included with this pull requests, as well as the following example config which is now supported, regardless of what config plugin you're using: ```json { "options": { "enable_monitor": "true" }, "packs": { "core_os_monitoring": { "version": "1.4.5", "discovery": [ "select pid from processes where name like '%osqueryd%';" ], "queries": { "kernel_modules": { "query": "SELECT name, size FROM kernel_modules;", "interval": 600 }, "system_controls": { "query": "SELECT * FROM system_controls;", "interval": 600, "snapshot": true, }, "usb_devices": { "query": "SELECT * FROM usb_devices;", "interval": 600 } } }, "osquery_internal_info": { "version": "1.4.5", "discovery": [ "select pid from processes where name like '%osqueryd%';" ], "queries": { "info": { "query": "select i.*, p.resident_size, p.user_time, p.system_time, time.minutes as counter from osquery_info i, processes p, time where p.pid = i.pid;", "interval": 60, "snapshot": true }, "registry": { "query": "SELECT * FROM osquery_registry;", "interval": 600, "snapshot": true }, "schedule": { "query": "select name, interval, executions, output_size, wall_time, (user_time/executions) as avg_user_time, (system_time/executions) as avg_system_time, average_memory from osquery_schedule;", "interval": 60, "snapshot": true } } } } } ``` The `osquery_packs` table was modified to remove the superfluous columns which could already have been found in `osquery_schedule`. Two more columns were added in their place, representing stats about pack's discovery query execution history. Notably, the internal API for the `osquery::Config` class has changed rather dramatically as apart of the refactoring. We think this is an improvement. While strictly adhering to the osquery config plugin interface will have avoided any compatibility errors, advanced users may notice compilation errors if they access config data directly. All internal users of the config have obviously been updated. Yet another reason to merge your code into mainline; we update it for you when we refactor!
2015-08-19 20:27:49 +00:00
namespace pt = boost::property_tree;
Major YARA refactor and enhancements 1. Rename yara_matches to yara_events. 2. Add support for Config::getParser(). - This returns a ConfigPluginRef, which is the ConfigParser for the given key. - Being able to get the parser is useful because the YARAConfigParserPlugin uses it to store the compiled rules as an attribute. 3. Finish rename and use ConfigParserPlugin. - Finish the table rename to yara_events. - Use the new ConfigParserPlugin interface to parse the YARA configuration. The file_paths and signatures are stored in the ConfigParserPlugin named "yara" under the key "yara". The rules are compiled and stored as a private attribute of the same ConfigParserPlugin object. Here is an example config using this new structure: { // Description of the YARA feature. "yara": { "signatures": { // Each key is an arbitrary group name to give the signatures listed "sig_group_1": [ "/Users/wxs/foo.sig", "/Users/wxs//bar.sig" ], "sig_group_2": [ "/Users/wxs/baz.sig" ] }, "file_paths": { // Each key is a key from file_paths // The value is a list of signature groups to run when an event fires // These will be watched for and scanned when the event framework // fire off an event to yara_events table "system_binaries": [ "sig_group_1" ], "tmp": [ "sig_group_1", "sig_group_2" ] } }, // Paths to watch for filesystem events "file_paths": { "system_binaries": [ "/usr/bin/%", "/usr/sbin/%" ], "tmp": [ "/Users/wxs/tmp/%%" ] } } - Currently the signature file must be an absolute path. 3. Move common YARA code to yara_utils. - In preparation for the yara table (different from yara_events) I'm moving the common YARA code into a separate place which is shared between the two tables. 4. Add yara table. - This allows you to do things like: ```sql select * from yara where path="/bin/ls" and sigfile="/tmp/foo.sig"; select * from yara where path="/bin/ls" and sig_group="sig_group_1"; ``` - The latter will use the signature grouping from the config. 5. Check for keys not existing.
2015-04-17 20:03:43 +00:00
namespace osquery {
void YARACompilerCallback(int error_level,
const char* file_name,
int line_number,
const char* message,
void* user_data);
2015-05-04 03:02:01 +00:00
Status compileSingleFile(const std::string& file, YR_RULES** rule);
Major YARA refactor and enhancements 1. Rename yara_matches to yara_events. 2. Add support for Config::getParser(). - This returns a ConfigPluginRef, which is the ConfigParser for the given key. - Being able to get the parser is useful because the YARAConfigParserPlugin uses it to store the compiled rules as an attribute. 3. Finish rename and use ConfigParserPlugin. - Finish the table rename to yara_events. - Use the new ConfigParserPlugin interface to parse the YARA configuration. The file_paths and signatures are stored in the ConfigParserPlugin named "yara" under the key "yara". The rules are compiled and stored as a private attribute of the same ConfigParserPlugin object. Here is an example config using this new structure: { // Description of the YARA feature. "yara": { "signatures": { // Each key is an arbitrary group name to give the signatures listed "sig_group_1": [ "/Users/wxs/foo.sig", "/Users/wxs//bar.sig" ], "sig_group_2": [ "/Users/wxs/baz.sig" ] }, "file_paths": { // Each key is a key from file_paths // The value is a list of signature groups to run when an event fires // These will be watched for and scanned when the event framework // fire off an event to yara_events table "system_binaries": [ "sig_group_1" ], "tmp": [ "sig_group_1", "sig_group_2" ] } }, // Paths to watch for filesystem events "file_paths": { "system_binaries": [ "/usr/bin/%", "/usr/sbin/%" ], "tmp": [ "/Users/wxs/tmp/%%" ] } } - Currently the signature file must be an absolute path. 3. Move common YARA code to yara_utils. - In preparation for the yara table (different from yara_events) I'm moving the common YARA code into a separate place which is shared between the two tables. 4. Add yara table. - This allows you to do things like: ```sql select * from yara where path="/bin/ls" and sigfile="/tmp/foo.sig"; select * from yara where path="/bin/ls" and sig_group="sig_group_1"; ``` - The latter will use the signature grouping from the config. 5. Check for keys not existing.
2015-04-17 20:03:43 +00:00
Status handleRuleFiles(const std::string& category,
const pt::ptree& rule_files,
std::map<std::string, YR_RULES*>& rules);
Major YARA refactor and enhancements 1. Rename yara_matches to yara_events. 2. Add support for Config::getParser(). - This returns a ConfigPluginRef, which is the ConfigParser for the given key. - Being able to get the parser is useful because the YARAConfigParserPlugin uses it to store the compiled rules as an attribute. 3. Finish rename and use ConfigParserPlugin. - Finish the table rename to yara_events. - Use the new ConfigParserPlugin interface to parse the YARA configuration. The file_paths and signatures are stored in the ConfigParserPlugin named "yara" under the key "yara". The rules are compiled and stored as a private attribute of the same ConfigParserPlugin object. Here is an example config using this new structure: { // Description of the YARA feature. "yara": { "signatures": { // Each key is an arbitrary group name to give the signatures listed "sig_group_1": [ "/Users/wxs/foo.sig", "/Users/wxs//bar.sig" ], "sig_group_2": [ "/Users/wxs/baz.sig" ] }, "file_paths": { // Each key is a key from file_paths // The value is a list of signature groups to run when an event fires // These will be watched for and scanned when the event framework // fire off an event to yara_events table "system_binaries": [ "sig_group_1" ], "tmp": [ "sig_group_1", "sig_group_2" ] } }, // Paths to watch for filesystem events "file_paths": { "system_binaries": [ "/usr/bin/%", "/usr/sbin/%" ], "tmp": [ "/Users/wxs/tmp/%%" ] } } - Currently the signature file must be an absolute path. 3. Move common YARA code to yara_utils. - In preparation for the yara table (different from yara_events) I'm moving the common YARA code into a separate place which is shared between the two tables. 4. Add yara table. - This allows you to do things like: ```sql select * from yara where path="/bin/ls" and sigfile="/tmp/foo.sig"; select * from yara where path="/bin/ls" and sig_group="sig_group_1"; ``` - The latter will use the signature grouping from the config. 5. Check for keys not existing.
2015-04-17 20:03:43 +00:00
int YARACallback(int message, void *message_data, void *user_data);
/**
* @brief A simple ConfigParserPlugin for a "yara" dictionary key.
*
* A straight forward ConfigParserPlugin that requests a single "yara" key.
* This stores a rather trivial "yara" data key. The accessor will be
* redundant since this is so simple:
*
* Pseudo-code:
* getParser("yara")->getKey("yara");
*/
class YARAConfigParserPlugin : public ConfigParserPlugin {
public:
/// Request a single "yara" top level key.
std::vector<std::string> keys() { return {"yara"}; }
// Retrieve compiled rules.
std::map<std::string, YR_RULES*>& rules() { return rules_; }
Major YARA refactor and enhancements 1. Rename yara_matches to yara_events. 2. Add support for Config::getParser(). - This returns a ConfigPluginRef, which is the ConfigParser for the given key. - Being able to get the parser is useful because the YARAConfigParserPlugin uses it to store the compiled rules as an attribute. 3. Finish rename and use ConfigParserPlugin. - Finish the table rename to yara_events. - Use the new ConfigParserPlugin interface to parse the YARA configuration. The file_paths and signatures are stored in the ConfigParserPlugin named "yara" under the key "yara". The rules are compiled and stored as a private attribute of the same ConfigParserPlugin object. Here is an example config using this new structure: { // Description of the YARA feature. "yara": { "signatures": { // Each key is an arbitrary group name to give the signatures listed "sig_group_1": [ "/Users/wxs/foo.sig", "/Users/wxs//bar.sig" ], "sig_group_2": [ "/Users/wxs/baz.sig" ] }, "file_paths": { // Each key is a key from file_paths // The value is a list of signature groups to run when an event fires // These will be watched for and scanned when the event framework // fire off an event to yara_events table "system_binaries": [ "sig_group_1" ], "tmp": [ "sig_group_1", "sig_group_2" ] } }, // Paths to watch for filesystem events "file_paths": { "system_binaries": [ "/usr/bin/%", "/usr/sbin/%" ], "tmp": [ "/Users/wxs/tmp/%%" ] } } - Currently the signature file must be an absolute path. 3. Move common YARA code to yara_utils. - In preparation for the yara table (different from yara_events) I'm moving the common YARA code into a separate place which is shared between the two tables. 4. Add yara table. - This allows you to do things like: ```sql select * from yara where path="/bin/ls" and sigfile="/tmp/foo.sig"; select * from yara where path="/bin/ls" and sig_group="sig_group_1"; ``` - The latter will use the signature grouping from the config. 5. Check for keys not existing.
2015-04-17 20:03:43 +00:00
Status setUp();
Major YARA refactor and enhancements 1. Rename yara_matches to yara_events. 2. Add support for Config::getParser(). - This returns a ConfigPluginRef, which is the ConfigParser for the given key. - Being able to get the parser is useful because the YARAConfigParserPlugin uses it to store the compiled rules as an attribute. 3. Finish rename and use ConfigParserPlugin. - Finish the table rename to yara_events. - Use the new ConfigParserPlugin interface to parse the YARA configuration. The file_paths and signatures are stored in the ConfigParserPlugin named "yara" under the key "yara". The rules are compiled and stored as a private attribute of the same ConfigParserPlugin object. Here is an example config using this new structure: { // Description of the YARA feature. "yara": { "signatures": { // Each key is an arbitrary group name to give the signatures listed "sig_group_1": [ "/Users/wxs/foo.sig", "/Users/wxs//bar.sig" ], "sig_group_2": [ "/Users/wxs/baz.sig" ] }, "file_paths": { // Each key is a key from file_paths // The value is a list of signature groups to run when an event fires // These will be watched for and scanned when the event framework // fire off an event to yara_events table "system_binaries": [ "sig_group_1" ], "tmp": [ "sig_group_1", "sig_group_2" ] } }, // Paths to watch for filesystem events "file_paths": { "system_binaries": [ "/usr/bin/%", "/usr/sbin/%" ], "tmp": [ "/Users/wxs/tmp/%%" ] } } - Currently the signature file must be an absolute path. 3. Move common YARA code to yara_utils. - In preparation for the yara table (different from yara_events) I'm moving the common YARA code into a separate place which is shared between the two tables. 4. Add yara table. - This allows you to do things like: ```sql select * from yara where path="/bin/ls" and sigfile="/tmp/foo.sig"; select * from yara where path="/bin/ls" and sig_group="sig_group_1"; ``` - The latter will use the signature grouping from the config. 5. Check for keys not existing.
2015-04-17 20:03:43 +00:00
private:
// Store compiled rules in a map (group => rules).
std::map<std::string, YR_RULES *> rules_;
/// Store the signatures and file_paths and compile the rules.
[fix #1390] query pack re-org This commit contains the features specified in #1390 as well as a refactoring of the general osquery configuration code. The API for the config plugins hasn't changed, although now there's a `genPack` method that config plugins can implement. If a plugin doesn't implement `genPack`, then the map<string, string> format cannot be used. The default config plugin, the filesystem plugin, now implements `genPack`, so existing query packs code will continue to work as it always has. Now many other config plugins can implement custom pack handling for what makes sense in their context. `genPacks` is not a pure virtual, so it doesn't have to be implemented in your plugin if you don't want to use it. Also, more importantly, all config plugins can use the standard inline pack format if they want to use query packs. Which is awesome. For more information, refer to #1390, the documentation and the doxygen comments included with this pull requests, as well as the following example config which is now supported, regardless of what config plugin you're using: ```json { "options": { "enable_monitor": "true" }, "packs": { "core_os_monitoring": { "version": "1.4.5", "discovery": [ "select pid from processes where name like '%osqueryd%';" ], "queries": { "kernel_modules": { "query": "SELECT name, size FROM kernel_modules;", "interval": 600 }, "system_controls": { "query": "SELECT * FROM system_controls;", "interval": 600, "snapshot": true, }, "usb_devices": { "query": "SELECT * FROM usb_devices;", "interval": 600 } } }, "osquery_internal_info": { "version": "1.4.5", "discovery": [ "select pid from processes where name like '%osqueryd%';" ], "queries": { "info": { "query": "select i.*, p.resident_size, p.user_time, p.system_time, time.minutes as counter from osquery_info i, processes p, time where p.pid = i.pid;", "interval": 60, "snapshot": true }, "registry": { "query": "SELECT * FROM osquery_registry;", "interval": 600, "snapshot": true }, "schedule": { "query": "select name, interval, executions, output_size, wall_time, (user_time/executions) as avg_user_time, (system_time/executions) as avg_system_time, average_memory from osquery_schedule;", "interval": 60, "snapshot": true } } } } } ``` The `osquery_packs` table was modified to remove the superfluous columns which could already have been found in `osquery_schedule`. Two more columns were added in their place, representing stats about pack's discovery query execution history. Notably, the internal API for the `osquery::Config` class has changed rather dramatically as apart of the refactoring. We think this is an improvement. While strictly adhering to the osquery config plugin interface will have avoided any compatibility errors, advanced users may notice compilation errors if they access config data directly. All internal users of the config have obviously been updated. Yet another reason to merge your code into mainline; we update it for you when we refactor!
2015-08-19 20:27:49 +00:00
Status update(const std::map<std::string, pt::ptree>& config);
Major YARA refactor and enhancements 1. Rename yara_matches to yara_events. 2. Add support for Config::getParser(). - This returns a ConfigPluginRef, which is the ConfigParser for the given key. - Being able to get the parser is useful because the YARAConfigParserPlugin uses it to store the compiled rules as an attribute. 3. Finish rename and use ConfigParserPlugin. - Finish the table rename to yara_events. - Use the new ConfigParserPlugin interface to parse the YARA configuration. The file_paths and signatures are stored in the ConfigParserPlugin named "yara" under the key "yara". The rules are compiled and stored as a private attribute of the same ConfigParserPlugin object. Here is an example config using this new structure: { // Description of the YARA feature. "yara": { "signatures": { // Each key is an arbitrary group name to give the signatures listed "sig_group_1": [ "/Users/wxs/foo.sig", "/Users/wxs//bar.sig" ], "sig_group_2": [ "/Users/wxs/baz.sig" ] }, "file_paths": { // Each key is a key from file_paths // The value is a list of signature groups to run when an event fires // These will be watched for and scanned when the event framework // fire off an event to yara_events table "system_binaries": [ "sig_group_1" ], "tmp": [ "sig_group_1", "sig_group_2" ] } }, // Paths to watch for filesystem events "file_paths": { "system_binaries": [ "/usr/bin/%", "/usr/sbin/%" ], "tmp": [ "/Users/wxs/tmp/%%" ] } } - Currently the signature file must be an absolute path. 3. Move common YARA code to yara_utils. - In preparation for the yara table (different from yara_events) I'm moving the common YARA code into a separate place which is shared between the two tables. 4. Add yara table. - This allows you to do things like: ```sql select * from yara where path="/bin/ls" and sigfile="/tmp/foo.sig"; select * from yara where path="/bin/ls" and sig_group="sig_group_1"; ``` - The latter will use the signature grouping from the config. 5. Check for keys not existing.
2015-04-17 20:03:43 +00:00
};
}