valitydev/osquery-1
14
0
Fork 0
mirror of https://github.com/valitydev/osquery-1.git synced 2024-11-07 09:58:54 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Implement YARA table.

Browse Source 
Currently only for OS X, will port to others soon.

Also need to add tests.

Remove old comment and add loading message.

Implement YARA table for Linux.

Use mask properly.

Use the various masks to specify the kinds of events we are interested
in. This removes the need to do the dirty "DELETED" check when the event
fires.

Make getYARAFiles return a const map.

Switch to LOG(WARNING) and emit error number.

Add vim .swp files to .gitignore.

Add yara_utils.(c|h).

Start to condense common code between the Linux and Darwin YARA tables
into a yara_utils.h. Right now it includes a function to compile rules
and store the results back in the map, indexed by category. It also has
the callback used by YARA when a rule is processed. I can not move much
more than that for the row creation code because the structures used in
the event callback are slightly different.

Include a better error message.

The errors are still printed by the compiler callback, but this will
allow my future work to return a Status from the event initialization to
print a useful message in summary.

Make Subscriber init() return Status.

Each EventSubscriber::init() now returns a Status. If the init() fails
for any reason the EventSubscriber is still stored but the failure is
tracked.

EventSubscribers now have a state member, which represents the current
state of the subscriber. The current supported states are:
uninitialized, running, paused, failed. Currently the only meaningful
ones are running and failed, but I put paused in there as a
forward-looking feature.

Subscriptions now have a subscriber_name member. This is used in
EventPublisherPlugin::fire() as a lookup to get the EventSubscriber and
check the state. If the EventSubscriber is not running the event will
not fire.

Only the EventSubscribers on OS X are using this. I'll do the Linux
implementation next.

Chase the init() changes to Linux.

This brings the Linux YARA table in line with the OS X one.

Require a EventSubscriberID when creating a subscription.

Now that Subscriptions are "tied" to EventSubscribers you must create a
Subscription with the name of the Subscriber it is for. This is because
when the event fires the list of Subscriptions is walked and the name is
used to lookup the EventSubscriber and make sure it is in the running
state.

Fix various tests.

Some tests would fire an event with only a Subscription, which is no
longer a valid thing to do. For these tests an EventSubscription is
created and registered in the EventFactory.

When Subscriptions are created pass the name of the EventSubscriber to
them. In some cases where no event is ever fired it is fine to pass a
bogus name.

Fix inotify tests.

Move a test down so the class is defined and make sure to create an
EventSubscriber and use it properly.

Add support for yara to provision.sh.

Right now this grabs yara 3.3.0 and applies the patch to fix min() and max(),
which is commit fc4696c8b725be1ac099d340359c8d550d116041 in the yara repo.

This has been tested under Ubuntu 14.04 only.

Remove NOMINMAX.

This is no longer necessary after the patch was backported to 3.3.0.

Revert "Add support for yara to provision.sh."

This reverts commit a8bd371498c0979f070adeff23d05571882ac3f1.

Use vendored YARA code in third-party.

This switches to using the YARA code contained in third-party, including
the patch to fix min/max macros.

Fix mismerge.

Remove unused function after merge.

Well, soon to be unused as soon as I fix up the Linux YARA table. ;)

Chase config changes.

Make the Linux YARA table use ConfigDataInstance along with files() and
yaraFiles().
This commit is contained in:
Wesley Shields 2015-03-10 09:22:16 -04:00 committed by Teddy Reed
parent d19eef1c76
commit a9644d22c2
23 changed files with 698 additions and 106 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
.gitignore vendored
View File

@ -53,3 +53,4 @@ doxygen/latex
# Editors
*~
\#*\#
*.swp

40
CMake/FindYARA.cmake Normal file
View File

@ -0,0 +1,40 @@
include(FindPackageHandleStandardArgs)
if(POLICY CMP0054)
cmake_policy(SET CMP0054 NEW)
endif()
set(YARA_VERSION "3.3.0")
set(YARA_ROOT_DIR "${CMAKE_BINARY_DIR}/third-party/yara-${YARA_VERSION}")
set(YARA_SOURCE_DIR "${CMAKE_SOURCE_DIR}/third-party/yara-${YARA_VERSION}")
set(YARA_PATCH_01 "${CMAKE_SOURCE_DIR}/third-party/yara-${YARA_VERSION}/fc4696c8b725be1ac099d340359c8d550d116041.diff")
INCLUDE(ExternalProject)
ExternalProject_Add(
yara
SOURCE_DIR ${YARA_SOURCE_DIR}
INSTALL_DIR ${YARA_ROOT_DIR}
PATCH_COMMAND patch -p1 < ${YARA_PATCH_01}
CONFIGURE_COMMAND ${YARA_SOURCE_DIR}/configure
CC=/usr/bin/clang CXX=/usr/bin/clang++ --prefix=${YARA_ROOT_DIR}
BUILD_COMMAND make
BUILD_IN_SOURCE 1
INSTALL_COMMAND make install
LOG_CONFIGURE ON
LOG_INSTALL ON
LOG_BUILD ON
)
ExternalProject_Add_Step(
yara bootstrap
COMMAND sh ./bootstrap.sh
DEPENDERS configure
WORKING_DIRECTORY ${YARA_SOURCE_DIR}
)
set(YARA_INCLUDE_DIR "${YARA_ROOT_DIR}/include")
set(YARA_INCLUDE_DIRS ${YARA_INCLUDE_DIR})
set(YARA_LIBRARY "${YARA_ROOT_DIR}/lib/libyara.a")
set(YARA_LIBRARIES ${YARA_LIBRARY})

2
CMakeLists.txt
View File

@ -159,6 +159,7 @@ find_package(RocksDB REQUIRED)
find_package(Snappy REQUIRED)
find_package(Sqlite3 REQUIRED)
find_package(Thrift 0.9.1 REQUIRED)
find_package(YARA REQUIRED)
enable_testing()
@ -167,6 +168,7 @@ include(Packages)
include(Thrift)
include_directories("${GLOG_INCLUDE_DIRS}")
include_directories("${YARA_INCLUDE_DIRS}")
include_directories("${CMAKE_SOURCE_DIR}/include")
include_directories("${CMAKE_SOURCE_DIR}")
include_directories("/usr/local/include")

6
include/osquery/config.h
View File

@ -42,6 +42,7 @@ struct ConfigData {
std::map<std::string, ScheduledQuery> schedule;
std::map<std::string, std::string> options;
std::map<std::string, std::vector<std::string> > files;
std::map<std::string, std::vector<std::string> > yara;
pt::ptree all_data;
};
@ -182,6 +183,11 @@ class ConfigDataInstance {
return Config::getInstance().data_.files;
}
/// Helper accessor for Config::data_.yara.
const std::map<std::string, std::vector<std::string> >& yara() {
return Config::getInstance().data_.yara;
}
/// Helper accessor for Config::data_.all_data.
const pt::ptree& data() { return Config::getInstance().data_.all_data; }

36
include/osquery/events.h
View File

@ -133,6 +133,9 @@ extern const std::vector<size_t> kEventTimeLists;
*/
struct Subscription {
public:
// EventSubscriber name.
std::string subscriber_name;
/// An EventPublisher%-specific SubscriptionContext.
SubscriptionContextRef context;
/// An EventSubscription member EventCallback method.
@ -140,12 +143,18 @@ struct Subscription {
/// A pointer to possible extra data
void* user_data;
static SubscriptionRef create() { return std::make_shared<Subscription>(); }
static SubscriptionRef create(const EventSubscriberID name_id) {
auto subscription = std::make_shared<Subscription>();
subscription->subscriber_name = name_id;
return subscription;
}
static SubscriptionRef create(const SubscriptionContextRef& mc,
static SubscriptionRef create(const EventSubscriberID name_id,
const SubscriptionContextRef& mc,
EventCallback ec = 0,
void* user_data = nullptr) {
auto subscription = std::make_shared<Subscription>();
subscription->subscriber_name = name_id;
subscription->context = mc;
subscription->callback = ec;
subscription->user_data = user_data;
@ -611,6 +620,7 @@ class EventFactory {
* @return Was the SubscriptionContext appropriate for the EventPublisher.
*/
static Status addSubscription(EventPublisherID& type_id,
const EventSubscriberID name_id,
const SubscriptionContextRef& mc,
EventCallback cb = 0,
void* user_data = nullptr);
@ -701,6 +711,20 @@ class EventFactory {
std::vector<std::shared_ptr<boost::thread> > threads_;
};
/**
* EventSubscribers can be in various states. They are:
*
* Uninitialized: The default state, uninitialized.
* Running: Subscriber is ready for events.
* Paused: Subscriber was successfully initialized but not currently accepting
* events.
* Failed: Subscriber failed to initialize or is otherwise offline.
*/
#define EVENT_SUBSCRIBER_UNINITIALIZED 0
#define EVENT_SUBSCRIBER_RUNNING 1
#define EVENT_SUBSCRIBER_PAUSED 2
#define EVENT_SUBSCRIBER_FAILED 3
/**
* @brief An interface binding Subscriptions, event response, and table
*generation.
@ -719,6 +743,7 @@ class EventSubscriber : public EventSubscriberPlugin {
protected:
typedef typename PUB::SCRef SCRef;
typedef typename PUB::ECRef ECRef;
int state_ = EVENT_SUBSCRIBER_UNINITIALIZED;
public:
/**
@ -727,7 +752,7 @@ class EventSubscriber : public EventSubscriberPlugin {
* When the EventSubscriber%'s `init` method is called you are assured the
* EventPublisher has `setUp` and is ready to subscription for events.
*/
virtual void init() {}
virtual Status init() { return Status(0, "OK"); }
/// Helper function to call the publisher's templated subscription generator.
SCRef createSubscriptionContext() const {
@ -754,12 +779,15 @@ class EventSubscriber : public EventSubscriberPlugin {
// EventSubscriber and a single parameter placeholder (the EventContext).
auto cb = std::bind(base_entry, self, _1, _2);
// Add a subscription using the callable and SubscriptionContext.
EventFactory::addSubscription(type(), sc, cb, user_data);
EventFactory::addSubscription(type(), self->name(), sc, cb, user_data);
}
/// Helper EventPublisher string type accessor.
EventPublisherID type() const { return BaseEventPublisher::getType<PUB>(); }
int state() const { return state_; }
void state(int state) { state_ = state; }
private:
FRIEND_TEST(EventsTests, test_event_sub);
FRIEND_TEST(EventsTests, test_event_sub_subscribe);

1
osquery/CMakeLists.txt
View File

@ -18,6 +18,7 @@ set(OSQUERY_LIBS
${GFLAGS_LIBRARY}
${OPENSSL_CRYPTO_LIBRARY}
${OPENSSL_SSL_LIBRARY}
${YARA_LIBRARY}
readline
pthread

29
osquery/config/config.cpp
View File

@ -107,10 +107,7 @@ inline void mergeAdditional(const tree_node& node, ConfigData& conf) {
conf.all_data.add_child("additional_monitoring." + node.first, node.second);
// Support special merging of file paths.
if (node.first != "file_paths") {
return;
}
if (node.first == "file_paths") {
for (const auto& category : node.second) {
for (const auto& path : category.second) {
resolveFilePattern(path.second.data(),
@ -118,6 +115,20 @@ inline void mergeAdditional(const tree_node& node, ConfigData& conf) {
REC_LIST_FOLDERS | REC_EVENT_OPT);
}
}
} else if (node.first == "yara") {
for (const auto& category : node.second) {
// Todo: the yara category key must come after "file_paths".
if (conf.files.find(category.first) == conf.files.end()) {
continue;
}
for (const auto& file : category.second) {
conf.yara[category.first].push_back(
file.second.get_value<std::string>());
}
}
} else {
// Unknown additional monitoring key.
}
}
// inline void mergeScheduledQuery(const tree_node& node, ConfigData& conf) {
@ -176,12 +187,12 @@ void Config::mergeConfig(const std::string& source, ConfigData& conf) {
}
}
if (tree.count("options") > 0) {
for (const auto& option : tree.get_child("options")) {
mergeOption(option, conf);
Status Config::getMD5(std::string& hash_string) {
std::string config_string;
auto s = genConfig(config_string);
if (!s.ok()) {
return s;
}
}
}
Status Config::getMD5(std::string& hash_string) {
// Request an accessor to our own config, outside of an update.

65
osquery/events/darwin/fsevents_tests.cpp
View File

@ -104,7 +104,7 @@ TEST_F(FSEventsTests, test_fsevents_add_subscription_missing_path) {
auto mc = std::make_shared<FSEventsSubscriptionContext>();
mc->path = "/this/path/is/fake";
auto subscription = Subscription::create(mc);
auto subscription = Subscription::create("TestSubscriber", mc);
auto status = EventFactory::addSubscription("fsevents", subscription);
EXPECT_TRUE(status.ok());
EventFactory::deregisterEventPublisher("fsevents");
@ -118,7 +118,7 @@ TEST_F(FSEventsTests, test_fsevents_add_subscription_success) {
auto mc = std::make_shared<FSEventsSubscriptionContext>();
mc->path = "/";
auto subscription = Subscription::create(mc);
auto subscription = Subscription::create("TestSubscriber", mc);
auto status = EventFactory::addSubscription("fsevents", subscription);
EXPECT_TRUE(status.ok());
@ -129,7 +129,7 @@ TEST_F(FSEventsTests, test_fsevents_add_subscription_success) {
// A duplicate subscription will work.
auto mc_dup = std::make_shared<FSEventsSubscriptionContext>();
mc_dup->path = "/";
auto subscription_dup = Subscription::create(mc_dup);
auto subscription_dup = Subscription::create("TestSubscriber", mc_dup);
status = EventFactory::addSubscription("fsevents", subscription_dup);
EXPECT_TRUE(status.ok());
@ -139,36 +139,12 @@ TEST_F(FSEventsTests, test_fsevents_add_subscription_success) {
EventFactory::deregisterEventPublisher("fsevents");
}
TEST_F(FSEventsTests, test_fsevents_run) {
// Assume event type is registered.
event_pub_ = std::make_shared<FSEventsEventPublisher>();
EventFactory::registerEventPublisher(event_pub_);
// Create a subscriptioning context
auto mc = std::make_shared<FSEventsSubscriptionContext>();
mc->path = kRealTestPath;
EventFactory::addSubscription("fsevents", Subscription::create(mc));
// Create an event loop thread (similar to main)
boost::thread temp_thread(EventFactory::run, "fsevents");
EXPECT_TRUE(event_pub_->numEvents() == 0);
// Cause an fsevents event(s) by writing to the watched path.
CreateEvents();
// Wait for the thread's run loop to select.
WaitForEvents(kMaxEventLatency);
EXPECT_TRUE(event_pub_->numEvents() > 0);
EventFactory::end();
}
class TestFSEventsEventSubscriber
: public EventSubscriber<FSEventsEventPublisher> {
DECLARE_SUBSCRIBER("TestFSEventsEventSubscriber");
public:
void init() { callback_count_ = 0; }
Status init() { callback_count_ = 0; return Status(0, "OK"); }
Status SimpleCallback(const FSEventsEventContextRef& ec,
const void* user_data) {
callback_count_ += 1;
@ -210,13 +186,41 @@ class TestFSEventsEventSubscriber
std::vector<std::string> actions_;
};
TEST_F(FSEventsTests, test_fsevents_run) {
// Assume event type is registered.
event_pub_ = std::make_shared<FSEventsEventPublisher>();
EventFactory::registerEventPublisher(event_pub_);
// Create a subscriber.
auto sub = std::make_shared<TestFSEventsEventSubscriber>();
EventFactory::registerEventSubscriber(sub);
// Create a subscriptioning context
auto mc = std::make_shared<FSEventsSubscriptionContext>();
mc->path = kRealTestPath;
EventFactory::addSubscription("fsevents", Subscription::create("TestFSEventsEventSubscriber", mc));
// Create an event loop thread (similar to main)
boost::thread temp_thread(EventFactory::run, "fsevents");
EXPECT_TRUE(event_pub_->numEvents() == 0);
// Cause an fsevents event(s) by writing to the watched path.
CreateEvents();
// Wait for the thread's run loop to select.
WaitForEvents(kMaxEventLatency);
EXPECT_TRUE(event_pub_->numEvents() > 0);
EventFactory::end();
}
TEST_F(FSEventsTests, test_fsevents_fire_event) {
// Assume event type is registered.
StartEventLoop();
// Simulate registering an event subscriber.
auto sub = std::make_shared<TestFSEventsEventSubscriber>();
sub->init();
auto status = sub->init();
// Create a subscriptioning context, note the added Event to the symbol
auto sc = sub->GetSubscription(0);
@ -237,9 +241,10 @@ TEST_F(FSEventsTests, test_fsevents_event_action) {
// Simulate registering an event subscriber.
auto sub = std::make_shared<TestFSEventsEventSubscriber>();
sub->init();
auto status = sub->init();
auto sc = sub->GetSubscription(0);
EventFactory::registerEventSubscriber(sub);
sub->subscribe(&TestFSEventsEventSubscriber::Callback, sc, nullptr);
CreateEvents();
sub->WaitForEvents(kMaxEventLatency);

23
osquery/events/events.cpp
View File

@ -69,8 +69,11 @@ void EventPublisherPlugin::fire(const EventContextRef& ec, EventTime time) {
}
for (const auto& subscription : subscriptions_) {
auto es = EventFactory::getEventSubscriber(subscription->subscriber_name);
if (es->state() == EVENT_SUBSCRIBER_RUNNING) {
fireCallback(subscription, ec);
}
}
}
std::vector<std::string> EventSubscriberPlugin::getIndexes(EventTime start,
@ -479,22 +482,31 @@ Status EventFactory::registerEventSubscriber(const PluginRef& sub) {
}
if (specialized_sub == nullptr || specialized_sub.get() == nullptr) {
return Status(0, "Invalid subscriber");
return Status(1, "Invalid subscriber");
}
// Let the module initialize any Subscriptions.
specialized_sub->init();
auto status = specialized_sub->init();
auto& ef = EventFactory::getInstance();
ef.event_subs_[specialized_sub->name()] = specialized_sub;
// Set state of subscriber.
if (!status.ok()) {
specialized_sub->state(EVENT_SUBSCRIBER_FAILED);
return Status(1, status.getMessage());
} else {
specialized_sub->state(EVENT_SUBSCRIBER_RUNNING);
return Status(0, "OK");
}
}
Status EventFactory::addSubscription(EventPublisherID& type_id,
const EventSubscriberID name_id,
const SubscriptionContextRef& mc,
EventCallback cb,
void* user_data) {
auto subscription = Subscription::create(mc, cb, user_data);
auto subscription = Subscription::create(name_id, mc, cb, user_data);
return EventFactory::addSubscription(type_id, subscription);
}
@ -614,7 +626,10 @@ void attachEvents() {
const auto& subscribers = Registry::all("event_subscriber");
for (const auto& subscriber : subscribers) {
EventFactory::registerEventSubscriber(subscriber.second);
auto status = EventFactory::registerEventSubscriber(subscriber.second);
if (!status.ok()) {
LOG(ERROR) << "Error registering subscriber: " << status.getMessage();
}
}
}
}

12
osquery/events/events_tests.cpp
View File

@ -34,6 +34,7 @@ class EventsTests : public ::testing::Test {
// The most basic event publisher uses useless Subscription/Event.
class BasicEventPublisher
: public EventPublisher<SubscriptionContext, EventContext> {};
class AnotherBasicEventPublisher
: public EventPublisher<SubscriptionContext, EventContext> {};
@ -137,7 +138,7 @@ TEST_F(EventsTests, test_create_subscription) {
// Make sure a subscription cannot be added for a non-existent event type.
// Note: It normally would not make sense to create a blank subscription.
auto subscription = Subscription::create();
auto subscription = Subscription::create("FakeSubscriber");
auto status = EventFactory::addSubscription("FakePublisher", subscription);
EXPECT_FALSE(status.ok());
@ -156,7 +157,7 @@ TEST_F(EventsTests, test_multiple_subscriptions) {
auto pub = std::make_shared<BasicEventPublisher>();
EventFactory::registerEventPublisher(pub);
auto subscription = Subscription::create();
auto subscription = Subscription::create("subscriber");
status = EventFactory::addSubscription("publisher", subscription);
status = EventFactory::addSubscription("publisher", subscription);
@ -232,7 +233,7 @@ TEST_F(EventsTests, test_custom_subscription) {
sc->smallest = -1;
// Step 3, add the subscription to the event type
status = EventFactory::addSubscription("TestPublisher", sc);
status = EventFactory::addSubscription("TestPublisher", "TestSubscriber", sc);
EXPECT_TRUE(status.ok());
EXPECT_EQ(pub->numSubscriptions(), 1);
@ -355,7 +356,8 @@ TEST_F(EventsTests, test_fire_event) {
auto pub = std::make_shared<BasicEventPublisher>();
status = EventFactory::registerEventPublisher(pub);
auto subscription = Subscription::create();
auto sub = std::make_shared<FakeEventSubscriber>();
auto subscription = Subscription::create("FakeSubscriber");
subscription->callback = TestTheeCallback;
status = EventFactory::addSubscription("publisher", subscription);
@ -364,7 +366,7 @@ TEST_F(EventsTests, test_fire_event) {
pub->fire(ec, 0);
EXPECT_EQ(kBellHathTolled, 1);
auto second_subscription = Subscription::create();
auto second_subscription = Subscription::create("FakeSubscriber");
status = EventFactory::addSubscription("publisher", second_subscription);
// Now there are two subscriptions (one sans callback).

73
osquery/events/linux/inotify_tests.cpp
View File

@ -64,7 +64,7 @@ class INotifyTests : public testing::Test {
mc->path = path;
mc->mask = mask;
EventFactory::addSubscription("inotify", mc, ec);
EventFactory::addSubscription("inotify", "TestSubscriber", mc, ec);
}
bool WaitForEvents(int max, int num_events = 0) {
@ -126,7 +126,7 @@ TEST_F(INotifyTests, test_inotify_add_subscription_missing_path) {
auto mc = std::make_shared<INotifySubscriptionContext>();
mc->path = "/this/path/is/fake";
auto subscription = Subscription::create(mc);
auto subscription = Subscription::create("TestSubscriber", mc);
auto status = EventFactory::addSubscription("inotify", subscription);
EXPECT_TRUE(status.ok());
EventFactory::deregisterEventPublisher("inotify");
@ -140,48 +140,18 @@ TEST_F(INotifyTests, test_inotify_add_subscription_success) {
auto mc = std::make_shared<INotifySubscriptionContext>();
mc->path = "/";
auto subscription = Subscription::create(mc);
auto subscription = Subscription::create("TestSubscriber", mc);
auto status = EventFactory::addSubscription("inotify", subscription);
EXPECT_TRUE(status.ok());
EventFactory::deregisterEventPublisher("inotify");
}
TEST_F(INotifyTests, test_inotify_run) {
// Assume event type is registered.
event_pub_ = std::make_shared<INotifyEventPublisher>();
auto status = EventFactory::registerEventPublisher(event_pub_);
EXPECT_TRUE(status.ok());
// Create a temporary file to watch, open writeable
FILE* fd = fopen(kRealTestPath.c_str(), "w");
// Create a subscriptioning context
auto mc = std::make_shared<INotifySubscriptionContext>();
mc->path = kRealTestPath;
status = EventFactory::addSubscription("inotify", Subscription::create(mc));
EXPECT_TRUE(status.ok());
// Create an event loop thread (similar to main)
boost::thread temp_thread(EventFactory::run, "inotify");
EXPECT_TRUE(event_pub_->numEvents() == 0);
// Cause an inotify event by writing to the watched path.
fputs("inotify", fd);
fclose(fd);
// Wait for the thread's run loop to select.
WaitForEvents(kMaxEventLatency);
EXPECT_TRUE(event_pub_->numEvents() > 0);
EventFactory::end();
temp_thread.join();
}
class TestINotifyEventSubscriber
: public EventSubscriber<INotifyEventPublisher> {
DECLARE_SUBSCRIBER("TestINotifyEventSubscriber");
public:
void init() { callback_count_ = 0; }
Status init() { callback_count_ = 0; return Status(0, "OK"); }
Status SimpleCallback(const INotifyEventContextRef& ec,
const void* user_data) {
callback_count_ += 1;
@ -227,6 +197,41 @@ class TestINotifyEventSubscriber
std::vector<std::string> actions_;
};
TEST_F(INotifyTests, test_inotify_run) {
// Assume event type is registered.
event_pub_ = std::make_shared<INotifyEventPublisher>();
auto status = EventFactory::registerEventPublisher(event_pub_);
EXPECT_TRUE(status.ok());
// Create a temporary file to watch, open writeable
FILE* fd = fopen(kRealTestPath.c_str(), "w");
// Create a subscriber.
auto sub = std::make_shared<TestINotifyEventSubscriber>();
EventFactory::registerEventSubscriber(sub);
// Create a subscriptioning context
auto mc = std::make_shared<INotifySubscriptionContext>();
mc->path = kRealTestPath;
status = EventFactory::addSubscription("inotify", Subscription::create("TestINotifyEventSubscriber", mc));
EXPECT_TRUE(status.ok());
// Create an event loop thread (similar to main)
boost::thread temp_thread(EventFactory::run, "inotify");
EXPECT_TRUE(event_pub_->numEvents() == 0);
// Cause an inotify event by writing to the watched path.
fputs("inotify", fd);
fclose(fd);
// Wait for the thread's run loop to select.
WaitForEvents(kMaxEventLatency);
EXPECT_TRUE(event_pub_->numEvents() > 0);
EventFactory::end();
temp_thread.join();
}
TEST_F(INotifyTests, test_inotify_fire_event) {
// Assume event type is registered.
StartEventLoop();

4
osquery/tables/CMakeLists.txt
View File

@ -10,6 +10,8 @@ if(APPLE)
events/darwin/passwd_changes.cpp
events/darwin/file_changes.cpp
events/darwin/hardware_events.cpp
events/darwin/yara.cpp
events/yara_utils.cpp
networking/darwin/routes.cpp
system/darwin/acpi_tables.cpp
system/darwin/ad_config.cpp
@ -61,6 +63,8 @@ else()
events/linux/hardware_events.cpp
events/linux/passwd_changes.cpp
events/linux/file_changes.cpp
events/linux/yara.cpp
events/yara_utils.cpp
networking/linux/arp_cache.cpp
networking/linux/process_open_sockets.cpp
networking/linux/routes.cpp

10
osquery/tables/events/darwin/file_changes.cpp
View File

@ -32,7 +32,7 @@ class FileChangesEventSubscriber
DECLARE_SUBSCRIBER("file_changes");
public:
void init();
Status init();
/**
* @brief This exports a single Callback for INotifyEventPublisher events.
@ -54,9 +54,9 @@ class FileChangesEventSubscriber
*/
REGISTER(FileChangesEventSubscriber, "event_subscriber", "file_changes");
void FileChangesEventSubscriber::init() {
ConfigDataInstance config;
for (const auto& element_kv : config.files()) {
Status FileChangesEventSubscriber::init() {
const auto& file_map = Config::getWatchedFiles();
for (const auto& element_kv : file_map) {
for (const auto& file : element_kv.second) {
VLOG(1) << "Added listener to: " << file;
auto mc = createSubscriptionContext();
@ -65,6 +65,8 @@ void FileChangesEventSubscriber::init() {
(void*)(&element_kv.first));
}
}
return Status(0, "OK");
}
Status FileChangesEventSubscriber::Callback(const FSEventsEventContextRef& ec,

6
osquery/tables/events/darwin/hardware_events.cpp
View File

@ -24,19 +24,21 @@ class HardwareEventSubscriber : public EventSubscriber<IOKitHIDEventPublisher> {
DECLARE_SUBSCRIBER("hardware_events");
public:
void init();
Status init();
Status Callback(const IOKitHIDEventContextRef& ec, const void* user_data);
};
REGISTER(HardwareEventSubscriber, "event_subscriber", "hardware_events");
void HardwareEventSubscriber::init() {
Status HardwareEventSubscriber::init() {
auto subscription = createSubscriptionContext();
// We don't want hardware value changes.
subscription->values = false;
subscribe(&HardwareEventSubscriber::Callback, subscription, nullptr);
return Status(0, "OK");
}
Status HardwareEventSubscriber::Callback(const IOKitHIDEventContextRef& ec,

6
osquery/tables/events/darwin/passwd_changes.cpp
View File

@ -34,7 +34,7 @@ class PasswdChangesEventSubscriber
DECLARE_SUBSCRIBER("passwd_changes");
public:
void init();
Status init();
/**
* @brief This exports a single Callback for INotifyEventPublisher events.
@ -56,12 +56,14 @@ class PasswdChangesEventSubscriber
*/
REGISTER(PasswdChangesEventSubscriber, "event_subscriber", "passwd_changes");
void PasswdChangesEventSubscriber::init() {
Status PasswdChangesEventSubscriber::init() {
for (const auto& path : kDarwinPasswdPaths) {
auto mc = createSubscriptionContext();
mc->path = path;
subscribe(&PasswdChangesEventSubscriber::Callback, mc, nullptr);
}
return Status(0, "OK");
}
Status PasswdChangesEventSubscriber::Callback(const FSEventsEventContextRef& ec,

129
osquery/tables/events/darwin/yara.cpp Normal file
View File

@ -0,0 +1,129 @@
/*
* Copyright (c) 2015, Wesley Shields
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
*
*/
#include <map>
#include <string>
#include <osquery/config.h>
#include <osquery/logger.h>
#include <yara.h>
#include "osquery/events/darwin/fsevents.h"
#include "osquery/tables/events/yara_utils.h"
namespace osquery {
namespace tables {
/**
* @brief Track YARA matches to files.
*/
class YARAEventSubscriber : public EventSubscriber<FSEventsEventPublisher> {
DECLARE_SUBSCRIBER("yara");
public:
Status init();
private:
std::map<std::string, YR_RULES *> rules;
/**
* @brief This exports a single Callback for FSEventsEventPublisher events.
*
* @param ec The Callback type receives an EventContextRef substruct
* for the FSEventsEventPublisher declared in this EventSubscriber subclass.
*
* @return Status
*/
Status Callback(const FSEventsEventContextRef& ec, const void* user_data);
};
/**
* @brief Each EventSubscriber must register itself so the init method is
* called.
*
* This registers YARAEventSubscriber into the osquery EventSubscriber
* pseudo-plugin registry.
*/
REGISTER(YARAEventSubscriber, "event_subscriber", "yara");
Status YARAEventSubscriber::init() {
Status status;
int result = yr_initialize();
if (result != ERROR_SUCCESS) {
LOG(WARNING) << "Unable to initalize YARA (" << result << ").";
return Status(1, "Unable to initalize YARA.");
}
const auto& yara_map = Config::getYARAFiles();
const auto& file_map = Config::getWatchedFiles();
// yara_map has a key of the category and a vector of rule files to load.
// file_map has a key of the category and a vector of files to watch. Use
// yara_map to get the category and subscribe to each file in file_map
// with that category. Then load each YARA rule file from yara_map.
for (const auto& element : yara_map) {
// Subscribe to each file for the given key (category).
for (const auto& file : file_map.find(element.first)->second) {
VLOG(1) << "Added YARA listener to: " << file;
auto mc = createSubscriptionContext();
mc->path = file;
mc->mask = kFSEventStreamEventFlagItemCreated |
kFSEventStreamEventFlagItemModified;
subscribe(&YARAEventSubscriber::Callback, mc, (void*)(&element.first));
}
// Attempt to compile the rules for this category.
status = handleRuleFiles(element.first, element.second, &rules);
if (!status.ok()) {
VLOG(1) << "Error: " << status.getMessage();
return status;
}
}
return Status(0, "OK");
}
Status YARAEventSubscriber::Callback(const FSEventsEventContextRef& ec,
const void* user_data) {
Row r;
r["action"] = ec->action;
r["time"] = ec->time_string;
r["target_path"] = ec->path;
if (user_data != nullptr) {
r["category"] = *(std::string*)user_data;
} else {
r["category"] = "Undefined";
}
r["transaction_id"] = INTEGER(ec->fsevent_id);
// These are default values, to be updated in YARACallback.
r["count"] = INTEGER(0);
r["matches"] = std::string("");
int result = yr_rules_scan_file(rules[*(std::string*)user_data],
ec->path.c_str(),
SCAN_FLAGS_FAST_MODE,
YARACallback,
(void*) &r,
0);
if (result != ERROR_SUCCESS) {
return Status(1, "YARA error: " + std::to_string(result));
}
if (ec->action != "") {
add(r, ec->time);
}
return Status(0, "OK");
}
}
}

4
osquery/tables/events/linux/file_changes.cpp
View File

@ -32,7 +32,7 @@ class FileChangesEventSubscriber
DECLARE_SUBSCRIBER("file_changes");
public:
void init();
Status init();
/**
* @brief This exports a single Callback for INotifyEventPublisher events.
@ -67,6 +67,8 @@ void FileChangesEventSubscriber::init() {
(void*)(&element_kv.first));
}
}
return Status(0, "OK");
}
Status FileChangesEventSubscriber::Callback(const INotifyEventContextRef& ec,

5
osquery/tables/events/linux/hardware_events.cpp
View File

@ -27,18 +27,19 @@ class HardwareEventSubscriber : public EventSubscriber<UdevEventPublisher> {
DECLARE_SUBSCRIBER("hardware_events");
public:
void init();
Status init();
Status Callback(const UdevEventContextRef& ec, const void* user_data);
};
REGISTER(HardwareEventSubscriber, "event_subscriber", "hardware_events");
void HardwareEventSubscriber::init() {
Status HardwareEventSubscriber::init() {
auto subscription = createSubscriptionContext();
subscription->action = UDEV_EVENT_ACTION_ALL;
subscribe(&HardwareEventSubscriber::Callback, subscription, nullptr);
return Status(0, "OK");
}
Status HardwareEventSubscriber::Callback(const UdevEventContextRef& ec,

5
osquery/tables/events/linux/passwd_changes.cpp
View File

@ -30,7 +30,7 @@ class PasswdChangesEventSubscriber
DECLARE_SUBSCRIBER("passwd_changes");
public:
void init();
Status init();
/**
* @brief This exports a single Callback for INotifyEventPublisher events.
@ -52,11 +52,12 @@ class PasswdChangesEventSubscriber
*/
REGISTER(PasswdChangesEventSubscriber, "event_subscriber", "passwd_changes");
void PasswdChangesEventSubscriber::init() {
Status PasswdChangesEventSubscriber::init() {
auto mc = createSubscriptionContext();
mc->path = "/etc/passwd";
mc->mask = IN_ATTRIB | IN_MODIFY | IN_DELETE | IN_CREATE;
subscribe(&PasswdChangesEventSubscriber::Callback, mc, nullptr);
return Status(0, "OK");
}
Status PasswdChangesEventSubscriber::Callback(const INotifyEventContextRef& ec,

132
osquery/tables/events/linux/yara.cpp Normal file
View File

@ -0,0 +1,132 @@
/*
* Copyright (c) 2015, Wesley Shields
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
*
*/
#include <map>
#include <string>
#include <osquery/config.h>
#include <osquery/logger.h>
#include <yara.h>
#include "osquery/events/linux/inotify.h"
#include "osquery/tables/events/yara_utils.h"
namespace osquery {
namespace tables {
/**
* @brief Track YARA matches to files.
*/
class YARAEventSubscriber : public EventSubscriber<INotifyEventPublisher> {
DECLARE_SUBSCRIBER("yara");
public:
Status init();
private:
std::map<std::string, YR_RULES *> rules;
/**
* @brief This exports a single Callback for INotifyEventPublisher
* events.
*
* @param ec The Callback type receives an EventContextRef substruct
* for the INotifyEventPublisher declared in this EventSubscriber subclass.
*
* @return Status
*/
Status Callback(const INotifyEventContextRef& ec, const void* user_data);
};
/**
* @brief Each EventSubscriber must register itself so the init method is
* called.
*
* This registers YARAEventSubscriber into the osquery EventSubscriber
* pseudo-plugin registry.
*/
REGISTER(YARAEventSubscriber, "event_subscriber", "yara");
Status YARAEventSubscriber::init() {
Status status;
ConfigDataInstance config;
int result = yr_initialize();
if (result != ERROR_SUCCESS) {
LOG(WARNING) << "Unable to initalize YARA (" << result << ").";
return Status(1, "Unable to initalize YARA.");
}
const auto& yara_map = config.yaraFiles();
const auto& file_map = config.files();
// yara_map has a key of the category and a vector of rule files to load.
// file_map has a key of the category and a vector of files to watch. Use
// yara_map to get the category and subscribe to each file in file_map
// with that category. Then load each YARA rule file from yara_map.
for (const auto& element : yara_map) {
// Subscribe to each file for the given key (category).
for (const auto& file : file_map.find(element.first)->second) {
VLOG(1) << "Added YARA listener to: " << file;
auto mc = createSubscriptionContext();
mc->path = file;
mc->recursive = true;
mc->mask = IN_CREATE | IN_CLOSE_WRITE | IN_MODIFY;
subscribe(&YARAEventSubscriber::Callback, mc, (void*)(&element.first));
}
// Attempt to compile the rules for this category.
status = handleRuleFiles(element.first, element.second, &rules);
if (!status.ok()) {
VLOG(1) << "Error: " << status.getMessage();
return status;
}
}
return Status(0, "OK");
}
Status YARAEventSubscriber::Callback(const INotifyEventContextRef& ec,
const void* user_data) {
Row r;
r["action"] = ec->action;
r["time"] = ec->time_string;
r["target_path"] = ec->path;
if (user_data != nullptr) {
r["category"] = *(std::string*)user_data;
} else {
r["category"] = "Undefined";
}
r["transaction_id"] = INTEGER(ec->event->cookie);
// These are default values, to be updated in YARACallback.
r["count"] = INTEGER(0);
r["matches"] = std::string("");
int result = yr_rules_scan_file(rules[*(std::string*)user_data],
ec->path.c_str(),
SCAN_FLAGS_FAST_MODE,
YARACallback,
(void*) &r,
0);
if (result != ERROR_SUCCESS) {
VLOG(1) << "ERROR: " << std::to_string(result);
return Status(1, "YARA error: " + std::to_string(result));
}
if (ec->action != "") {
add(r, ec->time);
}
return Status(0, "OK");
}
}
}

151
osquery/tables/events/yara_utils.cpp Normal file
View File

@ -0,0 +1,151 @@
/*
* Copyright (c) 2015, Wesley Shields
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
*
*/
#include <osquery/tables.h>
#include <osquery/logger.h>
#include <yara.h>
#include "osquery/tables/events/yara_utils.h"
namespace osquery {
namespace tables {
/**
* The callback used when there are compilation problems in the rules.
*/
void YARACompilerCallback(int error_level,
const char* file_name,
int line_number,
const char* message,
void* user_data) {
if (error_level == YARA_ERROR_LEVEL_ERROR) {
VLOG(1) << file_name << "(" << line_number << "): error: " << message;
}
else {
VLOG(1) << file_name << "(" << line_number << "): warning: " << message;
}
}
Status handleRuleFiles(std::string category,
std::vector<std::string> rule_files,
std::map<std::string, YR_RULES *> *rules) {
int result;
YR_COMPILER *compiler = nullptr;
bool compiled = false;
result = yr_compiler_create(&compiler);
if (result != ERROR_SUCCESS) {
VLOG(1) << "Could not create compiler: " + std::to_string(result);
return Status(1, "Could not create compiler: " + std::to_string(result));
}
yr_compiler_set_callback(compiler, YARACompilerCallback, NULL);
for (const auto& rule : rule_files) {
YR_RULES *tmp_rules;
VLOG(1) << "Loading " << rule;
// First attempt to load the file, in case it is saved (pre-compiled)
// rules. Sadly there is no way to load multiple compiled rules in
// succession. This means that:
//
// saved1, saved2
//
// results in saved2 being the only file used.
//
// Also, mixing source and saved rules results in the saved rules being
// overridden by the combination of the source rules once compiled, e.g.:
//
// file1, saved1
//
// result in file1 being the only file used.
//
// If you want to use saved rule files you must have them all in a single
// file. This is easy to accomplish with yarac(1).
result = yr_rules_load(rule.c_str(), &tmp_rules);
if (result != ERROR_SUCCESS && result != ERROR_INVALID_FILE) {
yr_compiler_destroy(compiler);
return Status(1, "Error loading YARA rules: " + std::to_string(result));
} else if (result == ERROR_SUCCESS) {
// If there are already rules there, destroy them and put new ones in.
if (rules->count(category) > 0) {
yr_rules_destroy((*rules)[category]);
}
(*rules)[category] = tmp_rules;
} else {
compiled = true;
// Try to compile the rules.
FILE *rule_file = fopen(rule.c_str(), "r");
if (rule_file == nullptr) {
VLOG(1) << "Could not open file: " << rule;
yr_compiler_destroy(compiler);
return Status(1, "Could not open file: " + rule);
}
int errors = yr_compiler_add_file(compiler,
rule_file,
NULL,
rule.c_str());
fclose(rule_file);
rule_file = nullptr;
if (errors > 0) {
yr_compiler_destroy(compiler);
// Errors printed via callback.
return Status(1, "Compilation errors.");
}
}
}
if (compiled) {
// All the rules for this category have been compiled, save them in
// the map.
result = yr_compiler_get_rules(compiler, &((*rules)[category]));
if (result != ERROR_SUCCESS) {
VLOG(1) << "Insufficient memory to get rules.";
yr_compiler_destroy(compiler);
return Status(1, "Insufficient memory to get rules");
}
}
if (compiler != nullptr) {
yr_compiler_destroy(compiler);
compiler = nullptr;
}
return Status(0, "OK");
}
/**
* This is the YARA callback. Used to store matching rules in the row which is
* passed in as user_data.
*/
int YARACallback(int message, void *message_data, void *user_data) {
if (message == CALLBACK_MSG_RULE_MATCHING) {
Row *r = (Row *) user_data;
YR_RULE *rule = (YR_RULE *) message_data;
if ((*r)["matches"].length() > 0) {
(*r)["matches"] += "," + std::string(rule->identifier);
} else {
(*r)["matches"] = std::string(rule->identifier);
}
(*r)["count"] = INTEGER(std::stoi((*r)["count"]) + 1);
}
return CALLBACK_CONTINUE;
}
}
}

37
osquery/tables/events/yara_utils.h Normal file
View File

@ -0,0 +1,37 @@
/*
* Copyright (c) 2015, Wesley Shields
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
*
*/
#include <map>
#include <string>
#include <vector>
#include <yara.h>
#include <osquery/status.h>
namespace osquery {
namespace tables {
/**
* Common initilization function. Compile vector of rule_files and save result
* in rules map, indexed by category.
*/
Status handleRuleFiles(std::string category,
std::vector<std::string> rule_files,
std::map<std::string, YR_RULES *> *rules);
/**
* This is the callback for YARA.
*/
int YARACallback(int message, void *message_data, void *user_data);
}
}

13
osquery/tables/specs/x/yara.table Normal file
View File

@ -0,0 +1,13 @@
table_name("yara")
description("Track YARA matches for files specified in configuation data.")
schema([
Column("target_path", TEXT, "The path scanned"),
Column("category", TEXT, "The category of the file"),
Column("time", TEXT, "Time of the scan"),
Column("action", TEXT, "Change action (UPDATE, REMOVE, etc)"),
Column("transaction_id", BIGINT, "ID used during bulk update"),
Column("matches", TEXT, "List of YARA matches"),
Column("count", INTEGER, "Number of YARA matches"),
])
attributes(event_subscriber=True)
implementation("yara@yara::genTable")