Numeric monitoring system concept (#4626)

Just an interface and simple implementation dumping points to file on disk.
And I add also few monitoring records to some places of osquery code as an example.

Brief
Just an interface and simple implementation dumping points to file on disk.
And I add also few monitoring records to some places of osquery code as an example.

Motivation
osquery can monitor system health. But at some point we need to monitor the condition of osquery itself. Vast majority of interesting parameters can be represented by
numbers. How many queries it runs, how long does each query takes, what is the performance hit of each query, how long was last downtime and so on and so far. For obviou
s reason it hard to measure most of this parameters by external instrument. And it is almost impossible to evaluate it on production. But we can do it from inside of osquery.

What this PR is for
The systems like graphite or RRDtool can store and plot time-series data for us. We just have to
be able to feed data to it. We can create different plugins to be able to send data to different instruments. And we need some proper internal interface to all potential plugins. This PR is attempt to create generic interface.

Interface description
The most systems accept data as sequences of 2-dimensional points. One of the dimensions is value, the other is time. Each particular sequence has unique key, to be distinguished from the others.
Data descriptions for carbon. I have used this three parameters as an attributes of one monitoring point.

To send one point from some particular place in the code you just need to call the function record from namespace monitoring declared in the file include/osquery/num eric_monitoring.h with 3 arguments (path, value, time). Where path is the unique key of sequence; value is some interesting value to watch; time is the time of the point (can be omitted, current system time is the default vaule).

include/osquery/numeric_monitoring.h

@@ -0,0 +1,82 @@
+/**
+ *  Copyright (c) 2014-present, Facebook, Inc.
+ *  All rights reserved.
+ *
+ *  This source code is licensed under both the Apache 2.0 license (found in the
+ *  LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ *  in the COPYING file in the root directory of this source tree).
+ *  You may select, at your option, one of the above-listed licenses.
+ */
+
+#pragma once
+
+#include <chrono>
+#include <string>
+
+#include <osquery/core/conversions.h>
+#include <osquery/expected.h>
+
+namespace osquery {
+
+namespace monitoring {
+
+/**
+ * Types for clock and time point in monitoring plugin
+ */
+using Clock = std::chrono::system_clock;
+using TimePoint = Clock::time_point;
+
+using ValueType = long long int;
+
+enum class PreAggregationType {
+  None,
+  Sum,
+  Min,
+  Max,
+  // not existing PreAggregationType, upper limit definition
+  InvalidTypeUpperLimit,
+};
+
+/**
+ * @brief Record new point to numeric monitoring system.
+ *
+ * @param path A unique key in monitoring system. If you need to add some common
+ * prefix for all osquery points do it in the plugin code.
+ * @param value A numeric value of new point.
+ * @param pre_aggregation An preliminary aggregation type for this particular
+ * path @see PreAggregationType. It allows some numeric monitoring plugins
+ * pre-aggregate points before send it.
+ * @param time_point A time of new point, in vast majority of cases it is just
+ * a now time (default time).
+ *
+ * Common way to use it:
+ * @code{.cpp}
+ * monitoring::record("watched.parameter.path",
+ *                    10.42,
+ *                    monitoring::PreAggregationType::Sum);
+ * @endcode
+ */
+void record(const std::string& path,
+            ValueType value,
+            PreAggregationType pre_aggregation = PreAggregationType::None,
+            TimePoint time_point = Clock::now());
+
+} // namespace monitoring
+
+/**
+ * Generic to convert PreAggregationType to string
+ */
+template <typename ToType>
+typename std::enable_if<std::is_same<std::string, ToType>::value, ToType>::type
+to(const monitoring::PreAggregationType& from);
+
+/**
+ * Generic to parse PreAggregationType from string
+ */
+template <typename ToType>
+typename std::enable_if<
+    std::is_same<monitoring::PreAggregationType, ToType>::value,
+    Expected<ToType, ConversionError>>::type
+tryTo(const std::string& from);
+
+} // namespace osquery

osquery/CMakeLists.txt

@@ -97,6 +97,8 @@ include(registry/CMakeLists.txt)
 include(sql/CMakeLists.txt)
 include(remote/CMakeLists.txt)
 include(utils/CMakeLists.txt)
+include(numeric_monitoring/CMakeLists.txt)
+
 if(NOT SKIP_CARVER)
   include(carver/CMakeLists.txt)
 endif()

osquery/core/init.cpp

@@ -44,6 +44,7 @@
 #include <osquery/filesystem.h>
 #include <osquery/flags.h>
 #include <osquery/logger.h>
+#include <osquery/numeric_monitoring/plugin_interface.h>
 #include <osquery/registry.h>
 #include <osquery/system.h>
 
@@ -176,6 +177,7 @@ namespace osquery {
 
 DECLARE_string(config_plugin);
 DECLARE_string(logger_plugin);
+DECLARE_string(numeric_monitoring_plugins);
 DECLARE_string(distributed_plugin);
 DECLARE_bool(config_check);
 DECLARE_bool(config_dump);
@@ -185,6 +187,7 @@ DECLARE_bool(disable_distributed);
 DECLARE_bool(disable_database);
 DECLARE_bool(disable_events);
 DECLARE_bool(disable_logging);
+DECLARE_bool(enable_numeric_monitoring);
 
 CLI_FLAG(bool, S, false, "Run as a shell process");
 CLI_FLAG(bool, D, false, "Run as a daemon process");
@@ -681,6 +684,11 @@ void Initializer::start() const {
     initActivePlugin("distributed", FLAGS_distributed_plugin);
   }
 
+  if (FLAGS_enable_numeric_monitoring) {
+    initActivePlugin(monitoring::registryName(),
+                     FLAGS_numeric_monitoring_plugins);
+  }
+
   // Start event threads.
   osquery::attachEvents();
   EventFactory::delay();

osquery/numeric_monitoring/CMakeLists.txt

@@ -0,0 +1,23 @@
+#  Copyright (c) 2014-present, Facebook, Inc.
+#  All rights reserved.
+#
+#  This source code is licensed under both the Apache 2.0 license (found in the
+#  LICENSE file in the root directory of this source tree) and the GPLv2 (found
+#  in the COPYING file in the root directory of this source tree).
+#  You may select, at your option, one of the above-listed licenses.
+
+target_sources(libosquery
+  PRIVATE
+    "${CMAKE_CURRENT_LIST_DIR}/numeric_monitoring.cpp"
+    "${CMAKE_CURRENT_LIST_DIR}/plugin_interface.cpp"
+)
+
+ADD_OSQUERY_LIBRARY_ADDITIONAL(
+  osquery_numeric_monitoring_plugins
+  "${CMAKE_CURRENT_LIST_DIR}/plugins/filesystem.cpp"
+)
+
+ADD_OSQUERY_TEST_ADDITIONAL(
+  "${CMAKE_CURRENT_LIST_DIR}/tests/numeric_monitoring_tests.cpp"
+  "${CMAKE_CURRENT_LIST_DIR}/plugins/tests/filesystem_tests.cpp"
+)

osquery/numeric_monitoring/numeric_monitoring.cpp

@@ -0,0 +1,115 @@
+/**
+ *  Copyright (c) 2014-present, Facebook, Inc.
+ *  All rights reserved.
+ *
+ *  This source code is licensed under both the Apache 2.0 license (found in the
+ *  LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ *  in the COPYING file in the root directory of this source tree).
+ *  You may select, at your option, one of the above-listed licenses.
+ */
+
+#include <unordered_map>
+
+#include <boost/format.hpp>
+
+#include <osquery/flags.h>
+#include <osquery/logger.h>
+#include <osquery/numeric_monitoring.h>
+#include <osquery/numeric_monitoring/plugin_interface.h>
+#include <osquery/registry_factory.h>
+
+namespace osquery {
+
+FLAG(bool,
+     enable_numeric_monitoring,
+     false,
+     "Enable numeric monitoring system");
+FLAG(string,
+     numeric_monitoring_plugins,
+     "filesystem",
+     "Coma separated numeric monitoring plugins names");
+
+namespace {
+
+using monitoring::PreAggregationType;
+
+template <typename KeyType, typename ValueType>
+inline auto reverseMap(const std::unordered_map<KeyType, ValueType>& straight) {
+  auto reversed = std::unordered_map<ValueType, KeyType>{};
+  for (const auto& item : straight) {
+    reversed.emplace(item.second, item.first);
+  }
+  return reversed;
+}
+
+const auto& getAggregationTypeToStringTable() {
+  const auto static table = std::unordered_map<PreAggregationType, std::string>{
+      {PreAggregationType::None, "none"},
+      {PreAggregationType::Sum, "sum"},
+      {PreAggregationType::Min, "min"},
+      {PreAggregationType::Max, "max"},
+  };
+  return table;
+}
+
+const auto& getStringToAggregationTypeTable() {
+  const auto static table = reverseMap(getAggregationTypeToStringTable());
+  return table;
+}
+
+} // namespace
+
+template <>
+std::string to<std::string>(const monitoring::PreAggregationType& from) {
+  auto it = getAggregationTypeToStringTable().find(from);
+  if (it == getAggregationTypeToStringTable().end()) {
+    LOG(ERROR) << "Unknown PreAggregationType "
+               << static_cast<std::underlying_type<PreAggregationType>::type>(
+                      from)
+               << " could not be converted to the string";
+    return "";
+  }
+  return it->second;
+}
+
+template <>
+Expected<monitoring::PreAggregationType, ConversionError>
+tryTo<monitoring::PreAggregationType>(const std::string& from) {
+  auto it = getStringToAggregationTypeTable().find(from);
+  if (it == getStringToAggregationTypeTable().end()) {
+    return createError(
+        ConversionError::InvalidArgument,
+        boost::str(
+            boost::format(
+                "Wrong string representation of `PreAggregationType`: \"%s\"") %
+            from));
+  }
+  return it->second;
+}
+
+namespace monitoring {
+
+void record(const std::string& path,
+            ValueType value,
+            PreAggregationType pre_aggregation,
+            TimePoint time_point) {
+  if (!FLAGS_enable_numeric_monitoring) {
+    return;
+  }
+  auto status = Registry::call(
+      registryName(),
+      FLAGS_numeric_monitoring_plugins,
+      {
+          {recordKeys().path, path},
+          {recordKeys().value, std::to_string(value)},
+          {recordKeys().timestamp,
+           std::to_string(time_point.time_since_epoch().count())},
+          {recordKeys().pre_aggregation, to<std::string>(pre_aggregation)},
+      });
+  if (!status.ok()) {
+    LOG(ERROR) << "Failed to send numeric monitoring record: " << status.what();
+  }
+}
+
+} // namespace monitoring
+} // namespace osquery

osquery/numeric_monitoring/plugin_interface.cpp

@@ -0,0 +1,52 @@
+/**
+ *  Copyright (c) 2014-present, Facebook, Inc.
+ *  All rights reserved.
+ *
+ *  This source code is licensed under both the Apache 2.0 license (found in the
+ *  LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ *  in the COPYING file in the root directory of this source tree).
+ *  You may select, at your option, one of the above-listed licenses.
+ */
+
+#include <osquery/numeric_monitoring/plugin_interface.h>
+#include <osquery/plugin.h>
+#include <osquery/registry_factory.h>
+
+namespace osquery {
+
+CREATE_REGISTRY(NumericMonitoringPlugin, monitoring::registryName());
+
+namespace monitoring {
+
+const char* registryName() {
+  static const auto name = "numeric_monitoring";
+  return name;
+}
+
+namespace {
+
+RecordKeys createRecordKeys() {
+  auto keys = RecordKeys{};
+  keys.path = "path";
+  keys.value = "value";
+  keys.timestamp = "timestamp";
+  keys.pre_aggregation = "pre_aggregation";
+  return keys;
+};
+
+} // namespace
+
+const RecordKeys& recordKeys() {
+  static const auto keys = createRecordKeys();
+  return keys;
+}
+
+} // namespace monitoring
+
+Status NumericMonitoringPlugin::call(const PluginRequest& request,
+                                     PluginResponse& response) {
+  // should be implemented in plugins
+  return Status();
+}
+
+} // namespace osquery

osquery/numeric_monitoring/plugin_interface.h

@@ -0,0 +1,50 @@
+/**
+ *  Copyright (c) 2014-present, Facebook, Inc.
+ *  All rights reserved.
+ *
+ *  This source code is licensed under both the Apache 2.0 license (found in the
+ *  LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ *  in the COPYING file in the root directory of this source tree).
+ *  You may select, at your option, one of the above-listed licenses.
+ */
+
+#pragma once
+
+#include <chrono>
+#include <string>
+
+#include <osquery/core.h>
+#include <osquery/expected.h>
+#include <osquery/plugin.h>
+#include <osquery/query.h>
+
+#include <osquery/numeric_monitoring.h>
+
+namespace osquery {
+
+namespace monitoring {
+
+struct RecordKeys {
+  std::string path;
+  std::string value;
+  std::string timestamp;
+  std::string pre_aggregation;
+};
+
+const RecordKeys& recordKeys();
+
+const char* registryName();
+
+} // namespace monitoring
+
+/**
+ * @brief Interface class for numeric monitoring system plugins.
+ * e.g. @see NumericMonitoringFilesystemPlugin from
+ * osquery/numeric_monitoring/plugins/filesystem.h
+ */
+class NumericMonitoringPlugin : public Plugin {
+ public:
+  Status call(const PluginRequest& request, PluginResponse& response) override;
+};
+
+} // namespace osquery

osquery/numeric_monitoring/plugins/filesystem.cpp

@@ -0,0 +1,96 @@
+/**
+ *  Copyright (c) 2014-present, Facebook, Inc.
+ *  All rights reserved.
+ *
+ *  This source code is licensed under both the Apache 2.0 license (found in the
+ *  LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ *  in the COPYING file in the root directory of this source tree).
+ *  You may select, at your option, one of the above-listed licenses.
+ */
+
+#include <boost/format.hpp>
+
+#include <osquery/logger.h>
+#include <osquery/registry_factory.h>
+
+#include <osquery/numeric_monitoring/plugins/filesystem.h>
+
+namespace fs = boost::filesystem;
+
+namespace osquery {
+
+FLAG(string,
+     numeric_monitoring_filesystem_path,
+     OSQUERY_LOG_HOME "numeric_monitoring.log",
+     "File to dump numeric monitoring records one per line. "
+     "The format of the line is <PATH><TAB><VALUE><TAB><TIMESTAMP>.");
+
+REGISTER(NumericMonitoringFilesystemPlugin,
+         monitoring::registryName(),
+         "filesystem");
+
+NumericMonitoringFilesystemPlugin::NumericMonitoringFilesystemPlugin()
+    : NumericMonitoringFilesystemPlugin(
+          FLAGS_numeric_monitoring_filesystem_path) {}
+
+NumericMonitoringFilesystemPlugin::NumericMonitoringFilesystemPlugin(
+    fs::path log_file_path
+)
+  : line_format_{
+      monitoring::recordKeys().path,
+      monitoring::recordKeys().value,
+      monitoring::recordKeys().timestamp,
+  }
+  , separator_{'\t'}
+  , log_file_path_(
+      std::move(log_file_path)
+  )
+{
+}
+
+Status NumericMonitoringFilesystemPlugin::formTheLine(
+    std::string& line, const PluginRequest& request) const {
+  for (const auto& key : line_format_) {
+    auto it = request.find(key);
+    if (it == request.end()) {
+      return Status(1, "Missing mandatory request field " + key);
+    }
+    line.append(it->second).push_back(separator_);
+  }
+  // remove last separator
+  line.pop_back();
+  return Status();
+}
+
+Status NumericMonitoringFilesystemPlugin::call(const PluginRequest& request,
+                                               PluginResponse& response) {
+  if (!isSetUp()) {
+    return Status(1, "NumericMonitoringFilesystemPlugin is not set up");
+  }
+  auto line = std::string{};
+  auto status = formTheLine(line, request);
+  if (status.ok()) {
+    std::unique_lock<std::mutex> lock(output_file_mutex_);
+    output_file_stream_ << line << std::endl;
+  }
+  return status;
+}
+
+Status NumericMonitoringFilesystemPlugin::setUp() {
+  output_file_stream_.open(log_file_path_.native(),
+                           std::ios::out | std::ios::app | std::ios::binary);
+  if (!output_file_stream_.is_open()) {
+    return Status(
+        1,
+        boost::str(boost::format(
+                       "Could not open file %s for numeric monitoring logs") %
+                   log_file_path_));
+  }
+  return Status();
+}
+
+bool NumericMonitoringFilesystemPlugin::isSetUp() const {
+  return output_file_stream_.is_open();
+}
+
+} // namespace osquery

osquery/numeric_monitoring/plugins/filesystem.h

@@ -0,0 +1,47 @@
+/**
+ *  Copyright (c) 2014-present, Facebook, Inc.
+ *  All rights reserved.
+ *
+ *  This source code is licensed under both the Apache 2.0 license (found in the
+ *  LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ *  in the COPYING file in the root directory of this source tree).
+ *  You may select, at your option, one of the above-listed licenses.
+ */
+
+#pragma once
+
+#include <fstream>
+#include <functional>
+#include <string>
+#include <vector>
+
+#include <boost/filesystem.hpp>
+
+#include <osquery/numeric_monitoring/plugin_interface.h>
+
+namespace osquery {
+
+class NumericMonitoringFilesystemPlugin : public NumericMonitoringPlugin {
+ public:
+  explicit NumericMonitoringFilesystemPlugin();
+  explicit NumericMonitoringFilesystemPlugin(
+      boost::filesystem::path log_file_path);
+
+  Status call(const PluginRequest& request, PluginResponse& response) override;
+
+  Status setUp() override;
+
+  bool isSetUp() const;
+
+ private:
+  Status formTheLine(std::string& line, const PluginRequest& request) const;
+
+ private:
+  const std::vector<std::string> line_format_;
+  const std::string::value_type separator_;
+  const boost::filesystem::path log_file_path_;
+  std::ofstream output_file_stream_;
+  std::mutex output_file_mutex_;
+};
+
+} // namespace osquery

osquery/numeric_monitoring/plugins/tests/filesystem_tests.cpp

@@ -0,0 +1,92 @@
+/**
+ *  Copyright (c) 2014-present, Facebook, Inc.
+ *  All rights reserved.
+ *
+ *  This source code is licensed under both the Apache 2.0 license (found in the
+ *  LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ *  in the COPYING file in the root directory of this source tree).
+ *  You may select, at your option, one of the above-listed licenses.
+ */
+
+#include <gtest/gtest.h>
+
+#include <boost/filesystem.hpp>
+
+#include <osquery/core/conversions.h>
+#include <osquery/filesystem.h>
+#include <osquery/logger.h>
+#include <osquery/registry_factory.h>
+
+#include <osquery/tests/test_util.h>
+
+#include "osquery/numeric_monitoring/plugins/filesystem.h"
+
+namespace fs = boost::filesystem;
+
+namespace osquery {
+
+DECLARE_string(numeric_monitoring_filesystem_path);
+
+class NumericMonitoringFilesystemPluginTests : public testing::Test {
+ public:
+  void SetUp() override {
+    old_flag_value_ = FLAGS_numeric_monitoring_filesystem_path;
+  }
+
+  void TearDown() override {
+    FLAGS_numeric_monitoring_filesystem_path = old_flag_value_;
+  }
+
+ protected:
+  std::string old_flag_value_;
+};
+
+TEST_F(NumericMonitoringFilesystemPluginTests, simple_workflow) {
+  const auto log_path =
+      fs::temp_directory_path() /
+      fs::unique_path(
+          "osquery.numeric_monitoring_filesystem_plugin_test.%%%%-%%%%%%.log");
+  FLAGS_numeric_monitoring_filesystem_path = log_path.string();
+  {
+    NumericMonitoringFilesystemPlugin plugin{};
+    ASSERT_FALSE(plugin.isSetUp());
+    ASSERT_TRUE(plugin.setUp().ok());
+    ASSERT_TRUE(plugin.isSetUp());
+    const auto path =
+        R"path(p !"#$%&'()*+,-./0127:;<=>?@0AZ[\]^_`4bcyz{|}~)path";
+    const auto value = double{1.5};
+    const auto tm = int{1051};
+    const auto request = PluginRequest{
+        {monitoring::recordKeys().path, path},
+        {monitoring::recordKeys().value, std::to_string(value)},
+        {monitoring::recordKeys().timestamp, std::to_string(tm)},
+    };
+    auto response = PluginResponse{};
+    EXPECT_TRUE(plugin.call(request, response).ok());
+    EXPECT_TRUE(plugin.call(request, response).ok());
+
+    ASSERT_TRUE(fs::exists(log_path));
+    ASSERT_FALSE(fs::is_empty(log_path));
+
+    auto fin =
+        std::ifstream(log_path.native(), std::ios::in | std::ios::binary);
+    auto line = std::string{};
+
+    std::getline(fin, line);
+    auto first_line = split(line, "\t");
+    EXPECT_EQ(first_line.size(), 3);
+    EXPECT_EQ(first_line[0], path);
+    EXPECT_NEAR(std::stod(first_line[1]), value, 0.00001);
+    EXPECT_EQ(std::stol(first_line[2]), tm);
+
+    std::getline(fin, line);
+    auto second_line = split(line, "\t");
+    EXPECT_EQ(second_line.size(), 3);
+    EXPECT_EQ(second_line[0], path);
+    EXPECT_NEAR(std::stod(second_line[1]), value, 0.00001);
+    EXPECT_EQ(std::stol(second_line[2]), tm);
+  }
+  fs::remove(log_path);
+}
+
+} // namespace osquery

osquery/numeric_monitoring/tests/numeric_monitoring_tests.cpp

@@ -0,0 +1,58 @@
+/**
+ *  Copyright (c) 2014-present, Facebook, Inc.
+ *  All rights reserved.
+ *
+ *  This source code is licensed under both the Apache 2.0 license (found in the
+ *  LICENSE file in the root directory of this source tree) and the GPLv2 (found
+ *  in the COPYING file in the root directory of this source tree).
+ *  You may select, at your option, one of the above-listed licenses.
+ */
+
+#include <limits>
+
+#include <gtest/gtest.h>
+
+#include <boost/filesystem.hpp>
+
+#include <osquery/core/conversions.h>
+#include <osquery/filesystem.h>
+#include <osquery/logger.h>
+#include <osquery/registry_factory.h>
+
+#include <osquery/tests/test_util.h>
+
+#include "include/osquery/numeric_monitoring.h"
+
+namespace osquery {
+
+namespace {
+void testAggrTypeToStringAndBack(const monitoring::PreAggregationType& aggrType,
+                                 const std::string& aggrTypeStrRepr) {
+  auto str = to<std::string>(aggrType);
+  EXPECT_EQ(str, aggrTypeStrRepr);
+
+  auto bRet = tryTo<monitoring::PreAggregationType>(str);
+  EXPECT_FALSE(bRet.isError());
+  EXPECT_EQ(bRet.get(), aggrType);
+}
+} // namespace
+
+GTEST_TEST(NumericMonitoringTests, PreAggregationTypeToStringAndBack) {
+  testAggrTypeToStringAndBack(monitoring::PreAggregationType::None, "none");
+  testAggrTypeToStringAndBack(monitoring::PreAggregationType::Sum, "sum");
+  testAggrTypeToStringAndBack(monitoring::PreAggregationType::Min, "min");
+  testAggrTypeToStringAndBack(monitoring::PreAggregationType::Max, "max");
+}
+
+GTEST_TEST(NumericMonitoringTests, PreAggregationTypeToStringRecall) {
+  // let's make sure we have string representation for every PreAggregationType
+  using UnderType = std::underlying_type<monitoring::PreAggregationType>::type;
+  const auto upper_limit = static_cast<UnderType>(
+      monitoring::PreAggregationType::InvalidTypeUpperLimit);
+  for (auto i = UnderType{}; i < upper_limit; ++i) {
+    auto e = static_cast<monitoring::PreAggregationType>(i);
+    EXPECT_FALSE(to<std::string>(e).empty());
+  }
+}
+
+} // namespace osquery