Pre-aggregation cache implementation for numeric monitoring plugins.
For the most of monitoring data some aggregations are going to be applied on the user side. To do analytics you rarely need to see all points on the graph. It means for such paths (unique keys for points subsets) particular points means not much. And to reduce a disk usage and a network traffic some pre-aggreagation could be applied on osquery side.
This PR is implementation of such pre-aggreagtion.
It based of PR #4626
Just an interface and simple implementation dumping points to file on disk.
And I add also few monitoring records to some places of osquery code as an example.
Brief
Just an interface and simple implementation dumping points to file on disk.
And I add also few monitoring records to some places of osquery code as an example.
Motivation
osquery can monitor system health. But at some point we need to monitor the condition of osquery itself. Vast majority of interesting parameters can be represented by
numbers. How many queries it runs, how long does each query takes, what is the performance hit of each query, how long was last downtime and so on and so far. For obviou
s reason it hard to measure most of this parameters by external instrument. And it is almost impossible to evaluate it on production. But we can do it from inside of osquery.
What this PR is for
The systems like graphite or RRDtool can store and plot time-series data for us. We just have to
be able to feed data to it. We can create different plugins to be able to send data to different instruments. And we need some proper internal interface to all potential plugins. This PR is attempt to create generic interface.
Interface description
The most systems accept data as sequences of 2-dimensional points. One of the dimensions is value, the other is time. Each particular sequence has unique key, to be distinguished from the others.
Data descriptions for carbon. I have used this three parameters as an attributes of one monitoring point.
To send one point from some particular place in the code you just need to call the function record from namespace monitoring declared in the file include/osquery/num eric_monitoring.h with 3 arguments (path, value, time). Where path is the unique key of sequence; value is some interesting value to watch; time is the time of the point (can be omitted, current system time is the default vaule).
To create an error human readable message should be provided among other argmunts.
Which is good to better understanding what happend by log records.
To make it more informative user in most cases should put in those message some data (numbers, strings etc.).
This operator will help us to avoid using verbose constructions like boost::format or std::ostringstream or something similar to format a proper error message.
We will be able just to "stream" in a created error any "printable" variables from the context.
Additionaly we will be able to use "fancy" tools for streams like boost::io::quoted or std::hex to format messages.
Example:
```c++
createError(SystemErorr::NoSuchFile, "Could not read pidfile: ")
<< boost::io::quoted(pidfile_path)
<< " " << read_status.toString();
```
* adding bash_session logging
* adding genShellHistoryFromBashSessions
updated to include new function for adding bash_sessions
* adding genShellHistoryFromBashSessions and tests and header
updated test use canonical for filepath
updated to include new function for adding bash_sessions
* Connecte expected.h and error.h tests to build system
% ./build/darwin/osquery/osquery_tests --gtest_filter='Expected*'
Note: Google Test filter = Expected*
[==========] Running 0 tests from 0 test cases.
[==========] 0 tests from 0 test cases ran. (0 ms total)
[ PASSED ] 0 tests.
% ./build/darwin/osquery/osquery_tests --gtest_filter='Expected*'
Note: Google Test filter = Expected*
[==========] Running 2 tests from 2 test cases.
[----------] Global test environment set-up.
[----------] 1 test from ExpectedValueTest
[ RUN ] ExpectedValueTest.initialization
[ OK ] ExpectedValueTest.initialization (0 ms)
[----------] 1 test from ExpectedValueTest (0 ms total)
[----------] 1 test from ExpectedPointerTest
[ RUN ] ExpectedPointerTest.initialization
[ OK ] ExpectedPointerTest.initialization (0 ms)
[----------] 1 test from ExpectedPointerTest (0 ms total)
[----------] Global test environment tear-down
[==========] 2 tests from 2 test cases ran. (0 ms total)
[ PASSED ] 2 tests.
* Check error message and name of error enum separatelly
Different compilers produce different names for C++ classes
Consider this PR as a cosmetic one.
Creating Status class object in the code is not so clear. It is not so obvious that defatult costructed Status is success. Also it is not obvious that status with zero code is success and non-zero is failure.
To fix it I created 2 static methods to make construction of some particular status clear to reader.
* Use assert to check code in Status::failure in debut mode
* Rename success_code constant to kSuccessCode
according to style guide
If the shell history file does not contain a timestamps for the lines
osquery will miss the time in rows and will show an confusing error
about attempt to convert empty string to INTEGER.
```
% head -n 3 ~/.zsh_history
ls
cd source
ls
```
```
osquery> select * from shell_history limit 1;
I0621 11:56:37.804193 2629124992 virtual_table.cpp:292] Error casting time () to INTEGER
+------------+------+---------+-------------------------------+
| uid | time | command | history_file |
+------------+------+---------+-------------------------------+
| 1868255265 | | exit | /home/akindyakov/.zsh_history |
+------------+------+---------+-------------------------------+
```
So, default value for the time in shell history can solve the problem.
