* to<Status>(Expected) implementation
To make a transition from Status to Expected a bit easier.
For transition time there must be a lot of conversions from Expected to Status and back.
Conversion from Status to Expected is not simple, for many reasons.
Conversion from Expected to Status is trivial, but a bit verbose:
```c++
if (exp.isError()) {
return Status::failure(
exp.getError().getFullMessage()
);
} else {
return Status::success();
}
```
I'd suggest using more laconic, clear and explicit way to convert.
- removed boost_to_std_shared_ptr and std_to_boost_shared_ptr as far as we don't use boost::shared_ptr in osquery anymore.
- removed replaceAll
- and removed useless includes
`tryTo<>` generics for string to integer conversion
The first approach to substitute all `safeStrto*` conversions to `tryTo<>` generics.
Thare are some advantages in using templates here:
- Destination value type explicitly takes a part in call syntax.
- You could use it other template code
Also I have removed `safeStrtoi` from the code as an example of usage.
Pre-aggregation cache implementation for numeric monitoring plugins.
For the most of monitoring data some aggregations are going to be applied on the user side. To do analytics you rarely need to see all points on the graph. It means for such paths (unique keys for points subsets) particular points means not much. And to reduce a disk usage and a network traffic some pre-aggreagation could be applied on osquery side.
This PR is implementation of such pre-aggreagtion.
It based of PR #4626
Just an interface and simple implementation dumping points to file on disk.
And I add also few monitoring records to some places of osquery code as an example.
Brief
Just an interface and simple implementation dumping points to file on disk.
And I add also few monitoring records to some places of osquery code as an example.
Motivation
osquery can monitor system health. But at some point we need to monitor the condition of osquery itself. Vast majority of interesting parameters can be represented by
numbers. How many queries it runs, how long does each query takes, what is the performance hit of each query, how long was last downtime and so on and so far. For obviou
s reason it hard to measure most of this parameters by external instrument. And it is almost impossible to evaluate it on production. But we can do it from inside of osquery.
What this PR is for
The systems like graphite or RRDtool can store and plot time-series data for us. We just have to
be able to feed data to it. We can create different plugins to be able to send data to different instruments. And we need some proper internal interface to all potential plugins. This PR is attempt to create generic interface.
Interface description
The most systems accept data as sequences of 2-dimensional points. One of the dimensions is value, the other is time. Each particular sequence has unique key, to be distinguished from the others.
Data descriptions for carbon. I have used this three parameters as an attributes of one monitoring point.
To send one point from some particular place in the code you just need to call the function record from namespace monitoring declared in the file include/osquery/num eric_monitoring.h with 3 arguments (path, value, time). Where path is the unique key of sequence; value is some interesting value to watch; time is the time of the point (can be omitted, current system time is the default vaule).
To create an error human readable message should be provided among other argmunts.
Which is good to better understanding what happend by log records.
To make it more informative user in most cases should put in those message some data (numbers, strings etc.).
This operator will help us to avoid using verbose constructions like boost::format or std::ostringstream or something similar to format a proper error message.
We will be able just to "stream" in a created error any "printable" variables from the context.
Additionaly we will be able to use "fancy" tools for streams like boost::io::quoted or std::hex to format messages.
Example:
```c++
createError(SystemErorr::NoSuchFile, "Could not read pidfile: ")
<< boost::io::quoted(pidfile_path)
<< " " << read_status.toString();
```
* adding bash_session logging
* adding genShellHistoryFromBashSessions
updated to include new function for adding bash_sessions
* adding genShellHistoryFromBashSessions and tests and header
updated test use canonical for filepath
updated to include new function for adding bash_sessions
* Connecte expected.h and error.h tests to build system
% ./build/darwin/osquery/osquery_tests --gtest_filter='Expected*'
Note: Google Test filter = Expected*
[==========] Running 0 tests from 0 test cases.
[==========] 0 tests from 0 test cases ran. (0 ms total)
[ PASSED ] 0 tests.
% ./build/darwin/osquery/osquery_tests --gtest_filter='Expected*'
Note: Google Test filter = Expected*
[==========] Running 2 tests from 2 test cases.
[----------] Global test environment set-up.
[----------] 1 test from ExpectedValueTest
[ RUN ] ExpectedValueTest.initialization
[ OK ] ExpectedValueTest.initialization (0 ms)
[----------] 1 test from ExpectedValueTest (0 ms total)
[----------] 1 test from ExpectedPointerTest
[ RUN ] ExpectedPointerTest.initialization
[ OK ] ExpectedPointerTest.initialization (0 ms)
[----------] 1 test from ExpectedPointerTest (0 ms total)
[----------] Global test environment tear-down
[==========] 2 tests from 2 test cases ran. (0 ms total)
[ PASSED ] 2 tests.
* Check error message and name of error enum separatelly
Different compilers produce different names for C++ classes
Consider this PR as a cosmetic one.
Creating Status class object in the code is not so clear. It is not so obvious that defatult costructed Status is success. Also it is not obvious that status with zero code is success and non-zero is failure.
To fix it I created 2 static methods to make construction of some particular status clear to reader.
* Use assert to check code in Status::failure in debut mode
* Rename success_code constant to kSuccessCode
according to style guide
If the shell history file does not contain a timestamps for the lines
osquery will miss the time in rows and will show an confusing error
about attempt to convert empty string to INTEGER.
```
% head -n 3 ~/.zsh_history
ls
cd source
ls
```
```
osquery> select * from shell_history limit 1;
I0621 11:56:37.804193 2629124992 virtual_table.cpp:292] Error casting time () to INTEGER
+------------+------+---------+-------------------------------+
| uid | time | command | history_file |
+------------+------+---------+-------------------------------+
| 1868255265 | | exit | /home/akindyakov/.zsh_history |
+------------+------+---------+-------------------------------+
```
So, default value for the time in shell history can solve the problem.
osquery itself does not care about unicode validity in table columns,
just takes it "as is". It definetely makes sense, because it could be broken.
But thrift extensions interface for python do it.
If, for instance, shell history contains broken unicode test `python_test_example_queries`
will fail.
```bash
% sed -n '5277p' < ~/.zsh_history | xxd -b [146]
00000000: 11000011 10000011 10111111 01101100 01110011 00001010 ...ls.
```
* dispatcher race conditions
dispatcher had 2 race condition.
In joinServices it was accessing service_threads_ with different lock(join_lock). However, if by that time new service was added baad things would happen :) .
Also dispatcher was accessing services_.size() without the lock. ( If by that time service was removed or joined bad things would happen)
* InterruptableRunnable RunnerInterruptPoint redesign
There were several inefficiencies in the old version of RunnerInterruptPoint and InterruptableRunnable.
1) RunnerInterruptPoint was throwing the exception when interrupted, however, the exception was always ignored.
2) InterruptableRunnable used the read-write lock, however only write lock was used.
3) InterruptableRunnable InterruptableRunnable, stored almost similar variable stop_, interrupted_.
4) std::atomic<bool> interrupted_ was used with locks, even though it was accessed by default safest access mode memory_order_seq_cst. So no additional cache invalidation was needed.
5) InterruptableRunnable contained code(in method interrupted() and variables bypass_check_, checked) just for testing. Which was slowing down method interrupted().
6) Some more confusing things. notify_all was not needed, as only one thread could be waiting for the conditional variable. RunnerInterruptPoint:: pause(void) looks ambiguous and that's why was not used anywhere.
I resolved all these problems by merging InterruptableRunnable and RunnerInterruptPoint into the InterruptableRunnable.
1) No use of the exception.
2) 4) Simple mutex, which is only used for pauseMilli. InterruptableRunnable::interrupted and InterruptableRunnable::interrupt function lock-free.
3) Single variable interrupted_.
5) Made InterruptableRunnable::interrupt virtual. Tests override interrupt to make things testable.
6) change to notify_one and removed pause without the specific time.
split(string,string,size_t) contained bug, it was joining on every delimiter, which would result to unusual outcome. However, test could not detect this problem as delim.size() was 1. It turned out, that this split is not used anywhere having delim.size() > 1, so completely fixing bug by changing signature of the method to split(string,char,size_t)
* Ephemeral native support for int
Ephemeral supports int without serialization.
Dump get and put pushed from database.cpp to the database plugin.
Basic support for rocksDB and sqlite
* status messages
* style and compilation problems fix
* const int& to int for database get
* const int& to int for database put
* more specific exception type of boost variant get
* eohemeral improvements
code repetition reduced
map operations reduced
error message improved
#everything, next in the following.
* safeStrtoi implementation and use
* code formatting
* atoi accepts base int type
* error message add to status
* compilation error and format correction
* rocksDB simplify
unnecessary private variables moved to functions.
Find method replaced by std::find
* reverting back some changes
* code move revert
* Basic db int type support
Added API and tests for int types in the database abstraction.
For now, it's just the wrapper over the string type. In the following commits, will utilize specific database capabilites for additional performance.
* source formatting corrected
* hardened tests
* Simple solution to reduce drift time in loop of scheduled queries #4301
based on measuring time of loop step and reduce sleep time on it.
* Change schedule time drift compensation policy
Sleep less time or do not sleep at all if there is non-zero accumulated drift.
Also new flag [schedule_max_drift] was added to make it configurable.
* Add test to check time drift accumulation and fix up code according review comments
