Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5545
This is a JSON serializing formatter for schemer. It represents C++ object as JSON object according to defined in C++ class schema. The implementation based on rapidjson library. It is very simple - just print all key:value pairs directly to rapidjson stream (it could be files stream or string stream). The second method of formatter converts C++ objects directly to JSON in string.
Two methods with the same name: `osquery::schemer::toJson`
Reviewed By: SAlexandru
Differential Revision: D14663996
fbshipit-source-id: 7a902b0777c9094faf39ee3111bbc09d8c3c7a55
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5544
This is a framework to declare a serialization and deserialization
schema for C++ classes. The schema can be used by different implementations
to represent C++ object as data-interchange format or to parse an object from
formatted representation.
It works without macro or any verbose transformation code. All you need to
do to be able to serialize and deserialize some C++ class is to define a
template static method `discloseSchema` in the class. Which describe all
members that have to be read by serializer and written by deserializer.
Everything else is a job of schemer formatters.
Support of specific types, including nested types support depends on the formatter implementation.
1. One place to describe class members instead of two different methods for serializer and deserializer.
2. One way do describe schema for many formatters (JSON, TOML, hasher, etc).
3. Schemer also sets the order of elements of class, therefore even binary formatters can use it, just ignoring the names.
4. It is simple - just one method, nothing more. Therefore there is zero dependencies. Everything is in formatters.
- JSON serializer/deserializer.
- Hasher - it helps to impelement transformation of osquery events in hash form for probabilistic filters.
Reviewed By: SAlexandru
Differential Revision: D14663949
fbshipit-source-id: effcc7e91a85514cee700a0d5484ab4d317ac199
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5550
This fixes a potential memory leak in keychain_items similar to D14567925. This leak was identified with the default options for OSS infer.
Reviewed By: guliashvili
Differential Revision: D14641455
fbshipit-source-id: 3fd776fe8a2af300435907ecd1d7746b3045066e
Summary: To clean it up, they are parts of the same experimental subsystem and shoul be close to each other in experimental directory.
Reviewed By: marekcirkos
Differential Revision: D14668951
fbshipit-source-id: 404be1f3c24699557edc0b9e5ada823ea04533f1
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5542
We replace deprecated calls to the Status class with newer ones. This will stop Lint from nagging me every time I open these files. There should be no change to functionality.
Specifically, we use a codemod on `.cpp` files to replace any instance of `return Status();` with `return Status::success();`.
Reviewed By: guliashvili
Differential Revision: D14652922
fbshipit-source-id: 243576195ed201d6b82fc942a785742c6b01cf83
Summary:
Iterating through a string is no longer allowed, and `range(0, len(_))` and `range(len(_))`
are equivalent. Switch to the shorter, more commonly used form
Reviewrs: #sentinel
Reviewed By: philipjameson
Differential Revision: D14657008
fbshipit-source-id: 1aabcbf168896bd0ee64b0d4eb17a72d6863aab2
Summary: Every time a differential query runs we save the results to the database. We were serializing those results with or without types based on the `FLAGS_log_numerics_as_numbers` flag. However we always collect results with proper types therefore, on subsequent executions of the same query, the new results (typed) were being compared with the results retrieved from the DB (untyped if FLAGS_log_numerics_as_numbers == false), causing all results to be "different" and therefore osquery to report all results as "added" (and eventually all old results as "removed").
Reviewed By: SAlexandru
Differential Revision: D14669476
fbshipit-source-id: 8abc68cbcac90c73bd92fc8d34572ba3ee2f2c75
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5543
This diff removes two vestigial functions from the osquery core. There are no references to these functions except for the unit test, which we also update.
Reviewed By: marekcirkos
Differential Revision: D14664005
fbshipit-source-id: 4e176014c803e5ec8cebdf0bd064874b483c8f76
Summary: setName was not used, other than the test, testing setName. Removing it to make class immutable
Reviewed By: jessek
Differential Revision: D14640959
fbshipit-source-id: d684939ed31e8ad275650847fadb2fb8c11b6fdc
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5538
This bug was found using the OSS version of infer with default options.
Reviewed By: guliashvili
Differential Revision: D14567925
fbshipit-source-id: 64272880464577207552e3436754b7f6d32ef4ec
Summary:
osquery returned status 1(error) always when osqueryd service install uninstall was requested by the flags.
Even if install/uninstall was successfull.
Add extra logging in case of success.
Exit with status 0 when successfully installed/uninstalled
Reviewed By: jessek
Differential Revision: D14622771
fbshipit-source-id: c0c7569a2b13434e2ad28ba92cba308024b044ec
Summary:
User could have requested install and uninstall of osqueryd service together. However it does not make sense and actually osquery would only install the service.
This additional check, will exit osquery in case when install and uninstall are requested together.
Reviewed By: jessek
Differential Revision: D14622634
fbshipit-source-id: b88dc519bd1ef38653eb1660712bbf2e75bad36f
Summary:
8 out of 234 tests in the integration tests have TEST_F(classname, sanity) others have TEST_F(classname, test_sanity).
Making them all the same.
Reviewed By: jessek
Differential Revision: D14627709
fbshipit-source-id: d3ea6569d59fd3885004bde9421a0f1feafdb36c
Summary:
Replaced fprint(stderr with std::cerr
Other than that, could not find any use of
```
```
Reviewed By: akindyakov, fmanco
Differential Revision: D14621228
fbshipit-source-id: e996b2507757dc6af710e7ab0e25fc8136bbb92e
Summary: Somehow, still looking for how, D14577520 broke communication between osqueryi/osqueryd -S and extension. Revert.
Reviewed By: akindyakov
Differential Revision: D14620885
fbshipit-source-id: 98392f6e1e5da1b0ea68ee54dc00b3bbcd687315
Summary: just switchin off some windows specific code on non-windows platforms
Reviewed By: guliashvili
Differential Revision: D14597685
fbshipit-source-id: 8b058ac1f9db8e4455e1b29e9e277d04468c2d64
Summary: Right now it blocks us, because build on macox doesn't work. Fix will take some time - lots of changes. But on the other hand value of such change in tests is nearly zero. So, let's just mute it.
Reviewed By: guliashvili
Differential Revision: D14597262
fbshipit-source-id: adaacc003f49647e255001bb84cc0e71273cd486
Summary: It requires some effort to fix warnings properly, let's just mute them for now.
Reviewed By: marekcirkos
Differential Revision: D14596617
fbshipit-source-id: 184937d15cc186afe045131d6d648cdbfb4ad574
Summary:
Working to offering thrift service validating queries. It will be used by query canary service and helldriver.
For that, I need to execute osqueryd -S with the given query. However, osqueryd -S (with extension) crashes while it exits. I can filter out crash output, but it means, I will not be able to catch if query really crashes the osqueryd.
I also don't want to really fix osquery watcher, as we have the watcher 2.0 as an intern project.
Fixing the current problem by stopping osquery to ignore FLAGS_disable_watchdog flag when extension is avaliable.
Reviewed By: marekcirkos
Differential Revision: D14577520
fbshipit-source-id: 588c31319b875b8af9d73da6d691f22bcfdb8fc9
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5532
Ok, this is overcomplicated code for signal handling. Besides untested magic
with signals it makes quite a lot of assumptions about program state when it is
called. For instance, it uses instance of Watcher and Dispatcher singleton
which can be already destroyed or corrupted at the that time.
This complexity cause rare unpredictable and hardly reproducible problems, that
lead to data losses, cross process deadlocks, crashes of osqueryd because of
crash an extensions and os on ans so far.
Couple of examples.
When extensions killed by signal it 2 time from 3 lead to crash of osqueryd. It
very well reproducible but quite not very well diagnosable.
Another example is std::terminate called at shutdown of extensions. Something
could not be destroyed correctly and call std::terminate, which is already bad
but sometimes reasonable. After it everything goes to signal handler which use
already destroyed Watcher and cause another std::terminate. It leads to
recursive signal handler call and eventually to stack overflow.
Solution.
I believe it could be fixed less radical, absolutely. It takes some time,
though. And because it will cost us a lot there is a question: do we really
need it here? I could not find a place in osquery code where it is important to
finish something before shutdown. Moreover it is pretty rare case when code
handle shutdown condition correctly. That makes me think that we don't need
signal handling at all or at least not like this.
Reviewed By: mkareta
Differential Revision: D14458287
fbshipit-source-id: daf8388323e4dacabeee3338d185577335503da3
Summary:
Most paths can be generated, so let's do that instead of keeping big dictionaries.
By now we're down to only a couple of variables that need to be defined, namely the base path for the bcfg files and the toolchains per platform.
Reviewed By: marekcirkos
Differential Revision: D14425056
fbshipit-source-id: 47c935972318602752f50a7b584175322ef27cb6
Summary: system.py will need to support different distros. Make it more generic to simplify that.
Reviewed By: marekcirkos
Differential Revision: D14282633
fbshipit-source-id: e333ac5f29ad8ff074b4c1132ee27c3bfdfda7d8
Summary: Before, we sent the user and kernel time in a separate time series. They are usefull separately, in order to measure which queries are syscall extensive. However, for measuring the total time we need them together. We have wall time, but user + kernel time is more precise as they measure resource usage of the thread.
Reviewed By: jessek
Differential Revision: D14502715
fbshipit-source-id: 7a03ec5b1d7b328e9657d94f752d33e7aa771e97
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5530
This was the behaviour pre-#5083 and is a more sane behaviour for a shell since we usually don't want to see all the verbosity.
Reviewed By: marekcirkos
Differential Revision: D14469709
fbshipit-source-id: e0235c3cfcd51d77053da48ae35cfe75678e77c3
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5531
Last diff in series to migrate to a new interface. More details in diff: D14405326 or PR #5523
There is a changes to remove old interface and a bit more migration (things was landed in a middle).
Reviewed By: guliashvili
Differential Revision: D14477768
fbshipit-source-id: 572791cb6929edf46d5a8bfe9416e03dc85eb96f
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5515
This diff augments the unit test which determines if a query should execute on this client. We add more test cases for different platforms, including the ones which are no longer officially supported, `centos` and `ubuntu`. These values remain equivalent to `linux`. That is, a query with either of those values specified for the platform will be executed on any Linux system.
Reviewed By: guliashvili
Differential Revision: D14430282
fbshipit-source-id: 64a47ded0ef769d927753b9d75a378b25a2391bd
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5528
by using config option `cxx.filepath_length_limited=true`. Because unfortunately there is very low limit for file path length on windows up to win10 (260 chars).
Reviewed By: KapJI
Differential Revision: D14460635
fbshipit-source-id: f63fc564766b49c2d4fb5f1c2bb7015592ab17e1
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5522
Add "cmdline" attribute for every event along with PID of observable process.
**Why cache**? I did that with a LRU caching because we expect receiving events so often, that so slow operation as reading a file is going to slow everything down.
**Why we should retrieve `cmdline` at the moment of receiving event**? It's is important to retrieve anything about process at the moment of receiving event. Because process can exit at any moment and system drops process metainformation shortly after it. Which means osquery will never get that information.
**Why only `cmdline`**? Because it is important and we need it already. Should we provide more information about process, like parent or start time, we just add it easily.
Reviewed By: SAlexandru
Differential Revision: D14426129
fbshipit-source-id: 8824cc4de7e8bd8f75e42e8f8829c4e1dadd4f18
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5521
Implementation based on `std::unordered_map` and `std::list`, without any
age-bits or any sort of optimisations. If later we realize we need something
more powerfull we will use some fancy OSS lib for it.
I need it as a cache of "hot" processes in syscall traicing. To read cmdline
from file in `/proc/<pid>/cmdline` less often and preserve cmdline or any other
info about process for the events that came when process is already gone.
Reviewed By: SAlexandru
Differential Revision: D14424352
fbshipit-source-id: 472cf8b542bab2921393b9d2a126c254c791404a
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5520
I need it to add cmdline attribute to the syscall tracing events. I think it
could be useful somewhere else in osquery (for instance in implementation of
table `processess`), so I put it under `osquery/utils/system/linux`.
Reviewed By: SAlexandru
Differential Revision: D14421472
fbshipit-source-id: 384a38f74b629d3622c48085bd3679174b70b2b4
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5519
This is very simple usage of system calls {kill, setuid} tracing. Enable it if cli flag `--enable_experimental_tracing` is specified.
All received events from the kernel will be serialised to JSON string and sent to experimental events streaming registry. Without any preprocessing or filtering.
I'm going to use this prototype to estimate performance hit of this subsystem in small fraction of our deployment.
Reviewed By: marekcirkos
Differential Revision: D14406173
fbshipit-source-id: 5ee7bd296216ad83f564aaca671a6db46739aaa7
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5527
for the sake of xCode, because it doesn't tolerate duplicated target names
Reviewed By: marekcirkos
Differential Revision: D14455350
fbshipit-source-id: c6fa8bfde87855d473884c2aa0bd664929e4757b
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5524
So let's get rid of the rest usecases of createError with message as an argument in order to remove it completely.
Reviewed By: jessek
Differential Revision: D14437933
fbshipit-source-id: acfb1a2a2a79e03d7cb650b305ee8ad3b6218d2a
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5518
method count of `std::chrono::duration::count` is not a constexpr so that means `kPollTimeout` could not be constexpr either. Let's make it just const member of the class PerfOutputPoll.
Reviewed By: jessek
Differential Revision: D14406162
fbshipit-source-id: c09a701206937e7a4bc9aeae8ef63495188dced1
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5517
It is experimental registry to stream events from osquery. It is separated from logger registry, which is semantically very similar, on purpose.
Firstly, because we are still not sure about exact way to process, filter, deliver events. I think the best way to test things is to implement pipeline for the events separately from the logger for now and may be merge them later. It's easier to merge things than split them.
Secondary, current implementation of logger is not very performant with plenty of additional functionality. Since we going to evaluate performance hit of new type of events (linux syscall tracing) it is not a good idea to connect it to sophisticated system with well known performance problems.
Reviewed By: jessek
Differential Revision: D14404665
fbshipit-source-id: bb98848d998669bd90bd4f5cf61981247bd537d6
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5516
for the experimental code. Which could be not very well tested or unstable. To be able to merge some not-yet-ready changes in order to discuss them or tests on the small fraction of the deployment.
Reviewed By: jessek
Differential Revision: D14404652
fbshipit-source-id: 010aadb7dbd56a9e2bec31a071141288bcee5311
Summary: Having two configuration files makes it harder to manage system.py so move this to the same file and to the toolchain generation script. This will allow us to automatically determine toolchain path as well in the future.
Reviewed By: marekcirkos
Differential Revision: D14425055
fbshipit-source-id: fdc017f2cc55a2efbb33cdf17df64df620eb11b8
Summary: Simplify the implementation since we can't support multiple flavors at this time anyway. This will also make it easier to support multiple distros later.
Reviewed By: SAlexandru
Differential Revision: D14282635
fbshipit-source-id: cb0262aac55c7a97149ca3432a70ddf71fca8f3b
Summary: Reduce differences to internal version to make it easier to diff. These will eventually be the same script but we need to do this step by step.
Reviewed By: SAlexandru
Differential Revision: D14282634
fbshipit-source-id: 322c0bf1b1750bf234d4d14b813c6c5aea3becbd
Summary:
There is unnecessary redundancy of how error message now is created.
It could be with string argument:
return createError(ConversionError::InvalidArgument,
"Wrong string representation of boolean ",
prop_exp.takeError())
<< boost::io::quoted(from);
And it could be without it:
return createError(ConversionError::InvalidArgument, prop_exp.takeError())
<< "Wrong string representation of boolean "
<< boost::io::quoted(from);
So, my suggestion is to make it uniform - use only second option and form error
string only with operator<<. This diff introduce function overload without
message within argument list and some usage examples. If everyone ok with it,
I'll remove all cases of 3 args form usage in next diff.
Reviewed By: mkareta
Differential Revision: D14405326
fbshipit-source-id: 47074de1f078e512c95518439557adadcf83cbd3
Summary:
This is a follow-up to #5400, fixing the `boot_partition` column.
Previous versions of the table would consistently provide a false negative for bootable drives, thanks to a mismatch in format between `Win32_DiskPartition.DeviceID` and `Win32_LogicalDisk.DeviceID`. This change replaces the use of `Win32_DiskPartition` with `Win32_BootConfiguration` and moves the query to the top-level, fixing the bug and removing the need to make `N` WMI requests for `N` system drives.
Pull Request resolved: https://github.com/facebook/osquery/pull/5477
Differential Revision: D14386989
Pulled By: akindyakov
fbshipit-source-id: 3843f760dd41dc5810048ccf969dc4b0d4674cdc
Summary:
Hi!
This PR refactors the WmiRequest and WmiResultItem classes to use smart pointers to manage their WMI objects. Previously, this code used raw pointers for these objects. The destructors needed to ensure to free these objects, and the move constructors needed to manually transfer the pointer, and set the src pointer to `nullptr`.
Now, `unique_ptr`s are used for most of the objects. `shared_ptr` is used for the `IWbemServices` pointer in preparation for a future PR which will add an identical shared services pointer to the `WmiResultItem` class, for implementing some new functionality there. For reference, you can see that future PR here at https://github.com/trailofbits/osquery-pr/pull/8/.
Conversion to smart pointers for these types requires a custom deleter, so there are some extra `using` statements and custom deletion functions that you'll find. With smart pointers in place, the destructors can be simplified to completely remove the free'ing logic, and the move constructor can be simplified to simply use `std::move`.
Apart from these, the only other part of this PR is adjusting some of the logic in the `WmiRequest` constructor to adjust to these new smart pointers. Specifically, we introduce some locals which are use for the winapi calls, and then `reset` the smart pointers using those locals.
Thanks!
Pull Request resolved: https://github.com/facebook/osquery/pull/5492
Differential Revision: D14368418
Pulled By: akindyakov
fbshipit-source-id: 5a6755aada6ee937f4b6b7cfef6dfd4863bc40fb
