PR osquery/osquery#5464 makes it impossible to call plugins
which are only registered in the core, as its happening with osquery/osquery#5839.
What happens when this is working is that the filesystem plugin is not present
in the extension registry, so when Registry::call tries to search for a plugin
that provides the "config" feature, it won't find one and the resulting plugin name,
and later item_name, will be empty.
This results in the RegistryInterface::call reroute the call to the core,
through Thrift, which will end up in the plugin correctly answering.
Change the message that is logged when a required extension is not
loaded or found from "Extension not autoloaded: <extension id>" to
"Required extension not found or not loaded: <extension id>".
Change the message that is logged when the extension manager has an
error happening during startup from "Cannot start extension manager:
<error>" to "An error occurred during extension manager startup: <error>"
Add the possibility of startExtensionManager returning with an error
if the Dispatcher failed to properly add the ExtensionManagerWatcher
or ExtensionManagerRunner.
Fallback to logging the startExtensionManager error message with a
severity of INFO only if extensions are required to be disabled.
Add tests for starting the extension manager with a nonexistent
socket path and a nonexistent required extension.
Issue osquery/osquery#5679
This defines a new CMake option: OSQUERY_FUZZ, which appends compiler
and linker flags for libFuzzer to be effective.
There is one example harness that will fuzz config parsing.
Add a way to compile third-party libraries from source instead of downloading prebuilt ones.
Each library source code is downloaded with git into a submodule at configure time,
in response to the find_package(library_name) CMake call,
except for OpenSSL where the official source archive is used.
Each submodule is attached to a release tag on its own upstream repository.
All the libraries are built using CMake directly, except for OpenSSL which uses a formula system,
which permits to build libraries with a separate build system
when there's no easy way to integrate it directly with CMake.
This new dependency system determines which library is fetched from where using the concept of "layers".
Currently we have three of them: source, formula, facebook,
where the last layer represents the pre-built libraries.
The provided order will be used when looking for libraries.
A system to patch submodule source code has been added and it's currently used with googletest, libudev and util-linux.
Patches should be put under libraries/cmake/source/<library name>/patches/<submodule>,
where <submodule> is often one and is "src", but in other cases, like AWS,
there are multiple with a more specific name.
If for whatever reason the submodule cloning or the patching fails,
the submodule has to be unregistered and its folder should be cleared.
This should be achievable with "git submodule deinit -f <submodule path>"
Following some other changes on existing functionality:
- Changed the CMake variable BUILD_TESTING to OSQUERY_BUILD_TESTS
to avoid enabling tests on third party libraries.
Due to an issue with glog the BUILD_TESTING variable
will be always forced to OFF.
- Moved compiler and linker flags to their own file cmake/flags.cmake
- Moved all the third-party CMakeLists.txt used for pre-built libraries under libraries/cmake/facebook
- Added the --exclude-folders option to tools/format-check.py and tools/git-clang-format.py,
so that it's possible to ignore any third party library source code.
- The format and format_check target use the new --exclude-folders option
to exclude libraries/cmake/source from formatting.
- The test and osquery binaries are properly compiled with PIE (osquery/osquery#5611)
Co-authored-by: Stefano Bonicatti <stefano.bonicatti@gmail.com>
Co-authored-by: Teddy Reed <teddy@casualhacking.io>
Issue highlighted by asan activated in PR osquery/osquery#5628
Ensure WrappedMessage has no padding since its members are written
consecutively in the buffer.
Also use memcpy when retrieving a WrappedMessage from a buffer, since
it could be written at a misaligned address.
PR: osquery/osquery#5665
Issue highlighted by asan activated in PR osquery/osquery#5628
EFI_DEVICE_PATH_PROTOCOL and HARDDRIVE_DEVICE_PATH were using
the wrong alignment/padding, since on disk they are written
with no padding.
PR: osquery/osquery#5665
Issue highlighted by asan activated in PR osquery/osquery#5628
Imprecisions between float -> double -> json -> double -> float
lead to out of range values been saved into a float variable.
Since json has only the notion of doubles as floating point numbers,
it's better to require to use them.
Also forced the json parser to parse floating point numbers
with full precision, otherwise the test testing for precision would fail.
PR: osquery/osquery#5665
Issue highlighted by asan activated in PR osquery/osquery#5628
When shifting left or right a byte, that must be positive, so
ensure it is.
Light cleanup of a bugged and unused function.
A deeper look into the table implementation is needed.
PR: osquery/osquery#5665
Issue highlighted by asan activated in PR osquery/osquery#5628
Do not try to read the destination address of a netmask if such address
is a default route.
PR: osquery/osquery#5665
Issue highlighted by asan activated in PR osquery/osquery#5628
Reading a 8 byte field from a 4 byte aligned struct needs to be
done with memcpy.
PR: osquery/osquery#5665
Issue highlighted by asan activated in PR osquery/osquery#5628
Do not use memcpy to copy strings around.
Also, ifa_name size is not guaranteed to be IFNAMSIZ.
PR: osquery/osquery#5665
Issue highlighted by asan activated in PR osquery/osquery#5628
Do not use clear() on a vector inizialized with a fixed size to clear
it of its contents when using it as a char buffer.
PR: osquery/osquery#5665
When proactively searching disk for personal certificates, there is no
need to filter system accounts (SYSTEM, Local Service, etc) anymore
because `findUserPersonalCertsOnDisk` is now capable of handling those
accounts by dynamically finding a user's home dir (as rather than
constructing a hard coded path).
This now makes the table even more complete; any certificates found in
the system accounts directories will always be found. Previously they
could be found but only if there was a store location other than the
`Users` store location that had a system store string that looked like
`S-1-5-18\My` or `.DEFAULT\My`.
It does not look like the EC2 tables are available on Windows
(due to spec-file configuration).
Please see https://osquery.io/schema/3.3.2#ec2_instance_tags
This removes the EC2 library linking requirement until we revisit this
feature.
Add OSQUERY_VERSION cache variable to force the version to be set at
that value.
Add OSQUERY_VERSION_AUTODECTED cache variable to show the osquery
version that has been detected through git
Add OSQUERY_VERSION_SUFFIX cache variable to append its value to the
final version, if the version was autodected.
Add OSQUERY_VERSION_INTERNAL variable to globally store the final
osquery version, to be used in other CMake scripts.
Add OSQUERY_VERSION_COMPONENTS variable to globally store the components
of the final osquery version. Currently used by the packaging system.
Modify overwrite_cache_variable so that's possible to specify the type.
The version is always autodetected using "git describe --tags --always --dirty"
and OSQUERY_VERSION_SUFFIX is appended to it, unless OSQUERY_VERSION is manually set.
Incorporates suggestions by @directionless.
CMake part of osquery/osquery#5615
* Initial implementation
* Use case insensitive comparisons for all service names
Fixes a bug where certificates for services that correspond to Local Service or
Network Service may not have their sids appear correctly. This is because the
services table is inconsistent with its user_account column.
* Make service name cache query-local
Previously, the service name cache existed for the lifetime of the
osquery process, which made it susceptible to stale reads if a service
restarted under a different user during osquery's lifetime. Now the
cache is created for each query. Also refactor it to directly map to the
sid, rather than the account name, which removes the need to translate
from account name to sid every row.
* Fix reference to destroyed object
Previously, getCurrentUserInfo took a reference to data from a local
vector, whose data is free'd after the function. This refactors the code
to use a unique_ptr (similar to how getSidFromUsername) does it.
The code was failing on Windows because the '\Windows\%' pattern
is relative and presumes that the Windows folder is on the same
drive (C:\) as the test process.
This might not be true, so we find where precisely is the Windows
directory and use the full path to it as a pattern.
The failing test checks that were testing the equal and LIKE operator
have been moved to a new test, test_table_constraints, since they
are not related to joins.
Fix UsersTest.test_sanity on Windows.
uid and gid were returned as int (while they normally are unsigned int)
and converted to signed integers in the table row.
This is wrong because beyond uid and gid not being ints,
they are taken from the RID part of the SID which in some cases,
like for a Service SID, it can have a value higher than then maximum
value of an int, so in the end the number shown in table is negative.
Now they are returned as uint32_t and converted as BIGINTs for the table
that uses them.
Fix other functions return values and conversions depending on the meaning of
the value.
On Windows stick to its specific types where possible.
Convert CRLF to LF on some of the files modified.
Calling GetLastError() will not return the error code for the Windows registry APIs,
since they return a LSTATUS value which already the error code.
This also fixes the RegistryTablesTest.test_registry_non_existing_key
test case which was incorrectly expecting success when querying
for a non-existent registry key.
Ported from https://github.com/osql/osql/pull/50
Only define BOOST_ASIO_DISABLE_STD_STRING_VIEW.
We shouldn't define BOOST_ASIO_HAS_STD_STRING_VIEW,
because even if we define BOOST_ASIO_DISABLE_STD_STRING_VIEW
the first define will actually enable parts of code that will use string_view.
This won't work on Windows and in general, string_view should not be
used unless compiling with C++17.
The hack has been also added to a test that was previously missed.
Taken from osql-experimental.
- Change CMake code license to the one present in osquery right now
- Package metadata doesn't mention Trail of Bits or osql anymore
- Set specific ACLs for the osqueryd on Windows when packaging
- Remove LLVM_INSTALL_PATH support on macOS, since we are using AppleClang
- Remove OSQUERY_SOURCE_DIR variable need and source in a submodule support
- Add targets format_check and format to check code formatting and
format it with clang-format
- Do not warn about not using Clang on macOS when using AppleClang
Summary:
I'm going to it as one of the decorators for streaming events. To be able to associate event with a certain machine on the backend side.
Why a new directory in osquery/? There are some libraries that depend on
core/database and any monster modules. To be able to keep utils simple,
lightweight and independent we should not put stuff like filesystem or network
under utils/ directory. But we need a single place to put all system assosiated
libs. So, let's make a `system` in the root of osquery.
Reviewed By: guliashvili
Differential Revision: D14706186
fbshipit-source-id: d40fde3872ca6b6677a1d8f89cfd8eda63c6b83d
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5553
Infer flagged this use of `unique_ptr` as potentially leaking the FD. I do not think this is the case but there is a missing check for a failed `open`. I am unsure of what the `ioctl` would do in this case.
I removed the custom decl to make the logic in this table easier (opinion).
Reviewed By: guliashvili
Differential Revision: D14700412
fbshipit-source-id: cfcfe9d2629aa27f18d39058830cb5c63881f163
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5552
This fixes a potential resource leak in the `syslog` event type. If the process is unable to `flock` then the FD is reset (overwritten to -1) and not closed.
Reviewed By: guliashvili
Differential Revision: D14700407
fbshipit-source-id: 092a22c3059d29aca748ee23ba30c5301cd065af
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5547
Since this diff an object of a class with defined schema (see type trait
schemer::has_schema) are allowed as memebers of anoter class with schema.
Example. C++ classes:
```
class Simple {
int alpha = 1;
public:
template <typename Archive, typename ValueType>
static void discloseSchema(Archive& a, ValueType& value) {
schemer::record(a, "alpha", value.alpha);
}
};
class Nested {
Frist beta;
int gama = 2;
public:
template <typename Archive, typename ValueType>
static void discloseSchema(Archive& a, ValueType& value) {
schemer::record(a, "beta", value.beta);
schemer::record(a, "gama", value.gama);
}
};
```
Json representation of `Nested`:
```
{
"beta": {
"alpha": 1
},
"gama": 2
}
```
Reviewed By: SAlexandru
Differential Revision: D14683589
fbshipit-source-id: 1f9e2f862d2bf64be166a717e49cf0f470f8ee36
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5548
This is a JSON deserializing formatter for schemer. It parse C++ object from
JSON object according to defined in C++ class schema. The implementation based
on rapidjson library.
Two methods with the same name: `osquery::schemer::fromJson`
Reviewed By: SAlexandru
Differential Revision: D14664162
fbshipit-source-id: ba73490dc19127a04a43b7ba1807d1f1a0bf4f43
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5545
This is a JSON serializing formatter for schemer. It represents C++ object as JSON object according to defined in C++ class schema. The implementation based on rapidjson library. It is very simple - just print all key:value pairs directly to rapidjson stream (it could be files stream or string stream). The second method of formatter converts C++ objects directly to JSON in string.
Two methods with the same name: `osquery::schemer::toJson`
Reviewed By: SAlexandru
Differential Revision: D14663996
fbshipit-source-id: 7a902b0777c9094faf39ee3111bbc09d8c3c7a55
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5544
This is a framework to declare a serialization and deserialization
schema for C++ classes. The schema can be used by different implementations
to represent C++ object as data-interchange format or to parse an object from
formatted representation.
It works without macro or any verbose transformation code. All you need to
do to be able to serialize and deserialize some C++ class is to define a
template static method `discloseSchema` in the class. Which describe all
members that have to be read by serializer and written by deserializer.
Everything else is a job of schemer formatters.
Support of specific types, including nested types support depends on the formatter implementation.
1. One place to describe class members instead of two different methods for serializer and deserializer.
2. One way do describe schema for many formatters (JSON, TOML, hasher, etc).
3. Schemer also sets the order of elements of class, therefore even binary formatters can use it, just ignoring the names.
4. It is simple - just one method, nothing more. Therefore there is zero dependencies. Everything is in formatters.
- JSON serializer/deserializer.
- Hasher - it helps to impelement transformation of osquery events in hash form for probabilistic filters.
Reviewed By: SAlexandru
Differential Revision: D14663949
fbshipit-source-id: effcc7e91a85514cee700a0d5484ab4d317ac199
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5550
This fixes a potential memory leak in keychain_items similar to D14567925. This leak was identified with the default options for OSS infer.
Reviewed By: guliashvili
Differential Revision: D14641455
fbshipit-source-id: 3fd776fe8a2af300435907ecd1d7746b3045066e
Summary: To clean it up, they are parts of the same experimental subsystem and shoul be close to each other in experimental directory.
Reviewed By: marekcirkos
Differential Revision: D14668951
fbshipit-source-id: 404be1f3c24699557edc0b9e5ada823ea04533f1
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5542
We replace deprecated calls to the Status class with newer ones. This will stop Lint from nagging me every time I open these files. There should be no change to functionality.
Specifically, we use a codemod on `.cpp` files to replace any instance of `return Status();` with `return Status::success();`.
Reviewed By: guliashvili
Differential Revision: D14652922
fbshipit-source-id: 243576195ed201d6b82fc942a785742c6b01cf83
Summary: Every time a differential query runs we save the results to the database. We were serializing those results with or without types based on the `FLAGS_log_numerics_as_numbers` flag. However we always collect results with proper types therefore, on subsequent executions of the same query, the new results (typed) were being compared with the results retrieved from the DB (untyped if FLAGS_log_numerics_as_numbers == false), causing all results to be "different" and therefore osquery to report all results as "added" (and eventually all old results as "removed").
Reviewed By: SAlexandru
Differential Revision: D14669476
fbshipit-source-id: 8abc68cbcac90c73bd92fc8d34572ba3ee2f2c75
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5543
This diff removes two vestigial functions from the osquery core. There are no references to these functions except for the unit test, which we also update.
Reviewed By: marekcirkos
Differential Revision: D14664005
fbshipit-source-id: 4e176014c803e5ec8cebdf0bd064874b483c8f76
Summary: setName was not used, other than the test, testing setName. Removing it to make class immutable
Reviewed By: jessek
Differential Revision: D14640959
fbshipit-source-id: d684939ed31e8ad275650847fadb2fb8c11b6fdc
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5538
This bug was found using the OSS version of infer with default options.
Reviewed By: guliashvili
Differential Revision: D14567925
fbshipit-source-id: 64272880464577207552e3436754b7f6d32ef4ec
Summary:
osquery returned status 1(error) always when osqueryd service install uninstall was requested by the flags.
Even if install/uninstall was successfull.
Add extra logging in case of success.
Exit with status 0 when successfully installed/uninstalled
Reviewed By: jessek
Differential Revision: D14622771
fbshipit-source-id: c0c7569a2b13434e2ad28ba92cba308024b044ec
Summary:
User could have requested install and uninstall of osqueryd service together. However it does not make sense and actually osquery would only install the service.
This additional check, will exit osquery in case when install and uninstall are requested together.
Reviewed By: jessek
Differential Revision: D14622634
fbshipit-source-id: b88dc519bd1ef38653eb1660712bbf2e75bad36f
Summary:
Replaced fprint(stderr with std::cerr
Other than that, could not find any use of
```
```
Reviewed By: akindyakov, fmanco
Differential Revision: D14621228
fbshipit-source-id: e996b2507757dc6af710e7ab0e25fc8136bbb92e
Summary: Somehow, still looking for how, D14577520 broke communication between osqueryi/osqueryd -S and extension. Revert.
Reviewed By: akindyakov
Differential Revision: D14620885
fbshipit-source-id: 98392f6e1e5da1b0ea68ee54dc00b3bbcd687315
Summary: It requires some effort to fix warnings properly, let's just mute them for now.
Reviewed By: marekcirkos
Differential Revision: D14596617
fbshipit-source-id: 184937d15cc186afe045131d6d648cdbfb4ad574
Summary:
Working to offering thrift service validating queries. It will be used by query canary service and helldriver.
For that, I need to execute osqueryd -S with the given query. However, osqueryd -S (with extension) crashes while it exits. I can filter out crash output, but it means, I will not be able to catch if query really crashes the osqueryd.
I also don't want to really fix osquery watcher, as we have the watcher 2.0 as an intern project.
Fixing the current problem by stopping osquery to ignore FLAGS_disable_watchdog flag when extension is avaliable.
Reviewed By: marekcirkos
Differential Revision: D14577520
fbshipit-source-id: 588c31319b875b8af9d73da6d691f22bcfdb8fc9
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5532
Ok, this is overcomplicated code for signal handling. Besides untested magic
with signals it makes quite a lot of assumptions about program state when it is
called. For instance, it uses instance of Watcher and Dispatcher singleton
which can be already destroyed or corrupted at the that time.
This complexity cause rare unpredictable and hardly reproducible problems, that
lead to data losses, cross process deadlocks, crashes of osqueryd because of
crash an extensions and os on ans so far.
Couple of examples.
When extensions killed by signal it 2 time from 3 lead to crash of osqueryd. It
very well reproducible but quite not very well diagnosable.
Another example is std::terminate called at shutdown of extensions. Something
could not be destroyed correctly and call std::terminate, which is already bad
but sometimes reasonable. After it everything goes to signal handler which use
already destroyed Watcher and cause another std::terminate. It leads to
recursive signal handler call and eventually to stack overflow.
Solution.
I believe it could be fixed less radical, absolutely. It takes some time,
though. And because it will cost us a lot there is a question: do we really
need it here? I could not find a place in osquery code where it is important to
finish something before shutdown. Moreover it is pretty rare case when code
handle shutdown condition correctly. That makes me think that we don't need
signal handling at all or at least not like this.
Reviewed By: mkareta
Differential Revision: D14458287
fbshipit-source-id: daf8388323e4dacabeee3338d185577335503da3
Summary: Before, we sent the user and kernel time in a separate time series. They are usefull separately, in order to measure which queries are syscall extensive. However, for measuring the total time we need them together. We have wall time, but user + kernel time is more precise as they measure resource usage of the thread.
Reviewed By: jessek
Differential Revision: D14502715
fbshipit-source-id: 7a03ec5b1d7b328e9657d94f752d33e7aa771e97
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5530
This was the behaviour pre-#5083 and is a more sane behaviour for a shell since we usually don't want to see all the verbosity.
Reviewed By: marekcirkos
Differential Revision: D14469709
fbshipit-source-id: e0235c3cfcd51d77053da48ae35cfe75678e77c3
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5531
Last diff in series to migrate to a new interface. More details in diff: D14405326 or PR #5523
There is a changes to remove old interface and a bit more migration (things was landed in a middle).
Reviewed By: guliashvili
Differential Revision: D14477768
fbshipit-source-id: 572791cb6929edf46d5a8bfe9416e03dc85eb96f
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5515
This diff augments the unit test which determines if a query should execute on this client. We add more test cases for different platforms, including the ones which are no longer officially supported, `centos` and `ubuntu`. These values remain equivalent to `linux`. That is, a query with either of those values specified for the platform will be executed on any Linux system.
Reviewed By: guliashvili
Differential Revision: D14430282
fbshipit-source-id: 64a47ded0ef769d927753b9d75a378b25a2391bd
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5522
Add "cmdline" attribute for every event along with PID of observable process.
**Why cache**? I did that with a LRU caching because we expect receiving events so often, that so slow operation as reading a file is going to slow everything down.
**Why we should retrieve `cmdline` at the moment of receiving event**? It's is important to retrieve anything about process at the moment of receiving event. Because process can exit at any moment and system drops process metainformation shortly after it. Which means osquery will never get that information.
**Why only `cmdline`**? Because it is important and we need it already. Should we provide more information about process, like parent or start time, we just add it easily.
Reviewed By: SAlexandru
Differential Revision: D14426129
fbshipit-source-id: 8824cc4de7e8bd8f75e42e8f8829c4e1dadd4f18
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5521
Implementation based on `std::unordered_map` and `std::list`, without any
age-bits or any sort of optimisations. If later we realize we need something
more powerfull we will use some fancy OSS lib for it.
I need it as a cache of "hot" processes in syscall traicing. To read cmdline
from file in `/proc/<pid>/cmdline` less often and preserve cmdline or any other
info about process for the events that came when process is already gone.
Reviewed By: SAlexandru
Differential Revision: D14424352
fbshipit-source-id: 472cf8b542bab2921393b9d2a126c254c791404a
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5520
I need it to add cmdline attribute to the syscall tracing events. I think it
could be useful somewhere else in osquery (for instance in implementation of
table `processess`), so I put it under `osquery/utils/system/linux`.
Reviewed By: SAlexandru
Differential Revision: D14421472
fbshipit-source-id: 384a38f74b629d3622c48085bd3679174b70b2b4
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5519
This is very simple usage of system calls {kill, setuid} tracing. Enable it if cli flag `--enable_experimental_tracing` is specified.
All received events from the kernel will be serialised to JSON string and sent to experimental events streaming registry. Without any preprocessing or filtering.
I'm going to use this prototype to estimate performance hit of this subsystem in small fraction of our deployment.
Reviewed By: marekcirkos
Differential Revision: D14406173
fbshipit-source-id: 5ee7bd296216ad83f564aaca671a6db46739aaa7
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5527
for the sake of xCode, because it doesn't tolerate duplicated target names
Reviewed By: marekcirkos
Differential Revision: D14455350
fbshipit-source-id: c6fa8bfde87855d473884c2aa0bd664929e4757b
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5524
So let's get rid of the rest usecases of createError with message as an argument in order to remove it completely.
Reviewed By: jessek
Differential Revision: D14437933
fbshipit-source-id: acfb1a2a2a79e03d7cb650b305ee8ad3b6218d2a
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5518
method count of `std::chrono::duration::count` is not a constexpr so that means `kPollTimeout` could not be constexpr either. Let's make it just const member of the class PerfOutputPoll.
Reviewed By: jessek
Differential Revision: D14406162
fbshipit-source-id: c09a701206937e7a4bc9aeae8ef63495188dced1
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5517
It is experimental registry to stream events from osquery. It is separated from logger registry, which is semantically very similar, on purpose.
Firstly, because we are still not sure about exact way to process, filter, deliver events. I think the best way to test things is to implement pipeline for the events separately from the logger for now and may be merge them later. It's easier to merge things than split them.
Secondary, current implementation of logger is not very performant with plenty of additional functionality. Since we going to evaluate performance hit of new type of events (linux syscall tracing) it is not a good idea to connect it to sophisticated system with well known performance problems.
Reviewed By: jessek
Differential Revision: D14404665
fbshipit-source-id: bb98848d998669bd90bd4f5cf61981247bd537d6
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5516
for the experimental code. Which could be not very well tested or unstable. To be able to merge some not-yet-ready changes in order to discuss them or tests on the small fraction of the deployment.
Reviewed By: jessek
Differential Revision: D14404652
fbshipit-source-id: 010aadb7dbd56a9e2bec31a071141288bcee5311
Summary:
There is unnecessary redundancy of how error message now is created.
It could be with string argument:
return createError(ConversionError::InvalidArgument,
"Wrong string representation of boolean ",
prop_exp.takeError())
<< boost::io::quoted(from);
And it could be without it:
return createError(ConversionError::InvalidArgument, prop_exp.takeError())
<< "Wrong string representation of boolean "
<< boost::io::quoted(from);
So, my suggestion is to make it uniform - use only second option and form error
string only with operator<<. This diff introduce function overload without
message within argument list and some usage examples. If everyone ok with it,
I'll remove all cases of 3 args form usage in next diff.
Reviewed By: mkareta
Differential Revision: D14405326
fbshipit-source-id: 47074de1f078e512c95518439557adadcf83cbd3
Summary:
This is a follow-up to #5400, fixing the `boot_partition` column.
Previous versions of the table would consistently provide a false negative for bootable drives, thanks to a mismatch in format between `Win32_DiskPartition.DeviceID` and `Win32_LogicalDisk.DeviceID`. This change replaces the use of `Win32_DiskPartition` with `Win32_BootConfiguration` and moves the query to the top-level, fixing the bug and removing the need to make `N` WMI requests for `N` system drives.
Pull Request resolved: https://github.com/facebook/osquery/pull/5477
Differential Revision: D14386989
Pulled By: akindyakov
fbshipit-source-id: 3843f760dd41dc5810048ccf969dc4b0d4674cdc
Summary:
Hi!
This PR refactors the WmiRequest and WmiResultItem classes to use smart pointers to manage their WMI objects. Previously, this code used raw pointers for these objects. The destructors needed to ensure to free these objects, and the move constructors needed to manually transfer the pointer, and set the src pointer to `nullptr`.
Now, `unique_ptr`s are used for most of the objects. `shared_ptr` is used for the `IWbemServices` pointer in preparation for a future PR which will add an identical shared services pointer to the `WmiResultItem` class, for implementing some new functionality there. For reference, you can see that future PR here at https://github.com/trailofbits/osquery-pr/pull/8/.
Conversion to smart pointers for these types requires a custom deleter, so there are some extra `using` statements and custom deletion functions that you'll find. With smart pointers in place, the destructors can be simplified to completely remove the free'ing logic, and the move constructor can be simplified to simply use `std::move`.
Apart from these, the only other part of this PR is adjusting some of the logic in the `WmiRequest` constructor to adjust to these new smart pointers. Specifically, we introduce some locals which are use for the winapi calls, and then `reset` the smart pointers using those locals.
Thanks!
Pull Request resolved: https://github.com/facebook/osquery/pull/5492
Differential Revision: D14368418
Pulled By: akindyakov
fbshipit-source-id: 5a6755aada6ee937f4b6b7cfef6dfd4863bc40fb
Summary:
Example of use case for this close function that may fail, send network request or something similar,
but you invoke this function on end of app lifetime, so there it not make sense to handle error and you want to treat call as oneway
Reviewed By: marekcirkos
Differential Revision: D14365081
fbshipit-source-id: 5a1067c6b874d62c283b01f1c936ac8de67719ca
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5498
to keep in sync headers namespace and real path to target
Reviewed By: jessek, marekcirkos
Differential Revision: D14322951
fbshipit-source-id: 3e99cb5e9227529520c0ee1cac0c3d755f594ed8
Summary:
In my testing of Osquery, I noticed the `path` field for `firefox_addons` is always blank. After inspecting the structure of the JSON and the code in Osquery, I believe the key to look up path should be renamed from `descriptor` to `path`.
Pull Request resolved: https://github.com/facebook/osquery/pull/5462
Differential Revision: D14301220
Pulled By: akindyakov
fbshipit-source-id: b4b56d7026736344da585aada83d5fea6e249f1c
Summary:
Add log_numerics_as_numbers flag.
Internal SQLite query has method that returns QueryDataTyped (instead of QueryData), which are boost::variant<string, double, long long>.
Ints are encoded as such if and only if new log_numerics_as_numbers flag is set to true.
Reviewed By: fmanco
Differential Revision: D13778323
fbshipit-source-id: 7d7bb31781486f63fcc088cd479d3b6f255a5cb4
Summary: Please do not use message of Status for anything except logging!
Reviewed By: fmanco
Differential Revision: D14281607
fbshipit-source-id: e48ed3d79de19acb1d86885d0acfab781e5058aa
Summary: Add query monitoring without specifying the scheme. It will help us in the simpler aggregation.
Reviewed By: marekcirkos
Differential Revision: D14280109
fbshipit-source-id: d70971c9ebe1e15c66f36e1490b7d8e198a68a7a
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5485
Initial steps to separate plugins from the rest of osquery. On the long run separating plugins will provide more build flexibility such that we can have configurable builds that include only the bits and pieces we actually ne
er deployment. Reducing the attack surface, possibility of supply chain attacks, binary size, etc.
Move numeric monitoring
Reviewed By: guliashvili
Differential Revision: D14259758
fbshipit-source-id: 1016fc28a0052417d658b6ce1cb3368e56597a7b
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5483
Initial steps to separate plugins from the rest of osquery. On the long run separating plugins will provide more build flexibility such that we can have configurable builds that include only the bits and pieces we actually need per deployment. Reducing the attack surface, possibility of supply chain attacks, binary size, etc.
Move killswitch
Reviewed By: guliashvili
Differential Revision: D14259760
fbshipit-source-id: deaaa148fac25b3f534a8881a1ea9eda0ec6bdc9
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5491
Initial implementation had a half-baked and broken wait() implementation to support multiple consumers. It does not make sense to support multiple consumers anyway, so let's remove that functionality and simplify the implementation.
Reviewed By: mkareta
Differential Revision: D14279796
fbshipit-source-id: d31fff69a64c0e45a9af8cd97bb0488c111c6b68
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5487
I got tired of Lint telling me to update the calls to the Status class, so I wrote some codemods to update the vast majority of them:
$ cd ~/fbsource
$ codemod -d xplat/osquery/oss/osquery --extensions cpp,h "return Status\(0\, \".*\"\);" "return Status::success();"
**Blindly accepted all changes**.
FWIW, I tried to do something similar with the failure return values, but the unit tests were failing afterward.
Reviewed By: guliashvili
Differential Revision: D14278739
fbshipit-source-id: b023f6e969be697eeb67d66ea3ad10d16c69ff0c
Summary:
Super tiny PR to fix some spelling. This is the first of a few incremental PRs for some improvements to WMI!
Pull Request resolved: https://github.com/facebook/osquery/pull/5471
Reviewed By: marekcirkos
Differential Revision: D14240440
Pulled By: jessek
fbshipit-source-id: 9a6c23c3aa31bda15972dc79bbad07640727d19a
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5482
We allow people using a debug build of osqueryd to skip verification of the server's TLS certificate. This allow people to connect to a devserver, for example, when testing the Graph API endpoints used by osquery.
This mode broke at some point when we stopped defining the `DEBUG` constant when building osqueryd in debug mode. This diff updates the code so that we use the constant `NDEBUG`, which is actually defined for release builds. For example, for OS X, see the build configs at https://fburl.com/ywr1tyuk and https://fburl.com/mte9ajvr.
We also update the code to ensure the command line option `--tls_allow_unsafe` allows a person to connect to a dev server.
Reviewed By: obelisk
Differential Revision: D14260226
fbshipit-source-id: c0f2f1c1f4c8dd5e3e1eaa6e1a52ab2a4ff765d4
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5475
osquery build doesn't use folly for now so this facebook specific routine is dead and should be removed or moved to another place.
Reviewed By: fmanco, guliashvili
Differential Revision: D14242160
fbshipit-source-id: f2a8942906ddc1d7d9bb47395b1081ad2e88712d
Summary: to be able to preserve old python namespace for backward compatibility and satisfy fbcode naming rules
Reviewed By: jessek, fmanco
Differential Revision: D14242579
fbshipit-source-id: b50ba0df6ad75752f651f0bf642f3278de363874
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5478
This makes it easier to update the osquery version and simplifies cxx.bzl by removing osquery specific preprocessor flags.
This will also make rebuilding osquery after changing versions faster, since the flags are now only defined for the headers which need them.
Reviewed By: akindyakov
Differential Revision: D14183142
fbshipit-source-id: 396d550f5b35a1d294fee802d2364cd9f7ab1d7a
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5474
To be able to exclude config:update plugin from osquery plugin sdk.
Reviewed By: guliashvili
Differential Revision: D14241686
fbshipit-source-id: e378718b1e79e3dbe314a676945c18d15d054a3d
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5473
to be able to exclude dependency on enroll:tls plugin from plugins_sdk.
Only plugin registration was moved by now. The actual plugin code will be moved a bit later.
Reviewed By: guliashvili
Differential Revision: D14241687
fbshipit-source-id: 814d52719ff27bd1283bcaecdc912fca3830ff5e
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5472
To get rid of dependency on `plugins/config:tls_config` from `osquery/remote/enroll/tls_enroll`.
To be able to remove dependency on `plugins/config:tls_config` from plugins_sdk.
Reviewed By: guliashvili
Differential Revision: D14241685
fbshipit-source-id: 27cda0be9d0b6dc142374cc3a9e7c40a4fa97b94
Summary:
This introduces two new (Windows-only) columns to the `logged_in_users` table:
* `sid` corresponds to the logged in user's security identifier, used to uniquely identify the user and their permissions on the local system.
* `registry_hive` corresponds to the user's HKU registry hive, used to look up per-user configuration information.
I've updated the integration tests to test for these columns on Windows only. Please let me know if there's anything else I can do!
Pull Request resolved: https://github.com/facebook/osquery/pull/5454
Differential Revision: D14195466
Pulled By: fmanco
fbshipit-source-id: def9c362fac1b5a68b68f826916daafee224295b
Summary:
This generally refactors the `logical_drives` table on Windows to conform more closely to C++11 idioms. It also enables the integration test for `logical_drives`.
See #5367. I'll open a PR for the boot partition fixes once this is merged.
cc akindyakov guliashvili
Pull Request resolved: https://github.com/facebook/osquery/pull/5400
Differential Revision: D14131722
Pulled By: fmanco
fbshipit-source-id: c3077da48147a9880ce08925d165e5d1da363bb9
Summary:
Let's create a fbcode wrapper for osuery plugin SDK to wrap it up with fbthrift implementation from fbcode. Also we should trick the build system and add empty function `registerForeignTables()` to avoid including all osquery builtin tables implementation.
Since that, SDK suppose to be full functional and ready to be used to build facebook PROD extensions in fbcode with dependency on it (see next diff).
Reviewed By: fmanco
Differential Revision: D14224205
fbshipit-source-id: c08cbb374f2d68f6b93637809b97dcf0159c9e08
Summary: To be able to build with different `thrift_impl` modules in extensions (like facebook does with fbthrift) I removed implementation from `extensions` lib and explicitly used for `osqueryd` and for all tests. This allows us to remove `thrift_impl` from sdk dependencies list. If you need it - please use explicitly as I did for `osqueryd`. If you want to use your own implementation of `thrift_impl` - build it and link against it and sdk simultaneously.
Reviewed By: marekcirkos
Differential Revision: D14224206
fbshipit-source-id: 218dc05d87e1ffaca37783185fc672a20684d757
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5466
To be able to use other implementation (like `fbthrift_impl.cpp`) instead of `thrift_impl.cpp`, if required.
Reviewed By: marekcirkos
Differential Revision: D14208053
fbshipit-source-id: ba379ea85fc66deef517763eaca896e88729d68e
Summary:
tables: re-architect the processes table to not use WMI queries
This PR re-writes the processes table to no longer rely on WMI queries. We do this to ensure that the processes table is robust and performant, as we rely on this table for the watcher process to regulate the worker processes. Further, this table further employs the selective column generation introduced to the processes table on darwin to allow us to more selectively generate column data only when necessary.
Lastly, this table removes a couple of extraneous columns from the Windows table schema, as these column values are achievable via sqlite logic. Specifically:
* `elapsed_time`: to get this column one can use the `start_time` column along with the sqlite built in `strftime` function:
```
osquery> select name, (strftime('%s', 'now') - start_time) as uptime from processes limit 5;
+------------------+--------+
| name | uptime |
+------------------+--------+
| [System Process] | |
| System | 7131 |
| Registry | 7154 |
| smss.exe | 7131 |
| csrss.exe | 7127 |
+------------------+--------+
```
* `percent_processor_time`: to obtain the total time that a process has been executing on a system, one can add together the `user_time` and `system_time` together to get the same value returned by the WMI value, the difference being that the sumation of `user_time` and `system_time` are in milliseconds whereas the values returned from WMI are in 100 nanosecond ticks:
osquery `percent_processor_time`:
```
osquery> select pid, name, (user_time + system_time) as percent_processor_time from processes where name = 'osqueryd.exe';
I1112 21:54:11.809412 2152 processes.cpp:461] Failed to open handle to process 0 with 203
+-------+--------------+------------------------+
| pid | name | percent_processor_time |
+-------+--------------+------------------------+
| 3892 | osqueryd.exe | 5952 |
| 808 | osqueryd.exe | 0 |
| 5892 | osqueryd.exe | 52374 |
| 11688 | osqueryd.exe | 280 |
+-------+--------------+------------------------+
```
Same data obtained with WMI.
```
λ Get-WmiObject -Query "Select IDProcess, Name, PercentProcessorTime from Win32_PerfRawData_PerfProc_Process" | Where-Object {$_.Name -like 'osqueryd*'}
IDProcess : 3892
Name : osqueryd
PercentProcessorTime : 59531250
IDProcess : 808
Name : osqueryd#1
PercentProcessorTime : 0
IDProcess : 5892
Name : osqueryd#2
PercentProcessorTime : 523750000
IDProcess : 11688
Name : osqueryd#3
PercentProcessorTime : 2812500
```
Some additional notes, just to verify we're getting all data:
```
C:\Users\Nick\work\repos\osquery [windows-process-rearch ↑]
λ .\build\windows10\osquery\RelWithDebInfo\osqueryd.exe -S --json "select count(*) from processes;"
[
{"count(*)":"160"}
]
C:\Users\Nick\work\repos\osquery [windows-process-rearch ↑]
λ C:\ProgramData\osquery\osqueryd\osqueryd.exe -S --json "select count(*) from processes;"
[
{"count(*)":"160"}
]
```
Also we achieve a considerable speed up from this re-arch:
```
C:\Users\Nick\work\repos\osquery [windows-process-rearch ↑]
λ (Measure-Command {.\build\windows10\osquery\RelWithDebInfo\osqueryd.exe -S --json "select count(*) from processes;"}).Milliseconds
168
C:\Users\Nick\work\repos\osquery [windows-process-rearch ↑]
λ (Measure-Command {C:\ProgramData\osquery\osqueryd\osqueryd.exe -S --json "select count(*) from processes;"}).Milliseconds
223
```
Pull Request resolved: https://github.com/facebook/osquery/pull/5293
Reviewed By: fmanco
Differential Revision: D13561337
Pulled By: muffins
fbshipit-source-id: 61435611d34d2cfd5f61ea52512a9dc208d259a5
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5464
While attempting to debug this issue I have discovered we are attempting to look up an empty item name in the registry. This doesn't make sense as we are clearly passing in an item name. This diff doesn't solve the issue, but it does add an error message for when it happens.
While we're here, appeasing Lint by using a more modern call to the `Status` return type.
Reviewed By: guliashvili
Differential Revision: D14207654
fbshipit-source-id: a53035ea84357f9ffe1d216b66e116257f291a7f
Summary: Trim whitespace in Darwin system_info table from hardware_serial column.
Reviewed By: jessek
Differential Revision: D14207715
fbshipit-source-id: 2948b8687c0c40d43f13ab42ddc4bbfec8e59a48
Summary:
This PR fixes#5106 by trimming the whitespace we get from the data returned from the macOS API. At first I thought this was an osquery regression, but going as far back as 2.x this looks like it has always been a problem.
Before the fix...
```
echo "select hardware_model, hardware_vendor, hardware_version from system_info;" | osqueryi --json | jq
[
{
"hardware_model": "MacBookPro13,3 ",
"hardware_vendor": "Apple Inc. ",
"hardware_version": "1.0 "
}
]
```
After the fix... (note the lack of trailing spaces)
```
echo "select hardware_version, hardware_vendor, hardware_model from system_info;" | /Users/jmeller/source/osquery/build/darwin10.13/osquery/osqueryi --json | jq
[
{
"hardware_model": "MacBookPro13,3",
"hardware_vendor": "Apple Inc.",
"hardware_version": "1.0"
}
]
```
Pull Request resolved: https://github.com/facebook/osquery/pull/5167
Reviewed By: jessek
Differential Revision: D14207353
Pulled By: guliashvili
fbshipit-source-id: 4012e04b38e2b5bfc299684baf86d0ab34cd42f8
Summary: as far as they are actually part of glibc, that would be less confusing. Later it will help us to deal with fully static build.
Reviewed By: marekcirkos
Differential Revision: D14183689
fbshipit-source-id: 8b712286e171305a49be6703b846f6f935b7dcc2
Summary: to be able to proxy it inside `osquery_tp_target` function.
Reviewed By: marekcirkos
Differential Revision: D14182918
fbshipit-source-id: d56b7135664ecd5afcc87b1c1cd2cde5394368a0
Summary:
It cause implicit dependency to osquery/core:core, that cause circular dependency :(. The point is to get rid of it. Let's use just exit() instead.
As far as we don't have decent implementation of Initializer::shutdown it will not cause us any more problems.
We can live with it for a while.
Reviewed By: marekcirkos
Differential Revision: D14182915
fbshipit-source-id: 22c54cde910ea29b1fb19d96598db381b9d1c60d
Summary:
This fixes a regression introduced in D13586036 .
The tailer downstream sometimes determines type by trying to parse the string as an int. We want this to consistently fail to do so when the value is in fact a float.
Reviewed By: SAlexandru
Differential Revision: D14161125
fbshipit-source-id: 8e2bb9c59559a5774813a7c7777257742f885c35
Summary: only events.cpp required by plugin_sdk, everything else is not necessary
Reviewed By: marekcirkos
Differential Revision: D14131365
fbshipit-source-id: 18e101b3e549d8403aa22ee3d5a5fadd2b296c82
Summary:
I didn't find a way to refer file in fbcode thrift_library from xplat. So, let's copy it.
There is something wrong with a namespaces in python - I'll fix it later. This is not a problem for now - file is not used.
Reviewed By: marekcirkos
Differential Revision: D14124542
fbshipit-source-id: 1060e23675a8086af494aa93037e1c613f250c71
Summary:
As far as this file is only for internal build, to be able to build osquery sdk against fbthrift I moved it to fbcode from oss.
Later if we want to osqeruy work with fbthrift we can copy/move it again
Reviewed By: marekcirkos
Differential Revision: D14124448
fbshipit-source-id: aaeadb40a6fac19621423b225cf6d522607d1313
Summary:
Hi! This PR adds a new column called `product_version` to the file table, which is only
populated when queries are done on Windows. It is a very minimal PR that uses an existing helper function (`windowsGetFileVersion`) to populate the column.
The column is not named `file_version`, despite the name of the helper function because the underlying data retrieved by that helper function is actually the `dwProductVersion*` fields of the `VS_FIXEDFILEINFO` struct. In the future, if we want to add a column that _actually_ contains the results of the `dwFileVersion*` fields, we can add a new column called `file_version` without modifying existing functionality.
Pull Request resolved: https://github.com/facebook/osquery/pull/5431
Differential Revision: D14169773
Pulled By: fmanco
fbshipit-source-id: 6fa7c92425fc92aa2e83a37383b1a8c796b17644
Summary: It cause problems with compiling against tp2 because in tp2 there is no such target. As far as I can see there is no necessity to depend on that target separatelly from gtest. So, the simplest solution is to remove it
Reviewed By: SAlexandru
Differential Revision: D14149801
fbshipit-source-id: fe0fcd0593bded7c8f2c6fd247b1d52bd29bac9f
Summary: This test is flaky on some systems. Need to investigate and fix.
Reviewed By: guliashvili
Differential Revision: D14149533
fbshipit-source-id: 7df570c84c66cad8bdc67a1286d91a7c7e8e3502
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5458
Use ASSERTs where necessary. ASSERTs are fatal and terminate the test immediately, so should be used when subsequent tests aren't meaningful in case of previous failures. Only use EXPECTED when it makes sense for the tests to continue.
Reviewed By: jessek
Differential Revision: D14138382
fbshipit-source-id: aae0cd8640e602ac467861303457f9cf017bc5e8
