While in the end the osqueryd binary will require almost all libraries
built and linked, having each intermediate library depend on the whole
set of AWS libraries might result in higher compiling time.
Especially since on some platforms not all libraries are used.
This should speed up the compilation speed on Windows.
There is a bug in RapidJSON 1.1.0 that results in wild pointer reads
when parsing exponents. We want to prevent regressing.
This requires updating the Buck/pre-built libraries.
Right now, there is no way to tell what platform osquery is running on. We have `os_version.platform` and `os_version.platform_like`, but they are highly inconsistent, and require a lot of custom parsing to use. We should expose the underlying platform bitmask.
This adds the `platform_mask` to the `osquery_info` table.
This replaces https://github.com/osquery/osquery/pull/5488
Add a way to compile third-party libraries from source instead of downloading prebuilt ones.
Each library source code is downloaded with git into a submodule at configure time,
in response to the find_package(library_name) CMake call,
except for OpenSSL where the official source archive is used.
Each submodule is attached to a release tag on its own upstream repository.
All the libraries are built using CMake directly, except for OpenSSL which uses a formula system,
which permits to build libraries with a separate build system
when there's no easy way to integrate it directly with CMake.
This new dependency system determines which library is fetched from where using the concept of "layers".
Currently we have three of them: source, formula, facebook,
where the last layer represents the pre-built libraries.
The provided order will be used when looking for libraries.
A system to patch submodule source code has been added and it's currently used with googletest, libudev and util-linux.
Patches should be put under libraries/cmake/source/<library name>/patches/<submodule>,
where <submodule> is often one and is "src", but in other cases, like AWS,
there are multiple with a more specific name.
If for whatever reason the submodule cloning or the patching fails,
the submodule has to be unregistered and its folder should be cleared.
This should be achievable with "git submodule deinit -f <submodule path>"
Following some other changes on existing functionality:
- Changed the CMake variable BUILD_TESTING to OSQUERY_BUILD_TESTS
to avoid enabling tests on third party libraries.
Due to an issue with glog the BUILD_TESTING variable
will be always forced to OFF.
- Moved compiler and linker flags to their own file cmake/flags.cmake
- Moved all the third-party CMakeLists.txt used for pre-built libraries under libraries/cmake/facebook
- Added the --exclude-folders option to tools/format-check.py and tools/git-clang-format.py,
so that it's possible to ignore any third party library source code.
- The format and format_check target use the new --exclude-folders option
to exclude libraries/cmake/source from formatting.
- The test and osquery binaries are properly compiled with PIE (osquery/osquery#5611)
Co-authored-by: Stefano Bonicatti <stefano.bonicatti@gmail.com>
Co-authored-by: Teddy Reed <teddy@casualhacking.io>
Issue highlighted by asan activated in PR osquery/osquery#5628
Ensure WrappedMessage has no padding since its members are written
consecutively in the buffer.
Also use memcpy when retrieving a WrappedMessage from a buffer, since
it could be written at a misaligned address.
PR: osquery/osquery#5665
Issue highlighted by asan activated in PR osquery/osquery#5628
Imprecisions between float -> double -> json -> double -> float
lead to out of range values been saved into a float variable.
Since json has only the notion of doubles as floating point numbers,
it's better to require to use them.
Also forced the json parser to parse floating point numbers
with full precision, otherwise the test testing for precision would fail.
PR: osquery/osquery#5665
Add OSQUERY_VERSION cache variable to force the version to be set at
that value.
Add OSQUERY_VERSION_AUTODECTED cache variable to show the osquery
version that has been detected through git
Add OSQUERY_VERSION_SUFFIX cache variable to append its value to the
final version, if the version was autodected.
Add OSQUERY_VERSION_INTERNAL variable to globally store the final
osquery version, to be used in other CMake scripts.
Add OSQUERY_VERSION_COMPONENTS variable to globally store the components
of the final osquery version. Currently used by the packaging system.
Modify overwrite_cache_variable so that's possible to specify the type.
The version is always autodetected using "git describe --tags --always --dirty"
and OSQUERY_VERSION_SUFFIX is appended to it, unless OSQUERY_VERSION is manually set.
Incorporates suggestions by @directionless.
CMake part of osquery/osquery#5615
Taken from osql-experimental.
- Change CMake code license to the one present in osquery right now
- Package metadata doesn't mention Trail of Bits or osql anymore
- Set specific ACLs for the osqueryd on Windows when packaging
- Remove LLVM_INSTALL_PATH support on macOS, since we are using AppleClang
- Remove OSQUERY_SOURCE_DIR variable need and source in a submodule support
- Add targets format_check and format to check code formatting and
format it with clang-format
- Do not warn about not using Clang on macOS when using AppleClang
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5547
Since this diff an object of a class with defined schema (see type trait
schemer::has_schema) are allowed as memebers of anoter class with schema.
Example. C++ classes:
```
class Simple {
int alpha = 1;
public:
template <typename Archive, typename ValueType>
static void discloseSchema(Archive& a, ValueType& value) {
schemer::record(a, "alpha", value.alpha);
}
};
class Nested {
Frist beta;
int gama = 2;
public:
template <typename Archive, typename ValueType>
static void discloseSchema(Archive& a, ValueType& value) {
schemer::record(a, "beta", value.beta);
schemer::record(a, "gama", value.gama);
}
};
```
Json representation of `Nested`:
```
{
"beta": {
"alpha": 1
},
"gama": 2
}
```
Reviewed By: SAlexandru
Differential Revision: D14683589
fbshipit-source-id: 1f9e2f862d2bf64be166a717e49cf0f470f8ee36
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5548
This is a JSON deserializing formatter for schemer. It parse C++ object from
JSON object according to defined in C++ class schema. The implementation based
on rapidjson library.
Two methods with the same name: `osquery::schemer::fromJson`
Reviewed By: SAlexandru
Differential Revision: D14664162
fbshipit-source-id: ba73490dc19127a04a43b7ba1807d1f1a0bf4f43
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5545
This is a JSON serializing formatter for schemer. It represents C++ object as JSON object according to defined in C++ class schema. The implementation based on rapidjson library. It is very simple - just print all key:value pairs directly to rapidjson stream (it could be files stream or string stream). The second method of formatter converts C++ objects directly to JSON in string.
Two methods with the same name: `osquery::schemer::toJson`
Reviewed By: SAlexandru
Differential Revision: D14663996
fbshipit-source-id: 7a902b0777c9094faf39ee3111bbc09d8c3c7a55
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5544
This is a framework to declare a serialization and deserialization
schema for C++ classes. The schema can be used by different implementations
to represent C++ object as data-interchange format or to parse an object from
formatted representation.
It works without macro or any verbose transformation code. All you need to
do to be able to serialize and deserialize some C++ class is to define a
template static method `discloseSchema` in the class. Which describe all
members that have to be read by serializer and written by deserializer.
Everything else is a job of schemer formatters.
Support of specific types, including nested types support depends on the formatter implementation.
1. One place to describe class members instead of two different methods for serializer and deserializer.
2. One way do describe schema for many formatters (JSON, TOML, hasher, etc).
3. Schemer also sets the order of elements of class, therefore even binary formatters can use it, just ignoring the names.
4. It is simple - just one method, nothing more. Therefore there is zero dependencies. Everything is in formatters.
- JSON serializer/deserializer.
- Hasher - it helps to impelement transformation of osquery events in hash form for probabilistic filters.
Reviewed By: SAlexandru
Differential Revision: D14663949
fbshipit-source-id: effcc7e91a85514cee700a0d5484ab4d317ac199
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5542
We replace deprecated calls to the Status class with newer ones. This will stop Lint from nagging me every time I open these files. There should be no change to functionality.
Specifically, we use a codemod on `.cpp` files to replace any instance of `return Status();` with `return Status::success();`.
Reviewed By: guliashvili
Differential Revision: D14652922
fbshipit-source-id: 243576195ed201d6b82fc942a785742c6b01cf83
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5531
Last diff in series to migrate to a new interface. More details in diff: D14405326 or PR #5523
There is a changes to remove old interface and a bit more migration (things was landed in a middle).
Reviewed By: guliashvili
Differential Revision: D14477768
fbshipit-source-id: 572791cb6929edf46d5a8bfe9416e03dc85eb96f
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5521
Implementation based on `std::unordered_map` and `std::list`, without any
age-bits or any sort of optimisations. If later we realize we need something
more powerfull we will use some fancy OSS lib for it.
I need it as a cache of "hot" processes in syscall traicing. To read cmdline
from file in `/proc/<pid>/cmdline` less often and preserve cmdline or any other
info about process for the events that came when process is already gone.
Reviewed By: SAlexandru
Differential Revision: D14424352
fbshipit-source-id: 472cf8b542bab2921393b9d2a126c254c791404a
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5520
I need it to add cmdline attribute to the syscall tracing events. I think it
could be useful somewhere else in osquery (for instance in implementation of
table `processess`), so I put it under `osquery/utils/system/linux`.
Reviewed By: SAlexandru
Differential Revision: D14421472
fbshipit-source-id: 384a38f74b629d3622c48085bd3679174b70b2b4
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5527
for the sake of xCode, because it doesn't tolerate duplicated target names
Reviewed By: marekcirkos
Differential Revision: D14455350
fbshipit-source-id: c6fa8bfde87855d473884c2aa0bd664929e4757b
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5524
So let's get rid of the rest usecases of createError with message as an argument in order to remove it completely.
Reviewed By: jessek
Differential Revision: D14437933
fbshipit-source-id: acfb1a2a2a79e03d7cb650b305ee8ad3b6218d2a
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5518
method count of `std::chrono::duration::count` is not a constexpr so that means `kPollTimeout` could not be constexpr either. Let's make it just const member of the class PerfOutputPoll.
Reviewed By: jessek
Differential Revision: D14406162
fbshipit-source-id: c09a701206937e7a4bc9aeae8ef63495188dced1
Summary:
There is unnecessary redundancy of how error message now is created.
It could be with string argument:
return createError(ConversionError::InvalidArgument,
"Wrong string representation of boolean ",
prop_exp.takeError())
<< boost::io::quoted(from);
And it could be without it:
return createError(ConversionError::InvalidArgument, prop_exp.takeError())
<< "Wrong string representation of boolean "
<< boost::io::quoted(from);
So, my suggestion is to make it uniform - use only second option and form error
string only with operator<<. This diff introduce function overload without
message within argument list and some usage examples. If everyone ok with it,
I'll remove all cases of 3 args form usage in next diff.
Reviewed By: mkareta
Differential Revision: D14405326
fbshipit-source-id: 47074de1f078e512c95518439557adadcf83cbd3
Summary:
Example of use case for this close function that may fail, send network request or something similar,
but you invoke this function on end of app lifetime, so there it not make sense to handle error and you want to treat call as oneway
Reviewed By: marekcirkos
Differential Revision: D14365081
fbshipit-source-id: 5a1067c6b874d62c283b01f1c936ac8de67719ca
Summary:
Add log_numerics_as_numbers flag.
Internal SQLite query has method that returns QueryDataTyped (instead of QueryData), which are boost::variant<string, double, long long>.
Ints are encoded as such if and only if new log_numerics_as_numbers flag is set to true.
Reviewed By: fmanco
Differential Revision: D13778323
fbshipit-source-id: 7d7bb31781486f63fcc088cd479d3b6f255a5cb4
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5483
Initial steps to separate plugins from the rest of osquery. On the long run separating plugins will provide more build flexibility such that we can have configurable builds that include only the bits and pieces we actually need per deployment. Reducing the attack surface, possibility of supply chain attacks, binary size, etc.
Move killswitch
Reviewed By: guliashvili
Differential Revision: D14259760
fbshipit-source-id: deaaa148fac25b3f534a8881a1ea9eda0ec6bdc9
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5487
I got tired of Lint telling me to update the calls to the Status class, so I wrote some codemods to update the vast majority of them:
$ cd ~/fbsource
$ codemod -d xplat/osquery/oss/osquery --extensions cpp,h "return Status\(0\, \".*\"\);" "return Status::success();"
**Blindly accepted all changes**.
FWIW, I tried to do something similar with the failure return values, but the unit tests were failing afterward.
Reviewed By: guliashvili
Differential Revision: D14278739
fbshipit-source-id: b023f6e969be697eeb67d66ea3ad10d16c69ff0c
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5478
This makes it easier to update the osquery version and simplifies cxx.bzl by removing osquery specific preprocessor flags.
This will also make rebuilding osquery after changing versions faster, since the flags are now only defined for the headers which need them.
Reviewed By: akindyakov
Differential Revision: D14183142
fbshipit-source-id: 396d550f5b35a1d294fee802d2364cd9f7ab1d7a
Summary: To be able to build with different `thrift_impl` modules in extensions (like facebook does with fbthrift) I removed implementation from `extensions` lib and explicitly used for `osqueryd` and for all tests. This allows us to remove `thrift_impl` from sdk dependencies list. If you need it - please use explicitly as I did for `osqueryd`. If you want to use your own implementation of `thrift_impl` - build it and link against it and sdk simultaneously.
Reviewed By: marekcirkos
Differential Revision: D14224206
fbshipit-source-id: 218dc05d87e1ffaca37783185fc672a20684d757
Summary:
This fixes a regression introduced in D13586036 .
The tailer downstream sometimes determines type by trying to parse the string as an int. We want this to consistently fail to do so when the value is in fact a float.
Reviewed By: SAlexandru
Differential Revision: D14161125
fbshipit-source-id: 8e2bb9c59559a5774813a7c7777257742f885c35
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5452
As suggested in another diff, this diff updates the language we use to describe the osquery licensing terms. We are changing all instances of
//This source code is licensed as defined on the LICENSE file found in the root directory of this source tree.//
to
//This source code is licensed in accordance with the terms specified in the LICENSE file found in the root directory of this source tree.//
We accomplish this with a codemod:
$ codemod -md xplat/osquery/oss --extensions cpp,h,in,py,sh,mm,ps1 "(.\s+)This source code is licensed as defined on the LICENSE file found in the(.*)root directory of this source tree\." "\1This source code is licensed in accordance with the terms specified in\2the LICENSE file found in the root directory of this source tree."
Reviewed By: fmanco
Differential Revision: D14131290
fbshipit-source-id: 52c90da342263e2a80f5a678ecd760c19cf7513e
Summary:
Initial steps to separate plugins from the rest of osquery. On the long run separating plugins will provide more build flexibility such that we can have configurable builds that include only the bits and pieces we actually need per deployment. Reducing the attack surface, possibility of supply chain attacks, binary size, etc.
Move config and config_parser plugins
Reviewed By: marekcirkos
Differential Revision: D14119102
fbshipit-source-id: 0bc956398b3829c6f1013b38ebba2f0fc1071a93
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5408
Move out c++ enums to std::string conversion function from
osquery/utils/error.h to separate module. To be able to use it somewhere else.
Reviewed By: guliashvili
Differential Revision: D13896772
fbshipit-source-id: 0a9f6327d5b2f115ce688446a67677879411eb1f
