Add a way to compile third-party libraries from source instead of downloading prebuilt ones.
Each library source code is downloaded with git into a submodule at configure time,
in response to the find_package(library_name) CMake call,
except for OpenSSL where the official source archive is used.
Each submodule is attached to a release tag on its own upstream repository.
All the libraries are built using CMake directly, except for OpenSSL which uses a formula system,
which permits to build libraries with a separate build system
when there's no easy way to integrate it directly with CMake.
This new dependency system determines which library is fetched from where using the concept of "layers".
Currently we have three of them: source, formula, facebook,
where the last layer represents the pre-built libraries.
The provided order will be used when looking for libraries.
A system to patch submodule source code has been added and it's currently used with googletest, libudev and util-linux.
Patches should be put under libraries/cmake/source/<library name>/patches/<submodule>,
where <submodule> is often one and is "src", but in other cases, like AWS,
there are multiple with a more specific name.
If for whatever reason the submodule cloning or the patching fails,
the submodule has to be unregistered and its folder should be cleared.
This should be achievable with "git submodule deinit -f <submodule path>"
Following some other changes on existing functionality:
- Changed the CMake variable BUILD_TESTING to OSQUERY_BUILD_TESTS
to avoid enabling tests on third party libraries.
Due to an issue with glog the BUILD_TESTING variable
will be always forced to OFF.
- Moved compiler and linker flags to their own file cmake/flags.cmake
- Moved all the third-party CMakeLists.txt used for pre-built libraries under libraries/cmake/facebook
- Added the --exclude-folders option to tools/format-check.py and tools/git-clang-format.py,
so that it's possible to ignore any third party library source code.
- The format and format_check target use the new --exclude-folders option
to exclude libraries/cmake/source from formatting.
- The test and osquery binaries are properly compiled with PIE (osquery/osquery#5611)
Co-authored-by: Stefano Bonicatti <stefano.bonicatti@gmail.com>
Co-authored-by: Teddy Reed <teddy@casualhacking.io>
Taken from osql-experimental.
- Change CMake code license to the one present in osquery right now
- Package metadata doesn't mention Trail of Bits or osql anymore
- Set specific ACLs for the osqueryd on Windows when packaging
- Remove LLVM_INSTALL_PATH support on macOS, since we are using AppleClang
- Remove OSQUERY_SOURCE_DIR variable need and source in a submodule support
- Add targets format_check and format to check code formatting and
format it with clang-format
- Do not warn about not using Clang on macOS when using AppleClang
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5542
We replace deprecated calls to the Status class with newer ones. This will stop Lint from nagging me every time I open these files. There should be no change to functionality.
Specifically, we use a codemod on `.cpp` files to replace any instance of `return Status();` with `return Status::success();`.
Reviewed By: guliashvili
Differential Revision: D14652922
fbshipit-source-id: 243576195ed201d6b82fc942a785742c6b01cf83
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5524
So let's get rid of the rest usecases of createError with message as an argument in order to remove it completely.
Reviewed By: jessek
Differential Revision: D14437933
fbshipit-source-id: acfb1a2a2a79e03d7cb650b305ee8ad3b6218d2a
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5485
Initial steps to separate plugins from the rest of osquery. On the long run separating plugins will provide more build flexibility such that we can have configurable builds that include only the bits and pieces we actually ne
er deployment. Reducing the attack surface, possibility of supply chain attacks, binary size, etc.
Move numeric monitoring
Reviewed By: guliashvili
Differential Revision: D14259758
fbshipit-source-id: 1016fc28a0052417d658b6ce1cb3368e56597a7b
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5483
Initial steps to separate plugins from the rest of osquery. On the long run separating plugins will provide more build flexibility such that we can have configurable builds that include only the bits and pieces we actually need per deployment. Reducing the attack surface, possibility of supply chain attacks, binary size, etc.
Move killswitch
Reviewed By: guliashvili
Differential Revision: D14259760
fbshipit-source-id: deaaa148fac25b3f534a8881a1ea9eda0ec6bdc9
Summary: To be able to build with different `thrift_impl` modules in extensions (like facebook does with fbthrift) I removed implementation from `extensions` lib and explicitly used for `osqueryd` and for all tests. This allows us to remove `thrift_impl` from sdk dependencies list. If you need it - please use explicitly as I did for `osqueryd`. If you want to use your own implementation of `thrift_impl` - build it and link against it and sdk simultaneously.
Reviewed By: marekcirkos
Differential Revision: D14224206
fbshipit-source-id: 218dc05d87e1ffaca37783185fc672a20684d757
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5452
As suggested in another diff, this diff updates the language we use to describe the osquery licensing terms. We are changing all instances of
//This source code is licensed as defined on the LICENSE file found in the root directory of this source tree.//
to
//This source code is licensed in accordance with the terms specified in the LICENSE file found in the root directory of this source tree.//
We accomplish this with a codemod:
$ codemod -md xplat/osquery/oss --extensions cpp,h,in,py,sh,mm,ps1 "(.\s+)This source code is licensed as defined on the LICENSE file found in the(.*)root directory of this source tree\." "\1This source code is licensed in accordance with the terms specified in\2the LICENSE file found in the root directory of this source tree."
Reviewed By: fmanco
Differential Revision: D14131290
fbshipit-source-id: 52c90da342263e2a80f5a678ecd760c19cf7513e
Summary:
Initial steps to separate plugins from the rest of osquery. On the long run separating plugins will provide more build flexibility such that we can have configurable builds that include only the bits and pieces we actually need per deployment. Reducing the attack surface, possibility of supply chain attacks, binary size, etc.
Move config and config_parser plugins
Reviewed By: marekcirkos
Differential Revision: D14119102
fbshipit-source-id: 0bc956398b3829c6f1013b38ebba2f0fc1071a93
Summary: For some monitoring, we need to include machine identity. Two example, I found is hostname or similar hostnames schema name.
Reviewed By: SAlexandru
Differential Revision: D13880705
fbshipit-source-id: e1d0238f4981adad1554d73f0ef6e5ef65a98c33
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5375
LICENSE is now defined in a single file on the root of the project, update the
header to contain that information.
**Project LICENSE did not change.**
Reviewed By: akindyakov
Differential Revision: D13750575
fbshipit-source-id: 1e608a81b260b8395f9d008fc67f463160c1fc2b
Summary: Some tools does not support virtual headers namespaces and expect header path to match actual file path from project root. This diff will fix few namespaces in utils library
Reviewed By: guliashvili
Differential Revision: D13552878
fbshipit-source-id: 2a06f73550c69777bf73be73abdde297fe580583
Pre-aggregation cache implementation for numeric monitoring plugins.
For the most of monitoring data some aggregations are going to be applied on the user side. To do analytics you rarely need to see all points on the graph. It means for such paths (unique keys for points subsets) particular points means not much. And to reduce a disk usage and a network traffic some pre-aggreagation could be applied on osquery side.
This PR is implementation of such pre-aggreagtion.
It based of PR #4626
Just an interface and simple implementation dumping points to file on disk.
And I add also few monitoring records to some places of osquery code as an example.
Brief
Just an interface and simple implementation dumping points to file on disk.
And I add also few monitoring records to some places of osquery code as an example.
Motivation
osquery can monitor system health. But at some point we need to monitor the condition of osquery itself. Vast majority of interesting parameters can be represented by
numbers. How many queries it runs, how long does each query takes, what is the performance hit of each query, how long was last downtime and so on and so far. For obviou
s reason it hard to measure most of this parameters by external instrument. And it is almost impossible to evaluate it on production. But we can do it from inside of osquery.
What this PR is for
The systems like graphite or RRDtool can store and plot time-series data for us. We just have to
be able to feed data to it. We can create different plugins to be able to send data to different instruments. And we need some proper internal interface to all potential plugins. This PR is attempt to create generic interface.
Interface description
The most systems accept data as sequences of 2-dimensional points. One of the dimensions is value, the other is time. Each particular sequence has unique key, to be distinguished from the others.
Data descriptions for carbon. I have used this three parameters as an attributes of one monitoring point.
To send one point from some particular place in the code you just need to call the function record from namespace monitoring declared in the file include/osquery/num eric_monitoring.h with 3 arguments (path, value, time). Where path is the unique key of sequence; value is some interesting value to watch; time is the time of the point (can be omitted, current system time is the default vaule).
