The windows documentation mentions that ExitThread should not be
used in C++ code. When debugging these exit flows I saw exceptions
when running dtors.
When reviewing the shutdown code for Windows and POSIX there was a
lot of complexity. This change allows any part of code to request
a shutdown and simplifies the response.
We should expect the main thread to call ::waitForShutdown.
The exception is the shell, which waits in a repl.
This was determined to be the wrong approach to adding simple
killswitches. The intent was to quickly flip on/off features. It was
not widely adopted due to the dependencies killswitching has.
A different approach is to implement the same functions with something
simple like filesystem flags.
When a thread different from the main requests a shutdown
through Initializer::requestShutdown, it should not call
waitForShutdown; there's no reason to wait, moreover the function
doesn't only wait, but also actually stops other components and then
finally calls exit().
Since the main thread is already inside the waitForShutdown call
waiting on Dispatcher::joinServices or inside the shutdown() callable
on Windows, having a secondary thread do
the same work potentially at the same time is wrong.
Moreover calling exit() from a secondary thread is most of the time
incorrect.
The waitForShutdown function has been renamed to waitThenShutdown
to better represent what it's actually doing.
This commit creates a new osquery/main/harnesses directory and moves fuzz-config there.
It removes OSQUERY_FUZZ for a new option OSQUERY_ENABLE_FUZZER_SANITIZERS
It creates a new option OSQUERY_ENABLE_ADDRESS_SANITIZER
The following behaviors are intended:
OSQUERY_BUILD_TESTS=ON will build the fuzzing harness. This configuration is not
intended for fuzzing purposes. OSQUERY_ENABLE_FUZZER_SANITIZERS=ON will also build
the fuzzing harness. However if this variable is true, it also requires
OSQUERY_ENABLE_ADDRESS_SANITIZER=ON and either CMAKE_BUILD_TYPE=Release or
RelWithDebInfo This configuration is actually intended for fuzzing.
This defines a new CMake option: OSQUERY_FUZZ, which appends compiler
and linker flags for libFuzzer to be effective.
There is one example harness that will fuzz config parsing.
Add a way to compile third-party libraries from source instead of downloading prebuilt ones.
Each library source code is downloaded with git into a submodule at configure time,
in response to the find_package(library_name) CMake call,
except for OpenSSL where the official source archive is used.
Each submodule is attached to a release tag on its own upstream repository.
All the libraries are built using CMake directly, except for OpenSSL which uses a formula system,
which permits to build libraries with a separate build system
when there's no easy way to integrate it directly with CMake.
This new dependency system determines which library is fetched from where using the concept of "layers".
Currently we have three of them: source, formula, facebook,
where the last layer represents the pre-built libraries.
The provided order will be used when looking for libraries.
A system to patch submodule source code has been added and it's currently used with googletest, libudev and util-linux.
Patches should be put under libraries/cmake/source/<library name>/patches/<submodule>,
where <submodule> is often one and is "src", but in other cases, like AWS,
there are multiple with a more specific name.
If for whatever reason the submodule cloning or the patching fails,
the submodule has to be unregistered and its folder should be cleared.
This should be achievable with "git submodule deinit -f <submodule path>"
Following some other changes on existing functionality:
- Changed the CMake variable BUILD_TESTING to OSQUERY_BUILD_TESTS
to avoid enabling tests on third party libraries.
Due to an issue with glog the BUILD_TESTING variable
will be always forced to OFF.
- Moved compiler and linker flags to their own file cmake/flags.cmake
- Moved all the third-party CMakeLists.txt used for pre-built libraries under libraries/cmake/facebook
- Added the --exclude-folders option to tools/format-check.py and tools/git-clang-format.py,
so that it's possible to ignore any third party library source code.
- The format and format_check target use the new --exclude-folders option
to exclude libraries/cmake/source from formatting.
- The test and osquery binaries are properly compiled with PIE (osquery/osquery#5611)
Co-authored-by: Stefano Bonicatti <stefano.bonicatti@gmail.com>
Co-authored-by: Teddy Reed <teddy@casualhacking.io>
Taken from osql-experimental.
- Change CMake code license to the one present in osquery right now
- Package metadata doesn't mention Trail of Bits or osql anymore
- Set specific ACLs for the osqueryd on Windows when packaging
- Remove LLVM_INSTALL_PATH support on macOS, since we are using AppleClang
- Remove OSQUERY_SOURCE_DIR variable need and source in a submodule support
- Add targets format_check and format to check code formatting and
format it with clang-format
- Do not warn about not using Clang on macOS when using AppleClang
Summary:
osquery returned status 1(error) always when osqueryd service install uninstall was requested by the flags.
Even if install/uninstall was successfull.
Add extra logging in case of success.
Exit with status 0 when successfully installed/uninstalled
Reviewed By: jessek
Differential Revision: D14622771
fbshipit-source-id: c0c7569a2b13434e2ad28ba92cba308024b044ec
Summary:
User could have requested install and uninstall of osqueryd service together. However it does not make sense and actually osquery would only install the service.
This additional check, will exit osquery in case when install and uninstall are requested together.
Reviewed By: jessek
Differential Revision: D14622634
fbshipit-source-id: b88dc519bd1ef38653eb1660712bbf2e75bad36f
Summary:
Replaced fprint(stderr with std::cerr
Other than that, could not find any use of
```
```
Reviewed By: akindyakov, fmanco
Differential Revision: D14621228
fbshipit-source-id: e996b2507757dc6af710e7ab0e25fc8136bbb92e
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5519
This is very simple usage of system calls {kill, setuid} tracing. Enable it if cli flag `--enable_experimental_tracing` is specified.
All received events from the kernel will be serialised to JSON string and sent to experimental events streaming registry. Without any preprocessing or filtering.
I'm going to use this prototype to estimate performance hit of this subsystem in small fraction of our deployment.
Reviewed By: marekcirkos
Differential Revision: D14406173
fbshipit-source-id: 5ee7bd296216ad83f564aaca671a6db46739aaa7
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5517
It is experimental registry to stream events from osquery. It is separated from logger registry, which is semantically very similar, on purpose.
Firstly, because we are still not sure about exact way to process, filter, deliver events. I think the best way to test things is to implement pipeline for the events separately from the logger for now and may be merge them later. It's easier to merge things than split them.
Secondary, current implementation of logger is not very performant with plenty of additional functionality. Since we going to evaluate performance hit of new type of events (linux syscall tracing) it is not a good idea to connect it to sophisticated system with well known performance problems.
Reviewed By: jessek
Differential Revision: D14404665
fbshipit-source-id: bb98848d998669bd90bd4f5cf61981247bd537d6
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5485
Initial steps to separate plugins from the rest of osquery. On the long run separating plugins will provide more build flexibility such that we can have configurable builds that include only the bits and pieces we actually ne
er deployment. Reducing the attack surface, possibility of supply chain attacks, binary size, etc.
Move numeric monitoring
Reviewed By: guliashvili
Differential Revision: D14259758
fbshipit-source-id: 1016fc28a0052417d658b6ce1cb3368e56597a7b
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5483
Initial steps to separate plugins from the rest of osquery. On the long run separating plugins will provide more build flexibility such that we can have configurable builds that include only the bits and pieces we actually need per deployment. Reducing the attack surface, possibility of supply chain attacks, binary size, etc.
Move killswitch
Reviewed By: guliashvili
Differential Revision: D14259760
fbshipit-source-id: deaaa148fac25b3f534a8881a1ea9eda0ec6bdc9
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5475
osquery build doesn't use folly for now so this facebook specific routine is dead and should be removed or moved to another place.
Reviewed By: fmanco, guliashvili
Differential Revision: D14242160
fbshipit-source-id: f2a8942906ddc1d7d9bb47395b1081ad2e88712d
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5474
To be able to exclude config:update plugin from osquery plugin sdk.
Reviewed By: guliashvili
Differential Revision: D14241686
fbshipit-source-id: e378718b1e79e3dbe314a676945c18d15d054a3d
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5473
to be able to exclude dependency on enroll:tls plugin from plugins_sdk.
Only plugin registration was moved by now. The actual plugin code will be moved a bit later.
Reviewed By: guliashvili
Differential Revision: D14241687
fbshipit-source-id: 814d52719ff27bd1283bcaecdc912fca3830ff5e
Summary: To be able to build with different `thrift_impl` modules in extensions (like facebook does with fbthrift) I removed implementation from `extensions` lib and explicitly used for `osqueryd` and for all tests. This allows us to remove `thrift_impl` from sdk dependencies list. If you need it - please use explicitly as I did for `osqueryd`. If you want to use your own implementation of `thrift_impl` - build it and link against it and sdk simultaneously.
Reviewed By: marekcirkos
Differential Revision: D14224206
fbshipit-source-id: 218dc05d87e1ffaca37783185fc672a20684d757
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5452
As suggested in another diff, this diff updates the language we use to describe the osquery licensing terms. We are changing all instances of
//This source code is licensed as defined on the LICENSE file found in the root directory of this source tree.//
to
//This source code is licensed in accordance with the terms specified in the LICENSE file found in the root directory of this source tree.//
We accomplish this with a codemod:
$ codemod -md xplat/osquery/oss --extensions cpp,h,in,py,sh,mm,ps1 "(.\s+)This source code is licensed as defined on the LICENSE file found in the(.*)root directory of this source tree\." "\1This source code is licensed in accordance with the terms specified in\2the LICENSE file found in the root directory of this source tree."
Reviewed By: fmanco
Differential Revision: D14131290
fbshipit-source-id: 52c90da342263e2a80f5a678ecd760c19cf7513e
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5449
Initial steps to separate plugins from the rest of osquery. On the long run separating plugins will provide more build flexibility such that we can have configurable builds that include only the bits and pieces we actually need per deployment. Reducing the attack surface, possibility of supply chain attacks, binary size, etc.
Move killswitch
Move test declaration to it's own BUCK file for consistency with the rest of the project.
Reviewed By: marekcirkos
Differential Revision: D14121618
fbshipit-source-id: 3e30e57befed4387585ed553ec087fdf8db6efc3
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5448
Initial steps to separate plugins from the rest of osquery. On the long run separating plugins will provide more build flexibility such that we can have configurable builds that include only the bits and pieces we actually need per deployment. Reducing the attack surface, possibility of supply chain attacks, binary size, etc.
Move logger
Reviewed By: marekcirkos
Differential Revision: D14121620
fbshipit-source-id: cef15e7cc354cbe597c6c6878ee63ff09b5fb06d
Summary:
Initial steps to separate plugins from the rest of osquery. On the long run separating plugins will provide more build flexibility such that we can have configurable builds that include only the bits and pieces we actually need per deployment. Reducing the attack surface, possibility of supply chain attacks, binary size, etc.
Move distributed
Reviewed By: marekcirkos
Differential Revision: D14121619
fbshipit-source-id: 9ad8a837450874e79a819ab4f11258ae24ec8014
Summary:
Initial steps to separate plugins from the rest of osquery. On the long run separating plugins will provide more build flexibility such that we can have configurable builds that include only the bits and pieces we actually need per deployment. Reducing the attack surface, possibility of supply chain attacks, binary size, etc.
Move config and config_parser plugins
Reviewed By: marekcirkos
Differential Revision: D14119102
fbshipit-source-id: 0bc956398b3829c6f1013b38ebba2f0fc1071a93
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5375
LICENSE is now defined in a single file on the root of the project, update the
header to contain that information.
**Project LICENSE did not change.**
Reviewed By: akindyakov
Differential Revision: D13750575
fbshipit-source-id: 1e608a81b260b8395f9d008fc67f463160c1fc2b
Summary:
While running `misspell` on a different codebase. I happened to notice that some misspellings in the osquery code base. So, I fixed them
Pull Request resolved: https://github.com/facebook/osquery/pull/5256
Reviewed By: guliashvili
Differential Revision: D13670897
Pulled By: fmanco
fbshipit-source-id: 5d33d858284955c376e8c3980acdf366d4edf3d3
Summary: Some tools does not support virtual headers namespaces and expect header path to match actual file path from project root. This diff will fix few namespaces in utils library
Reviewed By: guliashvili
Differential Revision: D13552878
fbshipit-source-id: 2a06f73550c69777bf73be73abdde297fe580583
