This was determined to be the wrong approach to adding simple
killswitches. The intent was to quickly flip on/off features. It was
not widely adopted due to the dependencies killswitching has.
A different approach is to implement the same functions with something
simple like filesystem flags.
Add a way to compile third-party libraries from source instead of downloading prebuilt ones.
Each library source code is downloaded with git into a submodule at configure time,
in response to the find_package(library_name) CMake call,
except for OpenSSL where the official source archive is used.
Each submodule is attached to a release tag on its own upstream repository.
All the libraries are built using CMake directly, except for OpenSSL which uses a formula system,
which permits to build libraries with a separate build system
when there's no easy way to integrate it directly with CMake.
This new dependency system determines which library is fetched from where using the concept of "layers".
Currently we have three of them: source, formula, facebook,
where the last layer represents the pre-built libraries.
The provided order will be used when looking for libraries.
A system to patch submodule source code has been added and it's currently used with googletest, libudev and util-linux.
Patches should be put under libraries/cmake/source/<library name>/patches/<submodule>,
where <submodule> is often one and is "src", but in other cases, like AWS,
there are multiple with a more specific name.
If for whatever reason the submodule cloning or the patching fails,
the submodule has to be unregistered and its folder should be cleared.
This should be achievable with "git submodule deinit -f <submodule path>"
Following some other changes on existing functionality:
- Changed the CMake variable BUILD_TESTING to OSQUERY_BUILD_TESTS
to avoid enabling tests on third party libraries.
Due to an issue with glog the BUILD_TESTING variable
will be always forced to OFF.
- Moved compiler and linker flags to their own file cmake/flags.cmake
- Moved all the third-party CMakeLists.txt used for pre-built libraries under libraries/cmake/facebook
- Added the --exclude-folders option to tools/format-check.py and tools/git-clang-format.py,
so that it's possible to ignore any third party library source code.
- The format and format_check target use the new --exclude-folders option
to exclude libraries/cmake/source from formatting.
- The test and osquery binaries are properly compiled with PIE (osquery/osquery#5611)
Co-authored-by: Stefano Bonicatti <stefano.bonicatti@gmail.com>
Co-authored-by: Teddy Reed <teddy@casualhacking.io>
Taken from osql-experimental.
- Change CMake code license to the one present in osquery right now
- Package metadata doesn't mention Trail of Bits or osql anymore
- Set specific ACLs for the osqueryd on Windows when packaging
- Remove LLVM_INSTALL_PATH support on macOS, since we are using AppleClang
- Remove OSQUERY_SOURCE_DIR variable need and source in a submodule support
- Add targets format_check and format to check code formatting and
format it with clang-format
- Do not warn about not using Clang on macOS when using AppleClang
Summary:
Add log_numerics_as_numbers flag.
Internal SQLite query has method that returns QueryDataTyped (instead of QueryData), which are boost::variant<string, double, long long>.
Ints are encoded as such if and only if new log_numerics_as_numbers flag is set to true.
Reviewed By: fmanco
Differential Revision: D13778323
fbshipit-source-id: 7d7bb31781486f63fcc088cd479d3b6f255a5cb4
Summary: Add query monitoring without specifying the scheme. It will help us in the simpler aggregation.
Reviewed By: marekcirkos
Differential Revision: D14280109
fbshipit-source-id: d70971c9ebe1e15c66f36e1490b7d8e198a68a7a
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5483
Initial steps to separate plugins from the rest of osquery. On the long run separating plugins will provide more build flexibility such that we can have configurable builds that include only the bits and pieces we actually need per deployment. Reducing the attack surface, possibility of supply chain attacks, binary size, etc.
Move killswitch
Reviewed By: guliashvili
Differential Revision: D14259760
fbshipit-source-id: deaaa148fac25b3f534a8881a1ea9eda0ec6bdc9
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5487
I got tired of Lint telling me to update the calls to the Status class, so I wrote some codemods to update the vast majority of them:
$ cd ~/fbsource
$ codemod -d xplat/osquery/oss/osquery --extensions cpp,h "return Status\(0\, \".*\"\);" "return Status::success();"
**Blindly accepted all changes**.
FWIW, I tried to do something similar with the failure return values, but the unit tests were failing afterward.
Reviewed By: guliashvili
Differential Revision: D14278739
fbshipit-source-id: b023f6e969be697eeb67d66ea3ad10d16c69ff0c
Summary: To be able to build with different `thrift_impl` modules in extensions (like facebook does with fbthrift) I removed implementation from `extensions` lib and explicitly used for `osqueryd` and for all tests. This allows us to remove `thrift_impl` from sdk dependencies list. If you need it - please use explicitly as I did for `osqueryd`. If you want to use your own implementation of `thrift_impl` - build it and link against it and sdk simultaneously.
Reviewed By: marekcirkos
Differential Revision: D14224206
fbshipit-source-id: 218dc05d87e1ffaca37783185fc672a20684d757
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5452
As suggested in another diff, this diff updates the language we use to describe the osquery licensing terms. We are changing all instances of
//This source code is licensed as defined on the LICENSE file found in the root directory of this source tree.//
to
//This source code is licensed in accordance with the terms specified in the LICENSE file found in the root directory of this source tree.//
We accomplish this with a codemod:
$ codemod -md xplat/osquery/oss --extensions cpp,h,in,py,sh,mm,ps1 "(.\s+)This source code is licensed as defined on the LICENSE file found in the(.*)root directory of this source tree\." "\1This source code is licensed in accordance with the terms specified in\2the LICENSE file found in the root directory of this source tree."
Reviewed By: fmanco
Differential Revision: D14131290
fbshipit-source-id: 52c90da342263e2a80f5a678ecd760c19cf7513e
Summary:
Initial steps to separate plugins from the rest of osquery. On the long run separating plugins will provide more build flexibility such that we can have configurable builds that include only the bits and pieces we actually need per deployment. Reducing the attack surface, possibility of supply chain attacks, binary size, etc.
Move config and config_parser plugins
Reviewed By: marekcirkos
Differential Revision: D14119102
fbshipit-source-id: 0bc956398b3829c6f1013b38ebba2f0fc1071a93
Summary: Removing flag which was declared but never used. enable_monitor
Reviewed By: marekcirkos
Differential Revision: D13958265
fbshipit-source-id: 3a812330950b101abdbd83ada4afd5b262cabd26
Summary: For some monitoring, we need to include machine identity. Two example, I found is hostname or similar hostnames schema name.
Reviewed By: SAlexandru
Differential Revision: D13880705
fbshipit-source-id: e1d0238f4981adad1554d73f0ef6e5ef65a98c33
Summary: before this diff osqueryd was relying on linker order to use right main function, since gtest also contains main function
Reviewed By: guliashvili
Differential Revision: D13897622
fbshipit-source-id: d260b7496f513c7052f4db87c8e7ff9300493671
Summary: profiler file contained the CodeProfiler class, which is a bit odd. Thanks alecx
Reviewed By: akindyakov
Differential Revision: D13861320
fbshipit-source-id: 11d711fd1d15830d4cce0be084cc6da3b1c6589c
Summary:
Split query name and pack name with the delimiter(dot) which should not accrue inside their names(instead of _ or -).
Also log things per pack name
Reviewed By: SAlexandru
Differential Revision: D13854471
fbshipit-source-id: 181e7e19fcfb5d57a779cea6a2804eda09dc5a91
Summary:
Split query name and pack name with the delimiter(dot) which should not accrue inside their names(instead of _ or -).
Also log things per pack name
Reviewed By: mkareta
Differential Revision: D13854389
fbshipit-source-id: 35ba80813d98371dd36a83ec32aad677f24aa6eb
Summary: Also it causes debug failure on unchecked Expected when status is not ok
Reviewed By: mkareta
Differential Revision: D13859377
fbshipit-source-id: 596410350cb91d469dc0a19f0e8eec558e8627bd
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5375
LICENSE is now defined in a single file on the root of the project, update the
header to contain that information.
**Project LICENSE did not change.**
Reviewed By: akindyakov
Differential Revision: D13750575
fbshipit-source-id: 1e608a81b260b8395f9d008fc67f463160c1fc2b
Summary: Some tools does not support virtual headers namespaces and expect header path to match actual file path from project root. This diff will fix few namespaces in utils library
Reviewed By: guliashvili
Differential Revision: D13552878
fbshipit-source-id: 2a06f73550c69777bf73be73abdde297fe580583
Summary: As discussed in [an old PR](37ffdf8a48 (r213278358)) this is of questionable benefit and will get complicated when we change to typed values, so dropping.
Reviewed By: guliashvili
Differential Revision: D13631227
fbshipit-source-id: ada9b5434297d8c1c4b3e3855fe595faf5937bf6
* dispatcher race conditions
dispatcher had 2 race condition.
In joinServices it was accessing service_threads_ with different lock(join_lock). However, if by that time new service was added baad things would happen :) .
Also dispatcher was accessing services_.size() without the lock. ( If by that time service was removed or joined bad things would happen)
* InterruptableRunnable RunnerInterruptPoint redesign
There were several inefficiencies in the old version of RunnerInterruptPoint and InterruptableRunnable.
1) RunnerInterruptPoint was throwing the exception when interrupted, however, the exception was always ignored.
2) InterruptableRunnable used the read-write lock, however only write lock was used.
3) InterruptableRunnable InterruptableRunnable, stored almost similar variable stop_, interrupted_.
4) std::atomic<bool> interrupted_ was used with locks, even though it was accessed by default safest access mode memory_order_seq_cst. So no additional cache invalidation was needed.
5) InterruptableRunnable contained code(in method interrupted() and variables bypass_check_, checked) just for testing. Which was slowing down method interrupted().
6) Some more confusing things. notify_all was not needed, as only one thread could be waiting for the conditional variable. RunnerInterruptPoint:: pause(void) looks ambiguous and that's why was not used anywhere.
I resolved all these problems by merging InterruptableRunnable and RunnerInterruptPoint into the InterruptableRunnable.
1) No use of the exception.
2) 4) Simple mutex, which is only used for pauseMilli. InterruptableRunnable::interrupted and InterruptableRunnable::interrupt function lock-free.
3) Single variable interrupted_.
5) Made InterruptableRunnable::interrupt virtual. Tests override interrupt to make things testable.
6) change to notify_one and removed pause without the specific time.
* Simple solution to reduce drift time in loop of scheduled queries #4301
based on measuring time of loop step and reduce sleep time on it.
* Change schedule time drift compensation policy
Sleep less time or do not sleep at all if there is non-zero accumulated drift.
Also new flag [schedule_max_drift] was added to make it configurable.
* Add test to check time drift accumulation and fix up code according review comments
