CMake 3.16 doesn't set Python3_EXECUTABLE as cache variable anymore,
it has changed to _Python3_EXECUTABLE which is for internal use.
The module returns Python3_EXECUTABLE as a local variable,
so we share that value through a new custom cache variable,
OSQUERY_PYTHON_EXECUTABLE.
Both libraries use the xxhash library,
compiling its source files directly.
The version they use though it's different so to avoid the linker
resolve the collision by removing one of the two implementations,
we prefix the functions with the respective library name.
Moreover we make the xxhash API private, by declaring the static
so that any unused function is not included anymore in the binary
and only the used one are present.
test_base.py received several fixes and now supports and requires
a custom folder where to find the config files that the tests need.
It has been also updated to use Python 3.
Fixed the gen_api function in genapi.py missing the path to the spec
file when evaluating blacklisted tables.
Co-Authored-By: SS <sahil.suneja@trailofbits.com>
This is needed by some third party libraries and
when using Ninja on Windows.
CMake complains that it cannot find the internal variable
for CMAKE_ASM_MASM_COMPILE_OBJECT if it's not active for the whole
project.
CMake doesn't support reading and therefore copying Windows symlinks.
A recent update to Thrift creates a symlink in its tutorial folder,
which makes the copy after patch process fail.
As a workaround we avoid copying the tutorial folder,
since we don't care about it anyway.
While in the end the osqueryd binary will require almost all libraries
built and linked, having each intermediate library depend on the whole
set of AWS libraries might result in higher compiling time.
Especially since on some platforms not all libraries are used.
This should speed up the compilation speed on Windows.
CMake utility functions(generateOsqueryExtensionGroup,addOsqueryExtensioni{Ex},
addOsqueryModule) to add extensions.
Updates the boost library submodules needed for extensions build.
Use cache variable(OSQUERY_EXTENSION_GROUP_NAME) for the grouping of extensions
and fix the linkage with osquery interface library.
Instead of moving a patched submodule from source to build directory,
we copy it and then hard reset the submodule to its original HEAD.
Minor message change when a library fails to be imported.
During the first configuration, the git command is searched
when submodules are initialized and it's then saved into a cache variable.
If though we delete the build folder and run configure again,
the submodules won't be initialized again, so the git command
won't be cached and visible to CMake.
Therefore we need to search for it each time.
Cleanup dead/deprecated code.
Keep patched source in the build directory instead of the source.
A new variable with the format OSQUERY_<submodule name>_ROOT_DIR
is set to the directory of where the submodule is, in case it's patched.
A new option PATCH has been added to importSourceModule
to let the patching system know that the submodule has to be patched
and which is the main submodule folder.
Mitigates recent CVEs in SQLite. It was already this version on Linux.
Establish a pattern for migrating source dependency builds to macOS and Windows.
- Always link to libc++abi.a, dl and rt.
- Add OSQUERY_TOOLCHAIN_SYSROOT option which should contain
the path to the sysroot where the portable compiler and its libraries are in.
- Fix OpenSSL build with custom toolchain.
- Always include the custom toolchain cmake.
Unfortunately system name detection is done when project() is called
which is also when compiler detection is done, and we need the compiler
to be set before that, so we always include the cmake file.
- Do not use getrandom syscall in Boost, for glibc < 2.25 support.
- Remove usage of secure_getenv and getauxval in librpm.
- Update CI to use the toolchain.
- Reflect changes in the docs.
Add a way to compile third-party libraries from source instead of downloading prebuilt ones.
Each library source code is downloaded with git into a submodule at configure time,
in response to the find_package(library_name) CMake call,
except for OpenSSL where the official source archive is used.
Each submodule is attached to a release tag on its own upstream repository.
All the libraries are built using CMake directly, except for OpenSSL which uses a formula system,
which permits to build libraries with a separate build system
when there's no easy way to integrate it directly with CMake.
This new dependency system determines which library is fetched from where using the concept of "layers".
Currently we have three of them: source, formula, facebook,
where the last layer represents the pre-built libraries.
The provided order will be used when looking for libraries.
A system to patch submodule source code has been added and it's currently used with googletest, libudev and util-linux.
Patches should be put under libraries/cmake/source/<library name>/patches/<submodule>,
where <submodule> is often one and is "src", but in other cases, like AWS,
there are multiple with a more specific name.
If for whatever reason the submodule cloning or the patching fails,
the submodule has to be unregistered and its folder should be cleared.
This should be achievable with "git submodule deinit -f <submodule path>"
Following some other changes on existing functionality:
- Changed the CMake variable BUILD_TESTING to OSQUERY_BUILD_TESTS
to avoid enabling tests on third party libraries.
Due to an issue with glog the BUILD_TESTING variable
will be always forced to OFF.
- Moved compiler and linker flags to their own file cmake/flags.cmake
- Moved all the third-party CMakeLists.txt used for pre-built libraries under libraries/cmake/facebook
- Added the --exclude-folders option to tools/format-check.py and tools/git-clang-format.py,
so that it's possible to ignore any third party library source code.
- The format and format_check target use the new --exclude-folders option
to exclude libraries/cmake/source from formatting.
- The test and osquery binaries are properly compiled with PIE (osquery/osquery#5611)
Co-authored-by: Stefano Bonicatti <stefano.bonicatti@gmail.com>
Co-authored-by: Teddy Reed <teddy@casualhacking.io>
