- Add the possibility of running table logic inside a container
namespace, so that's possible to query it instead of the host.
Needs minor modifications to each table logic and how they use logging.
In practice it works by having a pid_with_namespace column, which should
contain pids that are in the same mount namespace of the container one
wants to query.
The worker receives that column as a constraint, prepares two unnamed
pipes for read/write communications with the future child, then forks
into a new process.
While the parent sends a query job to the just created child and then waits
for results, the child receives the job, takes all the values given in the
pid_with_namespace constraint, retrieves the fd of the mount namespace
under "/proc/<constraint pid>/ns/mnt", then switches to it.
Finally it runs the table logic, sending the results back to the parent
through the pipe with a JSON message.
Important to note that the logging in the table logic is not GLOG
directly, because in the child this is in an unknown state; a custom
logging system that resembles glog and that takes advantage of the
existing communication channel is used to send the messages in JSON
format to the parent, which will take care to forward to GLOG.
- Add FLAGS_keep_container_worker_open so that the process used for
accessing a container is kept open, until the queries are for the same
table; when the table changes, the process will be closed
and a new one created.
This is off by default, which means that a new process will be always
created.
- Implemented a way to run tests that require root separated
from the others.
The OSQUERY_BUILD_ROOT_TESTS has been added to requests such tests to
be built.
To run only tests which require a normal user, one has to use
`ctest -LE "root-required"`, while `sudo ctest -L "root-required"`
to run those who need root.
PR: osquery/osquery#6209
Remove a level of indirection when configuring and building formulas.
This should simplify working with them and also remove some issues
encountered when trying to build on Windows.
This new toolchain contains a newer LLVM version (9.0.1),
a fix for the scan-build scripts and it keeps the LLVM static libraries,
necessary to implement the new BPF framework and tables.
In the past the Windows agent changed the path where Python2
was installed; a special logic was added which should've tested
if the path existed, though it wasn't correct in the case
the powershell script is configured to be aborted at the first error.
Since the old path should not be present anymore,
we simply remove the logic and use the path we expect to exist.
With the increasing size of the build and the respective ccache
and sccache caches, the disk space sometimes is not enough
and the build fails.
This deletes the build folder as the last step since it shouldn't
be necessary anymore.
When the batch script that implements the build step has been
changed to stop the sccache server as the last command,
all build failures started to be ignored because the last command,
always succeeding, was clearing out the exit status.
Batch scripts do not have a global "exit on error" option,
so manually checking the error level and exiting with such error is needed.
* Add sccache to the Windows job
Also fix debug symbols on googletest,
so that it doesn't try to create a separate pdb.
* Fix uploading the submodule cache
sccache needs its cache to be uploaded as normal files,
not with a tar, otherwise the cache when extracted
will contain extraneous files and folders (PaxHeader).
The Files mode though by default ignores the .git folder,
so we have to use the .artifactignore file to unignore it,
and we have to copy in the cached path for the cache task to see it.
This will speed up the CMake configure phase since
it won't have to download the submodules data each time
through git, which is slower than downloading it
as an Azure Pipeline cache.
git on Windows supports cloning a repository with symlinks
without actually creating them; it will substitute them
with text files containing the target path.
This can potentially break the build of some of osquery
third party libraries.
Moreover this is the default on the CI,
which is different compared to the setup on local machines,
which means that it can hide other kind of issues.
* Fix packaging regression on Windows
Component variables should be set only for Linux and macOS,
otherwise on Windows the internal WiX IDs change and the
WiX fragment patch won't work.
Cleanup the version patch component, since WiX doesn't support
version components that are not digits.
* Add the rpm package to the CI Dockerfile, to test rpm packaging
* Fix TGZ generation on other platforms
* Add dpkg-dev and file packages to the CI Dockerfile
They are needed for creating DEB debug packages.
* Add a packaging steps to the CI
Set TGZ as a default packaging system when one cannot detected.
Fix generatePackageTarget indentation.
The test modifies the source code, appending "\n\n" at the end of each .cpp files.
Then runs the format_check target and verifies if the list of files to be formatted,
identified by the script, are the same ones previously modified.
- Always link to libc++abi.a, dl and rt.
- Add OSQUERY_TOOLCHAIN_SYSROOT option which should contain
the path to the sysroot where the portable compiler and its libraries are in.
- Fix OpenSSL build with custom toolchain.
- Always include the custom toolchain cmake.
Unfortunately system name detection is done when project() is called
which is also when compiler detection is done, and we need the compiler
to be set before that, so we always include the cmake file.
- Do not use getrandom syscall in Boost, for glibc < 2.25 support.
- Remove usage of secure_getenv and getauxval in librpm.
- Update CI to use the toolchain.
- Reflect changes in the docs.
Use the new CacheBeta task to cache and restore the ccache
directory, which greatly improves build times.
The cache is saved only if the job ends with success.
A pipeline variable CacheVersion present in the cache key
is used to invalidate all old caches if such a need arises.
Add a way to compile third-party libraries from source instead of downloading prebuilt ones.
Each library source code is downloaded with git into a submodule at configure time,
in response to the find_package(library_name) CMake call,
except for OpenSSL where the official source archive is used.
Each submodule is attached to a release tag on its own upstream repository.
All the libraries are built using CMake directly, except for OpenSSL which uses a formula system,
which permits to build libraries with a separate build system
when there's no easy way to integrate it directly with CMake.
This new dependency system determines which library is fetched from where using the concept of "layers".
Currently we have three of them: source, formula, facebook,
where the last layer represents the pre-built libraries.
The provided order will be used when looking for libraries.
A system to patch submodule source code has been added and it's currently used with googletest, libudev and util-linux.
Patches should be put under libraries/cmake/source/<library name>/patches/<submodule>,
where <submodule> is often one and is "src", but in other cases, like AWS,
there are multiple with a more specific name.
If for whatever reason the submodule cloning or the patching fails,
the submodule has to be unregistered and its folder should be cleared.
This should be achievable with "git submodule deinit -f <submodule path>"
Following some other changes on existing functionality:
- Changed the CMake variable BUILD_TESTING to OSQUERY_BUILD_TESTS
to avoid enabling tests on third party libraries.
Due to an issue with glog the BUILD_TESTING variable
will be always forced to OFF.
- Moved compiler and linker flags to their own file cmake/flags.cmake
- Moved all the third-party CMakeLists.txt used for pre-built libraries under libraries/cmake/facebook
- Added the --exclude-folders option to tools/format-check.py and tools/git-clang-format.py,
so that it's possible to ignore any third party library source code.
- The format and format_check target use the new --exclude-folders option
to exclude libraries/cmake/source from formatting.
- The test and osquery binaries are properly compiled with PIE (osquery/osquery#5611)
Co-authored-by: Stefano Bonicatti <stefano.bonicatti@gmail.com>
Co-authored-by: Teddy Reed <teddy@casualhacking.io>
A new Windows VM has been recently deployed that moves the
Python 2 installation, Windows Buck builds are now failing.
Support the old and new path for Python 2 and
support multiple minor versions of python instead of hardcoding one.
* Add an option to avoid generating debug symbols
A new option, OSQUERY_NO_DEBUG_SYMBOLS, has been added
to avoid generating debug symbols when building in Debug or RelWithDebInfo.
This is especially needed on the CI where the disk size is limited
and symbols are not necessary, because we cannot directly access
processes to debug them.
* Enable Debug builds on the CI
Enabled Debug builds on macOS and Linux; a special job has been created
to represent the combined build status of Release and Debug builds.
Also format the other long cmakeArgs line so that there's one argument
per line.
* Run Docker as privileged also in Debug builds
* Simplify azure pipelines configuration
Use strategy and matrix to avoid repeating the script
for Release and Debug builds.
Fix ebpfTests.sysEbpf_null_attr, ebpfTests.sysEbpf_create_map,
ebpfMapTests.int_key_int_value, ebpfMapTests.int_key_struct_value
by running Docker as privileged on Azure Pipelines.
Docker is used only to get a new distribution running, it's not used
for any security purpose, so there's no point in limiting it.
Apparently there's a bug in the visualization of the logs,
where color codes eat characters.
So we disable tests colored output until a fix on Azure Pipelines appears.
Formatting a bit more consistently the yaml file.
Originally the separation existed because there were multiple branches,
and only one of them was protected by PRs.
So broken commits could land and differentiating from broken build
or tests was useful.
This is not true anymore and PRs checks are per pipeline, not per job,
so the separation wouldn't make a difference.
