CMake utility functions(generateOsqueryExtensionGroup,addOsqueryExtensioni{Ex},
addOsqueryModule) to add extensions.
Updates the boost library submodules needed for extensions build.
Use cache variable(OSQUERY_EXTENSION_GROUP_NAME) for the grouping of extensions
and fix the linkage with osquery interface library.
* Fix packaging regression on Windows
Component variables should be set only for Linux and macOS,
otherwise on Windows the internal WiX IDs change and the
WiX fragment patch won't work.
Cleanup the version patch component, since WiX doesn't support
version components that are not digits.
* Add the rpm package to the CI Dockerfile, to test rpm packaging
* Fix TGZ generation on other platforms
* Add dpkg-dev and file packages to the CI Dockerfile
They are needed for creating DEB debug packages.
* Add a packaging steps to the CI
Set TGZ as a default packaging system when one cannot detected.
Fix generatePackageTarget indentation.
Since normally the tests are run on a more complete VM or physical host,
we create fake empty packages to install so that
the test DebPackages.test_sanity doesn't fail.
The test modifies the source code, appending "\n\n" at the end of each .cpp files.
Then runs the format_check target and verifies if the list of files to be formatted,
identified by the script, are the same ones previously modified.
Summary: This adds a few minor updates to the MSI build function. First
it keeps build artifacts located in `build/msi` for ease of access.
Further it updates component referneces to be x64 compatible. Finally it
updates a few path references to reflect the new repo layout, as well as
changes a few references for author and ownership information to be
osquery.
Test Plan: Will update with sample MSI builds for Cmake.
* deployment: Add Mozilla CA certificate store 2019-08-28
This is the cacerts file that was once proviced by the OpenSSL Homebrew
formula. The osquery packages in 3.x and previous provided this store as
the default / fallback.
The hash can be corroborated via:
https://curl.haxx.se/ca/cacert.pem.sha256
38b6230aa4bee062cd34ee0ff6da173250899642b1937fc130896290b6bd91e3
* install certs.pem directly from source
Add a way to compile third-party libraries from source instead of downloading prebuilt ones.
Each library source code is downloaded with git into a submodule at configure time,
in response to the find_package(library_name) CMake call,
except for OpenSSL where the official source archive is used.
Each submodule is attached to a release tag on its own upstream repository.
All the libraries are built using CMake directly, except for OpenSSL which uses a formula system,
which permits to build libraries with a separate build system
when there's no easy way to integrate it directly with CMake.
This new dependency system determines which library is fetched from where using the concept of "layers".
Currently we have three of them: source, formula, facebook,
where the last layer represents the pre-built libraries.
The provided order will be used when looking for libraries.
A system to patch submodule source code has been added and it's currently used with googletest, libudev and util-linux.
Patches should be put under libraries/cmake/source/<library name>/patches/<submodule>,
where <submodule> is often one and is "src", but in other cases, like AWS,
there are multiple with a more specific name.
If for whatever reason the submodule cloning or the patching fails,
the submodule has to be unregistered and its folder should be cleared.
This should be achievable with "git submodule deinit -f <submodule path>"
Following some other changes on existing functionality:
- Changed the CMake variable BUILD_TESTING to OSQUERY_BUILD_TESTS
to avoid enabling tests on third party libraries.
Due to an issue with glog the BUILD_TESTING variable
will be always forced to OFF.
- Moved compiler and linker flags to their own file cmake/flags.cmake
- Moved all the third-party CMakeLists.txt used for pre-built libraries under libraries/cmake/facebook
- Added the --exclude-folders option to tools/format-check.py and tools/git-clang-format.py,
so that it's possible to ignore any third party library source code.
- The format and format_check target use the new --exclude-folders option
to exclude libraries/cmake/source from formatting.
- The test and osquery binaries are properly compiled with PIE (osquery/osquery#5611)
Co-authored-by: Stefano Bonicatti <stefano.bonicatti@gmail.com>
Co-authored-by: Teddy Reed <teddy@casualhacking.io>
The Wix configuration sets the service as _critical_. As I understand it, this means failures to start osquery, are considered startup failures, and will trigger a reboot. As there are occasional windows bugs causing a failure to start, this seems unfortunate. I think hit someone on slack today
This PR changes that to _normal_
Docs are http://wixtoolset.org/documentation/manual/v3/xsd/wix/serviceinstall.html
Taken from osql-experimental.
- Change CMake code license to the one present in osquery right now
- Package metadata doesn't mention Trail of Bits or osql anymore
- Set specific ACLs for the osqueryd on Windows when packaging
- Remove LLVM_INSTALL_PATH support on macOS, since we are using AppleClang
- Remove OSQUERY_SOURCE_DIR variable need and source in a submodule support
- Add targets format_check and format to check code formatting and
format it with clang-format
- Do not warn about not using Clang on macOS when using AppleClang
Summary:
Iterating through a string is no longer allowed, and `range(0, len(_))` and `range(len(_))`
are equivalent. Switch to the shorter, more commonly used form
Reviewrs: #sentinel
Reviewed By: philipjameson
Differential Revision: D14657008
fbshipit-source-id: 1aabcbf168896bd0ee64b0d4eb17a72d6863aab2
Summary: Right now it blocks us, because build on macox doesn't work. Fix will take some time - lots of changes. But on the other hand value of such change in tests is nearly zero. So, let's just mute it.
Reviewed By: guliashvili
Differential Revision: D14597262
fbshipit-source-id: adaacc003f49647e255001bb84cc0e71273cd486
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5528
by using config option `cxx.filepath_length_limited=true`. Because unfortunately there is very low limit for file path length on windows up to win10 (260 chars).
Reviewed By: KapJI
Differential Revision: D14460635
fbshipit-source-id: f63fc564766b49c2d4fb5f1c2bb7015592ab17e1
Summary: Having two configuration files makes it harder to manage system.py so move this to the same file and to the toolchain generation script. This will allow us to automatically determine toolchain path as well in the future.
Reviewed By: marekcirkos
Differential Revision: D14425055
fbshipit-source-id: fdc017f2cc55a2efbb33cdf17df64df620eb11b8
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5490
We use functionality of this libraries, how did it work before?
Reviewed By: guliashvili
Differential Revision: D14280974
fbshipit-source-id: c3b0c2d8d570680460cdc5bbe80efc24467bcb93
Summary: You can now build with `buck build @<mode> osqueryd` for both internal and external build. Also changed NBTD to make use of this.
Reviewed By: marekcirkos
Differential Revision: D14279886
fbshipit-source-id: 1b61bdf254b3d980388e2f23384101c91bf51b20
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5478
This makes it easier to update the osquery version and simplifies cxx.bzl by removing osquery specific preprocessor flags.
This will also make rebuilding osquery after changing versions faster, since the flags are now only defined for the headers which need them.
Reviewed By: akindyakov
Differential Revision: D14183142
fbshipit-source-id: 396d550f5b35a1d294fee802d2364cd9f7ab1d7a
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5469
This way we can specify extra arguments that are going to be added to the library, like exported_preprocessor_flags which is required by some libraries.
Reviewed By: marekcirkos, akindyakov
Differential Revision: D14220787
fbshipit-source-id: 652954e297e49147dfc9f77db8181e2c0e9e123f
Summary:
We need to be able to build `fbcode` projects with dependencies to `fbsource/xplat/osquery/oss/sdk:plugin_sdk`. As far as osquery is a part of `fbsource` now it would be very useful to build against it, make a tests. Which will helps us a lot to develop faster, will unblock us to run tests for every change either to `xplat/osquery` and to `fbcode`, which going to prevent code from bugs and interface breaking (which happens now too often).
`osquery` is very platform dependent project, because it built internally at least for 4 OS: `linux`, `freebsd`, `windows`, `darwin`. `osquery` has its own third-party libraries located in `fbsource/xplat/osquery/third-party`.
Also we have internal osquery extension (`fb_osquery`) in `fbcode` built with strong dependency to `osquery` and with lots of dependencies to `fbcode` projects (e.g. scribe, GK, ODS, configurator, serivicerouter and more).
We could not build `fb_osquery` directly against `osquery` because build system restrictions and third-party dependencies collision.
- Add necessary for `fb_osquery` parts of `fbsource/xpat/osquery` to xplat whitelist.
- Make it possible to use `fbcode` `cpp_library` target definition for `fbsource/xplat/osquery` targets when they are used for `fbcode` build.
- Make a translation platform dependant osquery targets for fbcode platform independent build.
- Use `fbcode/tp` libs instead of `fbsource/xplat/osquery/tp` in case of `fbcode` build.
Differential Revision: D13991062
fbshipit-source-id: 1294825f1c5f991bd465e0e299b8e5ff67bbc543
Summary: This was patched with D13767582 and is already deployed
Reviewed By: mkareta
Differential Revision: D14124516
fbshipit-source-id: 30679472458f4ed9647adc117db4352b940cf1cf
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5452
As suggested in another diff, this diff updates the language we use to describe the osquery licensing terms. We are changing all instances of
//This source code is licensed as defined on the LICENSE file found in the root directory of this source tree.//
to
//This source code is licensed in accordance with the terms specified in the LICENSE file found in the root directory of this source tree.//
We accomplish this with a codemod:
$ codemod -md xplat/osquery/oss --extensions cpp,h,in,py,sh,mm,ps1 "(.\s+)This source code is licensed as defined on the LICENSE file found in the(.*)root directory of this source tree\." "\1This source code is licensed in accordance with the terms specified in\2the LICENSE file found in the root directory of this source tree."
Reviewed By: fmanco
Differential Revision: D14131290
fbshipit-source-id: 52c90da342263e2a80f5a678ecd760c19cf7513e
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5451
This diff adds a Facebook copyright header to the bzl files used in osquery. Ultimately we want to update the files in `tools/build_defs/oss/osquery/`, but those are generated files. This diff updates the source files which we use to generate those files.
Reviewed By: fmanco
Differential Revision: D14131483
fbshipit-source-id: 2230dc382c26530ccd0909882fe6193ee7c674fb
