There is a memory leak when a SQL statement is evaluated and returns an
error while stepping. An example includes returning sqlite3_result_error
from regexStringMatchFunc. The root cause is not finalizing a prepared
statement.
Add a way to compile third-party libraries from source instead of downloading prebuilt ones.
Each library source code is downloaded with git into a submodule at configure time,
in response to the find_package(library_name) CMake call,
except for OpenSSL where the official source archive is used.
Each submodule is attached to a release tag on its own upstream repository.
All the libraries are built using CMake directly, except for OpenSSL which uses a formula system,
which permits to build libraries with a separate build system
when there's no easy way to integrate it directly with CMake.
This new dependency system determines which library is fetched from where using the concept of "layers".
Currently we have three of them: source, formula, facebook,
where the last layer represents the pre-built libraries.
The provided order will be used when looking for libraries.
A system to patch submodule source code has been added and it's currently used with googletest, libudev and util-linux.
Patches should be put under libraries/cmake/source/<library name>/patches/<submodule>,
where <submodule> is often one and is "src", but in other cases, like AWS,
there are multiple with a more specific name.
If for whatever reason the submodule cloning or the patching fails,
the submodule has to be unregistered and its folder should be cleared.
This should be achievable with "git submodule deinit -f <submodule path>"
Following some other changes on existing functionality:
- Changed the CMake variable BUILD_TESTING to OSQUERY_BUILD_TESTS
to avoid enabling tests on third party libraries.
Due to an issue with glog the BUILD_TESTING variable
will be always forced to OFF.
- Moved compiler and linker flags to their own file cmake/flags.cmake
- Moved all the third-party CMakeLists.txt used for pre-built libraries under libraries/cmake/facebook
- Added the --exclude-folders option to tools/format-check.py and tools/git-clang-format.py,
so that it's possible to ignore any third party library source code.
- The format and format_check target use the new --exclude-folders option
to exclude libraries/cmake/source from formatting.
- The test and osquery binaries are properly compiled with PIE (osquery/osquery#5611)
Co-authored-by: Stefano Bonicatti <stefano.bonicatti@gmail.com>
Co-authored-by: Teddy Reed <teddy@casualhacking.io>
Taken from osql-experimental.
- Change CMake code license to the one present in osquery right now
- Package metadata doesn't mention Trail of Bits or osql anymore
- Set specific ACLs for the osqueryd on Windows when packaging
- Remove LLVM_INSTALL_PATH support on macOS, since we are using AppleClang
- Remove OSQUERY_SOURCE_DIR variable need and source in a submodule support
- Add targets format_check and format to check code formatting and
format it with clang-format
- Do not warn about not using Clang on macOS when using AppleClang
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5542
We replace deprecated calls to the Status class with newer ones. This will stop Lint from nagging me every time I open these files. There should be no change to functionality.
Specifically, we use a codemod on `.cpp` files to replace any instance of `return Status();` with `return Status::success();`.
Reviewed By: guliashvili
Differential Revision: D14652922
fbshipit-source-id: 243576195ed201d6b82fc942a785742c6b01cf83
Summary:
Add log_numerics_as_numbers flag.
Internal SQLite query has method that returns QueryDataTyped (instead of QueryData), which are boost::variant<string, double, long long>.
Ints are encoded as such if and only if new log_numerics_as_numbers flag is set to true.
Reviewed By: fmanco
Differential Revision: D13778323
fbshipit-source-id: 7d7bb31781486f63fcc088cd479d3b6f255a5cb4
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5483
Initial steps to separate plugins from the rest of osquery. On the long run separating plugins will provide more build flexibility such that we can have configurable builds that include only the bits and pieces we actually need per deployment. Reducing the attack surface, possibility of supply chain attacks, binary size, etc.
Move killswitch
Reviewed By: guliashvili
Differential Revision: D14259760
fbshipit-source-id: deaaa148fac25b3f534a8881a1ea9eda0ec6bdc9
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5487
I got tired of Lint telling me to update the calls to the Status class, so I wrote some codemods to update the vast majority of them:
$ cd ~/fbsource
$ codemod -d xplat/osquery/oss/osquery --extensions cpp,h "return Status\(0\, \".*\"\);" "return Status::success();"
**Blindly accepted all changes**.
FWIW, I tried to do something similar with the failure return values, but the unit tests were failing afterward.
Reviewed By: guliashvili
Differential Revision: D14278739
fbshipit-source-id: b023f6e969be697eeb67d66ea3ad10d16c69ff0c
Summary: To be able to build with different `thrift_impl` modules in extensions (like facebook does with fbthrift) I removed implementation from `extensions` lib and explicitly used for `osqueryd` and for all tests. This allows us to remove `thrift_impl` from sdk dependencies list. If you need it - please use explicitly as I did for `osqueryd`. If you want to use your own implementation of `thrift_impl` - build it and link against it and sdk simultaneously.
Reviewed By: marekcirkos
Differential Revision: D14224206
fbshipit-source-id: 218dc05d87e1ffaca37783185fc672a20684d757
Summary:
This fixes a regression introduced in D13586036 .
The tailer downstream sometimes determines type by trying to parse the string as an int. We want this to consistently fail to do so when the value is in fact a float.
Reviewed By: SAlexandru
Differential Revision: D14161125
fbshipit-source-id: 8e2bb9c59559a5774813a7c7777257742f885c35
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5452
As suggested in another diff, this diff updates the language we use to describe the osquery licensing terms. We are changing all instances of
//This source code is licensed as defined on the LICENSE file found in the root directory of this source tree.//
to
//This source code is licensed in accordance with the terms specified in the LICENSE file found in the root directory of this source tree.//
We accomplish this with a codemod:
$ codemod -md xplat/osquery/oss --extensions cpp,h,in,py,sh,mm,ps1 "(.\s+)This source code is licensed as defined on the LICENSE file found in the(.*)root directory of this source tree\." "\1This source code is licensed in accordance with the terms specified in\2the LICENSE file found in the root directory of this source tree."
Reviewed By: fmanco
Differential Revision: D14131290
fbshipit-source-id: 52c90da342263e2a80f5a678ecd760c19cf7513e
Summary:
Initial steps to separate plugins from the rest of osquery. On the long run separating plugins will provide more build flexibility such that we can have configurable builds that include only the bits and pieces we actually need per deployment. Reducing the attack surface, possibility of supply chain attacks, binary size, etc.
Move config and config_parser plugins
Reviewed By: marekcirkos
Differential Revision: D14119102
fbshipit-source-id: 0bc956398b3829c6f1013b38ebba2f0fc1071a93
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5445
This diff adds a Facebook copyright header to files in the osquery open source repository which:
* Facebook owns
* Do not currently have a Facebook copyright header
Reviewed By: marekcirkos
Differential Revision: D14122845
fbshipit-source-id: 5a0fea10189ec4ec893f7a036911fd51de0e01ae
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5422
We were just de-prioritizing type of queries not constraining required columns. However, when the query is just useless without specific constraint, sqlite suggestion is to return SQLITE_CONSTRAINT status.
Reviewed By: marekcirkos
Differential Revision: D13964562
fbshipit-source-id: ee0e5f8baf9abbf83c34f7a39d2b5bd705cbac6d
Summary:
Fixed crash in virtual tables that occurs after following steps:
1. sqlite opens VT with xCreate
2. during query executions invokes xFilter with multiple tables
3. Few tables accumulated in affectedTables
4. xDestroy called before finishing query (last step of query execution)
5. query execution finished, SQL instance try to cleanup affected tables, but they were already destroyed by xDestory
This is only hotfix for this crash and this code base require full memory management review in future
Reviewed By: SAlexandru
Differential Revision: D13917015
fbshipit-source-id: 15396e47e4c4e592cf30608a783bc80d560c776f
Summary: before this diff osqueryd was relying on linker order to use right main function, since gtest also contains main function
Reviewed By: guliashvili
Differential Revision: D13897622
fbshipit-source-id: d260b7496f513c7052f4db87c8e7ff9300493671
Summary: We went with 'long' rather than 'int64_t' in the JSON serialization methods that were added, so change our variant type to match
Reviewed By: marekcirkos
Differential Revision: D13675340
fbshipit-source-id: 1ccde4ce9f651fe68db968b367507aa67823c74f
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5375
LICENSE is now defined in a single file on the root of the project, update the
header to contain that information.
**Project LICENSE did not change.**
Reviewed By: akindyakov
Differential Revision: D13750575
fbshipit-source-id: 1e608a81b260b8395f9d008fc67f463160c1fc2b
Summary:
While running `misspell` on a different codebase. I happened to notice that some misspellings in the osquery code base. So, I fixed them
Pull Request resolved: https://github.com/facebook/osquery/pull/5256
Reviewed By: guliashvili
Differential Revision: D13670897
Pulled By: fmanco
fbshipit-source-id: 5d33d858284955c376e8c3980acdf366d4edf3d3
Summary: Some tools does not support virtual headers namespaces and expect header path to match actual file path from project root. This diff will fix few namespaces in utils library
Reviewed By: guliashvili
Differential Revision: D13552878
fbshipit-source-id: 2a06f73550c69777bf73be73abdde297fe580583
Summary:
Continuing to march toward low-overhead, type-safe table rows, this commit
changes `TableRow` to be an interface rather than simply an alias for `Row`.
Accordingly, `DynamicTableRow` becomes an implementation of that interface
backed by a `Row`. The few remaining pieces of code that treated `TableRow`s as
`Row`s now call methods on the `TableRow` interface. Subsequent commits will
add code generation for strongly-typed table-specific implementations of
`TableRow`.
(Adapted from https://github.com/facebook/osquery/pull/5198)
Reviewed By: guliashvili
Differential Revision: D13438015
fbshipit-source-id: 61d5547e878e519c9706f94f844aab9d3e553410
Summary:
Continuing to march toward low-overhead, type-safe table rows, this commit changes
much of the code that uses `TableRow`s to stop assuming that they're just `vector`s
by another name. (`TableRow` is on the way to becoming an interface with multiple
implementations.) They're now held in `unique_ptr`s (`TableRowHolder`). For cases
where we really want a `vector`-backed `TableRow` (mostly test code and extension
support), we have a factory function (`make_table_row`) and a helper class
(`DynamicTableRowHolder`) to make that smoother.
(Adapted from https://github.com/facebook/osquery/pull/5198)
Reviewed By: mkareta
Differential Revision: D13438016
fbshipit-source-id: 2de9ce46a64c0a067b5d3299c59bbe3ccacd4abe
Summary:
Continuing to march toward low-overhead, type-safe table rows, this commit introduces
a distinction between rows being returned from a table (`TableRows`) and as the
result of a query (`QueryData`). Right now the two are simply aliases for each other;
that will change shortly.
(Adapted from https://github.com/facebook/osquery/pull/5198)
Reviewed By: guliashvili
Differential Revision: D13438019
fbshipit-source-id: 6563fc8c372d9d6c4b05705943ddf39b42260feb
Summary:
Step towards completion of https://github.com/facebook/osquery/pull/4904
- Add boost variant (typed) counterparts for RowData, Row, and QueryData.
- Add a queryInternal to sqlite_util that returns QueryDataTyped.
- Replace the queryInternal that returns QueryData with one that's a simple wrapper around the new typed implementation.
Reviewed By: guliashvili
Differential Revision: D13586036
fbshipit-source-id: d1d9b56470fbfcfb5802de422f87d9b9d6da7fb7
Summary:
Hand port of https://github.com/facebook/osquery/pull/5154 to the new build system
I'm moving toward generating constants for each column so that we
can do used column lookups without having to do a string set lookup,
but first I need to have the used columns information plumbed through
as a bitfield.
Once the code generation is in and all cases have been migrated to it,
I'll remove the string variants of `isColumnUsed`/`isAnyColumnUsed`.
Reviewed By: guliashvili
Differential Revision: D13423615
fbshipit-source-id: 6a2afe7dad42942610dfe0f6f55bcee4603a25af
