On macOS it is possible to have interfaces that have no netmask
set, for example:
| utun1 | 100.xx.xx.xx | | | 100.xx.xx.xx | unknown |
This causes the interfaces test to fail. This change allows the
mask field to be blank and the test to pass on systems with this
type of interface configured.
The sockaddr for this interface has an ifa_netmask with sa_family
set to zero (AF_UNSPEC) which causes getnameinfo(3) to fail.
ifconfig does return a netmask for this interface but it's not
clear if the value is at all meaningful.
When requesting kSecClassIdentity items from the keychain sometimes
invalid items are returned. These cause errSecInvalidItemRef to
be returned from SecKeychainItemCopyAttributesAndData and result
in an empty row in the table. Catch the error and avoid returning
empty rows.
Add a way to compile third-party libraries from source instead of downloading prebuilt ones.
Each library source code is downloaded with git into a submodule at configure time,
in response to the find_package(library_name) CMake call,
except for OpenSSL where the official source archive is used.
Each submodule is attached to a release tag on its own upstream repository.
All the libraries are built using CMake directly, except for OpenSSL which uses a formula system,
which permits to build libraries with a separate build system
when there's no easy way to integrate it directly with CMake.
This new dependency system determines which library is fetched from where using the concept of "layers".
Currently we have three of them: source, formula, facebook,
where the last layer represents the pre-built libraries.
The provided order will be used when looking for libraries.
A system to patch submodule source code has been added and it's currently used with googletest, libudev and util-linux.
Patches should be put under libraries/cmake/source/<library name>/patches/<submodule>,
where <submodule> is often one and is "src", but in other cases, like AWS,
there are multiple with a more specific name.
If for whatever reason the submodule cloning or the patching fails,
the submodule has to be unregistered and its folder should be cleared.
This should be achievable with "git submodule deinit -f <submodule path>"
Following some other changes on existing functionality:
- Changed the CMake variable BUILD_TESTING to OSQUERY_BUILD_TESTS
to avoid enabling tests on third party libraries.
Due to an issue with glog the BUILD_TESTING variable
will be always forced to OFF.
- Moved compiler and linker flags to their own file cmake/flags.cmake
- Moved all the third-party CMakeLists.txt used for pre-built libraries under libraries/cmake/facebook
- Added the --exclude-folders option to tools/format-check.py and tools/git-clang-format.py,
so that it's possible to ignore any third party library source code.
- The format and format_check target use the new --exclude-folders option
to exclude libraries/cmake/source from formatting.
- The test and osquery binaries are properly compiled with PIE (osquery/osquery#5611)
Co-authored-by: Stefano Bonicatti <stefano.bonicatti@gmail.com>
Co-authored-by: Teddy Reed <teddy@casualhacking.io>
When run alone, using --gtest_filter to select the specific test,
it throws an exception because the tables are not registered.
It also wrongly checks for the query rows number to be >= 0 and then
proceeds to access the first one.
The table it tests has several issues, it doesn't properly check
for errors in several places and returns unexpected values in
the columns.
Moreover the code could also be improved to make testing possible
without passing through SQL queries, by separating the part
that uses WMI queries from the conversion to row results.
Calling GetLastError() will not return the error code for the Windows registry APIs,
since they return a LSTATUS value which already the error code.
This also fixes the RegistryTablesTest.test_registry_non_existing_key
test case which was incorrectly expecting success when querying
for a non-existent registry key.
Ported from https://github.com/osql/osql/pull/50
Only define BOOST_ASIO_DISABLE_STD_STRING_VIEW.
We shouldn't define BOOST_ASIO_HAS_STD_STRING_VIEW,
because even if we define BOOST_ASIO_DISABLE_STD_STRING_VIEW
the first define will actually enable parts of code that will use string_view.
This won't work on Windows and in general, string_view should not be
used unless compiling with C++17.
The hack has been also added to a test that was previously missed.
Taken from osql-experimental.
- Change CMake code license to the one present in osquery right now
- Package metadata doesn't mention Trail of Bits or osql anymore
- Set specific ACLs for the osqueryd on Windows when packaging
- Remove LLVM_INSTALL_PATH support on macOS, since we are using AppleClang
- Remove OSQUERY_SOURCE_DIR variable need and source in a submodule support
- Add targets format_check and format to check code formatting and
format it with clang-format
- Do not warn about not using Clang on macOS when using AppleClang
Summary:
8 out of 234 tests in the integration tests have TEST_F(classname, sanity) others have TEST_F(classname, test_sanity).
Making them all the same.
Reviewed By: jessek
Differential Revision: D14627709
fbshipit-source-id: d3ea6569d59fd3885004bde9421a0f1feafdb36c
Summary: just switchin off some windows specific code on non-windows platforms
Reviewed By: guliashvili
Differential Revision: D14597685
fbshipit-source-id: 8b058ac1f9db8e4455e1b29e9e277d04468c2d64
Summary:
Add log_numerics_as_numbers flag.
Internal SQLite query has method that returns QueryDataTyped (instead of QueryData), which are boost::variant<string, double, long long>.
Ints are encoded as such if and only if new log_numerics_as_numbers flag is set to true.
Reviewed By: fmanco
Differential Revision: D13778323
fbshipit-source-id: 7d7bb31781486f63fcc088cd479d3b6f255a5cb4
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5483
Initial steps to separate plugins from the rest of osquery. On the long run separating plugins will provide more build flexibility such that we can have configurable builds that include only the bits and pieces we actually need per deployment. Reducing the attack surface, possibility of supply chain attacks, binary size, etc.
Move killswitch
Reviewed By: guliashvili
Differential Revision: D14259760
fbshipit-source-id: deaaa148fac25b3f534a8881a1ea9eda0ec6bdc9
Summary:
This introduces two new (Windows-only) columns to the `logged_in_users` table:
* `sid` corresponds to the logged in user's security identifier, used to uniquely identify the user and their permissions on the local system.
* `registry_hive` corresponds to the user's HKU registry hive, used to look up per-user configuration information.
I've updated the integration tests to test for these columns on Windows only. Please let me know if there's anything else I can do!
Pull Request resolved: https://github.com/facebook/osquery/pull/5454
Differential Revision: D14195466
Pulled By: fmanco
fbshipit-source-id: def9c362fac1b5a68b68f826916daafee224295b
Summary:
This generally refactors the `logical_drives` table on Windows to conform more closely to C++11 idioms. It also enables the integration test for `logical_drives`.
See #5367. I'll open a PR for the boot partition fixes once this is merged.
cc akindyakov guliashvili
Pull Request resolved: https://github.com/facebook/osquery/pull/5400
Differential Revision: D14131722
Pulled By: fmanco
fbshipit-source-id: c3077da48147a9880ce08925d165e5d1da363bb9
Summary: To be able to build with different `thrift_impl` modules in extensions (like facebook does with fbthrift) I removed implementation from `extensions` lib and explicitly used for `osqueryd` and for all tests. This allows us to remove `thrift_impl` from sdk dependencies list. If you need it - please use explicitly as I did for `osqueryd`. If you want to use your own implementation of `thrift_impl` - build it and link against it and sdk simultaneously.
Reviewed By: marekcirkos
Differential Revision: D14224206
fbshipit-source-id: 218dc05d87e1ffaca37783185fc672a20684d757
Summary:
Hi! This PR adds a new column called `product_version` to the file table, which is only
populated when queries are done on Windows. It is a very minimal PR that uses an existing helper function (`windowsGetFileVersion`) to populate the column.
The column is not named `file_version`, despite the name of the helper function because the underlying data retrieved by that helper function is actually the `dwProductVersion*` fields of the `VS_FIXEDFILEINFO` struct. In the future, if we want to add a column that _actually_ contains the results of the `dwFileVersion*` fields, we can add a new column called `file_version` without modifying existing functionality.
Pull Request resolved: https://github.com/facebook/osquery/pull/5431
Differential Revision: D14169773
Pulled By: fmanco
fbshipit-source-id: 6fa7c92425fc92aa2e83a37383b1a8c796b17644
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5452
As suggested in another diff, this diff updates the language we use to describe the osquery licensing terms. We are changing all instances of
//This source code is licensed as defined on the LICENSE file found in the root directory of this source tree.//
to
//This source code is licensed in accordance with the terms specified in the LICENSE file found in the root directory of this source tree.//
We accomplish this with a codemod:
$ codemod -md xplat/osquery/oss --extensions cpp,h,in,py,sh,mm,ps1 "(.\s+)This source code is licensed as defined on the LICENSE file found in the(.*)root directory of this source tree\." "\1This source code is licensed in accordance with the terms specified in\2the LICENSE file found in the root directory of this source tree."
Reviewed By: fmanco
Differential Revision: D14131290
fbshipit-source-id: 52c90da342263e2a80f5a678ecd760c19cf7513e
Summary:
Initial steps to separate plugins from the rest of osquery. On the long run separating plugins will provide more build flexibility such that we can have configurable builds that include only the bits and pieces we actually need per deployment. Reducing the attack surface, possibility of supply chain attacks, binary size, etc.
Move config and config_parser plugins
Reviewed By: marekcirkos
Differential Revision: D14119102
fbshipit-source-id: 0bc956398b3829c6f1013b38ebba2f0fc1071a93
Summary: It fails on some platforms because of permissions, let's just check if file path is not empty and is absolute
Reviewed By: marekcirkos
Differential Revision: D14086996
fbshipit-source-id: 98068e4b93e6be12a2392345fa74b547d26a2d43
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5422
We were just de-prioritizing type of queries not constraining required columns. However, when the query is just useless without specific constraint, sqlite suggestion is to return SQLITE_CONSTRAINT status.
Reviewed By: marekcirkos
Differential Revision: D13964562
fbshipit-source-id: ee0e5f8baf9abbf83c34f7a39d2b5bd705cbac6d
Summary:
Fixed crash in virtual tables that occurs after following steps:
1. sqlite opens VT with xCreate
2. during query executions invokes xFilter with multiple tables
3. Few tables accumulated in affectedTables
4. xDestroy called before finishing query (last step of query execution)
5. query execution finished, SQL instance try to cleanup affected tables, but they were already destroyed by xDestory
This is only hotfix for this crash and this code base require full memory management review in future
Reviewed By: SAlexandru
Differential Revision: D13917015
fbshipit-source-id: 15396e47e4c4e592cf30608a783bc80d560c776f
Summary: We fixed the meaning of `start_time` in previous PR/diffs, let's check it in the integration test.
Reviewed By: SAlexandru
Differential Revision: D13918628
fbshipit-source-id: 60f4ff74eadfbe286dfb325d713389f01142a0f8
Summary: before this diff osqueryd was relying on linker order to use right main function, since gtest also contains main function
Reviewed By: guliashvili
Differential Revision: D13897622
fbshipit-source-id: d260b7496f513c7052f4db87c8e7ff9300493671
Summary:
This PR is the result of the discussion in a previous PR (#5348) after we determined account_policy_data was the wrong place for the column.
Add `is_hidden` column to the users and groups tables in macOS. `is_hidden` is populated by looking for the `dsAttrTypeNative:IsHidden` attribute in the OpenDirectory record for the user/group if the value is `1`, `True`, or `Yes` is_hidden is 1. If the value is anything else it's set to 0. Invalid values have the same affect as the attribute not existing at all.
The `dsAttrTypeNative:IsHidden` attribute controls whether a user account is is visible in the preferences panel similar to having a uid < 500.
One test failed when running buck test:
```
====STANDARD OUT====
tests/integration/tables/helper.cpp:159: Failure
Value of: boost::get<CustomCheckerType>(validator)(value)
Actual: false
Expected: true
Custom validator of the column "mask" with value "" failed
```
This also fails when I ran the test on the current experimental branch as well.
Important to note I had to remove the optimization on both the user and group tables that just called `getpwnam` if the query specified the `uid` or `gid` since the struct returned doesn't contain the `IsHidden` attribute. I'm not sure if or how much this will affect performance since I wasn't able to get the profiling to work with the new version (very likely I'm just doing it incorrectly).
Pull Request resolved: https://github.com/facebook/osquery/pull/5368
Differential Revision: D13862375
Pulled By: akindyakov
fbshipit-source-id: 1fec88a6ba71884f7e611e1d96ea00630c5be655
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5375
LICENSE is now defined in a single file on the root of the project, update the
header to contain that information.
**Project LICENSE did not change.**
Reviewed By: akindyakov
Differential Revision: D13750575
fbshipit-source-id: 1e608a81b260b8395f9d008fc67f463160c1fc2b
