Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5498
to keep in sync headers namespace and real path to target
Reviewed By: jessek, marekcirkos
Differential Revision: D14322951
fbshipit-source-id: 3e99cb5e9227529520c0ee1cac0c3d755f594ed8
Summary:
In my testing of Osquery, I noticed the `path` field for `firefox_addons` is always blank. After inspecting the structure of the JSON and the code in Osquery, I believe the key to look up path should be renamed from `descriptor` to `path`.
Pull Request resolved: https://github.com/facebook/osquery/pull/5462
Differential Revision: D14301220
Pulled By: akindyakov
fbshipit-source-id: b4b56d7026736344da585aada83d5fea6e249f1c
Summary:
Add log_numerics_as_numbers flag.
Internal SQLite query has method that returns QueryDataTyped (instead of QueryData), which are boost::variant<string, double, long long>.
Ints are encoded as such if and only if new log_numerics_as_numbers flag is set to true.
Reviewed By: fmanco
Differential Revision: D13778323
fbshipit-source-id: 7d7bb31781486f63fcc088cd479d3b6f255a5cb4
Summary: Please do not use message of Status for anything except logging!
Reviewed By: fmanco
Differential Revision: D14281607
fbshipit-source-id: e48ed3d79de19acb1d86885d0acfab781e5058aa
Summary: Add query monitoring without specifying the scheme. It will help us in the simpler aggregation.
Reviewed By: marekcirkos
Differential Revision: D14280109
fbshipit-source-id: d70971c9ebe1e15c66f36e1490b7d8e198a68a7a
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5485
Initial steps to separate plugins from the rest of osquery. On the long run separating plugins will provide more build flexibility such that we can have configurable builds that include only the bits and pieces we actually ne
er deployment. Reducing the attack surface, possibility of supply chain attacks, binary size, etc.
Move numeric monitoring
Reviewed By: guliashvili
Differential Revision: D14259758
fbshipit-source-id: 1016fc28a0052417d658b6ce1cb3368e56597a7b
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5483
Initial steps to separate plugins from the rest of osquery. On the long run separating plugins will provide more build flexibility such that we can have configurable builds that include only the bits and pieces we actually need per deployment. Reducing the attack surface, possibility of supply chain attacks, binary size, etc.
Move killswitch
Reviewed By: guliashvili
Differential Revision: D14259760
fbshipit-source-id: deaaa148fac25b3f534a8881a1ea9eda0ec6bdc9
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5491
Initial implementation had a half-baked and broken wait() implementation to support multiple consumers. It does not make sense to support multiple consumers anyway, so let's remove that functionality and simplify the implementation.
Reviewed By: mkareta
Differential Revision: D14279796
fbshipit-source-id: d31fff69a64c0e45a9af8cd97bb0488c111c6b68
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5490
We use functionality of this libraries, how did it work before?
Reviewed By: guliashvili
Differential Revision: D14280974
fbshipit-source-id: c3b0c2d8d570680460cdc5bbe80efc24467bcb93
Summary: You can now build with `buck build @<mode> osqueryd` for both internal and external build. Also changed NBTD to make use of this.
Reviewed By: marekcirkos
Differential Revision: D14279886
fbshipit-source-id: 1b61bdf254b3d980388e2f23384101c91bf51b20
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5487
I got tired of Lint telling me to update the calls to the Status class, so I wrote some codemods to update the vast majority of them:
$ cd ~/fbsource
$ codemod -d xplat/osquery/oss/osquery --extensions cpp,h "return Status\(0\, \".*\"\);" "return Status::success();"
**Blindly accepted all changes**.
FWIW, I tried to do something similar with the failure return values, but the unit tests were failing afterward.
Reviewed By: guliashvili
Differential Revision: D14278739
fbshipit-source-id: b023f6e969be697eeb67d66ea3ad10d16c69ff0c
Summary:
Super tiny PR to fix some spelling. This is the first of a few incremental PRs for some improvements to WMI!
Pull Request resolved: https://github.com/facebook/osquery/pull/5471
Reviewed By: marekcirkos
Differential Revision: D14240440
Pulled By: jessek
fbshipit-source-id: 9a6c23c3aa31bda15972dc79bbad07640727d19a
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5482
We allow people using a debug build of osqueryd to skip verification of the server's TLS certificate. This allow people to connect to a devserver, for example, when testing the Graph API endpoints used by osquery.
This mode broke at some point when we stopped defining the `DEBUG` constant when building osqueryd in debug mode. This diff updates the code so that we use the constant `NDEBUG`, which is actually defined for release builds. For example, for OS X, see the build configs at https://fburl.com/ywr1tyuk and https://fburl.com/mte9ajvr.
We also update the code to ensure the command line option `--tls_allow_unsafe` allows a person to connect to a dev server.
Reviewed By: obelisk
Differential Revision: D14260226
fbshipit-source-id: c0f2f1c1f4c8dd5e3e1eaa6e1a52ab2a4ff765d4
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5481
osquery registries is big part of SDK interface and it is important to make
sure SDK include them all.
Need of tests for non optional plugins can be not so obvious. All non-optional
plugins should not be included to SDK library. Because it causes to double
plugin registration when extensions process connects to `osqueryd`. That leads
to the crash of extensions process.
Unfortunatelly there is no explisit list of plugins included to SDK and there is
no way to check them at compile time. Thankfully it can be done at
runtime and we can make a test for it.
Reviewed By: fmanco
Differential Revision: D14261046
fbshipit-source-id: 0fb6618368ae7f735c648cc69fc3ea5f331123ad
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5480
as a hacky way to avoid including implementation of all tables to plugin sdk.
Reviewed By: fmanco
Differential Revision: D14261047
fbshipit-source-id: fec20c7241968bd051f7419f8ee7e10bebd7141d
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5475
osquery build doesn't use folly for now so this facebook specific routine is dead and should be removed or moved to another place.
Reviewed By: fmanco, guliashvili
Differential Revision: D14242160
fbshipit-source-id: f2a8942906ddc1d7d9bb47395b1081ad2e88712d
Summary: to be able to preserve old python namespace for backward compatibility and satisfy fbcode naming rules
Reviewed By: jessek, fmanco
Differential Revision: D14242579
fbshipit-source-id: b50ba0df6ad75752f651f0bf642f3278de363874
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5478
This makes it easier to update the osquery version and simplifies cxx.bzl by removing osquery specific preprocessor flags.
This will also make rebuilding osquery after changing versions faster, since the flags are now only defined for the headers which need them.
Reviewed By: akindyakov
Differential Revision: D14183142
fbshipit-source-id: 396d550f5b35a1d294fee802d2364cd9f7ab1d7a
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5474
To be able to exclude config:update plugin from osquery plugin sdk.
Reviewed By: guliashvili
Differential Revision: D14241686
fbshipit-source-id: e378718b1e79e3dbe314a676945c18d15d054a3d
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5473
to be able to exclude dependency on enroll:tls plugin from plugins_sdk.
Only plugin registration was moved by now. The actual plugin code will be moved a bit later.
Reviewed By: guliashvili
Differential Revision: D14241687
fbshipit-source-id: 814d52719ff27bd1283bcaecdc912fca3830ff5e
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5472
To get rid of dependency on `plugins/config:tls_config` from `osquery/remote/enroll/tls_enroll`.
To be able to remove dependency on `plugins/config:tls_config` from plugins_sdk.
Reviewed By: guliashvili
Differential Revision: D14241685
fbshipit-source-id: 27cda0be9d0b6dc142374cc3a9e7c40a4fa97b94
Summary:
This introduces two new (Windows-only) columns to the `logged_in_users` table:
* `sid` corresponds to the logged in user's security identifier, used to uniquely identify the user and their permissions on the local system.
* `registry_hive` corresponds to the user's HKU registry hive, used to look up per-user configuration information.
I've updated the integration tests to test for these columns on Windows only. Please let me know if there's anything else I can do!
Pull Request resolved: https://github.com/facebook/osquery/pull/5454
Differential Revision: D14195466
Pulled By: fmanco
fbshipit-source-id: def9c362fac1b5a68b68f826916daafee224295b
Summary:
This generally refactors the `logical_drives` table on Windows to conform more closely to C++11 idioms. It also enables the integration test for `logical_drives`.
See #5367. I'll open a PR for the boot partition fixes once this is merged.
cc akindyakov guliashvili
Pull Request resolved: https://github.com/facebook/osquery/pull/5400
Differential Revision: D14131722
Pulled By: fmanco
fbshipit-source-id: c3077da48147a9880ce08925d165e5d1da363bb9
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5470
Building without ROCKSDB_LITE causes osquery to crash due to a mismatch between the structs compiled into the pre-built library and the compiled osquery code.
Reviewed By: marekcirkos
Differential Revision: D14220786
fbshipit-source-id: 514365bf38edaf059c30fabe1695ca95f7d5b326
Summary:
Let's create a fbcode wrapper for osuery plugin SDK to wrap it up with fbthrift implementation from fbcode. Also we should trick the build system and add empty function `registerForeignTables()` to avoid including all osquery builtin tables implementation.
Since that, SDK suppose to be full functional and ready to be used to build facebook PROD extensions in fbcode with dependency on it (see next diff).
Reviewed By: fmanco
Differential Revision: D14224205
fbshipit-source-id: c08cbb374f2d68f6b93637809b97dcf0159c9e08
Summary: To be able to build with different `thrift_impl` modules in extensions (like facebook does with fbthrift) I removed implementation from `extensions` lib and explicitly used for `osqueryd` and for all tests. This allows us to remove `thrift_impl` from sdk dependencies list. If you need it - please use explicitly as I did for `osqueryd`. If you want to use your own implementation of `thrift_impl` - build it and link against it and sdk simultaneously.
Reviewed By: marekcirkos
Differential Revision: D14224206
fbshipit-source-id: 218dc05d87e1ffaca37783185fc672a20684d757
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5466
To be able to use other implementation (like `fbthrift_impl.cpp`) instead of `thrift_impl.cpp`, if required.
Reviewed By: marekcirkos
Differential Revision: D14208053
fbshipit-source-id: ba379ea85fc66deef517763eaca896e88729d68e
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5469
This way we can specify extra arguments that are going to be added to the library, like exported_preprocessor_flags which is required by some libraries.
Reviewed By: marekcirkos, akindyakov
Differential Revision: D14220787
fbshipit-source-id: 652954e297e49147dfc9f77db8181e2c0e9e123f
Summary:
tables: re-architect the processes table to not use WMI queries
This PR re-writes the processes table to no longer rely on WMI queries. We do this to ensure that the processes table is robust and performant, as we rely on this table for the watcher process to regulate the worker processes. Further, this table further employs the selective column generation introduced to the processes table on darwin to allow us to more selectively generate column data only when necessary.
Lastly, this table removes a couple of extraneous columns from the Windows table schema, as these column values are achievable via sqlite logic. Specifically:
* `elapsed_time`: to get this column one can use the `start_time` column along with the sqlite built in `strftime` function:
```
osquery> select name, (strftime('%s', 'now') - start_time) as uptime from processes limit 5;
+------------------+--------+
| name | uptime |
+------------------+--------+
| [System Process] | |
| System | 7131 |
| Registry | 7154 |
| smss.exe | 7131 |
| csrss.exe | 7127 |
+------------------+--------+
```
* `percent_processor_time`: to obtain the total time that a process has been executing on a system, one can add together the `user_time` and `system_time` together to get the same value returned by the WMI value, the difference being that the sumation of `user_time` and `system_time` are in milliseconds whereas the values returned from WMI are in 100 nanosecond ticks:
osquery `percent_processor_time`:
```
osquery> select pid, name, (user_time + system_time) as percent_processor_time from processes where name = 'osqueryd.exe';
I1112 21:54:11.809412 2152 processes.cpp:461] Failed to open handle to process 0 with 203
+-------+--------------+------------------------+
| pid | name | percent_processor_time |
+-------+--------------+------------------------+
| 3892 | osqueryd.exe | 5952 |
| 808 | osqueryd.exe | 0 |
| 5892 | osqueryd.exe | 52374 |
| 11688 | osqueryd.exe | 280 |
+-------+--------------+------------------------+
```
Same data obtained with WMI.
```
λ Get-WmiObject -Query "Select IDProcess, Name, PercentProcessorTime from Win32_PerfRawData_PerfProc_Process" | Where-Object {$_.Name -like 'osqueryd*'}
IDProcess : 3892
Name : osqueryd
PercentProcessorTime : 59531250
IDProcess : 808
Name : osqueryd#1
PercentProcessorTime : 0
IDProcess : 5892
Name : osqueryd#2
PercentProcessorTime : 523750000
IDProcess : 11688
Name : osqueryd#3
PercentProcessorTime : 2812500
```
Some additional notes, just to verify we're getting all data:
```
C:\Users\Nick\work\repos\osquery [windows-process-rearch ↑]
λ .\build\windows10\osquery\RelWithDebInfo\osqueryd.exe -S --json "select count(*) from processes;"
[
{"count(*)":"160"}
]
C:\Users\Nick\work\repos\osquery [windows-process-rearch ↑]
λ C:\ProgramData\osquery\osqueryd\osqueryd.exe -S --json "select count(*) from processes;"
[
{"count(*)":"160"}
]
```
Also we achieve a considerable speed up from this re-arch:
```
C:\Users\Nick\work\repos\osquery [windows-process-rearch ↑]
λ (Measure-Command {.\build\windows10\osquery\RelWithDebInfo\osqueryd.exe -S --json "select count(*) from processes;"}).Milliseconds
168
C:\Users\Nick\work\repos\osquery [windows-process-rearch ↑]
λ (Measure-Command {C:\ProgramData\osquery\osqueryd\osqueryd.exe -S --json "select count(*) from processes;"}).Milliseconds
223
```
Pull Request resolved: https://github.com/facebook/osquery/pull/5293
Reviewed By: fmanco
Differential Revision: D13561337
Pulled By: muffins
fbshipit-source-id: 61435611d34d2cfd5f61ea52512a9dc208d259a5
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5464
While attempting to debug this issue I have discovered we are attempting to look up an empty item name in the registry. This doesn't make sense as we are clearly passing in an item name. This diff doesn't solve the issue, but it does add an error message for when it happens.
While we're here, appeasing Lint by using a more modern call to the `Status` return type.
Reviewed By: guliashvili
Differential Revision: D14207654
fbshipit-source-id: a53035ea84357f9ffe1d216b66e116257f291a7f
Summary: Trim whitespace in Darwin system_info table from hardware_serial column.
Reviewed By: jessek
Differential Revision: D14207715
fbshipit-source-id: 2948b8687c0c40d43f13ab42ddc4bbfec8e59a48
Summary:
This PR fixes#5106 by trimming the whitespace we get from the data returned from the macOS API. At first I thought this was an osquery regression, but going as far back as 2.x this looks like it has always been a problem.
Before the fix...
```
echo "select hardware_model, hardware_vendor, hardware_version from system_info;" | osqueryi --json | jq
[
{
"hardware_model": "MacBookPro13,3 ",
"hardware_vendor": "Apple Inc. ",
"hardware_version": "1.0 "
}
]
```
After the fix... (note the lack of trailing spaces)
```
echo "select hardware_version, hardware_vendor, hardware_model from system_info;" | /Users/jmeller/source/osquery/build/darwin10.13/osquery/osqueryi --json | jq
[
{
"hardware_model": "MacBookPro13,3",
"hardware_vendor": "Apple Inc.",
"hardware_version": "1.0"
}
]
```
Pull Request resolved: https://github.com/facebook/osquery/pull/5167
Reviewed By: jessek
Differential Revision: D14207353
Pulled By: guliashvili
fbshipit-source-id: 4012e04b38e2b5bfc299684baf86d0ab34cd42f8
Summary: as far as they are actually part of glibc, that would be less confusing. Later it will help us to deal with fully static build.
Reviewed By: marekcirkos
Differential Revision: D14183689
fbshipit-source-id: 8b712286e171305a49be6703b846f6f935b7dcc2
Summary: to be able to proxy it inside `osquery_tp_target` function.
Reviewed By: marekcirkos
Differential Revision: D14182918
fbshipit-source-id: d56b7135664ecd5afcc87b1c1cd2cde5394368a0
Summary:
It cause implicit dependency to osquery/core:core, that cause circular dependency :(. The point is to get rid of it. Let's use just exit() instead.
As far as we don't have decent implementation of Initializer::shutdown it will not cause us any more problems.
We can live with it for a while.
Reviewed By: marekcirkos
Differential Revision: D14182915
fbshipit-source-id: 22c54cde910ea29b1fb19d96598db381b9d1c60d
Summary:
This fixes a regression introduced in D13586036 .
The tailer downstream sometimes determines type by trying to parse the string as an int. We want this to consistently fail to do so when the value is in fact a float.
Reviewed By: SAlexandru
Differential Revision: D14161125
fbshipit-source-id: 8e2bb9c59559a5774813a7c7777257742f885c35
Summary: only events.cpp required by plugin_sdk, everything else is not necessary
Reviewed By: marekcirkos
Differential Revision: D14131365
fbshipit-source-id: 18e101b3e549d8403aa22ee3d5a5fadd2b296c82
Summary:
I didn't find a way to refer file in fbcode thrift_library from xplat. So, let's copy it.
There is something wrong with a namespaces in python - I'll fix it later. This is not a problem for now - file is not used.
Reviewed By: marekcirkos
Differential Revision: D14124542
fbshipit-source-id: 1060e23675a8086af494aa93037e1c613f250c71
Summary:
As far as this file is only for internal build, to be able to build osquery sdk against fbthrift I moved it to fbcode from oss.
Later if we want to osqeruy work with fbthrift we can copy/move it again
Reviewed By: marekcirkos
Differential Revision: D14124448
fbshipit-source-id: aaeadb40a6fac19621423b225cf6d522607d1313
Summary:
We need to be able to build `fbcode` projects with dependencies to `fbsource/xplat/osquery/oss/sdk:plugin_sdk`. As far as osquery is a part of `fbsource` now it would be very useful to build against it, make a tests. Which will helps us a lot to develop faster, will unblock us to run tests for every change either to `xplat/osquery` and to `fbcode`, which going to prevent code from bugs and interface breaking (which happens now too often).
`osquery` is very platform dependent project, because it built internally at least for 4 OS: `linux`, `freebsd`, `windows`, `darwin`. `osquery` has its own third-party libraries located in `fbsource/xplat/osquery/third-party`.
Also we have internal osquery extension (`fb_osquery`) in `fbcode` built with strong dependency to `osquery` and with lots of dependencies to `fbcode` projects (e.g. scribe, GK, ODS, configurator, serivicerouter and more).
We could not build `fb_osquery` directly against `osquery` because build system restrictions and third-party dependencies collision.
- Add necessary for `fb_osquery` parts of `fbsource/xpat/osquery` to xplat whitelist.
- Make it possible to use `fbcode` `cpp_library` target definition for `fbsource/xplat/osquery` targets when they are used for `fbcode` build.
- Make a translation platform dependant osquery targets for fbcode platform independent build.
- Use `fbcode/tp` libs instead of `fbsource/xplat/osquery/tp` in case of `fbcode` build.
Differential Revision: D13991062
fbshipit-source-id: 1294825f1c5f991bd465e0e299b8e5ff67bbc543
Summary:
Hi! This PR adds a new column called `product_version` to the file table, which is only
populated when queries are done on Windows. It is a very minimal PR that uses an existing helper function (`windowsGetFileVersion`) to populate the column.
The column is not named `file_version`, despite the name of the helper function because the underlying data retrieved by that helper function is actually the `dwProductVersion*` fields of the `VS_FIXEDFILEINFO` struct. In the future, if we want to add a column that _actually_ contains the results of the `dwFileVersion*` fields, we can add a new column called `file_version` without modifying existing functionality.
Pull Request resolved: https://github.com/facebook/osquery/pull/5431
Differential Revision: D14169773
Pulled By: fmanco
fbshipit-source-id: 6fa7c92425fc92aa2e83a37383b1a8c796b17644
Summary: It cause problems with compiling against tp2 because in tp2 there is no such target. As far as I can see there is no necessity to depend on that target separatelly from gtest. So, the simplest solution is to remove it
Reviewed By: SAlexandru
Differential Revision: D14149801
fbshipit-source-id: fe0fcd0593bded7c8f2c6fd247b1d52bd29bac9f
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5459
This diff updates the osquery Code of Conduct policy to reference the Facebook Open Source Code of Conduct, https://code.fb.com/codeofconduct/.
Reviewed By: marekcirkos
Differential Revision: D14149368
fbshipit-source-id: 4e6ae7acf9367053268c5d8fb6289b5e94703604
Summary: This test is flaky on some systems. Need to investigate and fix.
Reviewed By: guliashvili
Differential Revision: D14149533
fbshipit-source-id: 7df570c84c66cad8bdc67a1286d91a7c7e8e3502
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5458
Use ASSERTs where necessary. ASSERTs are fatal and terminate the test immediately, so should be used when subsequent tests aren't meaningful in case of previous failures. Only use EXPECTED when it makes sense for the tests to continue.
Reviewed By: jessek
Differential Revision: D14138382
fbshipit-source-id: aae0cd8640e602ac467861303457f9cf017bc5e8
