Summary:
tables: re-architect the processes table to not use WMI queries
This PR re-writes the processes table to no longer rely on WMI queries. We do this to ensure that the processes table is robust and performant, as we rely on this table for the watcher process to regulate the worker processes. Further, this table further employs the selective column generation introduced to the processes table on darwin to allow us to more selectively generate column data only when necessary.
Lastly, this table removes a couple of extraneous columns from the Windows table schema, as these column values are achievable via sqlite logic. Specifically:
* `elapsed_time`: to get this column one can use the `start_time` column along with the sqlite built in `strftime` function:
```
osquery> select name, (strftime('%s', 'now') - start_time) as uptime from processes limit 5;
+------------------+--------+
| name | uptime |
+------------------+--------+
| [System Process] | |
| System | 7131 |
| Registry | 7154 |
| smss.exe | 7131 |
| csrss.exe | 7127 |
+------------------+--------+
```
* `percent_processor_time`: to obtain the total time that a process has been executing on a system, one can add together the `user_time` and `system_time` together to get the same value returned by the WMI value, the difference being that the sumation of `user_time` and `system_time` are in milliseconds whereas the values returned from WMI are in 100 nanosecond ticks:
osquery `percent_processor_time`:
```
osquery> select pid, name, (user_time + system_time) as percent_processor_time from processes where name = 'osqueryd.exe';
I1112 21:54:11.809412 2152 processes.cpp:461] Failed to open handle to process 0 with 203
+-------+--------------+------------------------+
| pid | name | percent_processor_time |
+-------+--------------+------------------------+
| 3892 | osqueryd.exe | 5952 |
| 808 | osqueryd.exe | 0 |
| 5892 | osqueryd.exe | 52374 |
| 11688 | osqueryd.exe | 280 |
+-------+--------------+------------------------+
```
Same data obtained with WMI.
```
λ Get-WmiObject -Query "Select IDProcess, Name, PercentProcessorTime from Win32_PerfRawData_PerfProc_Process" | Where-Object {$_.Name -like 'osqueryd*'}
IDProcess : 3892
Name : osqueryd
PercentProcessorTime : 59531250
IDProcess : 808
Name : osqueryd#1
PercentProcessorTime : 0
IDProcess : 5892
Name : osqueryd#2
PercentProcessorTime : 523750000
IDProcess : 11688
Name : osqueryd#3
PercentProcessorTime : 2812500
```
Some additional notes, just to verify we're getting all data:
```
C:\Users\Nick\work\repos\osquery [windows-process-rearch ↑]
λ .\build\windows10\osquery\RelWithDebInfo\osqueryd.exe -S --json "select count(*) from processes;"
[
{"count(*)":"160"}
]
C:\Users\Nick\work\repos\osquery [windows-process-rearch ↑]
λ C:\ProgramData\osquery\osqueryd\osqueryd.exe -S --json "select count(*) from processes;"
[
{"count(*)":"160"}
]
```
Also we achieve a considerable speed up from this re-arch:
```
C:\Users\Nick\work\repos\osquery [windows-process-rearch ↑]
λ (Measure-Command {.\build\windows10\osquery\RelWithDebInfo\osqueryd.exe -S --json "select count(*) from processes;"}).Milliseconds
168
C:\Users\Nick\work\repos\osquery [windows-process-rearch ↑]
λ (Measure-Command {C:\ProgramData\osquery\osqueryd\osqueryd.exe -S --json "select count(*) from processes;"}).Milliseconds
223
```
Pull Request resolved: https://github.com/facebook/osquery/pull/5293
Reviewed By: fmanco
Differential Revision: D13561337
Pulled By: muffins
fbshipit-source-id: 61435611d34d2cfd5f61ea52512a9dc208d259a5
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5464
While attempting to debug this issue I have discovered we are attempting to look up an empty item name in the registry. This doesn't make sense as we are clearly passing in an item name. This diff doesn't solve the issue, but it does add an error message for when it happens.
While we're here, appeasing Lint by using a more modern call to the `Status` return type.
Reviewed By: guliashvili
Differential Revision: D14207654
fbshipit-source-id: a53035ea84357f9ffe1d216b66e116257f291a7f
Summary: Trim whitespace in Darwin system_info table from hardware_serial column.
Reviewed By: jessek
Differential Revision: D14207715
fbshipit-source-id: 2948b8687c0c40d43f13ab42ddc4bbfec8e59a48
Summary:
This PR fixes#5106 by trimming the whitespace we get from the data returned from the macOS API. At first I thought this was an osquery regression, but going as far back as 2.x this looks like it has always been a problem.
Before the fix...
```
echo "select hardware_model, hardware_vendor, hardware_version from system_info;" | osqueryi --json | jq
[
{
"hardware_model": "MacBookPro13,3 ",
"hardware_vendor": "Apple Inc. ",
"hardware_version": "1.0 "
}
]
```
After the fix... (note the lack of trailing spaces)
```
echo "select hardware_version, hardware_vendor, hardware_model from system_info;" | /Users/jmeller/source/osquery/build/darwin10.13/osquery/osqueryi --json | jq
[
{
"hardware_model": "MacBookPro13,3",
"hardware_vendor": "Apple Inc.",
"hardware_version": "1.0"
}
]
```
Pull Request resolved: https://github.com/facebook/osquery/pull/5167
Reviewed By: jessek
Differential Revision: D14207353
Pulled By: guliashvili
fbshipit-source-id: 4012e04b38e2b5bfc299684baf86d0ab34cd42f8
Summary: as far as they are actually part of glibc, that would be less confusing. Later it will help us to deal with fully static build.
Reviewed By: marekcirkos
Differential Revision: D14183689
fbshipit-source-id: 8b712286e171305a49be6703b846f6f935b7dcc2
Summary: to be able to proxy it inside `osquery_tp_target` function.
Reviewed By: marekcirkos
Differential Revision: D14182918
fbshipit-source-id: d56b7135664ecd5afcc87b1c1cd2cde5394368a0
Summary:
It cause implicit dependency to osquery/core:core, that cause circular dependency :(. The point is to get rid of it. Let's use just exit() instead.
As far as we don't have decent implementation of Initializer::shutdown it will not cause us any more problems.
We can live with it for a while.
Reviewed By: marekcirkos
Differential Revision: D14182915
fbshipit-source-id: 22c54cde910ea29b1fb19d96598db381b9d1c60d
Summary:
This fixes a regression introduced in D13586036 .
The tailer downstream sometimes determines type by trying to parse the string as an int. We want this to consistently fail to do so when the value is in fact a float.
Reviewed By: SAlexandru
Differential Revision: D14161125
fbshipit-source-id: 8e2bb9c59559a5774813a7c7777257742f885c35
Summary: only events.cpp required by plugin_sdk, everything else is not necessary
Reviewed By: marekcirkos
Differential Revision: D14131365
fbshipit-source-id: 18e101b3e549d8403aa22ee3d5a5fadd2b296c82
Summary:
I didn't find a way to refer file in fbcode thrift_library from xplat. So, let's copy it.
There is something wrong with a namespaces in python - I'll fix it later. This is not a problem for now - file is not used.
Reviewed By: marekcirkos
Differential Revision: D14124542
fbshipit-source-id: 1060e23675a8086af494aa93037e1c613f250c71
Summary:
As far as this file is only for internal build, to be able to build osquery sdk against fbthrift I moved it to fbcode from oss.
Later if we want to osqeruy work with fbthrift we can copy/move it again
Reviewed By: marekcirkos
Differential Revision: D14124448
fbshipit-source-id: aaeadb40a6fac19621423b225cf6d522607d1313
Summary:
Hi! This PR adds a new column called `product_version` to the file table, which is only
populated when queries are done on Windows. It is a very minimal PR that uses an existing helper function (`windowsGetFileVersion`) to populate the column.
The column is not named `file_version`, despite the name of the helper function because the underlying data retrieved by that helper function is actually the `dwProductVersion*` fields of the `VS_FIXEDFILEINFO` struct. In the future, if we want to add a column that _actually_ contains the results of the `dwFileVersion*` fields, we can add a new column called `file_version` without modifying existing functionality.
Pull Request resolved: https://github.com/facebook/osquery/pull/5431
Differential Revision: D14169773
Pulled By: fmanco
fbshipit-source-id: 6fa7c92425fc92aa2e83a37383b1a8c796b17644
Summary: It cause problems with compiling against tp2 because in tp2 there is no such target. As far as I can see there is no necessity to depend on that target separatelly from gtest. So, the simplest solution is to remove it
Reviewed By: SAlexandru
Differential Revision: D14149801
fbshipit-source-id: fe0fcd0593bded7c8f2c6fd247b1d52bd29bac9f
Summary: This test is flaky on some systems. Need to investigate and fix.
Reviewed By: guliashvili
Differential Revision: D14149533
fbshipit-source-id: 7df570c84c66cad8bdc67a1286d91a7c7e8e3502
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5458
Use ASSERTs where necessary. ASSERTs are fatal and terminate the test immediately, so should be used when subsequent tests aren't meaningful in case of previous failures. Only use EXPECTED when it makes sense for the tests to continue.
Reviewed By: jessek
Differential Revision: D14138382
fbshipit-source-id: aae0cd8640e602ac467861303457f9cf017bc5e8
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5452
As suggested in another diff, this diff updates the language we use to describe the osquery licensing terms. We are changing all instances of
//This source code is licensed as defined on the LICENSE file found in the root directory of this source tree.//
to
//This source code is licensed in accordance with the terms specified in the LICENSE file found in the root directory of this source tree.//
We accomplish this with a codemod:
$ codemod -md xplat/osquery/oss --extensions cpp,h,in,py,sh,mm,ps1 "(.\s+)This source code is licensed as defined on the LICENSE file found in the(.*)root directory of this source tree\." "\1This source code is licensed in accordance with the terms specified in\2the LICENSE file found in the root directory of this source tree."
Reviewed By: fmanco
Differential Revision: D14131290
fbshipit-source-id: 52c90da342263e2a80f5a678ecd760c19cf7513e
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5449
Initial steps to separate plugins from the rest of osquery. On the long run separating plugins will provide more build flexibility such that we can have configurable builds that include only the bits and pieces we actually need per deployment. Reducing the attack surface, possibility of supply chain attacks, binary size, etc.
Move killswitch
Move test declaration to it's own BUCK file for consistency with the rest of the project.
Reviewed By: marekcirkos
Differential Revision: D14121618
fbshipit-source-id: 3e30e57befed4387585ed553ec087fdf8db6efc3
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5448
Initial steps to separate plugins from the rest of osquery. On the long run separating plugins will provide more build flexibility such that we can have configurable builds that include only the bits and pieces we actually need per deployment. Reducing the attack surface, possibility of supply chain attacks, binary size, etc.
Move logger
Reviewed By: marekcirkos
Differential Revision: D14121620
fbshipit-source-id: cef15e7cc354cbe597c6c6878ee63ff09b5fb06d
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5450
This file was originally written by wxsBSD in 2015. He has since joined Facebook and has graciously agreed to re-license this file to Facebook. This diff formalizes the relicensing by changing the copyright notice on the file. Note that wxsBSD still retains a copyright to all previous versions of the file.
Reviewed By: wxsBSD
Differential Revision: D14131447
fbshipit-source-id: 3148eafc0162a23b86e064a9784ea01b685164ef
Summary:
Initial steps to separate plugins from the rest of osquery. On the long run separating plugins will provide more build flexibility such that we can have configurable builds that include only the bits and pieces we actually need per deployment. Reducing the attack surface, possibility of supply chain attacks, binary size, etc.
Move distributed
Reviewed By: marekcirkos
Differential Revision: D14121619
fbshipit-source-id: 9ad8a837450874e79a819ab4f11258ae24ec8014
Summary:
Initial steps to separate plugins from the rest of osquery. On the long run separating plugins will provide more build flexibility such that we can have configurable builds that include only the bits and pieces we actually need per deployment. Reducing the attack surface, possibility of supply chain attacks, binary size, etc.
Move config and config_parser plugins
Reviewed By: marekcirkos
Differential Revision: D14119102
fbshipit-source-id: 0bc956398b3829c6f1013b38ebba2f0fc1071a93
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5445
This diff adds a Facebook copyright header to files in the osquery open source repository which:
* Facebook owns
* Do not currently have a Facebook copyright header
Reviewed By: marekcirkos
Differential Revision: D14122845
fbshipit-source-id: 5a0fea10189ec4ec893f7a036911fd51de0e01ae
Summary: Please never reuse tmp directory with the same name :(
Reviewed By: guliashvili
Differential Revision: D14066968
fbshipit-source-id: 164d0b9e6f34b102759bc5919dadc37197ff0798
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5438
currently the magic table is broken. libmagic which is used to generate this information needs a database/configuration file that it usually auto-finds.
Our libmagic library tries to open the following file ```open("/usr/local/osquery/Cellar/libmagic/5.32_200/share/misc/magic.mgc", O_RDONLY) = -1 ENOENT (No such file or directory)``` (you can generate this by using strace like ```trace -q -e trace=open ./buck-out/debug/gen/xplat/osquery/oss/osquery/osqueryd#gcc-5-glibc-2.23-clang -verbose -S "select * from magic where path = '/etc/passwd'"```).
How it auto-finds it I don't know 100%, but I guess it has something to with how the libmagic.so is actually build and installed. Basically this never works unless you are a developer on mac and used our previous build system.
I've updated the table to be able to specify the path to magic database file. If you don't specify it, I tried to check if one of the default files (files that should be present under /usr/share/ exists and use the first found). If all fail, I try the default one, but that most likely will fail.
Reviewed By: guliashvili
Differential Revision: D14066467
fbshipit-source-id: d9d2aca4829b2275e6792f974de1f2a7808dc321
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5428
This is a final diff to be able to track syscalls by using eBPF + kernel events. Basically that one and previous are about to join high level initialisation routine in one place.
Part of a linux tracing system, blueprint: [#5218](https://github.com/facebook/osquery/issues/5218)
Reviewed By: SAlexandru
Differential Revision: D13801093
fbshipit-source-id: db8503b0d42127281a975ff517600872e9ed4302
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5437
- test should not assume that fs::temp_directory_path() is always the same
- test should clean everithin up in TearDown() method
- tests should no depend on the order (test_decompression previously was depending on test_compression)
Reviewed By: mkareta
Differential Revision: D14064645
fbshipit-source-id: 653e2061c3de8e3fc30a4f0fc553831f22e62fb7
Summary:
This addresses a slight regression to ensure that we set `set_terminate_threads` on Windows. Without this flag being set, Windows threads will deadlock on exit as the boost managed io service threads never receive termination notifications.
I'm opening this PR up against the old master as I feel we should likely cut a 3.3.3, and I'm happy to re-open this PR against the upstream experimental as well, but we'll want a fix for this released as quickly as possible to Windows deployments.
Pull Request resolved: https://github.com/facebook/osquery/pull/5421
Reviewed By: marekcirkos
Differential Revision: D13972916
Pulled By: muffins
fbshipit-source-id: 55e3b23c80091d5fb51a97d1efc043b52dc48ba3
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5419
to load new programs, enable native events and attach program to them.
Reviewed By: SAlexandru
Differential Revision: D13787783
fbshipit-source-id: cfc001da15b343e5c80fd0ab6a276f263aa0ef7a
Summary: Removing flag which was declared but never used. enable_monitor
Reviewed By: marekcirkos
Differential Revision: D13958265
fbshipit-source-id: 3a812330950b101abdbd83ada4afd5b262cabd26
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5422
We were just de-prioritizing type of queries not constraining required columns. However, when the query is just useless without specific constraint, sqlite suggestion is to return SQLITE_CONSTRAINT status.
Reviewed By: marekcirkos
Differential Revision: D13964562
fbshipit-source-id: ee0e5f8baf9abbf83c34f7a39d2b5bd705cbac6d
Summary:
Fixed crash in virtual tables that occurs after following steps:
1. sqlite opens VT with xCreate
2. during query executions invokes xFilter with multiple tables
3. Few tables accumulated in affectedTables
4. xDestroy called before finishing query (last step of query execution)
5. query execution finished, SQL instance try to cleanup affected tables, but they were already destroyed by xDestory
This is only hotfix for this crash and this code base require full memory management review in future
Reviewed By: SAlexandru
Differential Revision: D13917015
fbshipit-source-id: 15396e47e4c4e592cf30608a783bc80d560c776f
Summary:
This diff adds Xcode support for osquery.
Part of this diff will be reverted in future after adding prebuilt library and platform deps support to buck.
To use it you need to build osquery in debug mode and then run buck with following flags:
--config osquery.xcode=true --config project.ide=xcode
Reviewed By: SAlexandru
Differential Revision: D13903315
fbshipit-source-id: 4d131964d7a61236f25d917dc060a2f3c3d782bc
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5412
Now on different platforms column `start_time` in `processes` table means different things. On Linux it is seconds since system boot, but it works correct only for some platforms, because the number of clock ticks per second was hardcoded. On windows it was abs unix time in seconds since Epoch. On macos it is a time in milliseconds (may be?) since system boot. On freeBSD as far as I can see it an abs time since boot, but also I'm not sure.
In order to make it consistent for all OS we changed to more convenient format - absolute time since Epoch. This commit is about macos.
Reviewed By: marekcirkos
Differential Revision: D13918625
fbshipit-source-id: eacb297358b36ce72cb0d5a7d9171553688ab2a3
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5414
Now on different platforms column `start_time` in `processes` table means different things. On Linux it is seconds since system boot, but it works correct only for some platforms, because the number of clock ticks per second was hardcoded. On windows it was abs unix time in seconds since Epoch. On macos it is a time in milliseconds (may be?) since system boot. On freeBSD as far as I can see it an abs time since boot, but also I'm not sure.
In order to make it consistent for all OS we changed to more convenient format - absolute time since Epoch. This commit is about Linux. Next diffs going to be about Darwin and freeBSD.
Reviewed By: guliashvili
Differential Revision: D13918626
fbshipit-source-id: a9cf0570dc6ac9fa125bc8233e9965c4e01566a6
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5417
Hash multimap based joiner with ability to perform clean up old unpaired events from time to time.
Part of a linux tracing system, blueprint: [#5218](https://github.com/facebook/osquery/issues/5218)
Reviewed By: SAlexandru
Differential Revision: D13761675
fbshipit-source-id: f4b17cbeed495b2a9e6616a005f001963849875e
Summary: For some monitoring, we need to include machine identity. Two example, I found is hostname or similar hostnames schema name.
Reviewed By: SAlexandru
Differential Revision: D13880705
fbshipit-source-id: e1d0238f4981adad1554d73f0ef6e5ef65a98c33
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5408
Move out c++ enums to std::string conversion function from
osquery/utils/error.h to separate module. To be able to use it somewhere else.
Reviewed By: guliashvili
Differential Revision: D13896772
fbshipit-source-id: 0a9f6327d5b2f115ce688446a67677879411eb1f
Summary: before this diff osqueryd was relying on linker order to use right main function, since gtest also contains main function
Reviewed By: guliashvili
Differential Revision: D13897622
fbshipit-source-id: d260b7496f513c7052f4db87c8e7ff9300493671
Summary:
This PR is the result of the discussion in a previous PR (#5348) after we determined account_policy_data was the wrong place for the column.
Add `is_hidden` column to the users and groups tables in macOS. `is_hidden` is populated by looking for the `dsAttrTypeNative:IsHidden` attribute in the OpenDirectory record for the user/group if the value is `1`, `True`, or `Yes` is_hidden is 1. If the value is anything else it's set to 0. Invalid values have the same affect as the attribute not existing at all.
The `dsAttrTypeNative:IsHidden` attribute controls whether a user account is is visible in the preferences panel similar to having a uid < 500.
One test failed when running buck test:
```
====STANDARD OUT====
tests/integration/tables/helper.cpp:159: Failure
Value of: boost::get<CustomCheckerType>(validator)(value)
Actual: false
Expected: true
Custom validator of the column "mask" with value "" failed
```
This also fails when I ran the test on the current experimental branch as well.
Important to note I had to remove the optimization on both the user and group tables that just called `getpwnam` if the query specified the `uid` or `gid` since the struct returned doesn't contain the `IsHidden` attribute. I'm not sure if or how much this will affect performance since I wasn't able to get the profiling to work with the new version (very likely I'm just doing it incorrectly).
Pull Request resolved: https://github.com/facebook/osquery/pull/5368
Differential Revision: D13862375
Pulled By: akindyakov
fbshipit-source-id: 1fec88a6ba71884f7e611e1d96ea00630c5be655
Summary: profiler file contained the CodeProfiler class, which is a bit odd. Thanks alecx
Reviewed By: akindyakov
Differential Revision: D13861320
fbshipit-source-id: 11d711fd1d15830d4cce0be084cc6da3b1c6589c
Summary:
Split query name and pack name with the delimiter(dot) which should not accrue inside their names(instead of _ or -).
Also log things per pack name
Reviewed By: SAlexandru
Differential Revision: D13854471
fbshipit-source-id: 181e7e19fcfb5d57a779cea6a2804eda09dc5a91
Summary:
Split query name and pack name with the delimiter(dot) which should not accrue inside their names(instead of _ or -).
Also log things per pack name
Reviewed By: mkareta
Differential Revision: D13854389
fbshipit-source-id: 35ba80813d98371dd36a83ec32aad677f24aa6eb
Summary: ScheduledQuery was declared both in query.h and in scheduled_query.h. Let's keep scheduled_query.h only.
Reviewed By: mkareta
Differential Revision: D13859371
fbshipit-source-id: 241b948d21fe619be360037a4e2e52a833a23b2b
Summary: In the interest of making changes smaller, I'm replicating stuff around QueryData to QueryDataTyped in intermediate commits. I'll come back and remove stuff that's no longer used when subsequent changes eliminate use of them.
Reviewed By: guliashvili
Differential Revision: D13741994
fbshipit-source-id: de0b00ddb0ad4b344d68ce799fe9fac759bb6199
Summary: Also it causes debug failure on unchecked Expected when status is not ok
Reviewed By: mkareta
Differential Revision: D13859377
fbshipit-source-id: 596410350cb91d469dc0a19f0e8eec558e8627bd
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5398
The file moved to a different location.
Reviewed By: marekcirkos
Differential Revision: D13817331
fbshipit-source-id: 6bd3947894daa712edae84b71502af997947b9b0
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5390
to eliminate the effect of dropping privileges in other threads causing poll-ing EINTR errors in thrift.
According to ref to [bugzilla.redhat](https://bugzilla.redhat.com/show_bug.cgi?id=473907) in case of changing privileges `glibc` sends SIGRT_1 to other threads which lead to poll be interrupted. On posix we can not have different credentials for thread of one process. Therefore the solution is either to do not use dropping privileges for the whole osquery process or patch all usages of poll in thrift code. I like first option more because playing with permissions of the whole `osqueryd` can cause unpredicted interferences between threads. For instance the same table can provide different results because some other thread dropping and regaining privileges at the same time.
So, the solution for now I'd like to suggest is remove dropping privileges from safe places like reading files with known hostnames or shell history files. And because we can not interact with apt/rpm/yum databases as root and should drop to none user for it I'd suggest to increase the number of attempts to poll in case of EINTR. It can significantly eliminate the problem for now.
To address the problem in issue: [#5326](https://github.com/facebook/osquery/issues/5326)
Thanks fmanco for the help to investigate this problem.
Reviewed By: fmanco
Differential Revision: D13781886
fbshipit-source-id: 4b1f2b7d20c925cc19ba79cc0a2906b65e815c0b
Summary: We went with 'long' rather than 'int64_t' in the JSON serialization methods that were added, so change our variant type to match
Reviewed By: marekcirkos
Differential Revision: D13675340
fbshipit-source-id: 1ccde4ce9f651fe68db968b367507aa67823c74f
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5385
Left shift with >= 31 steps was done to integer type. Using unisgned long long(1ULL) instead of the int (1).
Reviewed By: fmanco
Differential Revision: D13751355
fbshipit-source-id: 4564b33e2d26a0cb459ee86d180c0af492fa1f43
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5373
I made wrapper for the syscall perf_event_open to isolate syscall code definition and use `Expected<>` as a return value.
Reviewed By: mkareta
Differential Revision: D13622565
fbshipit-source-id: 616080a64376d4b042629513ad715b1d2458b8a2
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5375
LICENSE is now defined in a single file on the root of the project, update the
header to contain that information.
**Project LICENSE did not change.**
Reviewed By: akindyakov
Differential Revision: D13750575
fbshipit-source-id: 1e608a81b260b8395f9d008fc67f463160c1fc2b
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5370
Handler to enable event with certain type, get an id of the event and make sure it will be disables afterwards.
Part of a linux `syscalls` tracing system, blueprint: [#5218](https://github.com/facebook/osquery/issues/5218)
Reviewed By: mkareta
Differential Revision: D13621388
fbshipit-source-id: 8adfbe3cc1d87f70538851c5036eae51c93bede8
Summary:
This adds support for the `#includedir` and `#include` directives to the `sudoers` table, making `sudoers` behave more like the actual `sudo` rule parser:
* When an `includefile` directive is encountered, the referenced file will be parsed using the same rules as the top-level sudoers file.
* When an `includedir` directive is encountered, the referenced directory will be listed and each valid file within (i.e., each file *not* containing a `.` and *not* ending with `~`) will be parsed using the same rules as the top-level sudoers file.
* An additional `source` column tracks the file that provides the row's rule.
* Like `sudoers(5)`, nesting is limited to 128 individual files, with directory inclusions being counted once for each file they contain.
Pull Request resolved: https://github.com/facebook/osquery/pull/5350
Differential Revision: D13717394
Pulled By: akindyakov
fbshipit-source-id: 9659526f21e82c712c495caa80775b15d7e47e37
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5354
eBPF system call osquery wrapper with Expected as a return value and function to check if functionality is supported by current kernel.
Blueprint: [#5218](https://github.com/facebook/osquery/issues/5218)
Reviewed By: mkareta
Differential Revision: D13607442
fbshipit-source-id: 58be84a86aba3fe5e33ca5ab15418976fd36107c
Summary:
While running `misspell` on a different codebase. I happened to notice that some misspellings in the osquery code base. So, I fixed them
Pull Request resolved: https://github.com/facebook/osquery/pull/5256
Reviewed By: guliashvili
Differential Revision: D13670897
Pulled By: fmanco
fbshipit-source-id: 5d33d858284955c376e8c3980acdf366d4edf3d3
Summary:
I had previously attempted to land a fix to the extra newline that was trailing on Windows Event Logs, but had failed. This actually fixes the extra newline, which I verified. I'd love to have unit tests for this, however we don't currently have any unit tests setup for Windows Event Logging.
Pull Request resolved: https://github.com/facebook/osquery/pull/5271
Differential Revision: D13690844
Pulled By: fmanco
fbshipit-source-id: b7447fe2f6e2313c8e7f8765d636f7999d2b5909
Summary:
The set accidentally missed Saturday and Sunday days of the week. This just adds them in.
Fixes#5297.
Pull Request resolved: https://github.com/facebook/osquery/pull/5298
Reviewed By: guliashvili
Differential Revision: D13690812
Pulled By: fmanco
fbshipit-source-id: 1859d32ad635d63548435fed8e9da131530be5ce
Summary:
Add support for listing atom packages on Linux and Mac. Shouldn't be a problem adding support for Windows, but have no way to test it right now.
Lists package name, version, description, package path, license and homepage (see test).
Reviewed By: fiorix
Differential Revision: D13636097
fbshipit-source-id: f38a57128cedde2d027a0205588c8b563e2b188c
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5342
This sort of enumeration is required to pass posix system errors with `Expected<>`
Reviewed By: mkareta, fmanco
Differential Revision: D13608306
fbshipit-source-id: 0b787b12fac915f097dc310b616df507772da455
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5343
Just a parser for the semantic version in string. I gonna use it later to parse kernel version from the `int uname()` result.
Reviewed By: guliashvili
Differential Revision: D13607313
fbshipit-source-id: d6d01fe220f242d3811406dcd63d609feb0d6688
Summary: Currently osquery extension uses old osquery core SDK. So does not know, types other than none, sum, min, max(Others were added later) . Let's revert profiler reporting type until new SDK is generated, so that I'm not blocked in rolling out the profiler.
Reviewed By: akindyakov
Differential Revision: D13654143
fbshipit-source-id: 07ded5c6a1bc85d5c2ed983e7a55f3fff9826980
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5335
It was reported that osquery eventing mechanism uses too much disk space. As daebeike found it, event expiring was failing in some cases to be executed.
More specifically, expiration check was supposed to run every time EVENTS_CHECKPOINT number of events where added. However, in between the checks for expiration more than EVENTS_CHECKPOINT events could be added and no expiration would be executed. I suppose, this behaviour would be easily reproducible under the high load
Reviewed By: fmanco
Differential Revision: D13565250
fbshipit-source-id: 78bbad3f7aded4beb9e5f42bafd9184e9c2f8efb
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5346
Let's define win32 api version only inside of buck files, but not in cpp header
Reviewed By: guliashvili
Differential Revision: D13635704
fbshipit-source-id: cd978661ed6f733950363c2ac261811045263ed2
Summary: Some tools does not support virtual headers namespaces and expect header path to match actual file path from project root. This diff will fix few namespaces in utils library
Reviewed By: guliashvili
Differential Revision: D13552878
fbshipit-source-id: 2a06f73550c69777bf73be73abdde297fe580583
Summary:
On C++17 `NODISCARD` maps to a `[[nodiscard]]` attribute, which must
appear at the start of the function signature.
Reviewed By: j-calvert, guliashvili
Differential Revision: D13627821
fbshipit-source-id: d308650cf0188141a4942f528b7ec91825a0c318
Summary: As discussed in [an old PR](37ffdf8a48 (r213278358)) this is of questionable benefit and will get complicated when we change to typed values, so dropping.
Reviewed By: guliashvili
Differential Revision: D13631227
fbshipit-source-id: ada9b5434297d8c1c4b3e3855fe595faf5937bf6
Summary:
Continuing to march toward low-overhead, type-safe table rows, this commit
converts the Darwin `processes` table to using the generated row types.
My march concludes here, but there's a lot of work yet to be done with
migrating other tables.
(Adapted from https://github.com/facebook/osquery/pull/5199)
Reviewed By: guliashvili
Differential Revision: D13438014
fbshipit-source-id: 0e8365f26fe95131fe53ba3491cf07899600e997
Summary:
Continuing to march toward low-overhead, type-safe table rows, this commit
introduces the code generation for said rows. Nothing uses it yet; see the
next commit for that.
(Adapted from https://github.com/facebook/osquery/pull/5199)
Reviewed By: guliashvili
Differential Revision: D13438017
fbshipit-source-id: 959a6e092aee38d33e1c6539cbe14b85172c0135
Summary:
Continuing to march toward low-overhead, type-safe table rows, this commit
changes `TableRow` to be an interface rather than simply an alias for `Row`.
Accordingly, `DynamicTableRow` becomes an implementation of that interface
backed by a `Row`. The few remaining pieces of code that treated `TableRow`s as
`Row`s now call methods on the `TableRow` interface. Subsequent commits will
add code generation for strongly-typed table-specific implementations of
`TableRow`.
(Adapted from https://github.com/facebook/osquery/pull/5198)
Reviewed By: guliashvili
Differential Revision: D13438015
fbshipit-source-id: 61d5547e878e519c9706f94f844aab9d3e553410
Summary:
Continuing to march toward low-overhead, type-safe table rows, this commit changes
much of the code that uses `TableRow`s to stop assuming that they're just `vector`s
by another name. (`TableRow` is on the way to becoming an interface with multiple
implementations.) They're now held in `unique_ptr`s (`TableRowHolder`). For cases
where we really want a `vector`-backed `TableRow` (mostly test code and extension
support), we have a factory function (`make_table_row`) and a helper class
(`DynamicTableRowHolder`) to make that smoother.
(Adapted from https://github.com/facebook/osquery/pull/5198)
Reviewed By: mkareta
Differential Revision: D13438016
fbshipit-source-id: 2de9ce46a64c0a067b5d3299c59bbe3ccacd4abe
Summary:
Continuing to march toward low-overhead, type-safe table rows, this commit introduces
a distinction between rows being returned from a table (`TableRows`) and as the
result of a query (`QueryData`). Right now the two are simply aliases for each other;
that will change shortly.
(Adapted from https://github.com/facebook/osquery/pull/5198)
Reviewed By: guliashvili
Differential Revision: D13438019
fbshipit-source-id: 6563fc8c372d9d6c4b05705943ddf39b42260feb
Summary:
Step towards completion of https://github.com/facebook/osquery/pull/4904
- Add boost variant (typed) counterparts for RowData, Row, and QueryData.
- Add a queryInternal to sqlite_util that returns QueryDataTyped.
- Replace the queryInternal that returns QueryData with one that's a simple wrapper around the new typed implementation.
Reviewed By: guliashvili
Differential Revision: D13586036
fbshipit-source-id: d1d9b56470fbfcfb5802de422f87d9b9d6da7fb7
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5332
malloc might return null and memcpy will crash in that case
Reviewed By: akindyakov
Differential Revision: D13517060
fbshipit-source-id: 261f493823974dcc40a40139094eb4a98cd24dec
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5331
Let's use attributes to prevent such mistakes (see previous diff in the stack)
Reviewed By: guliashvili
Differential Revision: D13504146
fbshipit-source-id: 7cab2dd345599b036fa2a27bf682f2a159fd1c08
Summary:
Hand port of https://github.com/facebook/osquery/pull/5154 to the new build system
I'm moving toward generating constants for each column so that we
can do used column lookups without having to do a string set lookup,
but first I need to have the used columns information plumbed through
as a bitfield.
Once the code generation is in and all cases have been migrated to it,
I'll remove the string variants of `isColumnUsed`/`isAnyColumnUsed`.
Reviewed By: guliashvili
Differential Revision: D13423615
fbshipit-source-id: 6a2afe7dad42942610dfe0f6f55bcee4603a25af
Summary:
Manual port of https://github.com/facebook/osquery/pull/5153 to the new build system.
New methods to allow adding larger numeric types.
Reviewed By: guliashvili
Differential Revision: D13422612
fbshipit-source-id: 6b503553f05139351f803ea6fcc5c825e62c35ea
Summary:
This is a manual port of https://github.com/facebook/osquery/pull/5216 to the new build system.
This table allows you to fetch a list of running applications on OSX. Contains PID, Bundle ID and whether or not the application is in focus. (The latter can be used to figure out what application was being used at a given moment.)
Reviewed By: guliashvili
Differential Revision: D13422150
fbshipit-source-id: 236b28d9140a9a9062fb913815d0c9f1da21c8b5
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5318
This is most likely broken and is causing problems with some toolchains (e.g. clang on Ubuntu xenial).
Reviewed By: guliashvili
Differential Revision: D13415457
fbshipit-source-id: b4aa686913d3cfe6bb26089b0525144fae7cc928
Summary: These missing includes make the build fail with some toolchains.
Reviewed By: guliashvili
Differential Revision: D13414926
fbshipit-source-id: b9a7992e37751da81ec1f5a9055703bc1e3cc4bc
Summary: Switch off osquery/remote tests for windows, it takes time to fix it. I'll do it later.
Reviewed By: fmanco
Differential Revision: D13378357
fbshipit-source-id: 21077630864fc4a27ba65078c634e115875f3970
Summary: It turns out we already fixed that old issue
Reviewed By: guliashvili
Differential Revision: D13376876
fbshipit-source-id: a7822a1c79aa180e40f7ce54faa7c811b0d1b24b
Summary: Windows named pipe requires a bit different name format in compare to posix platforms.
Reviewed By: guliashvili
Differential Revision: D13376805
fbshipit-source-id: 443c8f963863adbacd1edb76945919f00a1a2e4e
The iptables uapi kernel headers are incompatible with C++. To avoid
depending on modified kernel headers query iptables from a C based
proxy. This is a temporary solution until the problem is fixed on
upstream Linux.
Directory creation function to osquery/filesystem.h
Status was used as a return value, as far as all function in this file operate with Status not Expected. Let's move from Status to Expected in another PR.
There are a lot of lookups in the maps the osquery code. Most of them are verbose and not-optimal with check if such key exists in the table before get access. Some of them consists error e.g.:
```c++
r["uid"] = row.count("uuid") > 0 ? row.at("uid") : "";
```
Introduced code will help to avoid the most of such problems.
FreeBSD supports renaming threads with pthread_np.
The difference with Linux or Darwin is that there's no error code:
"Because of the debugging nature of this function, all errors that may
appear inside are silently ignored"
This isn't really a problem because thread names are meant for debugging
and osquery does not check the retun value of `setThreadName()` anyway.
Test plan:
adrs@freebsd: procstat -t `pidof old_osqueryi`
PID TID COMM TDNAME CPU PRI STATE WCHAN
7612 100059 osqueryi - -1 152 sleep ttyin
7612 100162 osqueryi - -1 152 sleep uwait
7612 100163 osqueryi - -1 152 sleep select
adrs@freebsd: procstat -t `pidof osqueryi`
PID TID COMM TDNAME CPU PRI STATE WCHAN
7278 100151 osqueryi - -1 120 sleep ttyin
7278 100160 osqueryi ExtensionWatcher -1 120 sleep uwait
7278 100161 osqueryi ExtensionRunnerCore -1 131 sleep select
