* Initial implementation
* Use case insensitive comparisons for all service names
Fixes a bug where certificates for services that correspond to Local Service or
Network Service may not have their sids appear correctly. This is because the
services table is inconsistent with its user_account column.
* Make service name cache query-local
Previously, the service name cache existed for the lifetime of the
osquery process, which made it susceptible to stale reads if a service
restarted under a different user during osquery's lifetime. Now the
cache is created for each query. Also refactor it to directly map to the
sid, rather than the account name, which removes the need to translate
from account name to sid every row.
* Fix reference to destroyed object
Previously, getCurrentUserInfo took a reference to data from a local
vector, whose data is free'd after the function. This refactors the code
to use a unique_ptr (similar to how getSidFromUsername) does it.
The code was failing on Windows because the '\Windows\%' pattern
is relative and presumes that the Windows folder is on the same
drive (C:\) as the test process.
This might not be true, so we find where precisely is the Windows
directory and use the full path to it as a pattern.
The failing test checks that were testing the equal and LIKE operator
have been moved to a new test, test_table_constraints, since they
are not related to joins.
Fix UsersTest.test_sanity on Windows.
uid and gid were returned as int (while they normally are unsigned int)
and converted to signed integers in the table row.
This is wrong because beyond uid and gid not being ints,
they are taken from the RID part of the SID which in some cases,
like for a Service SID, it can have a value higher than then maximum
value of an int, so in the end the number shown in table is negative.
Now they are returned as uint32_t and converted as BIGINTs for the table
that uses them.
Fix other functions return values and conversions depending on the meaning of
the value.
On Windows stick to its specific types where possible.
Convert CRLF to LF on some of the files modified.
Calling GetLastError() will not return the error code for the Windows registry APIs,
since they return a LSTATUS value which already the error code.
This also fixes the RegistryTablesTest.test_registry_non_existing_key
test case which was incorrectly expecting success when querying
for a non-existent registry key.
Ported from https://github.com/osql/osql/pull/50
Only define BOOST_ASIO_DISABLE_STD_STRING_VIEW.
We shouldn't define BOOST_ASIO_HAS_STD_STRING_VIEW,
because even if we define BOOST_ASIO_DISABLE_STD_STRING_VIEW
the first define will actually enable parts of code that will use string_view.
This won't work on Windows and in general, string_view should not be
used unless compiling with C++17.
The hack has been also added to a test that was previously missed.
Taken from osql-experimental.
- Change CMake code license to the one present in osquery right now
- Package metadata doesn't mention Trail of Bits or osql anymore
- Set specific ACLs for the osqueryd on Windows when packaging
- Remove LLVM_INSTALL_PATH support on macOS, since we are using AppleClang
- Remove OSQUERY_SOURCE_DIR variable need and source in a submodule support
- Add targets format_check and format to check code formatting and
format it with clang-format
- Do not warn about not using Clang on macOS when using AppleClang
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5553
Infer flagged this use of `unique_ptr` as potentially leaking the FD. I do not think this is the case but there is a missing check for a failed `open`. I am unsure of what the `ioctl` would do in this case.
I removed the custom decl to make the logic in this table easier (opinion).
Reviewed By: guliashvili
Differential Revision: D14700412
fbshipit-source-id: cfcfe9d2629aa27f18d39058830cb5c63881f163
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5550
This fixes a potential memory leak in keychain_items similar to D14567925. This leak was identified with the default options for OSS infer.
Reviewed By: guliashvili
Differential Revision: D14641455
fbshipit-source-id: 3fd776fe8a2af300435907ecd1d7746b3045066e
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5542
We replace deprecated calls to the Status class with newer ones. This will stop Lint from nagging me every time I open these files. There should be no change to functionality.
Specifically, we use a codemod on `.cpp` files to replace any instance of `return Status();` with `return Status::success();`.
Reviewed By: guliashvili
Differential Revision: D14652922
fbshipit-source-id: 243576195ed201d6b82fc942a785742c6b01cf83
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5538
This bug was found using the OSS version of infer with default options.
Reviewed By: guliashvili
Differential Revision: D14567925
fbshipit-source-id: 64272880464577207552e3436754b7f6d32ef4ec
Summary: It requires some effort to fix warnings properly, let's just mute them for now.
Reviewed By: marekcirkos
Differential Revision: D14596617
fbshipit-source-id: 184937d15cc186afe045131d6d648cdbfb4ad574
Summary:
This is a follow-up to #5400, fixing the `boot_partition` column.
Previous versions of the table would consistently provide a false negative for bootable drives, thanks to a mismatch in format between `Win32_DiskPartition.DeviceID` and `Win32_LogicalDisk.DeviceID`. This change replaces the use of `Win32_DiskPartition` with `Win32_BootConfiguration` and moves the query to the top-level, fixing the bug and removing the need to make `N` WMI requests for `N` system drives.
Pull Request resolved: https://github.com/facebook/osquery/pull/5477
Differential Revision: D14386989
Pulled By: akindyakov
fbshipit-source-id: 3843f760dd41dc5810048ccf969dc4b0d4674cdc
Summary:
In my testing of Osquery, I noticed the `path` field for `firefox_addons` is always blank. After inspecting the structure of the JSON and the code in Osquery, I believe the key to look up path should be renamed from `descriptor` to `path`.
Pull Request resolved: https://github.com/facebook/osquery/pull/5462
Differential Revision: D14301220
Pulled By: akindyakov
fbshipit-source-id: b4b56d7026736344da585aada83d5fea6e249f1c
Summary: Please do not use message of Status for anything except logging!
Reviewed By: fmanco
Differential Revision: D14281607
fbshipit-source-id: e48ed3d79de19acb1d86885d0acfab781e5058aa
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5483
Initial steps to separate plugins from the rest of osquery. On the long run separating plugins will provide more build flexibility such that we can have configurable builds that include only the bits and pieces we actually need per deployment. Reducing the attack surface, possibility of supply chain attacks, binary size, etc.
Move killswitch
Reviewed By: guliashvili
Differential Revision: D14259760
fbshipit-source-id: deaaa148fac25b3f534a8881a1ea9eda0ec6bdc9
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5487
I got tired of Lint telling me to update the calls to the Status class, so I wrote some codemods to update the vast majority of them:
$ cd ~/fbsource
$ codemod -d xplat/osquery/oss/osquery --extensions cpp,h "return Status\(0\, \".*\"\);" "return Status::success();"
**Blindly accepted all changes**.
FWIW, I tried to do something similar with the failure return values, but the unit tests were failing afterward.
Reviewed By: guliashvili
Differential Revision: D14278739
fbshipit-source-id: b023f6e969be697eeb67d66ea3ad10d16c69ff0c
Summary:
This introduces two new (Windows-only) columns to the `logged_in_users` table:
* `sid` corresponds to the logged in user's security identifier, used to uniquely identify the user and their permissions on the local system.
* `registry_hive` corresponds to the user's HKU registry hive, used to look up per-user configuration information.
I've updated the integration tests to test for these columns on Windows only. Please let me know if there's anything else I can do!
Pull Request resolved: https://github.com/facebook/osquery/pull/5454
Differential Revision: D14195466
Pulled By: fmanco
fbshipit-source-id: def9c362fac1b5a68b68f826916daafee224295b
Summary:
This generally refactors the `logical_drives` table on Windows to conform more closely to C++11 idioms. It also enables the integration test for `logical_drives`.
See #5367. I'll open a PR for the boot partition fixes once this is merged.
cc akindyakov guliashvili
Pull Request resolved: https://github.com/facebook/osquery/pull/5400
Differential Revision: D14131722
Pulled By: fmanco
fbshipit-source-id: c3077da48147a9880ce08925d165e5d1da363bb9
Summary: To be able to build with different `thrift_impl` modules in extensions (like facebook does with fbthrift) I removed implementation from `extensions` lib and explicitly used for `osqueryd` and for all tests. This allows us to remove `thrift_impl` from sdk dependencies list. If you need it - please use explicitly as I did for `osqueryd`. If you want to use your own implementation of `thrift_impl` - build it and link against it and sdk simultaneously.
Reviewed By: marekcirkos
Differential Revision: D14224206
fbshipit-source-id: 218dc05d87e1ffaca37783185fc672a20684d757
Summary:
tables: re-architect the processes table to not use WMI queries
This PR re-writes the processes table to no longer rely on WMI queries. We do this to ensure that the processes table is robust and performant, as we rely on this table for the watcher process to regulate the worker processes. Further, this table further employs the selective column generation introduced to the processes table on darwin to allow us to more selectively generate column data only when necessary.
Lastly, this table removes a couple of extraneous columns from the Windows table schema, as these column values are achievable via sqlite logic. Specifically:
* `elapsed_time`: to get this column one can use the `start_time` column along with the sqlite built in `strftime` function:
```
osquery> select name, (strftime('%s', 'now') - start_time) as uptime from processes limit 5;
+------------------+--------+
| name | uptime |
+------------------+--------+
| [System Process] | |
| System | 7131 |
| Registry | 7154 |
| smss.exe | 7131 |
| csrss.exe | 7127 |
+------------------+--------+
```
* `percent_processor_time`: to obtain the total time that a process has been executing on a system, one can add together the `user_time` and `system_time` together to get the same value returned by the WMI value, the difference being that the sumation of `user_time` and `system_time` are in milliseconds whereas the values returned from WMI are in 100 nanosecond ticks:
osquery `percent_processor_time`:
```
osquery> select pid, name, (user_time + system_time) as percent_processor_time from processes where name = 'osqueryd.exe';
I1112 21:54:11.809412 2152 processes.cpp:461] Failed to open handle to process 0 with 203
+-------+--------------+------------------------+
| pid | name | percent_processor_time |
+-------+--------------+------------------------+
| 3892 | osqueryd.exe | 5952 |
| 808 | osqueryd.exe | 0 |
| 5892 | osqueryd.exe | 52374 |
| 11688 | osqueryd.exe | 280 |
+-------+--------------+------------------------+
```
Same data obtained with WMI.
```
λ Get-WmiObject -Query "Select IDProcess, Name, PercentProcessorTime from Win32_PerfRawData_PerfProc_Process" | Where-Object {$_.Name -like 'osqueryd*'}
IDProcess : 3892
Name : osqueryd
PercentProcessorTime : 59531250
IDProcess : 808
Name : osqueryd#1
PercentProcessorTime : 0
IDProcess : 5892
Name : osqueryd#2
PercentProcessorTime : 523750000
IDProcess : 11688
Name : osqueryd#3
PercentProcessorTime : 2812500
```
Some additional notes, just to verify we're getting all data:
```
C:\Users\Nick\work\repos\osquery [windows-process-rearch ↑]
λ .\build\windows10\osquery\RelWithDebInfo\osqueryd.exe -S --json "select count(*) from processes;"
[
{"count(*)":"160"}
]
C:\Users\Nick\work\repos\osquery [windows-process-rearch ↑]
λ C:\ProgramData\osquery\osqueryd\osqueryd.exe -S --json "select count(*) from processes;"
[
{"count(*)":"160"}
]
```
Also we achieve a considerable speed up from this re-arch:
```
C:\Users\Nick\work\repos\osquery [windows-process-rearch ↑]
λ (Measure-Command {.\build\windows10\osquery\RelWithDebInfo\osqueryd.exe -S --json "select count(*) from processes;"}).Milliseconds
168
C:\Users\Nick\work\repos\osquery [windows-process-rearch ↑]
λ (Measure-Command {C:\ProgramData\osquery\osqueryd\osqueryd.exe -S --json "select count(*) from processes;"}).Milliseconds
223
```
Pull Request resolved: https://github.com/facebook/osquery/pull/5293
Reviewed By: fmanco
Differential Revision: D13561337
Pulled By: muffins
fbshipit-source-id: 61435611d34d2cfd5f61ea52512a9dc208d259a5
Summary: Trim whitespace in Darwin system_info table from hardware_serial column.
Reviewed By: jessek
Differential Revision: D14207715
fbshipit-source-id: 2948b8687c0c40d43f13ab42ddc4bbfec8e59a48
Summary:
This PR fixes#5106 by trimming the whitespace we get from the data returned from the macOS API. At first I thought this was an osquery regression, but going as far back as 2.x this looks like it has always been a problem.
Before the fix...
```
echo "select hardware_model, hardware_vendor, hardware_version from system_info;" | osqueryi --json | jq
[
{
"hardware_model": "MacBookPro13,3 ",
"hardware_vendor": "Apple Inc. ",
"hardware_version": "1.0 "
}
]
```
After the fix... (note the lack of trailing spaces)
```
echo "select hardware_version, hardware_vendor, hardware_model from system_info;" | /Users/jmeller/source/osquery/build/darwin10.13/osquery/osqueryi --json | jq
[
{
"hardware_model": "MacBookPro13,3",
"hardware_vendor": "Apple Inc.",
"hardware_version": "1.0"
}
]
```
Pull Request resolved: https://github.com/facebook/osquery/pull/5167
Reviewed By: jessek
Differential Revision: D14207353
Pulled By: guliashvili
fbshipit-source-id: 4012e04b38e2b5bfc299684baf86d0ab34cd42f8
Summary: as far as they are actually part of glibc, that would be less confusing. Later it will help us to deal with fully static build.
Reviewed By: marekcirkos
Differential Revision: D14183689
fbshipit-source-id: 8b712286e171305a49be6703b846f6f935b7dcc2
Summary:
Hi! This PR adds a new column called `product_version` to the file table, which is only
populated when queries are done on Windows. It is a very minimal PR that uses an existing helper function (`windowsGetFileVersion`) to populate the column.
The column is not named `file_version`, despite the name of the helper function because the underlying data retrieved by that helper function is actually the `dwProductVersion*` fields of the `VS_FIXEDFILEINFO` struct. In the future, if we want to add a column that _actually_ contains the results of the `dwFileVersion*` fields, we can add a new column called `file_version` without modifying existing functionality.
Pull Request resolved: https://github.com/facebook/osquery/pull/5431
Differential Revision: D14169773
Pulled By: fmanco
fbshipit-source-id: 6fa7c92425fc92aa2e83a37383b1a8c796b17644
Summary: It cause problems with compiling against tp2 because in tp2 there is no such target. As far as I can see there is no necessity to depend on that target separatelly from gtest. So, the simplest solution is to remove it
Reviewed By: SAlexandru
Differential Revision: D14149801
fbshipit-source-id: fe0fcd0593bded7c8f2c6fd247b1d52bd29bac9f
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5452
As suggested in another diff, this diff updates the language we use to describe the osquery licensing terms. We are changing all instances of
//This source code is licensed as defined on the LICENSE file found in the root directory of this source tree.//
to
//This source code is licensed in accordance with the terms specified in the LICENSE file found in the root directory of this source tree.//
We accomplish this with a codemod:
$ codemod -md xplat/osquery/oss --extensions cpp,h,in,py,sh,mm,ps1 "(.\s+)This source code is licensed as defined on the LICENSE file found in the(.*)root directory of this source tree\." "\1This source code is licensed in accordance with the terms specified in\2the LICENSE file found in the root directory of this source tree."
Reviewed By: fmanco
Differential Revision: D14131290
fbshipit-source-id: 52c90da342263e2a80f5a678ecd760c19cf7513e
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5450
This file was originally written by wxsBSD in 2015. He has since joined Facebook and has graciously agreed to re-license this file to Facebook. This diff formalizes the relicensing by changing the copyright notice on the file. Note that wxsBSD still retains a copyright to all previous versions of the file.
Reviewed By: wxsBSD
Differential Revision: D14131447
fbshipit-source-id: 3148eafc0162a23b86e064a9784ea01b685164ef
Summary:
Initial steps to separate plugins from the rest of osquery. On the long run separating plugins will provide more build flexibility such that we can have configurable builds that include only the bits and pieces we actually need per deployment. Reducing the attack surface, possibility of supply chain attacks, binary size, etc.
Move config and config_parser plugins
Reviewed By: marekcirkos
Differential Revision: D14119102
fbshipit-source-id: 0bc956398b3829c6f1013b38ebba2f0fc1071a93
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5438
currently the magic table is broken. libmagic which is used to generate this information needs a database/configuration file that it usually auto-finds.
Our libmagic library tries to open the following file ```open("/usr/local/osquery/Cellar/libmagic/5.32_200/share/misc/magic.mgc", O_RDONLY) = -1 ENOENT (No such file or directory)``` (you can generate this by using strace like ```trace -q -e trace=open ./buck-out/debug/gen/xplat/osquery/oss/osquery/osqueryd#gcc-5-glibc-2.23-clang -verbose -S "select * from magic where path = '/etc/passwd'"```).
How it auto-finds it I don't know 100%, but I guess it has something to with how the libmagic.so is actually build and installed. Basically this never works unless you are a developer on mac and used our previous build system.
I've updated the table to be able to specify the path to magic database file. If you don't specify it, I tried to check if one of the default files (files that should be present under /usr/share/ exists and use the first found). If all fail, I try the default one, but that most likely will fail.
Reviewed By: guliashvili
Differential Revision: D14066467
fbshipit-source-id: d9d2aca4829b2275e6792f974de1f2a7808dc321
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5412
Now on different platforms column `start_time` in `processes` table means different things. On Linux it is seconds since system boot, but it works correct only for some platforms, because the number of clock ticks per second was hardcoded. On windows it was abs unix time in seconds since Epoch. On macos it is a time in milliseconds (may be?) since system boot. On freeBSD as far as I can see it an abs time since boot, but also I'm not sure.
In order to make it consistent for all OS we changed to more convenient format - absolute time since Epoch. This commit is about macos.
Reviewed By: marekcirkos
Differential Revision: D13918625
fbshipit-source-id: eacb297358b36ce72cb0d5a7d9171553688ab2a3
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5414
Now on different platforms column `start_time` in `processes` table means different things. On Linux it is seconds since system boot, but it works correct only for some platforms, because the number of clock ticks per second was hardcoded. On windows it was abs unix time in seconds since Epoch. On macos it is a time in milliseconds (may be?) since system boot. On freeBSD as far as I can see it an abs time since boot, but also I'm not sure.
In order to make it consistent for all OS we changed to more convenient format - absolute time since Epoch. This commit is about Linux. Next diffs going to be about Darwin and freeBSD.
Reviewed By: guliashvili
Differential Revision: D13918626
fbshipit-source-id: a9cf0570dc6ac9fa125bc8233e9965c4e01566a6
Summary:
This PR is the result of the discussion in a previous PR (#5348) after we determined account_policy_data was the wrong place for the column.
Add `is_hidden` column to the users and groups tables in macOS. `is_hidden` is populated by looking for the `dsAttrTypeNative:IsHidden` attribute in the OpenDirectory record for the user/group if the value is `1`, `True`, or `Yes` is_hidden is 1. If the value is anything else it's set to 0. Invalid values have the same affect as the attribute not existing at all.
The `dsAttrTypeNative:IsHidden` attribute controls whether a user account is is visible in the preferences panel similar to having a uid < 500.
One test failed when running buck test:
```
====STANDARD OUT====
tests/integration/tables/helper.cpp:159: Failure
Value of: boost::get<CustomCheckerType>(validator)(value)
Actual: false
Expected: true
Custom validator of the column "mask" with value "" failed
```
This also fails when I ran the test on the current experimental branch as well.
Important to note I had to remove the optimization on both the user and group tables that just called `getpwnam` if the query specified the `uid` or `gid` since the struct returned doesn't contain the `IsHidden` attribute. I'm not sure if or how much this will affect performance since I wasn't able to get the profiling to work with the new version (very likely I'm just doing it incorrectly).
Pull Request resolved: https://github.com/facebook/osquery/pull/5368
Differential Revision: D13862375
Pulled By: akindyakov
fbshipit-source-id: 1fec88a6ba71884f7e611e1d96ea00630c5be655
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5385
Left shift with >= 31 steps was done to integer type. Using unisgned long long(1ULL) instead of the int (1).
Reviewed By: fmanco
Differential Revision: D13751355
fbshipit-source-id: 4564b33e2d26a0cb459ee86d180c0af492fa1f43
Summary:
Pull Request resolved: https://github.com/facebook/osquery/pull/5375
LICENSE is now defined in a single file on the root of the project, update the
header to contain that information.
**Project LICENSE did not change.**
Reviewed By: akindyakov
Differential Revision: D13750575
fbshipit-source-id: 1e608a81b260b8395f9d008fc67f463160c1fc2b
Summary:
This adds support for the `#includedir` and `#include` directives to the `sudoers` table, making `sudoers` behave more like the actual `sudo` rule parser:
* When an `includefile` directive is encountered, the referenced file will be parsed using the same rules as the top-level sudoers file.
* When an `includedir` directive is encountered, the referenced directory will be listed and each valid file within (i.e., each file *not* containing a `.` and *not* ending with `~`) will be parsed using the same rules as the top-level sudoers file.
* An additional `source` column tracks the file that provides the row's rule.
* Like `sudoers(5)`, nesting is limited to 128 individual files, with directory inclusions being counted once for each file they contain.
Pull Request resolved: https://github.com/facebook/osquery/pull/5350
Differential Revision: D13717394
Pulled By: akindyakov
fbshipit-source-id: 9659526f21e82c712c495caa80775b15d7e47e37
