Hello, this largish PR adds shellbags support to osquery.
Shellbags is a complex (imo) windows Registry artifact that primarily keeps track of directories a user has browsed to (specifically directories accessed using Windows Explorer). By parsing shellbags its possible to recreate what directories a user accessed
Shellbags are composed of shellitems, this PR also adds support to parsing several shellitems, finally shellbags also contain FAT timestamps that show when a directory was created, modified, accessed, FAT timestamp parsing is also included in this PR
Example query of what shellbags looks like
```
osquery> select * from shellbags;
+-----------------------------------------------+--------------+-------------------------------------------------------------------------------------+---------------+--------------+---------------+-----------+--------------+
| sid | source | path | modified_time | created_time | accessed_time | mft_entry | mft_sequence |
+-----------------------------------------------+--------------+-------------------------------------------------------------------------------------+---------------+--------------+---------------+-----------+--------------+
| S-1-5-21-1079689790-2336414676-942872339-1001 | usrclass.dat | This PC\C:\Users\bob | 0 | 0 | 0 | 0 | 0 |
| S-1-5-21-1079689790-2336414676-942872339-1001 | usrclass.dat | This PC\C:\Users\bob\Downloads | 1571635108 | 1571620406 | 1571635108 | 3074 | 5 |
| S-1-5-21-1079689790-2336414676-942872339-1001 | usrclass.dat | This PC\C:\Users\bob\Projects | 0 | 0 | 0 | 0 | 0 |
| S-1-5-21-1079689790-2336414676-942872339-1001 | usrclass.dat | This PC\C:\Users\bob\Projects\osquery | 0 | 0 | 0 | 0 | 0 |
| S-1-5-21-1079689790-2336414676-942872339-1001 | usrclass.dat | This PC\C:\Users\bob\Projects\osquery\build | 0 | 0 | 0 | 0 | 0 |
| S-1-5-21-1079689790-2336414676-942872339-1001 | usrclass.dat | This PC\C:\Users\bob\Projects\osquery\build\osquery | 0 | 0 | 0 | 0 | 0 |
| S-1-5-21-1079689790-2336414676-942872339-1001 | usrclass.dat | This PC\C:\Users\bob\Projects\osquery\build\osquery\RelWithDebInfo | 0 | 0 | 0 | 0 | 0 |
| S-1-5-21-1079689790-2336414676-942872339-1001 | usrclass.dat | This PC\C:\Users\bob\Projects\osquery\osquery | 1578192498 | 1571701478 | 1578192498 | 495902 | 4 |
| S-1-5-21-1079689790-2336414676-942872339-1001 | usrclass.dat | This PC\C:\Users\bob\Projects\osquery\osquery\killswitch | 1578192406 | 1575859554 | 1578192406 | 707032 | 2 |
| S-1-5-21-1079689790-2336414676-942872339-1001 | usrclass.dat | This PC\C:\Users\bob\Projects\clamav-osquery | 1572045050 | 1572045050 | 1572045050 | 221518 | 14 |
| S-1-5-21-1079689790-2336414676-942872339-1001 | usrclass.dat | This PC\C:\Users\bob\Projects\clamav-osquery\extension_clamav | 1572045050 | 1572045050 | 1572045050 | 432733 | 8 |
| S-1-5-21-1079689790-2336414676-942872339-1001 | usrclass.dat | This PC\C:\Users\bob\Projects\clamav-osquery\extension_clamav\src | 1572045050 | 1572045050 | 1572045050 | 432736 | 11 |
| S-1-5-21-1079689790-2336414676-942872339-1001 | usrclass.dat | This PC\C:\Users\bob\.osquery | 1571706262 | 1571706212 | 1571706262 | 575462 | 4 |
| S-1-5-21-1079689790-2336414676-942872339-1001 | usrclass.dat | This PC\C:\Users\bob\AppData | 1571623328 | 1571623318 | 1571623328 | 206482 | 7 |
| S-1-5-21-1079689790-2336414676-942872339-1001 | usrclass.dat | This PC\C:\Users\bob\AppData\Local | 1571701908 | 1571623318 | 1571701908 | 206502 | 8 |
| S-1-5-21-1079689790-2336414676-942872339-1001 | usrclass.dat | This PC\C:\Users\bob\AppData\Local\Microsoft | 1593297370 | 1571623318 | 1593297370 | 206504 | 8 |
| S-1-5-21-1079689790-2336414676-942872339-1001 | usrclass.dat | This PC\C:\Users\bob\AppData\Local\Microsoft\Office | 1593297370 | 1593295160 | 1593297370 | 52684 | 60 |
| S-1-5-21-1079689790-2336414676-942872339-1001 | usrclass.dat | This PC\C:\Users\bob\AppData\Local\Microsoft\Office\16.0 | 1593297388 | 1593295160 | 1593297388 | 81742 | 11 |
| S-1-5-21-1079689790-2336414676-942872339-1001 | usrclass.dat | This PC\C:\Users\bob\AppData\Local\autopsy | 1612935328 | 1612935328 | 1612935328 | 37041 | 7 |
| S-1-5-21-1079689790-2336414676-942872339-1001 | usrclass.dat | This PC\C:\Users\bob\AppData\Local\autopsy\Cache | 1612935328 | 1612935328 | 1612935328 | 37104 | 8 |
| S-1-5-21-1079689790-2336414676-942872339-1001 | usrclass.dat | This PC\C:\Users\bob\super secret sensitive stuff | 1613198066 | 1613198066 | 1613198066 | 101729 | 123 |
| S-1-5-21-1079689790-2336414676-942872339-1001 | usrclass.dat | This PC\C:\Users\bob\super secret sensitive stuff\secret data | 1613198092 | 1613198092 | 1613198092 | 383733 | 38 |
| S-1-5-21-1079689790-2336414676-942872339-1001 | usrclass.dat | This PC\C:\Users\bob\super secret sensitive stuff\secret data\dont look secret data | 1613198108 | 1613198108 | 1613198108 | 383736 | 15 |
+-----------------------------------------------+--------------+-------------------------------------------------------------------------------------+---------------+--------------+---------------+-----------+--------------+
```
Due to the complexity of shellbags currently this PR does not support (or only has partial support) for the following shellitems:
* optical disc
* variable (partial support)
* mtp (partial support)
* user property view data (partial support)
I was not able to generate shellbag data (or only some data) for the above shellitems in my windows vms (tested on two different Windows 10 systems, Windows 8.1, and Windows Server 2019), if osquery encounters any unknown shellbag data it will log a warning and mark the shellitem as "[UNKNOWN SHELL FORMAT]" when building directory paths.
The main value of shellbags is reconstructing directories accessed as shown above, but this PR does include additional shellbag support such as FTP servers connected to via Windows Explorer, ZIP files opened, MTP devices (partial), and network shares browsed to via Windows Explorer
This PR is kind of large, let me know if there are any questions, suggestions for improvements, or issues, thanks!
Shellbags references:
[Shellitems](https://github.com/libyal/libfwsi/blob/main/documentation/Windows%20Shell%20Item%20format.asciidoc)
[Shellbags](https://www.magnetforensics.com/blog/forensic-analysis-of-windows-shellbags/)
[Property Stores](https://github.com/libyal/libfwps/blob/main/documentation/Windows%20Property%20Store%20format.asciidoc)
The "path" column for a systemd unit row can be empty.
Added missing possible values for the "status" column,
when the "type" column value is "systemd unit".
Removed "Login Item" as a possible value for the "type" column,
since now "Startup Item" is used.
Removed "disabled" as a possible value for the "status" column,
since it's not returned anymore and due to the type "Startup Item"
being either a script or a .desktop file, which do not have a disabled state;
if they need to be they'll just be removed
and they won't appear in the table anymore.
Separately check for the "status" column possible values
since they differ based on the "type" column value.
Added the OSQUERY_BUILD_AWS option due to
oss-fuzz CI failing to build the AWS EC2 library,
due to its high requirement of memory.
The fuzzers do not gain much from having these tables enabled,
moreover this can be used during development
to avoid building a possibly unneeded table.
Done some minor cleanups to remove logic duplication
and enable ec2 integration tests on all the platforms
that supports them.
The instruction doesn't exist on other processors, and while there
might be ways to retrieve the equivalent information, none directly
matches the x86 "cpuid". This takes out the table completely on
non-x86_64 architectures.
Co-Authored-By: Benjamin Herrenschmidt <benh@kernel.crashing.org>
The PR adds an on-demand version of the Windows events log table that can return the WEL entries from an arbitrary event provider. It supports selective filtering based on additional query parameters. An example query:
```
osquery> select * from windows_events_vtable where channel = 'Application' and eventid = 1000;
+-------------+--------------------------------+------+-------+----------------------------+----------------------------------------+---------+----------+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------+------+------+
| channel | datetime | task | level | provider_name | provider_guid | eventid | recordid | keywords | data | pid | tid |
+-------------+--------------------------------+------+-------+----------------------------+----------------------------------------+---------+----------+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------+------+------+
| Application | 2020-07-22T03:19:32.617052900Z | 0 | 4 | Microsoft-Windows-LoadPerf | {122EE297-BB47-41AE-B265-1CA8D1886D40} | 1000 | -1 | 0x8000000000000000 | {"UserData":{"EventXML":"","binaryData":"EC2C0000BC2D0000ED2C0000BD2D0000","binaryDataSize":"16","param1":"WmiApRpl","param2":"WmiApRpl"}} | 5824 | 5828 |
+-------------+--------------------------------+------+-------+----------------------------+----------------------------------------+---------+----------+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------+------+------+
```
This is for consistency with the decisions taken
for PR osquery/osquery#6413 and osquery/osquery#6414.
We also update the integration test helper to accept custom constraints,
in the case the table has required columns.
This simple helper takes a table name and
an already prepared rows map to add the container
columns, which are normally hidden, and call validate_rows on it.
We use this function in deb_packages, rpm_packages, npm_packages,
os_version integration tests.
