osquery-1/CHANGELOG.md

4.3.0

Git Commits

New Features / Under the Hood improvements

• Change verbosity of scheduled query execution messages from INFO to verbose only (#6271)
• Updated the unwanted-chrome-extensions queries to include all users, not the osquery process owner only (#6265)
• Check for errors in the return status of the extension tables and report them (#6108)
• First steps to properly support UTF8 strings on Windows (#6190)
• Display the undelying API error string when udev monitoring fails (#6186)
• Add the path column to the ATC generate specs (#6278)
• Add Kafka support to Microsoft Windows (#6095)
• Log a warning message if osquery fails to get the service description on Microsoft Windows (#6281)
• Make AWS kinesis status logging configurable (#6135)
• Add an integration test for the disk_info table (#6323)
• Use -1 for missing ppid in the process_events table (#6339)
• Remove error when converting empty numeric rows (#6371)
• Change verbosity from ERROR to INFO of access failures to system processes on Microsoft Windows (#6370)
• Make possible to get verbose messages from the dispatcher service management on Microsoft Windows too (#6369)

Build

• Fix codegen template for extension group (#6244)
• Update SQLite from 3.30.1-1 to 3.31.1 (#6252)
• Update the osquery-toolchain to version 1.1.0 which uses LLVM/Clang 9.0.1 (#6315)
• Update openssl to version 1.1.1f (#6302, #6359)
• Simplify formula-based third party libraries build (#6303)
• Removed the Buck build system (#6361)

Bug Fixes

• Fix CFNumber conversion when the type was a Float64/32 instead of a Double (#6273)
• Fix duplicate results being returned by the chrome_extensions table (#6277)
• Fix flaky ProcessOpenFilesTest.test_sanity (#6185)
• Fix the --database_dump flag for RocksDB not outputting anything (#6272)
• Fix the pci_devices table pci ids extraction in non-existing paths (#6297)
• Fix parsing an invalid decorators config (#6317)
• Fix flaky TLSConfigTests.test_runner_and_scheduler (#6308)
• Fix chromeExtensions.test_sanity (#6324)
• Fix broken Unicode filename searches on Microsoft Windows (#6291)
• Fix a use-after-free when sqlite attempts to access the entire rows data at the end of a query (#6328)
• Keep proc instance for test_base and test_osqueryd (#6335)
• Fix osquery not exiting when given check or dump requests (#6334)
• Fix process table cmdline parsing (#6340)
• Fix a crash when parsing files with libmagic (#6363)
• Fix a sporadic readFile API failure when using non-blocking I/O (#6368)
• Fix the MSI package not always installing in the system drive by default (#6379)
• Ensure the extensions uuid is never 0 (#6377)
• Fix a race condition making the watcher act as a worker on Microsoft Windows (#6372)
• Fix extensions tables detaching which was sometimes failing (#6373)
• Fix an issue with extensions re-registration (#6374)
• Fix a crash due to a race condition in accessing the iokit port on Darwin (Apple OS X) (#6380)

Hardening

• Limit SQL functions regex_match and regex_split regex size (#6267)
• Prevent a stack overflow when parsing deeply nested configs (#6325)

Table Changes

• Added table chrome_extension_content_scripts to All Platforms (#6140)
• Added table docker_container_fs_changes to POSIX-compatible Plaforms (#6178)
• Added table windows_security_center to Microsoft Windows (#6256)
• Added many new tables to Linux to query lxd (#6249)
• Added table screenlock to Darwin (Apple OS X) (#6243)
• Added table userassist to Microsoft Windows (#5539)
• Added column status (TEXT) to table deb_packages (#6341)
• Added many new columns to the curl_certificate table (#6176)
• Added table socket_events to Darwin (Apple OS X) (#6028)
• Added table hvci_status, previously inadvertly left out from the build, to Microsoft Windows (6378)

4.2.0

Git Commits

New Features / Under the Hood improvements

• TLS Testing infrastructure has been overhauled (#6170)
• Boost regex has been replaced with std (#6236)
• community_id_v1 added as a SQL function (#6211)

Build

• Fix format checking on Windows (#6188)
• Fix format folder exclusions for build checks (#6201)
• Fix the linking for extensions in build (#6219)
• Fix build to include windows optional features table (#6207)

Security Issues

• [CVE-2020-1887] osquery does not properly verify the SNI hostname (#6197)

Bug Fixes

• Carver no longer returns empty carves for hidden files (#6183)
• Address a race in the Dispatcher logic (#6145)
• Fix validation in 'last' table (#6147)
• Fix flaky logger testing (#6171)
• Fix JSON format assumptions in file_paths parsing (#6159)
• Fix windows WMI BSTR to be wstrings (#6175)
• Fix windows string <-> wstring conversion functions (#6187)
• Enable more intelligent path expansion on Windows (#6153)
• Fix heap buffer overflow in callDoubleFunc and powerFunc (#6225)

Table Changes

• Added table firefox_addons to All Platforms (#6200)
• Added table ssh_configs to All Platforms (#6161)
• Added table user_ssh_keys to All Platforms (#6161)
• Added table mdls to Darwin (Apple OS X) (#4825)
• Added table hvci_status to Microsoft Windows (#5426)
• Added table ntfs_journal_events to Microsoft Windows (#5371)
• Added table docker_image_layers to POSIX-compatible Plaforms (#6154)
• Added table process_open_pipes to POSIX-compatible Plaforms (#6142)
• Added table apparmor_profiles to Ubuntu, CentOS (#6138)
• Added table selinux_settings to Ubuntu, CentOS (#6118)
• Added column lock_status (INTEGER_TYPE) to table bitlocker_info (#6155)
• Added column percentage_encrypted (INTEGER_TYPE) to table bitlocker_info (#6155)
• Added column version (INTEGER_TYPE) to table bitlocker_info (#6155)
• Added column optional_permissions (TEXT_TYPE) to table chrome_extensions (#6115)
• Removed table firefox_addons from POSIX-compatible Plaforms (#6200)
• Removed table ssh_configs from POSIX-compatible Plaforms (#6161)
• Removed table user_ssh_keys from POSIX-compatible Plaforms (#6161)

4.1.2

Git Commits

New Features / Under the Hood improvements

• Add more tests throughout the codebase (#5908), (#6071), (#6126)
• The chrome_extensions table now supports Chromium and Brave (#6126)

Build

• Require Python 3.5 and greater (#6081), (#6120)
• Prepare Python tests for CI (lots of effort!) (#6068)
• Restore osqueryd integration test (#6116)

Bug Fixes

• Continue to use com.facebook.osquery.plist for Launch Daemon configuration (#6093)
• Update systemd service to use KillMode=control-group (#6096)
• RPM and DEB packages both have post-install scripts to reload systemd (#6097)
• Update Windows package build script to include cert bundle (#6114)
• Update table specs to fix constraints passing (#6103), (#6104), (#6105), (#6106), (#6122)

Table Changes

• Added tables azure_instance_tags and azure_instance_metadata to Linux and Microsoft Windows (#5434)
• Added column install_time (INTEGER_TYPE) to table rpm_packages (#6113)
• Added column bsd_flags (TEST_TYPE) to table file on Darwin (#5981)

4.1.1

Git Commits

New Features / Under the Hood improvements

• Improve nvram table to use input variable names (#6053)
• Improve apt_sources source detection (#6047)
• Change atom_packages to use user constraints (#6052)
• Re-enable required-column warning messages (#6038)

Build

• Migrate several libraries to the CMake source layer (#5902), (#6023)
• Update SQLite from 3.29.0-3 to 3.30.1-1 (#6020)
• Recommend building with MacOS 10.11 SDK (#6000)

Bug Fixes

• Fix Linux audit incorrect read and handle leak (#5959)
• Change "logNumericsAsNumbers" to "numerics" logger top-level key (#6002)
• Restore INDEX behavior for extensions (#6006)
• Fix potential JSON parsing issues in ATC plugin (#6029)
• Avoid scanning special files with YARA (#5971)
• Fix use-after-move in YARA subscriber (#6054)
• Handle relative redirects in internal HTTP clients (#6049)
• Apply options config parsing before others (#6050)

Table Changes

• Added table windows_optional_features to Microsoft Windows #5991)

4.1.0

Git Commits

New Features / Under the Hood improvements

• Restore extension SDK and build support (#5851)
• Documentation improvements (#5860), (#5852), (#5912), (#5954)
• Add more tests throughout the codebase (#5837), (#5832), (#5857), (#5864), (#5855), (#5869), (#5871), (#5885), (#5903), (#5879), (#5914), (#5941), (#5957)
• Allow configuration more Linux Audit settings using flags (#5953)
• Add logger_tls_max_lines flag (#5956)
• Add AWS Session Token support (#5944)

Build

• Lots of work on CPack-based packaging (#5809), (#5822), (#5823), (#5827), (#5780), (#5850), (#5843), (#5881), (#5825), (#5940), (#5951), (#5936)
• Lots of work porting Python2 to Python3 (#5846)
• Upgrade OpenSSL to 1.0.2t on all platforms (#5928)
• Use SQLite 3.29.0 on Windows and macOS (#5810)
• Use aws-sdk-cpp source-builds on Windows and macOS (#5889)
• Add various code quality checks and utilities (#5834), (#5730), (#5872)

Hardening

• Restore fuzzing harness and use oss-fuzz (#5844), (#5886), (#5910), (#5915), (#5923), (#5955), (#5963)
• Use newer RapidJSON and switch to safer iterative parsing (#5893), (#5913)

Bug Fixes

• Set Windows MSI ErrorControl to normal instead of critical (#5818)
• Wrap flagfile with quotes for Windows install flag (#5824)
• Improve submodule usages in CMake (#5850), (#5880), (#5892), (#5897), (#5907)
• Improve locking support in internal APIs (#5841), (#5906), (#5943), (#5944)
• Fixes for macOS application layer firewall tables (#5378)
• Fixes within BPF event tables (#5874)
• Refactor and improve PCI device tables on Linux (#5446)
• Implement PID indexing on Windows processes table (#5919)
• Improve WHERE IN() performance (#5924), (#5938)
• Improve the internal HTTP client (#5891), (#5946), (#5947)
• Fix Windows version codename lookup (#5887)

Table Changes

• Added table alf_services to Darwin (Apple OS X) (#5378)
• Added table connectivity to Microsoft Windows (#5500)
• Added table default_environment to Microsoft Windows (#5441)
• Added table windows_security_products to Microsoft Windows (#5479)
• Added column platform_mask (INTEGER_TYPE) to table osquery_info (#5898)

4.0.2

This release fixes crashes identified in 4.0.1. There are no changes in functionality.

Git Commits

Bug Fixes

• Fix configuration of AWS libraries to address crash in Linux (#5799)
• Remove RocksDB optimization causing crash (#5797)

4.0.1

This release has two major focuses. It is the first release since osquery transitioned to a Linux Foundation project.

It features a heavily reworked build system. This aims to provide flexibility and stability.

Git Commits

New Features / Under the Hood improvements

• Linux Audit process_events Implement support for fork/vfork/clone/execveat (#5701)
• New SQLite function regex_match to match across columns (#5444)
• LRU cache for syscall tracing (#5521)
• Basic tracing via eBPF on Linux (#5403, #5386, #5384)
• Experimental kill and setuid syscall tracing in Linux via eBPF (#5519)
• New eventing (ev2) framework (#5401)
• Improved table performance profiles (#5187)
• macOS query pack: detect SearchAwesome malware (#5713)
• macOS query pack: detect when a process is tapping keyboard event (#5345)

Build

• Refactor CMake build (#5604, #5627, #5630, (#5618), (#5619))
• Refactor third-party libraries to build from source on Linux (#5706)
• Add Azure Pipelines support for CI/CD (#5604, #5632, #5626, #5613, #5607, #5673, #5610)
• Add Buck as a build system (971bee44)
• Use urllib2 to automatically handle HTTP 301/302 redirections (#5612)
• Update MSI package to install to Program Files on Windows (#5579)
• Linux custom toolchain integration (#5759)

Harderning

• Link binaries with Full RELRO on Linux (#5748)
• Remove FTS features from SQLite (#5703) (#5702)
• Fix SQLite API usage errors (#5551)
• Fix issues reported by ASAN (#5665)
• Handle bad FDs in md_tables (#5553)
• Fix lock resource leak in events/syslog (#5552)
• Fix memory leak in macOS keychain_items and extended_attributes tables (#5550, #5538)
• Fix memory leak in genLoggedInUsers (Windows). Update WTSFreeMemoryEx to WTSFreeMemory (#5642)
• Fix potential null dereferences in smbios_tables (#5332)
• Fix osquery exiting with wrong status (3824c2e6)
• Add additional install and uninstall flag incompatibility check (85eb77a0)
• Fix warning with constants initialisation in magic (2a624f2f)
• Fix sign compare warning in file_compression (b93069b3)
• Refactored logical_drives table on Windows (#5400)
• Refactored core/windows/wmi to use smart pointers (#5492)
• Fixed various potential crashes in the virtual table implementaion (6ade85a5)
• Increase the amount of MaxRecvRetries for Thrift sockets (#5390)

Bug Fixes

• Fix the reading of the serial of a certificate (little-endian big int) (#5742)
• Fix bugs and update pathname variables in MSI package build script (#5733)
• Fix registry table exception closing an uninitialized key handle (#5718)
• Config views are now recreated on startup (#5732)
• Change MSI Service Error handling on Windows (#5467)
• Allow mounting SQLite DBs using WAL journaling with ATC (#5525, #5633)
• Fix mount table interacting with direct autofs (#5635)
• Fix HTTP Host Header to include port (#5576)
• Various fixes to the Windows certificates table and expansion to include Personal certificates (#5697), (#5696), (#5640), (#5631)
• Add optimization back to macOS users and groups (#5684)
• Do not return a row for macOS battery if no data is present (#5650)
• Fix several integer conversions in process_ops (#5614)
• Include weekends on the kernel_panics table (#5298)
• Fix key_strength bug for Windows certificates table (#5304)
• The interface column of routes table could be empty on Windows (bcf0ab8e)
• The name column of programs table could be empty on Windows (7bceba4b)
• Fix disable_watcher flag (08dc11b7)
• Populate path column correctly in firefox_addons table (#5462)
• Fix numeric monitoring plugin not being registered (#5484)
• Fix wrong error code returned when querying the Windows registry (#5621)
• Fix logical_drives boot partition detection (#5477)
• Replace sync calls by async within the HTTP client implementation (#5606)
• Fix RocksDB crash related to OptimizeForSmallDb (a31d7582)
• Fix bug in table column data validator (e3037331)
• Fix random port problem (a32ed7c4)
• Refactor battery table and return information even if advanced information is missing (6a64e353)

Table Changes

• Added table ibridge_info on macOS (Notebooks only) (#5707)
• Added table running_apps on macOS (#5216)
• Added table atom_packages on macOS and Linux (6d159d40)
• Remove EC2 tables on Windows (#5657)
• Added column win_timestamp to time table on Windows (3bbe6c51)
• Added column is_hidded to users and groups table on macOS (#5368)
• Added column profile to chrome_extensions table (#5213)
• Added column epoch to rpm_packages table on Linux (#5248)
• Added column sid to logged_in_users table on Windows (#5454)
• Added column registry_hive to logged_in_users table on Windows (#5454)
• Added column sid to certificates table on Windows (#5631)
• Added column store_location to certificates table on Windows (#5631)
• Added column store to certificates table on Windows (#5631)
• Added column username to certificates table on Windows (#5631)
• Added column store_id to certificates table on Windows (#5631)
• Added column product_version to file table on Windows (#5431)
• Added column source to sudoers table on POSIX systems (#5350)