Commit Graph

5158 Commits

Author SHA1 Message Date
seph
2c0da99016 Light refactor and fixes for the users and groups table
- Lightly update the logic, to make the code paths similar between
  no-constraint, and specified uid cases.
- Add tests to the group table.
2019-08-05 22:15:13 -04:00
Teddy Reed
1145140016 tables: Add optimization back to macOS users and groups
This PR supersedes #5669 with an alternate approach of moving forward
without a revert of #5368.
2019-08-05 22:15:13 -04:00
Alessandro Gario
4581ec42ce Remove deprecated build tools 2019-08-04 22:21:38 +02:00
Stefano Bonicatti
c2fde72fa6
Fix runningApps.test_sanity (#5670)
When run alone, using --gtest_filter to select the specific test,
it throws an exception because the tables are not registered.

It also wrongly checks for the query rows number to be >= 0 and then
proceeds to access the first one.
2019-07-29 15:47:30 +02:00
Stefano Bonicatti
f2514a347e
Fix python detection on Azure Pipelines (#5673)
A new Windows VM has been recently deployed that moves the
Python 2 installation, Windows Buck builds are now failing.

Support the old and new path for Python 2 and
support multiple minor versions of python instead of hardcoding one.
2019-07-29 10:26:23 +02:00
seph
debf92d943
Fix link in docs (#5666)
Update the hyperlink to tables.ht in the docs.
2019-07-24 21:19:33 -07:00
Teddy Reed
d7c7a1de88
Remove cloud tables from windows (#5657)
It does not look like the EC2 tables are available on Windows
(due to spec-file configuration).

Please see https://osquery.io/schema/3.3.2#ec2_instance_tags

This removes the EC2 library linking requirement until we revisit this
feature.
2019-07-23 08:13:03 -07:00
Stefano Bonicatti
bf8eb1c3b7
Specify compiler versions supported in BUILD.md (#5653)
Also add missing xcode-select --install in the CMake macOS guide

Fix osquery/osquery#5651
2019-07-23 16:51:29 +02:00
Teddy Reed
22d51ddc20 rpm: Check for glibc version and librpm secure_getenv
The librpm dependency may be configured with a glibc version < 2.17. If
so then it will expect __secure_getenv to be available.
2019-07-23 14:19:57 +02:00
Stefano Bonicatti
a5dfb3da7e
Improve osquery version detection (#5630)
Add OSQUERY_VERSION cache variable to force the version to be set at
that value.
Add OSQUERY_VERSION_AUTODECTED cache variable to show the osquery
version that has been detected through git
Add OSQUERY_VERSION_SUFFIX cache variable to append its value to the
final version, if the version was autodected.
Add OSQUERY_VERSION_INTERNAL variable to globally store the final
osquery version, to be used in other CMake scripts.
Add OSQUERY_VERSION_COMPONENTS variable to globally store the components
of the final osquery version. Currently used by the packaging system.

Modify overwrite_cache_variable so that's possible to specify the type.

The version is always autodetected using "git describe --tags --always --dirty"
and OSQUERY_VERSION_SUFFIX is appended to it, unless OSQUERY_VERSION is manually set.

Incorporates suggestions by @directionless.

CMake part of osquery/osquery#5615
2019-07-23 13:01:20 +02:00
seph
bf7e9f5c96
Don't return a battery row, if there are no results (#5650)
If there are no battery results, don't return a nil row. Return an empty set.
2019-07-20 21:23:05 -07:00
Stefano Bonicatti
9cf33c84f8 Restore AugeasTests test
PR: osquery/osquery#5629
2019-07-19 22:36:40 +02:00
Stefano Bonicatti
c56020cac3 Restore WindowsEventLogTests test
PR: osquery/osquery#5629
2019-07-19 22:36:40 +02:00
Stefano Bonicatti
36bb0b8b1e Restore ProcessTests test
Also add the same process name checks done on Windows to Linux.

PR: osquery/osquery#5629
2019-07-19 22:36:40 +02:00
Stefano Bonicatti
55e066322c Restore RegistryTablesTest test
PR: osquery/osquery#5629
2019-07-19 22:36:40 +02:00
seph
54f967a528
Change MSI Service Error handling (#5467)
The Wix configuration sets the service as _critical_. As I understand it, this means failures to start osquery, are considered startup failures, and will trigger a reboot. As there are occasional windows bugs causing a failure to start, this seems unfortunate. I think hit someone on slack today 

This PR changes that to _normal_

Docs are http://wixtoolset.org/documentation/manual/v3/xsd/wix/serviceinstall.html
2019-07-19 09:51:41 -07:00
seph
d10dab8443
Update WTSFreeMemoryEx to WTSFreeMemory (#5642)
Change `WTSFreeMemoryEx` to `WTSFreeMemory` as documented.

More context in:
- https://github.com/osquery/osquery/issues/4655
- http://redplait.blogspot.com/2018/06/interesting-case-of-memory-leak.html

Thanks for the bug report (redplait)[https://github.com/redplait]
2019-07-19 09:51:18 -07:00
Alex
f98403fcfc fixing the mixed line-endings causing PyAST error (#5645) 2019-07-18 20:10:30 -07:00
uptycs-nishant
37caa44787 crontab testcase fix (#5637) 2019-07-18 06:52:37 -07:00
Matteo Piano
880f003a09 allow mounting SQLite DBs using WAL journaling with ATC 2019-07-17 10:12:48 -07:00
uptycs-nishant
e5eb30ea5f Replacing sync calls by async ones and some cleanup (#5606) 2019-07-16 18:22:30 -07:00
Mark Mossberg
a60b940290 windows/certificates: Fix enumeration bugs, add columns (#5631)
* Initial implementation

* Use case insensitive comparisons for all service names

Fixes a bug where certificates for services that correspond to Local Service or
Network Service may not have their sids appear correctly. This is because the
services table is inconsistent with its user_account column.

* Make service name cache query-local

Previously, the service name cache existed for the lifetime of the
osquery process, which made it susceptible to stale reads if a service
restarted under a different user during osquery's lifetime. Now the
cache is created for each query. Also refactor it to directly map to the
sid, rather than the account name, which removes the need to translate
from account name to sid every row.

* Fix reference to destroyed object

Previously, getCurrentUserInfo took a reference to data from a local
vector, whose data is free'd after the function. This refactors the code
to use a unique_ptr (similar to how getSidFromUsername) does it.
2019-07-16 17:34:39 +02:00
Stefano Bonicatti
31e35ae276 Add buck builds on Azure Pipelines (#5632)
First part of osquery/osquery#5620

Remove travis.
2019-07-11 17:12:54 -07:00
Stefano Bonicatti
72c72b775f Explicitly add BUILD_TESTING CMake option (#5627)
This way is visible as a variable in the cache that can be set.
2019-07-10 03:57:10 +02:00
Stefano Bonicatti
08c36472cf Enable Debug builds on the CI (#5626)
* Add an option to avoid generating debug symbols

A new option, OSQUERY_NO_DEBUG_SYMBOLS, has been added
to avoid generating debug symbols when building in Debug or RelWithDebInfo.
This is especially needed on the CI where the disk size is limited
and symbols are not necessary, because we cannot directly access
processes to debug them.

* Enable Debug builds on the CI

Enabled Debug builds on macOS and Linux; a special job has been created
to represent the combined build status of Release and Debug builds.

Also format the other long cmakeArgs line so that there's one argument
per line.

* Run Docker as privileged also in Debug builds

* Simplify azure pipelines configuration

Use strategy and matrix to avoid repeating the script
for Release and Debug builds.
2019-07-10 00:32:26 +02:00
Stefano Bonicatti
2f681e7bdc Fix SystemsTablesTests.test_abstract_joins
The code was failing on Windows because the '\Windows\%' pattern
is relative and presumes that the Windows folder is on the same
drive (C:\) as the test process.
This might not be true, so we find where precisely is the Windows
directory and use the full path to it as a pattern.

The failing test checks that were testing the equal and LIKE operator
have been moved to a new test, test_table_constraints, since they
are not related to joins.
2019-07-09 12:59:45 +02:00
Stefano Bonicatti
dc02738db6 Disable InterfaceDetailsTest.test_sanity test on Windows
The table it tests has several issues, it doesn't properly check
for errors in several places and returns unexpected values in
the columns.
Moreover the code could also be improved to make testing possible
without passing through SQL queries, by separating the part
that uses WMI queries from the conversion to row results.
2019-07-09 12:59:45 +02:00
Stefano Bonicatti
65aa1cfa3f Fix several integer conversions in process_ops
Fix UsersTest.test_sanity on Windows.

uid and gid were returned as int (while they normally are unsigned int)
and converted to signed integers in the table row.
This is wrong because beyond uid and gid not being ints,
they are taken from the RID part of the SID which in some cases,
like for a Service SID, it can have a value higher than then maximum
value of an int, so in the end the number shown in table is negative.

Now they are returned as uint32_t and converted as BIGINTs for the table
that uses them.

Fix other functions return values and conversions depending on the meaning of
the value.
On Windows stick to its specific types where possible.

Convert CRLF to LF on some of the files modified.
2019-07-09 12:59:45 +02:00
Stefano Bonicatti
f34afd223c Fix ConfigTests.test_content_update
Do not checkout with carriage returns on Windows, otherwise it would
mess up with code that expect unix style files.
2019-07-09 12:59:45 +02:00
Stefano Bonicatti
6e928157b3 Fix ebpf tests failing.
Fix ebpfTests.sysEbpf_null_attr, ebpfTests.sysEbpf_create_map,
ebpfMapTests.int_key_int_value, ebpfMapTests.int_key_struct_value
by running Docker as privileged on Azure Pipelines.

Docker is used only to get a new distribution running, it's not used
for any security purpose, so there's no point in limiting it.
2019-07-09 12:59:45 +02:00
Nick Anderson
4eede7b00b
buck: fixing up buck config generation script for use on Windows (#5623) 2019-07-05 15:49:47 -07:00
Gustavo Grieco
cc1693a805 Fix typo in base64 log message (#5622) 2019-07-03 01:34:25 +02:00
Stefano Bonicatti
e7fde8ad8e Fix wrong error code returned when querying the Windows registry (#5621)
Calling GetLastError() will not return the error code for the Windows registry APIs,
since they return a LSTATUS value which already the error code.

This also fixes the RegistryTablesTest.test_registry_non_existing_key
test case which was incorrectly expecting success when querying
for a non-existent registry key.

Ported from https://github.com/osql/osql/pull/50
2019-07-01 22:23:02 +02:00
Stefano Bonicatti
d6a2f2f1df Fix detection of some headers on some IDEs (#5619)
Some IDE require that the headers are assigned to a target, so they know
which TU uses them and also that they are part of the project.
The option ADD_HEADERS_AS_SOURCES has been added, if it's ON,
we assign those headers as INTERFACE sources of the target that publish
them.
2019-06-30 23:24:56 +02:00
Stefano Bonicatti
961bc2c6ed
Fix prepare_for_ide target on macOS and Windows (#5618)
This is a hotfix to let the prepare_for_ide target work on Windows and
macOS, avoiding to download libudev that it's not needed and
doesn't exist for those platforms.

In theory for all the third-party libraries that do not exist for a
platform we should have an INTERFACE library as a substitute,
instead of avoiding to define any target.
We wait to do this change, because we are going to refactor how
dependencies are downloaded and imported.
2019-06-30 23:02:48 +02:00
Stefano Bonicatti
dc7ffb820a
Fix tests output and yaml formatting for Azure Pipelines (#5613)
Apparently there's a bug in the visualization of the logs,
where color codes eat characters.
So we disable tests colored output until a fix on Azure Pipelines appears.

Formatting a bit more consistently the yaml file.
2019-06-29 12:48:08 +02:00
Stefano Bonicatti
07216ac894
Add Azure Pipelines status badges to the README (#5607)
Also restore the old table format with all the useful project urls
2019-06-29 02:48:26 +02:00
Gustavo Grieco
b4bbcd274a Improve the downloader.py script to use urllib2 (#5612)
* Improve the downloader.py script to use urllib2

This will allow to automatically handle URL re-directions (e.g. 302)

* Better exception handling
2019-06-29 02:28:12 +02:00
Stefano Bonicatti
2addb87602
Merge Azure Pipelines build and test jobs (#5610)
Originally the separation existed because there were multiple branches,
and only one of them was protected by PRs.
So broken commits could land and differentiating from broken build
or tests was useful.

This is not true anymore and PRs checks are per pipeline, not per job,
so the separation wouldn't make a difference.
2019-06-28 08:59:14 -04:00
Will-Low
ae25976afd Fixing port logic (#5576) 2019-06-28 08:31:25 -04:00
Stefano Bonicatti
ce5fee3dd4 Update README and BUILD with CMake instructions 2019-06-26 21:49:06 -04:00
Stefano Bonicatti
f8e279b4b9 Add Azure Pipelines support
Taken from osql-experimental.

- Use AppleClang compiler for macOS

- Run format_check on Linux

- Run pipeline only on master
2019-06-26 21:49:06 -04:00
Stefano Bonicatti
826723c29a Fix boost asio string_view detection hack
Only define BOOST_ASIO_DISABLE_STD_STRING_VIEW.

We shouldn't define BOOST_ASIO_HAS_STD_STRING_VIEW,
because even if we define BOOST_ASIO_DISABLE_STD_STRING_VIEW
the first define will actually enable parts of code that will use string_view.
This won't work on Windows and in general, string_view should not be
used unless compiling with C++17.

The hack has been also added to a test that was previously missed.
2019-06-26 21:49:06 -04:00
Stefano Bonicatti
942878854b Add CMake support
Taken from osql-experimental.

- Change CMake code license to the one present in osquery right now

- Package metadata doesn't mention Trail of Bits or osql anymore

- Set specific ACLs for the osqueryd on Windows when packaging

- Remove LLVM_INSTALL_PATH support on macOS, since we are using AppleClang

- Remove OSQUERY_SOURCE_DIR variable need and source in a submodule support

- Add targets format_check and format to check code formatting and
  format it with clang-format

- Do not warn about not using Clang on macOS when using AppleClang
2019-06-26 21:49:06 -04:00
Alessandro Gario
33fbbecb5f Initial commit
Taken from osql-experimental.

Initial support for Linux and macOS.
2019-06-26 21:49:06 -04:00
seph
a2b8d40cbb
Initial codeowners (#5603)
This adds a codeowners file. The intent here, is to provide a route to opening up more people to vet and approve commits. Committers should be added to the [osquery/osquery-committers](https://github.com/orgs/osquery/teams/osquery-committers) team. This will grant them write permissions, and cause them to be notified on PRs.

We will need to _additionally_ toggle the codeowners button in master branch protections.

This does _not_ provided a clearly visible audit trail for who has been added to `osquery-committers` or a process for doing that.

Fixes: https://github.com/osquery/foundation/issues/1
2019-06-26 09:14:13 -04:00
Teddy Reed
a5692c194f travis: Wait for up to 30 minutes for buck build 2019-06-25 08:49:11 -04:00
Teddy Reed
597a0c613d buck: Remove quotes from project/buck_out config 2019-06-25 08:49:11 -04:00
Teddy Reed
23c9a87efe travis: Add homebrew update true and use adoptopenjdk8
Note that java8 is no longer provided by homebrew cask-versions:
homebrew-cask-versions/pull/7261
2019-06-25 08:49:11 -04:00
Teddy Reed
e6fe15eb49
macos: Add hack for boost asio string_view detection (#5592) 2019-06-23 01:05:42 -04:00