- Update links to documentation
- Update link to assets by editing `getAssetURL()`
- Prettier did some reformatting of the email template code which is resulting in a larger amount of lines changed
- Add policy.rego file defining authorization policies.
- Add Go integrations to evaluate Rego policies (via OPA).
- Add middleware to ensure requests without authorization check are rejected (guard against programmer error).
- Add authorization checks to most service endpoints.
- Move team-related service methods to `ee/server/service`.
- Instantiate different service on startup based on license key.
- Refactor service errors into separate package.
- Add support for running E2E tests in both Core and Basic tiers.
- Add config option for license key.
- Define license details data structure.
- Include license details in app config API responses.
Currently any non-empty value for `--license_key` behaves as though the
installation is licensed for `basic`. If the license key is empty,
`core` is returned.
Still to come is the appropriate parsing for the license key.
1. use [staticcheck](https://staticcheck.io/) to check the code, and fix some issues.
2. use `go fmt` to format the code.
3. use `go mod tidy` clean the go mod.
Instead of synchronously updating the seen_time column for a host on an update, batch these updates to be written together every 1 second.
This results in a ~33% reduction in MySQL CPU usage in a local test with 4,000 simulated hosts and MySQL running in Docker.
This adds the option to set up an S3 bucket as the storage backend for file carving (partially solving #111).
It works by using the multipart upload capabilities of S3 to maintain compatibility with the "upload in blocks" protocol that osquery uses. It does this basically replacing the carve_blocks table while still maintaining the metadata in the original place (it would probably be possible to rely completely on S3 by using object tagging at the cost of listing performance). To make this pluggable, I created a new field in the service struct dedicated to the CarveStore which, if no configuration for S3 is set up will be just a reference to the standard datastore, otherwise it will point to the S3 one (effectively this separation will allow in the future to add more backends).
This change optimizes live queries by pushing the computation of query
targets to the creation time of the query, and efficiently caching the
targets in Redis. This results in a huge performance improvement at both
steady-state, and when running live queries.
- Live queries are stored using a bitfield in Redis, and takes
advantage of bitfield operations to be extremely efficient.
- Only run Redis live query test when REDIS_TEST is set in environment
- Ensure that live queries are only sent to hosts when there is a client
listening for results. Addresses an existing issue in Fleet along with
appropriate cleanup for the refactored live query backend.
This change allows the images in Fleet emails to load properly from any
device with connectivity to github.com. Previously, emails might try to
load resources from a Kolide server not accessible from the email
client.
The asset URL will be based on the most recent git tag to accomodate
backwards-compatibility if the assets in the repo change.
Closes#1471
- Refactor configuration for logging to use separate plugins
- Move existing filesystem logging to filesystem plugin
- Create new AWS firehose plugin
- Update documentation around logging
This PR partially addresses #1456, providing SSO SAML support. The flow of the code is as follows.
A Kolide user attempts to access a protected resource and is directed to log in.
If SSO identity providers (IDP) have been configured by an admin, the user is presented with SSO log in.
The user selects SSO, which invokes a call the InitiateSSO passing the URL of the protected resource that the user was originally trying access. Kolide server loads the IDP metadata and caches it along with the URL. We then build an auth request URL for the IDP which is returned to the front end.
The IDP calls the server, invoking CallbackSSO with the auth response.
We extract the original request id from the response and use it to fetch the cached metadata and the URL. We check the signature of the response, and validate the timestamps. If everything passes we get the user id from the IDP response and use it to create a login session. We then build a page which executes some javascript that will write the token to web local storage, and redirect to the original URL.
I've created a test web page in tools/app/authtest.html that can be used to test and debug new IDP's which also illustrates how a front end would interact with the IDP and the server. This page can be loaded by starting Kolide with the environment variable KOLIDE_TEST_PAGE_PATH to the full path of the page and then accessed at https://localhost:8080/test
Made log rotation for osquery results and status logs optional. This required writing the logwriter package which is a drop in replacement for lumberjack. We still use lumberjack if the log rotation flag --osquery_enable_log_rotation flag is set. Note that the performance of the default is quite a bit better than lumberjack.
BenchmarkLogger-8 2000000 747 ns/op
BenchmarkLumberjack-8 1000000 1965 ns/op
PASS
BenchmarkLogger-8 2000000 731 ns/op
BenchmarkLumberjack-8 1000000 2040 ns/op
PASS
BenchmarkLogger-8 2000000 741 ns/op
BenchmarkLumberjack-8 1000000 1970 ns/op
PASS
BenchmarkLogger-8 2000000 737 ns/op
BenchmarkLumberjack-8 1000000 1930 ns/op
PASS
Notable refactoring:
- Use stdlib "context" in place of "golang.org/x/net/context"
- Go-kit no longer wraps errors, so we remove the unwrap in transport_error.go
- Use MakeHandler when setting up endpoint tests (fixes test bug caught during
this refactoring)
Closes#1411.
rotate osqueryd logs on SIGHUP
Closes#1256
Note: Sometimes the test fails to rotate the log on SIGHUP, although
that doesn't appear to be the case with a long running process.
After some discussion and debugging with @zwass we agreed to call
t.Log and come back to this issue at a later time.
* Initial scaffolding of the host summary endpoint
* inmem datastore implementation of GenerateHostStatusStatistics
* HostSummary docstring
* changing the url of the host summary endpoint
* datastore tests for GenerateHostStatusStatistics
* MySQL datastore implementation of GenerateHostStatusStatistics
* <= and >= to catch exact time edge case
* removing clock interface method
* lowercase error wraps
* removin superfluous whitespace
* use updated_at
* adding a seen_at column to the hosts table
* moving the update of seen_time to the caller
* using db.Get instead of db.Select
