Prior to this change, the live query status was calculated after each retrieved
result. This was fine at a low host count, but when more hosts are added, doing
this calculation becomes slower.
Now, the status is only calculated every 5 seconds, while results can be
streamed continuously.
Fixes#1895
An incorrect authorization check allowed non-admin users to modify the details of other users. We now enforce the appropriate authorization so that unprivileged users can only modify their own details.
Thanks to 'Quikke' for the report.
The ability to modify a users admin and enabled status was erroneously left in
place during development of https://github.com/kolide/fleet/pull/959. To
mitigate a privilege escalation vulnerability we need to ensure those values
can only be modified through the explicit methods.
This patch includes a unit test and fix for the vulnerability.
Thanks to 'Quikke' for submitting this vulnerability.
- Delete duplicate queries in packs created by the UI (because the duplicates
were causing undefined behavior). Now it is not possible to schedule
duplicates in the UI (but is in fleetctl).
- Fix bug in which packs created in UI could not be loaded by fleetctl.
Replaces the UI endpoints for creating and modifying labels. These were removed
in #1686 because we thought we were killing the UI.
Now labels can be created and edited in the UI again.
Replaces (and appropriately refactors) a number of endpoints that were removed long ago when we decided to kill the UI with the fleetctl release. We turned out not to do this, and now need to restore these missing endpoints.
This is not a straight up replacement of the existing code because of refactoring to the DB schemas that was also done in the migration.
Most of the replaced code was removed in #1670 and #1686.
Fixes#1811, fixes#1810
This PR adds support for getting resources by name.
```
$ fleetctl get queries
no queries found
$ fleetctl apply -f ./query.yaml
[+] applied 1 queries
$ fleetctl get queries
+-----------------+--------------------------------+--------------------------------+
| NAME | DESCRIPTION | QUERY |
+-----------------+--------------------------------+--------------------------------+
| osquery_version | The version of the Launcher | select launcher.version, |
| | and Osquery process | osquery.version from |
| | | kolide_launcher_info launcher, |
| | | osquery_info osquery; |
+-----------------+--------------------------------+--------------------------------+
$ fleetctl get query osquery_version
apiVersion: v1
kind: query
spec:
description: The version of the Launcher and Osquery process
name: osquery_version
query: select launcher.version, osquery.version from kolide_launcher_info launcher,
osquery_info osquery;
```
- Fix places where we accidentally return nil when we should return an error.
- Simplify interfaces/implementation of specialized errors
- Use more specific error messages
- Consistent JSON decoding
With the UI, deleting by ID made sense. With fleetctl, we now want to delete
by name. Transition only the methods used for spec related entities, as others
will be removed soon.
Previously decorators were stored in a separate table. Now they are stored
directly with the config so that they can be modified on a per-platform basis.
Delete now unused decorators code.
- Add new Apply spec methods for queries and packs
- Remove now extraneous datastore/service methods
- Remove import service (unused, and had many dependencies that this breaks)
- Refactor tests as appropriate
Include the appropriate values for removed and shard when generating config
to return to osqueryd.
Note: This was originally fixed and tested in the fleetctl branch (#1680), and
the fix is being cherry-picked into master without the test.
After discussion with @groob and @marpaia, we have decided that the service
methods should not be aware of any YAML/JSON definitions, and should work
directly with objects. The new pattern we will use will involve converting YAML
to JSON at the client, and then sending the JSON which will be decoded using
the familiar go-kit mechanisms before being passed to the service methods.
Instead of trying to decode and re-encode status logs, we now write them directly as they come in.
This change prevents future changes to the osquery status log file format (addition and deletion of fields ) from
affecting Fleet. A similar change was implemented in #1636 for result logs.
Closes#1664
Initially fleet decoded the incoming JSON sent to the log endpoint.
Then the log event would be written to a log writer by calling json.Encoder{}.Encode.
Re-encoding logs is lossy; whenever a new field is sent by osqueryd we don't keep up with them.
Instead of caring about the content of the OsqueryResultLog, fleet will now write all log results
exactly as sent to the server by osqueryd.
Closes#1632Closes#1615
This PR adds support for file integrity monitoring. This is done by providing a simplified API that can be used to PATCH/GET FIM configurations. There is also code to build the FIM configuration to send back to osquery. Each PATCH request, if successful, replaces Fleet's existing FIM configuration. For example:
curl -X "PATCH" "https://localhost:8080/api/v1/kolide/fim" \
-H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzZXNzaW9uX2tleSI6IkVhaFhvZWswMGtWSEdaTTNCWndIMnhpYWxkNWZpcVFDR2hEcW1HK2UySmRNOGVFVE1DeTNTaUlFWmhZNUxhdW1ueFZDV2JiR1Bwdm5TKzdyK3NJUzNnPT0ifQ.SDCHAUA1vTuWGjXtcQds2GZLM27HAAiOUhR4WvgvTNY" \
-H "Content-Type: application/json; charset=utf-8" \
-d $'{
"interval": 500,
"file_paths": {
"etc": [
"/etc/%%"
],
"users": [
"/Users/%/Library/%%",
"/Users/%/Documents/%%"
],
"usr": [
"/usr/bin/%%"
]
}
}'
Fix for issue where osquery sends empty strings where we expect integers in detail queries. We handle empty strings in these cases by changing them to "0" and then letting the different conversion functions change the "0" string into the appropriate integer type. This has been tested against running osquery hosts.
Closes#1521
Normally a Kolide user will always have at least two built in decorators that they can't delete through the UI so a situation with zero decorators should never happen; however, in the event we change this behavior in the future, or a user manually deletes decorators from the database the UI should handle an empty decorator set.
Closes issue #1475
The command line tool that uses this endpoint -> https://github.com/kolide/configimporter
* Added support for atomic imports and dry run imports
* Added code so that imports are idempotent
Closes issue #1456 This PR adds a single sign on option to the login form, exposes single sign on to the end user, and allows an admin user to set single sign on configuration options.
Closes#1502. This PR adds support for SSO to the new user creation process. An admin now has the option to select SSO when creating a new user. When the confirmation form is submitted, the user is automatically authenticated with the IDP, and if successful, is redirected to the Kolide home page. Password authentication, password change and password reset are not allowed for an SSO user.
This PR partially addresses #1456, providing SSO SAML support. The flow of the code is as follows.
A Kolide user attempts to access a protected resource and is directed to log in.
If SSO identity providers (IDP) have been configured by an admin, the user is presented with SSO log in.
The user selects SSO, which invokes a call the InitiateSSO passing the URL of the protected resource that the user was originally trying access. Kolide server loads the IDP metadata and caches it along with the URL. We then build an auth request URL for the IDP which is returned to the front end.
The IDP calls the server, invoking CallbackSSO with the auth response.
We extract the original request id from the response and use it to fetch the cached metadata and the URL. We check the signature of the response, and validate the timestamps. If everything passes we get the user id from the IDP response and use it to create a login session. We then build a page which executes some javascript that will write the token to web local storage, and redirect to the original URL.
I've created a test web page in tools/app/authtest.html that can be used to test and debug new IDP's which also illustrates how a front end would interact with the IDP and the server. This page can be loaded by starting Kolide with the environment variable KOLIDE_TEST_PAGE_PATH to the full path of the page and then accessed at https://localhost:8080/test
This PR contains a fix for a bug that turned up when I was testing configimporter. If the platform field is not specified, its supposed to default to all per the osquery configuration spec. The default was not properly implemented, and if the platform value was missing from the imported configuration it failed. The PR also added instructions to the api documentation describing how to import an osquery configuration.
Replaces the existing calculation that uses a global online interval. This method was lacking due to the fact that different hosts may have different checkin intervals set.
The new calculation uses `min(distributed_interval, config_tls_refresh) + 30` as the interval. This is calculated with the stored values for each host.
Closes#1321
Partially addresses #1456. This PR provides datastore support for SSO by creating a new entity IdentityProvider. This entity is an abstraction of the SAML IdentityProvider and contains the data needed to perform SAML authentication.
We now track the `config_tls_refresh`, `distributed_interval` and
`logger_tls_period` flag values for each host. Each value is updated by a
detail query agains the `osquery_flags` table, because they may be specified
outside of Kolide. The flags that can be specified within Kolide are also
updated when a config is returned to the host that changes their value.
This will enable us to do a more accurate per-host online status calculation as
discussed in #1419.
Made log rotation for osquery results and status logs optional. This required writing the logwriter package which is a drop in replacement for lumberjack. We still use lumberjack if the log rotation flag --osquery_enable_log_rotation flag is set. Note that the performance of the default is quite a bit better than lumberjack.
BenchmarkLogger-8 2000000 747 ns/op
BenchmarkLumberjack-8 1000000 1965 ns/op
PASS
BenchmarkLogger-8 2000000 731 ns/op
BenchmarkLumberjack-8 1000000 2040 ns/op
PASS
BenchmarkLogger-8 2000000 741 ns/op
BenchmarkLumberjack-8 1000000 1970 ns/op
PASS
BenchmarkLogger-8 2000000 737 ns/op
BenchmarkLumberjack-8 1000000 1930 ns/op
PASS
Closes issue #1388. The problem here is that previously, the reset button loaded a hard coded list of default options into the component state, instead of the proper behavior which is to reset the options to default values on the back end, and then load them back into the redux store. This PR adds a ResetOptions endpoint on the server, and wires up the UI so that it triggers the endpoint, then loads the default options from the backend server.
Closes issue #1390
There were quite a few places where UPDATES could fail silently because we weren't checking target rows where actually found where we expect them to be. In order to address this problem clientFoundRows was set in the sql driver configuration and checks for UPDATES were added to determine if matched rows were found where we expect them to be.
Push the calculation of target counts into the SQL query, rather than loading
all of the targets and then counting them. This provides a dramatic (>100x)
speedup in loading of the manage packs page when large numbers of hosts are
present.
Closes#1426
The seen time should only be updated once per request from the osquery agent to
the Kolide server. We now do that only in AuthenticateHost (which every request
besides enrollment must go through).
* Fix issue where config interval can be number or string
* Implemented @groob code review suggestions
* Added type assertions with graceful failure if something slips through validation
* Implemented code review changes per @zwass
Return `accelerate: 10` with distributed queries if we do not have host
details. This facilitates the host quickly joining all expected labels, as
`platform` gated label queries will not be returned until the detail queries
return with the platform.
Fixes#1421.
On CentOS6 there is a bug in which osquery incorrectly reports an empty string
for platform. This PR fixes our detection of centos in this case.
Fixes#1339
Notable refactoring:
- Use stdlib "context" in place of "golang.org/x/net/context"
- Go-kit no longer wraps errors, so we remove the unwrap in transport_error.go
- Use MakeHandler when setting up endpoint tests (fixes test bug caught during
this refactoring)
Closes#1411.
Improve the mechanism used to calculate whether or not hosts are online.
Previously, hosts were categorized as "online" if they had been seen within the past 30 minutes. To make the "online" status more representative of reality, hosts are marked "online" if the Kolide server has heard from them within two times the lowest polling interval as described by the Kolide-managed osquery configuration. For example, if you've configured osqueryd to check-in with Kolide every 10 seconds, only hosts that Kolide has heard from within the last 20 seconds will be marked "online".
rotate osqueryd logs on SIGHUP
Closes#1256
Note: Sometimes the test fails to rotate the log on SIGHUP, although
that doesn't appear to be the case with a long running process.
After some discussion and debugging with @zwass we agreed to call
t.Log and come back to this issue at a later time.
Use the [SockJS Protocol](https://github.com/sockjs/sockjs-protocol) to handle
bidirectional communication instead of plain websockets. This allows
distributed queries to function in situations in which they previously failed
(Load balancers not supporting websockets, issues with Safari and self-signed
certs, etc.).
Also includes fixes to the JS message handling logic where slightly different
message delivery semantics (when using XHR) were exposing bugs.
Fixes#1241, #1327.
Ensure that host network interfaces do not disappear when they (unexpectedly)
are returned with no updates from osquery. Add test to verify.
Fixes#1278
These decorators were removed in #953 due to an osquery bug. That bug is now
fixed, and we are adding the decorators back. We also now use `load` decorators
rather than `interval` decorators because they seem to function more reliably.
Previously we were using `build_platform`, which does not always properly
reflect the platform of the host running osquery. Now we should properly
retrieve the platform.
Fixes#1264
* Change email functionality
* Code review changes for @groob
* Name change per @groob
* Code review changes per @marpaia
Also added addition non-happy path tests to satisfy concerns by @groob
Add a hostname field to every distributed query result row in websocket response. By doing this calculation on the server, we're hoping to improve the rendering on the frontend client, which currently does the same calculation.
Closes#1079
add endpoint to serve the kolide certificate back to the user
The API will attempt to establish a TLS connection and fetch the certificate from the TLS ConnectionState.
The PEM encoded certificate will be served to the client in a JSON response as a base64 encoded string.
Closes#1012
* If a detail query doesn't return results, log problem but don't interrupt additional query processing
* changed arg to ingest func to logger interface
* correctly list packs in response
Using append was adding a default pack response to the list of packs
* handle unique index for packs that exist but are deleted
Previously, when determining which packs a host should get when it checked in, we were iterating each pack and only checking whether or not the host was apart of a label which was a target of the pack, but we were never checking whether or not the host had been added as a specific target of that pack. This PR makes the necessary modification to `svc.ListPacksForHost`.
Saving a new detail update time when the host details were not actually updated
caused detail updates to be missed. This PR fixes the existing test to catch
the bug, and fixes the bug.
* Simplifying SMTP Logic
This commit breaks the test email sending into it's own service method
(thus removing the capability from the API- if we want it back, we can
wire up another endpoint for just that). Additionally, error wrapping is
used through the new ModifyAppConfig service method to ensure that an
error or failed email will always result in an error while ensuring that
the submitted record always get committed (unless a serious error
happens).
* never wrap a nil error
* use err instead of individual errors
As of recently, osquery will report when a distributed query fails. We now
expose errors over the results websocket. When a query errored on the host, the
`error` key in the result will be non-null. Note that osquery currently doesn't
provide any details so the error string will always be "failed". I anticipate
that we will fix this and the string is included for future-proofing.
Successful result:
```
{
"type": "result",
"data": {
"distributed_query_execution_id": 15,
"host": {
... omitted ...
},
"rows": [
{
"hour": "1"
}
],
"error": null
}
}
```
Failed result:
```
{
"type": "result",
"data": {
"distributed_query_execution_id": 14,
"host": {
... omitted ...
},
"rows": [
],
"error": "failed"
}
}
```
* add a js validator that makes smtp server port required
* specifying that the InputField should be a number. this doesn't work, but i think that it should.
* casting the port as an int as a stop-gap fix
* email doesn't already have to be enabled to be enabled
* don't return the smtp password from the API
* show a fake placeholder password if the username is also set
* error type for @groob
Fixes#751
For #760
Race conditions were caused by running the test in parallel.
Also remove assertions which were no longer true. The RequestPasswordReset
method was refactored in #725, but because of the racy test, the assertions
which should've failed did not.
* initial scaffolding of a hostResponseForHost helper to consistently get all required values when returning hosts via the api
* Using the hostname as the display text
* remove err: nil
* groob comments
* pre-allocating the hostResponses slice
Permissions errors were preventing users from completing this flow
- Add separate endpoint for performing required password reset
- Rewrite frontend reset to use this endpoint
Fixes#792
Add logging middleware for more of the kolide Service interfaces.
This PR was created through code generation, however it's not likely that the logging middleware can all be continuously regenerated - we're likely to want to add method specific key/values to individual methods. Moving forward, logging middleware should be maintained when changes are made to a service interface method.
* Defining a concrete type for session tokens
* More rightish vc.IsLoggedIn()
* using type conversion instead of a method call
* include sessions in test viewer contexts
- Remove require password reset from ModifyUser and
RequestPasswordReset methods, and UserPayload struct
- Add new RequirePasswordReset method
- Refactor JS for new separate method
This PR separates the table migrations from the data population migrations. Table migrations run before data migrations.
Now, we have the ability to create the database tables without populating them with data. This can be useful for running "unit" tests against a MySQL store that doesn't have any pre-populated data. When performing real migrations, or for more "integration" style testing, the data migrations can also be executed.
Note there are some special cases that must be observed with these migrations, and the README is updated to reflect those.
* Initial scaffolding of the host summary endpoint
* inmem datastore implementation of GenerateHostStatusStatistics
* HostSummary docstring
* changing the url of the host summary endpoint
* datastore tests for GenerateHostStatusStatistics
* MySQL datastore implementation of GenerateHostStatusStatistics
* <= and >= to catch exact time edge case
* removing clock interface method
* lowercase error wraps
* removin superfluous whitespace
* use updated_at
* adding a seen_at column to the hosts table
* moving the update of seen_time to the caller
* using db.Get instead of db.Select
This PR adds the `host_ids` and `label_ids` field to the packs HTTP API so that one can operate on the hosts/labels which a pack is scheduled to be executed on. This replaces (and deletes) the `/api/v1/kolide/packs/123/labels/456` API in favor of `PATCH /api/v1/packs/123` and specifying the `label_ids` field. This also allows for bulk operations.
Consider the following API examples:
## Creating a pack with a known set of hosts and labels
The key addition is the `host_ids` and `label_ids` field in both the request and the response.
### Request
```
POST /api/v1/kolide/packs
```
```json
{
"name": "My new pack",
"description": "The newest of the packs",
"host_ids": [1, 2, 3],
"label_ids": [1, 3, 5]
}
```
### Response
```json
{
"pack": {
"id": 123,
"name": "My new pack",
"description": "The newest of the packs",
"platform": "",
"created_by": 1,
"disabled": false,
"query_count": 0,
"total_hosts_count": 5,
"host_ids": [1, 2, 3],
"label_ids": [1, 3, 5]
}
}
```
## Modifying the hosts and/or labels that a pack is scheduled to execute on
### Request
```
PATCH /api/v1/kolide/packs/123
```
```json
{
"host_ids": [1, 2, 3, 4, 5],
"label_ids": [1, 3, 5, 7]
}
```
### Response
```json
{
"pack": {
"id": 123,
"name": "My new pack",
"description": "The newest of the packs",
"platform": "",
"created_by": 1,
"disabled": false,
"query_count": 0,
"total_hosts_count": 5,
"host_ids": [1, 2, 3, 4, 5],
"label_ids": [1, 3, 5, 7]
}
}
```
close#633
keep the format for error returns consistent by always returning a
[]map[string]string for json errors. This simplifies the error handling
on the frontend.
Use "name":"base" as the name field for errors which do not have
a specific or known form field.
A distributed query campaign can be "orphaned" (left in the QueryRunning state)
if the Kolide server restarts while it is running, or other weirdness occurs.
When this happens, no subscribers are waiting to read results written by
osqueryd agents, but the agents continue to receive the query. Previously, this
would cause us to error on ingestion.
The new behavior will instead set the campaign to completed when it detects
that it is orphaned. This should prevent sending queries for which there is no
subscriber.
- New NoSubscriber error interface in pubsub
- Detect NoSubscriber errors and close campaigns
- Tests on pubsub and service methods
Fixes#695
with an exposed interface.
Not checking for a specific sentinel error reduces coupling between packages
and allows adding context like the resource ID and resource type.
* Moving query attributes from the query object to the pack-query relationship
* some additional tests
* http request parsing test
* QueryOptions in new test_util code
* initial scaffolding of new request structures
* service and datastore
* test outline
* l2 merge conflict scrub
* service tests for scheduled query service
* service and datastore tests
* most endpoints and transports
* order of values are not deterministic with inmem
* transport tests
* rename PackQuery to ScheduledQuery
* removing existing implementation of adding queries to packs
* accounting for the new argument to NewQuery
* fix alignment in sql query
* removing underscore
* add removed to the datastore
* removed differential from the schema
- Only expire sessions at reset request time when admin forces reset
- Expire sessions when reset completed
Prior to this, there was a possible DoS vector in which an attacker could
prevent a user from taking actions in the app by constantly requesting password
resets and expiring all the user's active sessions.
Fixes#612
- New datastore method for bulk deletion
- New service method calling this datastore method
- Endpoint, transport and handler connections for service method
Closes#389
- Add saved state to query (to differentiate queries explicitly saved from
those just run as distributed queries)
- Remove unique constraint on query name
Closes#390
- New route `/api/v1/kolide/results/{id}` with upgrade to websocket connection
- Query results pushed over websocket as they are received from pubsub
- Target totals updates pushed over websocket every second
- New datastore method to support retrieiving target totals
- Websocket package includes helpers and patterns for communicating over websockets
* Adds created_by attribute to packs
This PR also updated the distributed query code to use the pattern
established here (service checks context)
* add enable/disable state to packs
* add query_count to packs API responses
* add host_count to packs API responses (very, very poorly)
* pack description should not be required
* counting hosts in packs via mysql
* removing extraneous newline in test
* Switch case instead of if/if else
* add description to update query for SavePack method
* change AND to WHERE in query as per @zwass
* add ordering and list options as per @murphybytes' suggestion
* initial scaffolding
* pack info sidebar
* fixing the merge of the routes
* Remove radium from pack info sidepanel
* lint
* cards!
* redux entity config
* pack interface
* wiring up redux with fake dev data
* Add description attribute to packs
* move redux to top level page component to isolate data fetching
* initial scaffolding of all packs table
* adding redux entities back
* minimal
* alpha order in packs.js
* no newlines in HTML
* onclick handler to function on component class
* alpha order in router
* alpha order in paths.js
* no newline in side panel
* removing input field
* lint fixes
Removed Gorm, replaced it with Sqlx
* Added SQL bundling command to Makfile
* Using go-kit logger
* Added soft delete capability
* Changed SearchLabel to accept a variadic param for optional omit list
instead of array
* Gorm removed
* Refactor table structures to use CURRENT_TIMESTAMP mysql function
* Moved Inmem datastore into it's own package
* Updated README
* Implemented code review suggestions from @zwass
* Removed reference to Gorm from glide.yaml
The endpoint is only active if there are no users in the datastore.
While the endpoint is active, it also disables all the other API endpoints, and /config returns `{"require_setup":true}`
for #378
* Adds loadAll action to redux entity config
* API Client get invites
* Add invites to the user management page
* Updates user block styles on user management page
* Submit modal form on enter
* Modify details form styles
* Enter submits edit user form
* Removes unused admin dashboard page
* API Client - revoke invites
* Delete invite entities in redux
* Revoke invites from admin manage users page
* Show success flash message after user invite is revoked
- Introduce kolide.ListOptions to store pagination params (in the future it can
also store ordering/filtering params)
- Refactor service/datastore methods to take kolide.ListOptions
- Implement pagination
- Introduce a new pattern for defining/ingesting detail queries
- Add many relevant host details:
- Platform
- osquery Version
- Memory
- Hostname
- UUID
- OS Version
- Uptime
- Primary interface MAC
- Primary interface IP
- Fix parsing for inconsistent JSON schema returned from osquery
- Tests
- Establish a pattern for host authentication
- Establish a pattern for error JSON
- Add transport and make endpoint functions
- Fix discovered bugs + update tests
