Additional information is collected when host details are updated using
the queries specified in the Fleet configuration. This additional
information is then available in the host API responses.
This adds a SQL injection prevention for a case in which we cannot use
parameters in the query.
It is not clear that this was possible to exploit. If it was possible,
it would have required a valid login to the Fleet server.
Previously a Go package attempting to import Fleet packages would run
into an error like "server/kolide/emails.go:93:23: undefined: Asset".
This commit refactors bindata asset handling to allow importing Fleet as
a library without changing the typical developer experience.
- Add toggle to disable live queries in advanced settings
- Add new live query status endpoint (checks for disabled via config and Redis health)
- Update QueryPage UI to use new live query status endpoint
Implements #2140
Update the github.com/russellhaering/goxmldsig dependency and apply
the appropriate fixes for the API changes.
This is a preparation for integration with
github.com/AbGuthrie/goquery, which uses a newer version of the
dependency.
- Add logging for new campaigns
- Add logging for new query creations/modification/deletion
- Add usernames for logs found in labels, options, packs, osquery options, queries and scheduled queries where something is created, modified or deleted
- Add the server_url_prefix flag for configuring this functionality
- Add prefix handling to the server routes
- Refactor JS to use appropriate paths from modules
- Use JS template to get URL prefix into JS environment
- Update webpack config to support prefixing
Thanks to securityonion.net for sponsoring the development of this feature.
Closes#1661
This change allows the images in Fleet emails to load properly from any
device with connectivity to github.com. Previously, emails might try to
load resources from a Kolide server not accessible from the email
client.
The asset URL will be based on the most recent git tag to accomodate
backwards-compatibility if the assets in the repo change.
Closes#1471
Adds Google Cloud PubSub logging for status and results.
This also changes the Write interface for logging modules to add a context.Context (only used by pubsub currently).
When an osqueryd agent sends an enroll request it automatically sends
some details about the system. We now save these details which helps
ensure we send the correct platform config.
Closes#2065
When LOGIN authentication was used, Fleet would send SMTP credentials
even if the connection the SMTP server was not secured via TLS.
Copying the pattern used in the standard library PlainAuth
implementation, we now only send credentials when the connection is
secure or the server is localhost.
- The most active NIC will be picked even if a formerly more active
interface still exists (previously, a NIC would stay primary as long
as it existed).
- Ignore link-local and loopback interfaces when choosing the primary.
- Fix bugs in which update status of the primary interface could be
reported incorrectly.
Fixes#2020
Fixes a regression introduced in 2.1.0 in which separate log lines are
no longer output separated by a newline. Now log lines in both output
plugins will do so.
- Refactor configuration for logging to use separate plugins
- Move existing filesystem logging to filesystem plugin
- Create new AWS firehose plugin
- Update documentation around logging
Almost two years ago, we began referring to the project as Fleet, but there are
many occurences of the term "Kolide" throughout the UI and documentation. This
PR attempts to clear up those uses where it is easily achievable.
The term "Kolide" is used throughout the code as well, but modifying this would
be more likely to introduce bugs.
Brings the behavior of the server in line with the documentation, by using the
query name if the scheduled query name is not specified in a pack spec.
Closes#1990
This PR adds support for the SMTP LOGIN authentication method. Office 365 Exchange removed support for PLAIN authentication some time ago, and only supports LOGIN and an OAuth2 authentication method. This patch has been tested with a licensed O365 account. This method should also be usable with any other email server that advertises LOGIN in its 250-AUTH response.
Note: If using this with O365, the account used must not have MFA enabled.
Closes#1663
This commit:
- adds a new sub-command for fetching hosts to `fleetctl get` command.
Why?
- this allows for listing of all hosts via the fleetctl interface.
There may be additional attributes of the host that we'd like to
display, but this should be a good start.
Closes#1962
Prior to this change, the live query status was calculated after each retrieved
result. This was fine at a low host count, but when more hosts are added, doing
this calculation becomes slower.
Now, the status is only calculated every 5 seconds, while results can be
streamed continuously.
Fixes#1895
Avoids potential bugs in which soft-deleted entities are returned from database
queries (soft-deletion is now deprecated), but some records may still exist.
Fixes#1956
This should fix the loading of the all hosts page in cases where there are many
hosts and it overwhelms the number of parameters allowed in a prepared
statement. May also make that page load slightly quicker as it removes the
constraint from the query, but should return the same number of results.
Fixes#1939
The SMTP configuration could be used by an admin user to port scan the network
the Fleet server was running on. This commit reduces the information possible
to determine via this technique. A malicious admin can no longer determine
whether any TCP server is listening on a given port/address. They can only
determine ports and addresses where SMTP servers are running.
Thanks to 'quikke' for reporting this vulnerability.
An incorrect authorization check allowed non-admin users to modify the details of other users. We now enforce the appropriate authorization so that unprivileged users can only modify their own details.
Thanks to 'Quikke' for the report.
The ability to modify a users admin and enabled status was erroneously left in
place during development of https://github.com/kolide/fleet/pull/959. To
mitigate a privilege escalation vulnerability we need to ensure those values
can only be modified through the explicit methods.
This patch includes a unit test and fix for the vulnerability.
Thanks to 'Quikke' for submitting this vulnerability.
Packs can be targeted to individual hosts through the UI. This was supported
previously and was broken with refactoring in Fleet 2.0.
There is currently no support in the fleetctl format for targeting individual
hosts, but this could be added at a later date.
Fixes#1878
- Delete duplicate queries in packs created by the UI (because the duplicates
were causing undefined behavior). Now it is not possible to schedule
duplicates in the UI (but is in fleetctl).
- Fix bug in which packs created in UI could not be loaded by fleetctl.
- Add cascading deletes for scheduled_queries when queries are deleted
- Also add cascading deletes for scheduled_queries when packs are deleted
Fixes#1837
Replaces the UI endpoints for creating and modifying labels. These were removed
in #1686 because we thought we were killing the UI.
Now labels can be created and edited in the UI again.
Replaces (and appropriately refactors) a number of endpoints that were removed long ago when we decided to kill the UI with the fleetctl release. We turned out not to do this, and now need to restore these missing endpoints.
This is not a straight up replacement of the existing code because of refactoring to the DB schemas that was also done in the migration.
Most of the replaced code was removed in #1670 and #1686.
Fixes#1811, fixes#1810
This PR adds support for getting resources by name.
```
$ fleetctl get queries
no queries found
$ fleetctl apply -f ./query.yaml
[+] applied 1 queries
$ fleetctl get queries
+-----------------+--------------------------------+--------------------------------+
| NAME | DESCRIPTION | QUERY |
+-----------------+--------------------------------+--------------------------------+
| osquery_version | The version of the Launcher | select launcher.version, |
| | and Osquery process | osquery.version from |
| | | kolide_launcher_info launcher, |
| | | osquery_info osquery; |
+-----------------+--------------------------------+--------------------------------+
$ fleetctl get query osquery_version
apiVersion: v1
kind: query
spec:
description: The version of the Launcher and Osquery process
name: osquery_version
query: select launcher.version, osquery.version from kolide_launcher_info launcher,
osquery_info osquery;
```
- Fix places where we accidentally return nil when we should return an error.
- Simplify interfaces/implementation of specialized errors
- Use more specific error messages
- Consistent JSON decoding
With the UI, deleting by ID made sense. With fleetctl, we now want to delete
by name. Transition only the methods used for spec related entities, as others
will be removed soon.
Previously decorators were stored in a separate table. Now they are stored
directly with the config so that they can be modified on a per-platform basis.
Delete now unused decorators code.
The DMARC and DKIM email authentication systems both require the RFC822
From header to function. Kolide currently only includes the configured
sender address as the SMTP Envelop From address (e.g., the MAIL FROM
command). This patch also includes the configured sender address in the
RFC822 email From header which should allow these emails to pass both
DKIM and DMARC authentication.
See https://goo.gl/zuku4E.
> The most obvious remediation here is ensuring your SAML library is extracting
the full text of a given XML element when comments are present.
Our implementation asks for the innerxml of the NameID field, so it returns the
entire text including the comment (See https://goo.gl/KLLXof). By default Go's
XML parsing would return the text not including the comment (but including
further text after the comment). Both of these options prevent the
vulnerability.
- Add new Apply spec methods for queries and packs
- Remove now extraneous datastore/service methods
- Remove import service (unused, and had many dependencies that this breaks)
- Refactor tests as appropriate
Include the appropriate values for removed and shard when generating config
to return to osqueryd.
Note: This was originally fixed and tested in the fleetctl branch (#1680), and
the fix is being cherry-picked into master without the test.
After discussion with @groob and @marpaia, we have decided that the service
methods should not be aware of any YAML/JSON definitions, and should work
directly with objects. The new pattern we will use will involve converting YAML
to JSON at the client, and then sending the JSON which will be decoded using
the familiar go-kit mechanisms before being passed to the service methods.
Instead of trying to decode and re-encode status logs, we now write them directly as they come in.
This change prevents future changes to the osquery status log file format (addition and deletion of fields ) from
affecting Fleet. A similar change was implemented in #1636 for result logs.
Closes#1664
Initially fleet decoded the incoming JSON sent to the log endpoint.
Then the log event would be written to a log writer by calling json.Encoder{}.Encode.
Re-encoding logs is lossy; whenever a new field is sent by osqueryd we don't keep up with them.
Instead of caring about the content of the OsqueryResultLog, fleet will now write all log results
exactly as sent to the server by osqueryd.
Closes#1632Closes#1615
The launcher service implementation is an adapter around the TLS service.
All launcher methods that have an equivalent in TLS pass the business logic to the
TLS API.
Closes#1565
This PR adds support for file integrity monitoring. This is done by providing a simplified API that can be used to PATCH/GET FIM configurations. There is also code to build the FIM configuration to send back to osquery. Each PATCH request, if successful, replaces Fleet's existing FIM configuration. For example:
curl -X "PATCH" "https://localhost:8080/api/v1/kolide/fim" \
-H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzZXNzaW9uX2tleSI6IkVhaFhvZWswMGtWSEdaTTNCWndIMnhpYWxkNWZpcVFDR2hEcW1HK2UySmRNOGVFVE1DeTNTaUlFWmhZNUxhdW1ueFZDV2JiR1Bwdm5TKzdyK3NJUzNnPT0ifQ.SDCHAUA1vTuWGjXtcQds2GZLM27HAAiOUhR4WvgvTNY" \
-H "Content-Type: application/json; charset=utf-8" \
-d $'{
"interval": 500,
"file_paths": {
"etc": [
"/etc/%%"
],
"users": [
"/Users/%/Library/%%",
"/Users/%/Documents/%%"
],
"usr": [
"/usr/bin/%%"
]
}
}'
Add a gRPC server that will interact with osquery through Launcher. This endpoint will expose the osquery configuration suitable for use via the Launcher plugin, and collect log and query results.
Closes#1533
Since the SAML 2.0 spec doesn't say what characters are valid in an Entity ID and Active Directory doesn't like '=' signs in base64 encoded ID's I added code that generates ID's with a character set that we know works. Also, removed ProtocolBinding attribute from AuthRequest as is was forcing ADFS to use redirect binding when it should use post binding.
Closes#1532
Fixes error that was caused because there was a bug in processing nested assertions in a successful SAML response. This was not caught in the initial push of this code because the IDP's we tested against all sign the entire response document as opposed to parts of it. Thus the existing test cases didn't cover the code that dealt with nested assertions.
Fix for issue where osquery sends empty strings where we expect integers in detail queries. We handle empty strings in these cases by changing them to "0" and then letting the different conversion functions change the "0" string into the appropriate integer type. This has been tested against running osquery hosts.
Closes#1521
Normally a Kolide user will always have at least two built in decorators that they can't delete through the UI so a situation with zero decorators should never happen; however, in the event we change this behavior in the future, or a user manually deletes decorators from the database the UI should handle an empty decorator set.
Closes issue #1475
The command line tool that uses this endpoint -> https://github.com/kolide/configimporter
* Added support for atomic imports and dry run imports
* Added code so that imports are idempotent
Closes issue #1456 This PR adds a single sign on option to the login form, exposes single sign on to the end user, and allows an admin user to set single sign on configuration options.
Closes#1502. This PR adds support for SSO to the new user creation process. An admin now has the option to select SSO when creating a new user. When the confirmation form is submitted, the user is automatically authenticated with the IDP, and if successful, is redirected to the Kolide home page. Password authentication, password change and password reset are not allowed for an SSO user.
This PR partially addresses #1456, providing SSO SAML support. The flow of the code is as follows.
A Kolide user attempts to access a protected resource and is directed to log in.
If SSO identity providers (IDP) have been configured by an admin, the user is presented with SSO log in.
The user selects SSO, which invokes a call the InitiateSSO passing the URL of the protected resource that the user was originally trying access. Kolide server loads the IDP metadata and caches it along with the URL. We then build an auth request URL for the IDP which is returned to the front end.
The IDP calls the server, invoking CallbackSSO with the auth response.
We extract the original request id from the response and use it to fetch the cached metadata and the URL. We check the signature of the response, and validate the timestamps. If everything passes we get the user id from the IDP response and use it to create a login session. We then build a page which executes some javascript that will write the token to web local storage, and redirect to the original URL.
I've created a test web page in tools/app/authtest.html that can be used to test and debug new IDP's which also illustrates how a front end would interact with the IDP and the server. This page can be loaded by starting Kolide with the environment variable KOLIDE_TEST_PAGE_PATH to the full path of the page and then accessed at https://localhost:8080/test
This PR contains a fix for a bug that turned up when I was testing configimporter. If the platform field is not specified, its supposed to default to all per the osquery configuration spec. The default was not properly implemented, and if the platform value was missing from the imported configuration it failed. The PR also added instructions to the api documentation describing how to import an osquery configuration.
