- Maintain software inventory with detail queries.
- Associated database migrations.
- Feature flagged off by default (see documentation for details to turn on).
- Documentation.
- New test helper for slice element comparisons skipping ID.
Instead of synchronously updating the seen_time column for a host on an update, batch these updates to be written together every 1 second.
This results in a ~33% reduction in MySQL CPU usage in a local test with 4,000 simulated hosts and MySQL running in Docker.
- Migrate old admins to global admins
- Migrate old non-admins to global maintainers
- Remove old admin column
- Give initial user global admin privilege
- Comment out some tests (to be refactored for new permissions model later)
- Reorder migrations post-rebase
- Fix global_role in user payload
- Add teams/roles to invite entities
- Add teams/roles support to invite datastore methods
- Update tests
- Carry over team information from invite when creating user
Improves MySQL test time (on my 2020 MBP) to ~18s from ~125s.
- Use separate databases for each test to allow parallelization.
- Run migrations only once at beginning of tests and then reload
generated schema.
- Add `--innodb-file-per-table=OFF` for ~20% additional speedup.
- Fix issue with built-in labels showing multiple platforms when hosts
are reinstalled with new platform.
- Add Red Hat Linux built-in label.
- Display more labels by default in target selector.
Fixes#546, #553
- Use a non-loopback, non-link-local IPv4 address if available
- Otherwise use non-loopback, non-link-local IPv6 address if available
- Otherwise use any address
Closes#532
Prevent abuse of these endpoints with rate limiting backed by Redis. The
limits assigned should be appropriate for almost any Fleet deployment.
Closes#530
- Add a "Last fetched" column to the table on the Hosts page. This column uses the `detail_updated_at` property.
- Add a "Last fetched" timestamp to the Host details page.
- Adjust styles on _Host details_ page
- Return same error in all cases for login endpoint.
- Log error details in server logs.
- Make most login errors take ~1s to prevent timing attacks.
- Don't return forgot password errors.
- Log password errors in server logs.
- Make most forgot password requests take ~1s to prevent timing attacks.
Fixes#531
This saves a few seconds per test by only creating 10 hosts rather than
1000. The tests required no further changes as they were not using more
than the first few created hosts.
The AuthenticateHost loading of hosts accidentally dropped IP addresses,
which would cause the IP to be dropped on save under certain scenarios.
Also fixes a potential issue with flapping host additional info.
Fixes#358
The enrollment cooldown period was sometimes causing problems when
osquery (probably unintentionally, see
https://github.com/osquery/osquery/issues/6993) tried to enroll more
than once from the same osqueryd process.
We now set this to default to off and make it configurable. With #417
this feature may be unnecessary for most deployments.
Osquery now exposes more information during host enrollment than Fleet
previously handled. We can use this to provide more options to users in
problematic enrollment scenarios.
Users can configure --osquery_host_identifier in Fleet to set which
identifier is used to determine uniqueness of hosts. The
default (provided) replicates existing behavior in Fleet. For many
users, setting this to instance will provide better enrollment
stability.
Closes#373
Since the original logic was implemented, there have been some changes
in the way that config refreshes are configured. This commit reflects
those changes and should be backwards compatible.
Closes#357
Uses a LIKE clause to search for hosts matching the query against
columns `host_name`, `uuid`, `hardware_serial`, and `primary_ip`.
Introduces the `searchLike` helper to add the appropriate filters to the
SQL query.
Aliases `hostname` (`host_name`) and `memory` (`physical_memory`) when
used as keys for ordering in the API this allows for better consistency
on the frontend.
To be cleaned up further in #317
- Support both /api/v1/fleet and /api/v1/kolide routes in server.
- Add logging for use of deprecated routes.
- Rename routes in frontend JS.
- Rename routes and add notes in documentation.
In #212 these settings were updated and caused connectivity issues for
users in common environment configurations. The new changes are
aggressive (modern enforces TLS 1.3) and Mozilla indicates that
intermediate is an appropriate default. This will ensure better
compatibility for common deployments while still allowing the option to
use the strictest settings.
Document unintentional mismatched yaml key.
Fixes#269
- Fix the specific case that caused panic.
- Add panic handler around entire live query results handler. This will
prevent similar issues from causing crashes in the future.
Note that other endpoints already have panic handling but this one is
special due to the use of websockets.
- Make the preview directory in the default .fleet directory.
- Check for Docker daemon installed but not running.
- Add message for Chrome users on self-signed certs.
- Display login information on later invocations of command.
- Remove "Kolide" from error messages.
Closes#190
Part of #197
Somewhere around osquery 4.4.0 these messages were added to query
responses. We can now expose them to the API clients rather than using
the placeholder text.
Required for #192
On new installations we unintentionally set the enroll secret to empty
string during database migrations. The enroll secret would be reset
during the setup process. This fixes the migration to not create any
enroll secret until the setup process.
The current implementation of FleetDM doesn't support Docker secrets for supplying the MySQL password and JWT key. This PR provides the ability for a file path to read in secrets. The goal of this PR is to avoid storing secrets in a static config or in an environment variable.
Example config for Docker:
```yaml
mysql:
address: mysql:3306
database: fleet
username: fleet
password_path: /run/secrets/mysql-fleetdm-password
redis:
address: redis:6379
server:
address: 0.0.0.0:8080
cert: /run/secrets/fleetdm-tls-cert
key: /run/secrets/fleetdm-tls-key
auth:
jwt_key_path: /run/secrets/fleetdm-jwt-key
filesystem:
status_log_file: /var/log/osquery/status.log
result_log_file: /var/log/osquery/result.log
enable_log_rotation: true
logging:
json: true
```
This adds the option to set up an S3 bucket as the storage backend for file carving (partially solving #111).
It works by using the multipart upload capabilities of S3 to maintain compatibility with the "upload in blocks" protocol that osquery uses. It does this basically replacing the carve_blocks table while still maintaining the metadata in the original place (it would probably be possible to rely completely on S3 by using object tagging at the cost of listing performance). To make this pluggable, I created a new field in the service struct dedicated to the CarveStore which, if no configuration for S3 is set up will be just a reference to the standard datastore, otherwise it will point to the S3 one (effectively this separation will allow in the future to add more backends).
Mitigate
[CVE-2020-26273](https://github.com/osquery/osquery/security/advisories/GHSA-4g56-2482-x7q8)
by attempting to prevent users from executing or saving queries that use
the SQLite `ATTACH` command.
Users must still update to osquery 4.6.0 to ensure the functionality is
fully disabled in osquery.
This addresses an issue some users experienced in which performance
problems were encountered when hosts were "competing" for enrollment
using the same osquery host identifier. The issue is addressed by adding
a cooldown period for host enrollment, preventing the same (as judged by
osquery host identifier) host from enrolling more than once per minute.
When users end up in the problematic scenario, they will see quite a bit
of error logs due to this issue. For now that's probably a good thing as
users need to be aware of the lack of visibility. We can explore rate
limiting the logging if that becomes an issue for someone.
Fixes#102
The goal of this PR was to insert new colors, remove legacy colors, and update only the styles accordingly. The Nunito Sans Italic font was also added.
Later PRs will include layout, copy, and style change to individual components. These later changes will more exactly resemble the current mockups.
This is the second PR as part of the Fleet UI Refresh #38.
The goal of this PR was to insert all updated Fleet assets and remove all old assets. More style changes, including the exact sizing and placement of the new images, will occur in a future PR.
This is the first PR as a part of the Fleet UI Refresh #38.
Changes include:
- Add Nunito Sans font files and modify global styles to reflect the font change.
- Modify global font variables to reflect new sizing and weight naming conventions.
-- New sizing and naming conventions:
--- SIZE: xx-small: 12px, x-small: 14px, small: 16px, medium: 20px, large: 24px, x-large: 28px
--- WEIGHT: regular: 400, bold: 700
- Remove the old Oxygen font files.
Changes to other style sheets reflect the changes to the new font sizing and weight naming conventions for global variables. The changes don't necessarily use the correct size (as illustrated by mockups). Those "up to spec" sizing changes are to come.
Replace the now-deleted migration
server/datastore/mysql/migrations/data/20181119180000_DeleteSoftDeletedEntities.go
with a new migration containing the same timestamp. This allows Fleet to
see the appropriate migration state for users upgrading from previous
versions without actually modifying the DB.
Fixes#48
Adds endpoints and fleetctl commands to retrieve various debug profiles
from the Fleet server.
The best summary is from the help text:
```
fleetctl debug
NAME:
fleetctl debug - Tools for debugging Fleet
USAGE:
fleetctl debug command [command options] [arguments...]
COMMANDS:
profile Record a CPU profile from the Fleet server.
cmdline Get the command line used to invoke the Fleet server.
heap Report the allocated memory in the Fleet server.
goroutine Get stack traces of all goroutines (threads) in the Fleet server.
trace Record an execution trace on the Fleet server.
archive Create an archive with the entire suite of debug profiles.
OPTIONS:
--config value Path to the Fleet config file (default: "/Users/zwass/.fleet/config") [$CONFIG]
--context value Name of Fleet config context to use (default: "default") [$CONTEXT]
--help, -h show help
```
PR #9 unintentionally exposed the validation that prevented the @
character in usernames. We have decided there is no reason to block this
character.
Fixes#36
This is another error introduced in
https://github.com/kolide/fleet/pull/2327 we did not catch previously
due to insufficient unit test coverage. Test is now added.
- Add endpoints for osquery to register and continue a carve.
- Implement client functionality for retrieving carve details and contents in fleetctl.
- Add documentation on using file carving with Fleet.
Addresses kolide/fleet#1714
Changes in https://github.com/kolide/fleet/pull/2327 broke the MySQL
syntax for listing hosts with online status. This was not caught due to
the lack of a unit test for the functionality. This PR adds a unit test
and fixes the regression.
* Perform migration to delete any entries with `deleted` set, and
subsequently drop columns `deleted` and `deleted_at`.
* Remove `deleted` and `deleted_at` references.
Closes#2146
- Debounce frontend to reduce number of target searches in live query.
- More efficiently calculate label counts in live query and hosts
dashboard. Instead of using the (slow) CountHostsInTargets function,
retrieve the host counts while looking up the labels.
- Optimize targets search query. Removing the nested query retrieves the
same logical result set, but substantially optimizes MySQL CPU usage.
Testing indicates about a 50% reduction in MySQL CPU usage for the
frontend targets search API call after applying this change.
Getting a single host with `fleetctl get host foobar` will look up the
host with the matching hostname, uuid, osquery identifier, or node key,
and provide the full host details along with the labels the host is a
member of.
"Manual" labels can be specified by hostname, allowing users to specify
the membership of a label without having to use a dynamic query. See the
included documentation.
Label membership is now stored in the label_membership table. This is
done in preparation for adding "manual" labels, as previously label
membership was associated directly with label query executions.
Label queries are now all executed at the same time, rather than on
separate intervals. This simplifies the calculation of which distributed
queries need to be run when a host checks in.
This commit takes advantage of the existing pagination APIs in the Fleet
server, and provides additional APIs to support pagination in the web
UI. Doing this dramatically reduces the response sizes for requests from
the UI, and limits the performance impact of UI clients on the Fleet and
MySQL servers.
This change optimizes live queries by pushing the computation of query
targets to the creation time of the query, and efficiently caching the
targets in Redis. This results in a huge performance improvement at both
steady-state, and when running live queries.
- Live queries are stored using a bitfield in Redis, and takes
advantage of bitfield operations to be extremely efficient.
- Only run Redis live query test when REDIS_TEST is set in environment
- Ensure that live queries are only sent to hosts when there is a client
listening for results. Addresses an existing issue in Fleet along with
appropriate cleanup for the refactored live query backend.
Fleet used significant resources storing the full network interface
information for each host. This data was unused, except to get the
IP and MAC of the primary interface. With these changes, only those
pieces of data are stored.
- Calculate and store primary IP and MAC
- Remove transaction for storing full interfaces
- Update targets search to use new IP and MAC columns
- Update frontend to use new new columns
This PR removes unused types, code, DB tables, and associated migrations that are unused since Fleet 2.0.
An existing migration was refactored, and should remain compatible with both existing and new Fleet installations.
Additional information is collected when host details are updated using
the queries specified in the Fleet configuration. This additional
information is then available in the host API responses.
This adds a SQL injection prevention for a case in which we cannot use
parameters in the query.
It is not clear that this was possible to exploit. If it was possible,
it would have required a valid login to the Fleet server.
Previously a Go package attempting to import Fleet packages would run
into an error like "server/kolide/emails.go:93:23: undefined: Asset".
This commit refactors bindata asset handling to allow importing Fleet as
a library without changing the typical developer experience.
- Add toggle to disable live queries in advanced settings
- Add new live query status endpoint (checks for disabled via config and Redis health)
- Update QueryPage UI to use new live query status endpoint
Implements #2140
Update the github.com/russellhaering/goxmldsig dependency and apply
the appropriate fixes for the API changes.
This is a preparation for integration with
github.com/AbGuthrie/goquery, which uses a newer version of the
dependency.
- Add logging for new campaigns
- Add logging for new query creations/modification/deletion
- Add usernames for logs found in labels, options, packs, osquery options, queries and scheduled queries where something is created, modified or deleted
- Add the server_url_prefix flag for configuring this functionality
- Add prefix handling to the server routes
- Refactor JS to use appropriate paths from modules
- Use JS template to get URL prefix into JS environment
- Update webpack config to support prefixing
Thanks to securityonion.net for sponsoring the development of this feature.
Closes#1661
This change allows the images in Fleet emails to load properly from any
device with connectivity to github.com. Previously, emails might try to
load resources from a Kolide server not accessible from the email
client.
The asset URL will be based on the most recent git tag to accomodate
backwards-compatibility if the assets in the repo change.
Closes#1471
Adds Google Cloud PubSub logging for status and results.
This also changes the Write interface for logging modules to add a context.Context (only used by pubsub currently).
When an osqueryd agent sends an enroll request it automatically sends
some details about the system. We now save these details which helps
ensure we send the correct platform config.
Closes#2065
When LOGIN authentication was used, Fleet would send SMTP credentials
even if the connection the SMTP server was not secured via TLS.
Copying the pattern used in the standard library PlainAuth
implementation, we now only send credentials when the connection is
secure or the server is localhost.
- The most active NIC will be picked even if a formerly more active
interface still exists (previously, a NIC would stay primary as long
as it existed).
- Ignore link-local and loopback interfaces when choosing the primary.
- Fix bugs in which update status of the primary interface could be
reported incorrectly.
Fixes#2020
Fixes a regression introduced in 2.1.0 in which separate log lines are
no longer output separated by a newline. Now log lines in both output
plugins will do so.
- Refactor configuration for logging to use separate plugins
- Move existing filesystem logging to filesystem plugin
- Create new AWS firehose plugin
- Update documentation around logging
Almost two years ago, we began referring to the project as Fleet, but there are
many occurences of the term "Kolide" throughout the UI and documentation. This
PR attempts to clear up those uses where it is easily achievable.
The term "Kolide" is used throughout the code as well, but modifying this would
be more likely to introduce bugs.
Brings the behavior of the server in line with the documentation, by using the
query name if the scheduled query name is not specified in a pack spec.
Closes#1990
This PR adds support for the SMTP LOGIN authentication method. Office 365 Exchange removed support for PLAIN authentication some time ago, and only supports LOGIN and an OAuth2 authentication method. This patch has been tested with a licensed O365 account. This method should also be usable with any other email server that advertises LOGIN in its 250-AUTH response.
Note: If using this with O365, the account used must not have MFA enabled.
Closes#1663
This commit:
- adds a new sub-command for fetching hosts to `fleetctl get` command.
Why?
- this allows for listing of all hosts via the fleetctl interface.
There may be additional attributes of the host that we'd like to
display, but this should be a good start.
Closes#1962
Prior to this change, the live query status was calculated after each retrieved
result. This was fine at a low host count, but when more hosts are added, doing
this calculation becomes slower.
Now, the status is only calculated every 5 seconds, while results can be
streamed continuously.
Fixes#1895
Avoids potential bugs in which soft-deleted entities are returned from database
queries (soft-deletion is now deprecated), but some records may still exist.
Fixes#1956
This should fix the loading of the all hosts page in cases where there are many
hosts and it overwhelms the number of parameters allowed in a prepared
statement. May also make that page load slightly quicker as it removes the
constraint from the query, but should return the same number of results.
Fixes#1939
The SMTP configuration could be used by an admin user to port scan the network
the Fleet server was running on. This commit reduces the information possible
to determine via this technique. A malicious admin can no longer determine
whether any TCP server is listening on a given port/address. They can only
determine ports and addresses where SMTP servers are running.
Thanks to 'quikke' for reporting this vulnerability.
An incorrect authorization check allowed non-admin users to modify the details of other users. We now enforce the appropriate authorization so that unprivileged users can only modify their own details.
Thanks to 'Quikke' for the report.
The ability to modify a users admin and enabled status was erroneously left in
place during development of https://github.com/kolide/fleet/pull/959. To
mitigate a privilege escalation vulnerability we need to ensure those values
can only be modified through the explicit methods.
This patch includes a unit test and fix for the vulnerability.
Thanks to 'Quikke' for submitting this vulnerability.
Packs can be targeted to individual hosts through the UI. This was supported
previously and was broken with refactoring in Fleet 2.0.
There is currently no support in the fleetctl format for targeting individual
hosts, but this could be added at a later date.
Fixes#1878
- Delete duplicate queries in packs created by the UI (because the duplicates
were causing undefined behavior). Now it is not possible to schedule
duplicates in the UI (but is in fleetctl).
- Fix bug in which packs created in UI could not be loaded by fleetctl.
- Add cascading deletes for scheduled_queries when queries are deleted
- Also add cascading deletes for scheduled_queries when packs are deleted
Fixes#1837
Replaces the UI endpoints for creating and modifying labels. These were removed
in #1686 because we thought we were killing the UI.
Now labels can be created and edited in the UI again.
Replaces (and appropriately refactors) a number of endpoints that were removed long ago when we decided to kill the UI with the fleetctl release. We turned out not to do this, and now need to restore these missing endpoints.
This is not a straight up replacement of the existing code because of refactoring to the DB schemas that was also done in the migration.
Most of the replaced code was removed in #1670 and #1686.
Fixes#1811, fixes#1810
This PR adds support for getting resources by name.
```
$ fleetctl get queries
no queries found
$ fleetctl apply -f ./query.yaml
[+] applied 1 queries
$ fleetctl get queries
+-----------------+--------------------------------+--------------------------------+
| NAME | DESCRIPTION | QUERY |
+-----------------+--------------------------------+--------------------------------+
| osquery_version | The version of the Launcher | select launcher.version, |
| | and Osquery process | osquery.version from |
| | | kolide_launcher_info launcher, |
| | | osquery_info osquery; |
+-----------------+--------------------------------+--------------------------------+
$ fleetctl get query osquery_version
apiVersion: v1
kind: query
spec:
description: The version of the Launcher and Osquery process
name: osquery_version
query: select launcher.version, osquery.version from kolide_launcher_info launcher,
osquery_info osquery;
```
- Fix places where we accidentally return nil when we should return an error.
- Simplify interfaces/implementation of specialized errors
- Use more specific error messages
- Consistent JSON decoding
With the UI, deleting by ID made sense. With fleetctl, we now want to delete
by name. Transition only the methods used for spec related entities, as others
will be removed soon.
Previously decorators were stored in a separate table. Now they are stored
directly with the config so that they can be modified on a per-platform basis.
Delete now unused decorators code.
The DMARC and DKIM email authentication systems both require the RFC822
From header to function. Kolide currently only includes the configured
sender address as the SMTP Envelop From address (e.g., the MAIL FROM
command). This patch also includes the configured sender address in the
RFC822 email From header which should allow these emails to pass both
DKIM and DMARC authentication.
See https://goo.gl/zuku4E.
> The most obvious remediation here is ensuring your SAML library is extracting
the full text of a given XML element when comments are present.
Our implementation asks for the innerxml of the NameID field, so it returns the
entire text including the comment (See https://goo.gl/KLLXof). By default Go's
XML parsing would return the text not including the comment (but including
further text after the comment). Both of these options prevent the
vulnerability.
- Add new Apply spec methods for queries and packs
- Remove now extraneous datastore/service methods
- Remove import service (unused, and had many dependencies that this breaks)
- Refactor tests as appropriate
Include the appropriate values for removed and shard when generating config
to return to osqueryd.
Note: This was originally fixed and tested in the fleetctl branch (#1680), and
the fix is being cherry-picked into master without the test.
After discussion with @groob and @marpaia, we have decided that the service
methods should not be aware of any YAML/JSON definitions, and should work
directly with objects. The new pattern we will use will involve converting YAML
to JSON at the client, and then sending the JSON which will be decoded using
the familiar go-kit mechanisms before being passed to the service methods.
Instead of trying to decode and re-encode status logs, we now write them directly as they come in.
This change prevents future changes to the osquery status log file format (addition and deletion of fields ) from
affecting Fleet. A similar change was implemented in #1636 for result logs.
Closes#1664
Initially fleet decoded the incoming JSON sent to the log endpoint.
Then the log event would be written to a log writer by calling json.Encoder{}.Encode.
Re-encoding logs is lossy; whenever a new field is sent by osqueryd we don't keep up with them.
Instead of caring about the content of the OsqueryResultLog, fleet will now write all log results
exactly as sent to the server by osqueryd.
Closes#1632Closes#1615
