The enrollment cooldown period was sometimes causing problems when
osquery (probably unintentionally, see
https://github.com/osquery/osquery/issues/6993) tried to enroll more
than once from the same osqueryd process.
We now set this to default to off and make it configurable. With #417
this feature may be unnecessary for most deployments.
Uses a LIKE clause to search for hosts matching the query against
columns `host_name`, `uuid`, `hardware_serial`, and `primary_ip`.
Introduces the `searchLike` helper to add the appropriate filters to the
SQL query.
Somewhere around osquery 4.4.0 these messages were added to query
responses. We can now expose them to the API clients rather than using
the placeholder text.
Required for #192
This adds the option to set up an S3 bucket as the storage backend for file carving (partially solving #111).
It works by using the multipart upload capabilities of S3 to maintain compatibility with the "upload in blocks" protocol that osquery uses. It does this basically replacing the carve_blocks table while still maintaining the metadata in the original place (it would probably be possible to rely completely on S3 by using object tagging at the cost of listing performance). To make this pluggable, I created a new field in the service struct dedicated to the CarveStore which, if no configuration for S3 is set up will be just a reference to the standard datastore, otherwise it will point to the S3 one (effectively this separation will allow in the future to add more backends).
Mitigate
[CVE-2020-26273](https://github.com/osquery/osquery/security/advisories/GHSA-4g56-2482-x7q8)
by attempting to prevent users from executing or saving queries that use
the SQLite `ATTACH` command.
Users must still update to osquery 4.6.0 to ensure the functionality is
fully disabled in osquery.
This addresses an issue some users experienced in which performance
problems were encountered when hosts were "competing" for enrollment
using the same osquery host identifier. The issue is addressed by adding
a cooldown period for host enrollment, preventing the same (as judged by
osquery host identifier) host from enrolling more than once per minute.
When users end up in the problematic scenario, they will see quite a bit
of error logs due to this issue. For now that's probably a good thing as
users need to be aware of the lack of visibility. We can explore rate
limiting the logging if that becomes an issue for someone.
Fixes#102
- Add endpoints for osquery to register and continue a carve.
- Implement client functionality for retrieving carve details and contents in fleetctl.
- Add documentation on using file carving with Fleet.
Addresses kolide/fleet#1714
* Perform migration to delete any entries with `deleted` set, and
subsequently drop columns `deleted` and `deleted_at`.
* Remove `deleted` and `deleted_at` references.
Closes#2146
- Debounce frontend to reduce number of target searches in live query.
- More efficiently calculate label counts in live query and hosts
dashboard. Instead of using the (slow) CountHostsInTargets function,
retrieve the host counts while looking up the labels.
- Optimize targets search query. Removing the nested query retrieves the
same logical result set, but substantially optimizes MySQL CPU usage.
Testing indicates about a 50% reduction in MySQL CPU usage for the
frontend targets search API call after applying this change.
Getting a single host with `fleetctl get host foobar` will look up the
host with the matching hostname, uuid, osquery identifier, or node key,
and provide the full host details along with the labels the host is a
member of.
"Manual" labels can be specified by hostname, allowing users to specify
the membership of a label without having to use a dynamic query. See the
included documentation.
Label membership is now stored in the label_membership table. This is
done in preparation for adding "manual" labels, as previously label
membership was associated directly with label query executions.
Label queries are now all executed at the same time, rather than on
separate intervals. This simplifies the calculation of which distributed
queries need to be run when a host checks in.
This commit takes advantage of the existing pagination APIs in the Fleet
server, and provides additional APIs to support pagination in the web
UI. Doing this dramatically reduces the response sizes for requests from
the UI, and limits the performance impact of UI clients on the Fleet and
MySQL servers.
This change optimizes live queries by pushing the computation of query
targets to the creation time of the query, and efficiently caching the
targets in Redis. This results in a huge performance improvement at both
steady-state, and when running live queries.
- Live queries are stored using a bitfield in Redis, and takes
advantage of bitfield operations to be extremely efficient.
- Only run Redis live query test when REDIS_TEST is set in environment
- Ensure that live queries are only sent to hosts when there is a client
listening for results. Addresses an existing issue in Fleet along with
appropriate cleanup for the refactored live query backend.
Fleet used significant resources storing the full network interface
information for each host. This data was unused, except to get the
IP and MAC of the primary interface. With these changes, only those
pieces of data are stored.
- Calculate and store primary IP and MAC
- Remove transaction for storing full interfaces
- Update targets search to use new IP and MAC columns
- Update frontend to use new new columns
This PR removes unused types, code, DB tables, and associated migrations that are unused since Fleet 2.0.
An existing migration was refactored, and should remain compatible with both existing and new Fleet installations.
Additional information is collected when host details are updated using
the queries specified in the Fleet configuration. This additional
information is then available in the host API responses.
Previously a Go package attempting to import Fleet packages would run
into an error like "server/kolide/emails.go:93:23: undefined: Asset".
This commit refactors bindata asset handling to allow importing Fleet as
a library without changing the typical developer experience.
- Add toggle to disable live queries in advanced settings
- Add new live query status endpoint (checks for disabled via config and Redis health)
- Update QueryPage UI to use new live query status endpoint
Implements #2140
- Add logging for new campaigns
- Add logging for new query creations/modification/deletion
- Add usernames for logs found in labels, options, packs, osquery options, queries and scheduled queries where something is created, modified or deleted
- Add the server_url_prefix flag for configuring this functionality
- Add prefix handling to the server routes
- Refactor JS to use appropriate paths from modules
- Use JS template to get URL prefix into JS environment
- Update webpack config to support prefixing
Thanks to securityonion.net for sponsoring the development of this feature.
Closes#1661
This change allows the images in Fleet emails to load properly from any
device with connectivity to github.com. Previously, emails might try to
load resources from a Kolide server not accessible from the email
client.
The asset URL will be based on the most recent git tag to accomodate
backwards-compatibility if the assets in the repo change.
Closes#1471
Adds Google Cloud PubSub logging for status and results.
This also changes the Write interface for logging modules to add a context.Context (only used by pubsub currently).
When an osqueryd agent sends an enroll request it automatically sends
some details about the system. We now save these details which helps
ensure we send the correct platform config.
Closes#2065
- The most active NIC will be picked even if a formerly more active
interface still exists (previously, a NIC would stay primary as long
as it existed).
- Ignore link-local and loopback interfaces when choosing the primary.
- Fix bugs in which update status of the primary interface could be
reported incorrectly.
Fixes#2020
- Refactor configuration for logging to use separate plugins
- Move existing filesystem logging to filesystem plugin
- Create new AWS firehose plugin
- Update documentation around logging
Almost two years ago, we began referring to the project as Fleet, but there are
many occurences of the term "Kolide" throughout the UI and documentation. This
PR attempts to clear up those uses where it is easily achievable.
The term "Kolide" is used throughout the code as well, but modifying this would
be more likely to introduce bugs.
This PR adds support for the SMTP LOGIN authentication method. Office 365 Exchange removed support for PLAIN authentication some time ago, and only supports LOGIN and an OAuth2 authentication method. This patch has been tested with a licensed O365 account. This method should also be usable with any other email server that advertises LOGIN in its 250-AUTH response.
Note: If using this with O365, the account used must not have MFA enabled.
Closes#1663
