This addresses an issue some users experienced in which performance
problems were encountered when hosts were "competing" for enrollment
using the same osquery host identifier. The issue is addressed by adding
a cooldown period for host enrollment, preventing the same (as judged by
osquery host identifier) host from enrolling more than once per minute.
When users end up in the problematic scenario, they will see quite a bit
of error logs due to this issue. For now that's probably a good thing as
users need to be aware of the lack of visibility. We can explore rate
limiting the logging if that becomes an issue for someone.
Fixes#102
The goal of this PR was to insert new colors, remove legacy colors, and update only the styles accordingly. The Nunito Sans Italic font was also added.
Later PRs will include layout, copy, and style change to individual components. These later changes will more exactly resemble the current mockups.
This is the second PR as part of the Fleet UI Refresh #38.
The goal of this PR was to insert all updated Fleet assets and remove all old assets. More style changes, including the exact sizing and placement of the new images, will occur in a future PR.
This is the first PR as a part of the Fleet UI Refresh #38.
Changes include:
- Add Nunito Sans font files and modify global styles to reflect the font change.
- Modify global font variables to reflect new sizing and weight naming conventions.
-- New sizing and naming conventions:
--- SIZE: xx-small: 12px, x-small: 14px, small: 16px, medium: 20px, large: 24px, x-large: 28px
--- WEIGHT: regular: 400, bold: 700
- Remove the old Oxygen font files.
Changes to other style sheets reflect the changes to the new font sizing and weight naming conventions for global variables. The changes don't necessarily use the correct size (as illustrated by mockups). Those "up to spec" sizing changes are to come.
Replace the now-deleted migration
server/datastore/mysql/migrations/data/20181119180000_DeleteSoftDeletedEntities.go
with a new migration containing the same timestamp. This allows Fleet to
see the appropriate migration state for users upgrading from previous
versions without actually modifying the DB.
Fixes#48
Adds endpoints and fleetctl commands to retrieve various debug profiles
from the Fleet server.
The best summary is from the help text:
```
fleetctl debug
NAME:
fleetctl debug - Tools for debugging Fleet
USAGE:
fleetctl debug command [command options] [arguments...]
COMMANDS:
profile Record a CPU profile from the Fleet server.
cmdline Get the command line used to invoke the Fleet server.
heap Report the allocated memory in the Fleet server.
goroutine Get stack traces of all goroutines (threads) in the Fleet server.
trace Record an execution trace on the Fleet server.
archive Create an archive with the entire suite of debug profiles.
OPTIONS:
--config value Path to the Fleet config file (default: "/Users/zwass/.fleet/config") [$CONFIG]
--context value Name of Fleet config context to use (default: "default") [$CONTEXT]
--help, -h show help
```
PR #9 unintentionally exposed the validation that prevented the @
character in usernames. We have decided there is no reason to block this
character.
Fixes#36
This is another error introduced in
https://github.com/kolide/fleet/pull/2327 we did not catch previously
due to insufficient unit test coverage. Test is now added.
- Add endpoints for osquery to register and continue a carve.
- Implement client functionality for retrieving carve details and contents in fleetctl.
- Add documentation on using file carving with Fleet.
Addresses kolide/fleet#1714
Changes in https://github.com/kolide/fleet/pull/2327 broke the MySQL
syntax for listing hosts with online status. This was not caught due to
the lack of a unit test for the functionality. This PR adds a unit test
and fixes the regression.
* Perform migration to delete any entries with `deleted` set, and
subsequently drop columns `deleted` and `deleted_at`.
* Remove `deleted` and `deleted_at` references.
Closes#2146
- Debounce frontend to reduce number of target searches in live query.
- More efficiently calculate label counts in live query and hosts
dashboard. Instead of using the (slow) CountHostsInTargets function,
retrieve the host counts while looking up the labels.
- Optimize targets search query. Removing the nested query retrieves the
same logical result set, but substantially optimizes MySQL CPU usage.
Testing indicates about a 50% reduction in MySQL CPU usage for the
frontend targets search API call after applying this change.
Getting a single host with `fleetctl get host foobar` will look up the
host with the matching hostname, uuid, osquery identifier, or node key,
and provide the full host details along with the labels the host is a
member of.
"Manual" labels can be specified by hostname, allowing users to specify
the membership of a label without having to use a dynamic query. See the
included documentation.
Label membership is now stored in the label_membership table. This is
done in preparation for adding "manual" labels, as previously label
membership was associated directly with label query executions.
Label queries are now all executed at the same time, rather than on
separate intervals. This simplifies the calculation of which distributed
queries need to be run when a host checks in.
This commit takes advantage of the existing pagination APIs in the Fleet
server, and provides additional APIs to support pagination in the web
UI. Doing this dramatically reduces the response sizes for requests from
the UI, and limits the performance impact of UI clients on the Fleet and
MySQL servers.
This change optimizes live queries by pushing the computation of query
targets to the creation time of the query, and efficiently caching the
targets in Redis. This results in a huge performance improvement at both
steady-state, and when running live queries.
- Live queries are stored using a bitfield in Redis, and takes
advantage of bitfield operations to be extremely efficient.
- Only run Redis live query test when REDIS_TEST is set in environment
- Ensure that live queries are only sent to hosts when there is a client
listening for results. Addresses an existing issue in Fleet along with
appropriate cleanup for the refactored live query backend.
Fleet used significant resources storing the full network interface
information for each host. This data was unused, except to get the
IP and MAC of the primary interface. With these changes, only those
pieces of data are stored.
- Calculate and store primary IP and MAC
- Remove transaction for storing full interfaces
- Update targets search to use new IP and MAC columns
- Update frontend to use new new columns
This PR removes unused types, code, DB tables, and associated migrations that are unused since Fleet 2.0.
An existing migration was refactored, and should remain compatible with both existing and new Fleet installations.
Additional information is collected when host details are updated using
the queries specified in the Fleet configuration. This additional
information is then available in the host API responses.
This adds a SQL injection prevention for a case in which we cannot use
parameters in the query.
It is not clear that this was possible to exploit. If it was possible,
it would have required a valid login to the Fleet server.
Previously a Go package attempting to import Fleet packages would run
into an error like "server/kolide/emails.go:93:23: undefined: Asset".
This commit refactors bindata asset handling to allow importing Fleet as
a library without changing the typical developer experience.
- Add toggle to disable live queries in advanced settings
- Add new live query status endpoint (checks for disabled via config and Redis health)
- Update QueryPage UI to use new live query status endpoint
Implements #2140
Update the github.com/russellhaering/goxmldsig dependency and apply
the appropriate fixes for the API changes.
This is a preparation for integration with
github.com/AbGuthrie/goquery, which uses a newer version of the
dependency.
- Add logging for new campaigns
- Add logging for new query creations/modification/deletion
- Add usernames for logs found in labels, options, packs, osquery options, queries and scheduled queries where something is created, modified or deleted
- Add the server_url_prefix flag for configuring this functionality
- Add prefix handling to the server routes
- Refactor JS to use appropriate paths from modules
- Use JS template to get URL prefix into JS environment
- Update webpack config to support prefixing
Thanks to securityonion.net for sponsoring the development of this feature.
Closes#1661
This change allows the images in Fleet emails to load properly from any
device with connectivity to github.com. Previously, emails might try to
load resources from a Kolide server not accessible from the email
client.
The asset URL will be based on the most recent git tag to accomodate
backwards-compatibility if the assets in the repo change.
Closes#1471
Adds Google Cloud PubSub logging for status and results.
This also changes the Write interface for logging modules to add a context.Context (only used by pubsub currently).
When an osqueryd agent sends an enroll request it automatically sends
some details about the system. We now save these details which helps
ensure we send the correct platform config.
Closes#2065
When LOGIN authentication was used, Fleet would send SMTP credentials
even if the connection the SMTP server was not secured via TLS.
Copying the pattern used in the standard library PlainAuth
implementation, we now only send credentials when the connection is
secure or the server is localhost.
- The most active NIC will be picked even if a formerly more active
interface still exists (previously, a NIC would stay primary as long
as it existed).
- Ignore link-local and loopback interfaces when choosing the primary.
- Fix bugs in which update status of the primary interface could be
reported incorrectly.
Fixes#2020
Fixes a regression introduced in 2.1.0 in which separate log lines are
no longer output separated by a newline. Now log lines in both output
plugins will do so.
- Refactor configuration for logging to use separate plugins
- Move existing filesystem logging to filesystem plugin
- Create new AWS firehose plugin
- Update documentation around logging
Almost two years ago, we began referring to the project as Fleet, but there are
many occurences of the term "Kolide" throughout the UI and documentation. This
PR attempts to clear up those uses where it is easily achievable.
The term "Kolide" is used throughout the code as well, but modifying this would
be more likely to introduce bugs.
Brings the behavior of the server in line with the documentation, by using the
query name if the scheduled query name is not specified in a pack spec.
Closes#1990
This PR adds support for the SMTP LOGIN authentication method. Office 365 Exchange removed support for PLAIN authentication some time ago, and only supports LOGIN and an OAuth2 authentication method. This patch has been tested with a licensed O365 account. This method should also be usable with any other email server that advertises LOGIN in its 250-AUTH response.
Note: If using this with O365, the account used must not have MFA enabled.
Closes#1663
This commit:
- adds a new sub-command for fetching hosts to `fleetctl get` command.
Why?
- this allows for listing of all hosts via the fleetctl interface.
There may be additional attributes of the host that we'd like to
display, but this should be a good start.
Closes#1962
