- Debounce frontend to reduce number of target searches in live query.
- More efficiently calculate label counts in live query and hosts
dashboard. Instead of using the (slow) CountHostsInTargets function,
retrieve the host counts while looking up the labels.
- Optimize targets search query. Removing the nested query retrieves the
same logical result set, but substantially optimizes MySQL CPU usage.
Testing indicates about a 50% reduction in MySQL CPU usage for the
frontend targets search API call after applying this change.
Getting a single host with `fleetctl get host foobar` will look up the
host with the matching hostname, uuid, osquery identifier, or node key,
and provide the full host details along with the labels the host is a
member of.
"Manual" labels can be specified by hostname, allowing users to specify
the membership of a label without having to use a dynamic query. See the
included documentation.
Label membership is now stored in the label_membership table. This is
done in preparation for adding "manual" labels, as previously label
membership was associated directly with label query executions.
Label queries are now all executed at the same time, rather than on
separate intervals. This simplifies the calculation of which distributed
queries need to be run when a host checks in.
This commit takes advantage of the existing pagination APIs in the Fleet
server, and provides additional APIs to support pagination in the web
UI. Doing this dramatically reduces the response sizes for requests from
the UI, and limits the performance impact of UI clients on the Fleet and
MySQL servers.
This change optimizes live queries by pushing the computation of query
targets to the creation time of the query, and efficiently caching the
targets in Redis. This results in a huge performance improvement at both
steady-state, and when running live queries.
- Live queries are stored using a bitfield in Redis, and takes
advantage of bitfield operations to be extremely efficient.
- Only run Redis live query test when REDIS_TEST is set in environment
- Ensure that live queries are only sent to hosts when there is a client
listening for results. Addresses an existing issue in Fleet along with
appropriate cleanup for the refactored live query backend.
Fleet used significant resources storing the full network interface
information for each host. This data was unused, except to get the
IP and MAC of the primary interface. With these changes, only those
pieces of data are stored.
- Calculate and store primary IP and MAC
- Remove transaction for storing full interfaces
- Update targets search to use new IP and MAC columns
- Update frontend to use new new columns
This PR removes unused types, code, DB tables, and associated migrations that are unused since Fleet 2.0.
An existing migration was refactored, and should remain compatible with both existing and new Fleet installations.
Additional information is collected when host details are updated using
the queries specified in the Fleet configuration. This additional
information is then available in the host API responses.
This adds a SQL injection prevention for a case in which we cannot use
parameters in the query.
It is not clear that this was possible to exploit. If it was possible,
it would have required a valid login to the Fleet server.
- Add toggle to disable live queries in advanced settings
- Add new live query status endpoint (checks for disabled via config and Redis health)
- Update QueryPage UI to use new live query status endpoint
Implements #2140
- Add logging for new campaigns
- Add logging for new query creations/modification/deletion
- Add usernames for logs found in labels, options, packs, osquery options, queries and scheduled queries where something is created, modified or deleted
- Add the server_url_prefix flag for configuring this functionality
- Add prefix handling to the server routes
- Refactor JS to use appropriate paths from modules
- Use JS template to get URL prefix into JS environment
- Update webpack config to support prefixing
Thanks to securityonion.net for sponsoring the development of this feature.
Closes#1661
Almost two years ago, we began referring to the project as Fleet, but there are
many occurences of the term "Kolide" throughout the UI and documentation. This
PR attempts to clear up those uses where it is easily achievable.
The term "Kolide" is used throughout the code as well, but modifying this would
be more likely to introduce bugs.
Brings the behavior of the server in line with the documentation, by using the
query name if the scheduled query name is not specified in a pack spec.
Closes#1990
Avoids potential bugs in which soft-deleted entities are returned from database
queries (soft-deletion is now deprecated), but some records may still exist.
Fixes#1956
This should fix the loading of the all hosts page in cases where there are many
hosts and it overwhelms the number of parameters allowed in a prepared
statement. May also make that page load slightly quicker as it removes the
constraint from the query, but should return the same number of results.
Fixes#1939
Packs can be targeted to individual hosts through the UI. This was supported
previously and was broken with refactoring in Fleet 2.0.
There is currently no support in the fleetctl format for targeting individual
hosts, but this could be added at a later date.
Fixes#1878
- Delete duplicate queries in packs created by the UI (because the duplicates
were causing undefined behavior). Now it is not possible to schedule
duplicates in the UI (but is in fleetctl).
- Fix bug in which packs created in UI could not be loaded by fleetctl.
- Add cascading deletes for scheduled_queries when queries are deleted
- Also add cascading deletes for scheduled_queries when packs are deleted
Fixes#1837
Replaces the UI endpoints for creating and modifying labels. These were removed
in #1686 because we thought we were killing the UI.
Now labels can be created and edited in the UI again.
Replaces (and appropriately refactors) a number of endpoints that were removed long ago when we decided to kill the UI with the fleetctl release. We turned out not to do this, and now need to restore these missing endpoints.
This is not a straight up replacement of the existing code because of refactoring to the DB schemas that was also done in the migration.
Most of the replaced code was removed in #1670 and #1686.
Fixes#1811, fixes#1810
