* Add ability to apply a `policy` yaml document so that GitOps workflows can be used to create and
modify policies.
* Add ability to run a live query that returns 1,000+ results in the Fleet UI by adding
client-side pagination to the results table.
* Improve the accuracy of query platform compatibility detection by adding recognition for queries
with the `WITH` expression.
* Add ability to open a page in the Fleet UI in a new tab by "right-clicking" an item in the navigation.
* Improve the [live query API route (`GET /api/v1/queries/run`)](https://fleetdm.com/docs/using-fleet/rest-api#run-live-query) so that it successfully return results for Fleet
instances using a load balancer by reducing the wait period to 25 seconds.
* Improve performance of the Fleet UI by updating loading states and reducing the number of requests
made to the Fleet API.
* Improve performance of the MySQL database by updating the queries used to populate host vitals and
caching the results.
* Add [`read_timeout` Redis configuration
option](https://fleetdm.com/docs/deploying/configuration#redis-read-timeout) to customize the
maximum amount of time Fleet should wait to receive a response from a Redis server.
* Add [`write_timeout` Redis configuration
option](https://fleetdm.com/docs/deploying/configuration#redis-write-timeout) to customize the
maximum amount of time Fleet should wait to send a command to a Redis server.
* Fix a bug in which browser extensions (Google Chrome, Firefox, and Safari) were not included in
software inventory.
* Improve the security of the **Organization settings** page by preventing the browser from requesting
to save SMTP credentials.
* Fix a bug in which an existing pack's targets were not cleaned up after deleting hosts, labels, and teams.
* Fix a bug in which non-existent queries and policies would not return a 404 not found response.
### Performance
* Our testing demonstrated an increase in max devices served in our load test infrastructure to 70,000 from 60,000 in v4.8.0.
#### Load Test Infrastructure
* Fleet server
* AWS Fargate
* 2 tasks with 1024 CPU units and 2048 MiB of RAM.
* MySQL
* Amazon RDS
* db.r5.2xlarge
* Redis
* Amazon ElastiCache
* cache.m5.large with 2 replicas (no cluster mode)
#### What was changed to accomplish these improvements?
* Optimized the updating and fetching of host data to only send and receive the bare minimum data
needed.
* Reduced the number of times host information is updated by caching more data.
* Updated cleanup jobs and deletion logic.
#### Future improvements
* At maximum DB utilization, we found that some hosts fail to respond to live queries. Future releases of Fleet will improve upon this.
* Improve performance of the **Home** page by removing total hosts count from the "Software" table.
* Improve performance of the **Queries** page by adding pagination to the list of queries.
* Fix a bug in which the "Shell" column of the "Users" table on the **Host details** page would sometimes fail to update.
* Fix a bug in which a host's status could quickly alternate between "Online" and "Offline" by increasing the grace period for host status.
* Fix a bug in which some hosts would have a missing `host_seen_times` entry.
* Add an `after` parameter to the [`GET /hosts` API route](https://fleetdm.com/docs/using-fleet/rest-api#list-hosts) to allow for cursor pagination.
* Add a `disable_failing_policies` parameter to the [`GET /hosts` API route](https://fleetdm.com/docs/using-fleet/rest-api#list-hosts) to allow the API request to respond faster if failing policies count information is not needed.
* Fleet Premium: Add ability to filter aggregate host data such as platforms (macOS, Windows, and Linux) and status (online, offline, and new) the **Home** page. The aggregate host data is also available in the [`GET /host_summary API route`](https://fleetdm.com/docs/using-fleet/rest-api#get-hosts-summary).
* Fleet Premium: Add ability to move pending invited users between teams.
* Fleet Premium: Add `fleetctl updates rotate` command for rotation of keys in the updates system. The `fleetctl updates` command provides the ability to [self-manage an agent update server](https://fleetdm.com/docs/deploying/fleetctl-agent-updates).
* Enable the software inventory by default for new Fleet instances. The software inventory feature can be turned on or off using the [`enable_software_inventory` configuration option](https://fleetdm.com/docs/using-fleet/vulnerability-processing#setup).
* Update the JSON payload for the host status webhook by renaming the `"message"` property to `"text"` so that the payload can be received and displayed in Slack.
* Remove the deprecated `app_configs` table from Fleet's MySQL database. The `app_config_json` table has replaced it.
* Improve performance of the policies feature for Fleet instances with over 100,000 hosts.
* Add instructions in the Fleet UI for generating an osquery installer for macOS, Linux, or Windows. Documentation for generating an osquery installer and distributing the installer to your hosts to add them to Fleet can be found here on [fleetdm.com/docs](https://fleetdm.com/docs/using-fleet/adding-hosts)
* Add ability to see all the software, and filter by vulnerable software, installed across all your hosts on the **Home** page. Each software's `name`, `version`, `hosts_count`, `vulnerabilities`, and more is also available in the [`GET /software` API route](https://fleetdm.com/docs/using-fleet/rest-api#software) and `fleetctl get software` command.
* Add ability to add, edit, and delete enroll secrets on the **Hosts** page.
* Add ability to see aggregate host data such as platforms (macOS, Windows, and Linux) and status (online, offline, and new) the **Home** page.
* Add ability to see all of the queries scheduled to run on a specific host on the **Host details** page immediately after a query is added to a schedule or pack.
* Add a "Shell" column to the "Users" table on the **Host details** page so users can now be filtered to see only those who have logged in.
* Package osquery's `certs.pem` in `fleetctl package` to improve TLS compatibility.
* Add support for packaging an osquery flagfile with `fleetctl package --osquery-flagfile`.
* Use "Fleet osquery" rather than "Orbit osquery" in packages generated by `fleetctl package`.
* Clarify that a policy in Fleet is a yes or no question you can ask about your hosts by replacing "Passing" and "Failing" text with "Yes" and "No" respectively on the **Policies** page and **Host details** page.
* Add ability to see the original author of a query on the **Query** page.
* Improve the UI for the "Software" table and "Policies" table on the **Host details** page so that it's easier to pivot to see all hosts with a specific software installed or answering "No" to a specific policy.
* Fix a bug in which modifying a specific target for a live query, in target selector UI, would deselect a different target.
* Fix a bug in which the user was navigated to a non existent page, in the Fleet UI, after saving a pack.
* Fix a bug in which long software names in the "Software" table caused the bundle identifier tooltip to be inaccessible.
* Fleet Premium: Add a Team admin user role. This allows users to delegate the responsibility of managing team members in Fleet. Documentation for the permissions associated with the Team admin and other user roles can be found [here on fleetdm.com/docs](https://fleetdm.com/docs/using-fleet/permissions).
* Add Apache Kafka logging plugin. Documentation for configuring Kafka as a logging plugin can be found [here on fleetdm.com/docs](https://fleetdm.com/docs/deploying/configuration#kafka-rest-proxy-logging). Thank you to Joseph Macaulay for adding this capability.
* Add support for [MinIO](https://min.io/) as a file carving backend. Documentation for configuring MinIO as a file carving backend can be found [here on fleetdm.com/docs](https://fleetdm.com/docs/using-fleet/fleetctl-cli#minio). Thank you to Chandra Majumdar and Ben Edwards for adding this capability.
* Add support for generating a `.pkg` osquery installer on Linux without dependencies (beyond Docker) with the `fleetctl package` command.
* Improve the performance of vulnerability processing by making the process consume less RAM. Documentation for the vulnerability processing feature can be found [here on fleetdm.com/docs](https://fleetdm.com/docs/using-fleet/vulnerability-processing).
* Add the ability to run a live query and receive results using only the Fleet REST API with a `GET /api/v1/fleet/queries/run` API route. Documentation for this new API route can be found [here on fleetdm.com/docs](https://fleetdm.com/docs/using-fleet/rest-api#run-live-query).
* Add ability to see whether a specific host is "Passing" or "Failing" a policy on the **Host details** page. This information is also exposed in the `GET api/v1/fleet/hosts/{id}` API route. In Fleet, a policy is a "yes" or "no" question you can ask of all your hosts.
* Add the ability to quickly see the total number of "Failing" policies for a particular host on the **Hosts** page with a new "Issues" column. Total "Issues" are also revealed on a specific host's **Host details** page.
* Add the ability to see which platforms (macOS, Windows, Linux) a specific query is compatible with. The compatibility detected by Fleet is estimated based on the osquery tables used in the query.
* Add the ability to see whether your queries have a "Minimal," "Considerable," or "Excessive" performance impact on your hosts. Query performance information is only collected when a query runs as a scheduled query.
* Running a "Minimal" query, very frequently, has little to no impact on your host's performance.
* Running a "Considerable" query, frequently, can have a noticeable impact on your host's performance.
* Running an "Excessive" query, even infrequently, can have a significant impact on your host’s performance.
* Add the ability to see a list of hosts that have a specific software version installed by selecting a software version on a specific host's **Host details** page. Software inventory is currently under a feature flag. To enable this feature flag, check out the [feature flag documentation](./docs/02-Deploying/03-Configuration.md#feature-flags).
* Add the ability to see all vulnerable software detected across all your hosts with the `GET /api/v1/fleet/software` API route. Documentation for this new API route can be found [here on fleetdm.com/docs](https://fleetdm.com/docs/using-fleet/rest-api#software).
* Add the ability to see the exact number of hosts that selected filters on the **Hosts** page. This ability is also available when using the `GET api/v1/fleet/hosts/count` API route.
* Add ability to automatically "Refetch" host vitals for a particular host without manually reloading the page.
* Add ability to connect to Redis with TLS. Documentation for configuring Fleet to use a TLS connection to the Redis server can be found [here on fleetdm.com/docs](https://fleetdm.com/docs/deploying/configuration#redis-use-tls).
* Add `cluster_read_from_replica` Redis to specify whether or not to prefer readying from a replica when possible. Documentation for this configuration option can be found [here on fleetdm.com/docs](https://fleetdm.com/docs/deploying/configuration#redis-cluster-read-from-replica).
* Improve database performance by reducing the amount of MySQL database queries when a host checks in.
* Fix a bug in which users with the global maintainer role could not edit or save queries. In, Fleet 4.0.0, the Admin, Maintainer, and Observer user roles were introduced. Documentation for the permissions associated with each role can be found [here on fleetdm.com/docs](https://fleetdm.com/docs/using-fleet/permissions).
* Fix a bug in which policies were checked about every second and add a `policy_update_interval` osquery configuration option. Documentation for this configuration option can be found [here on fleetdm.com/docs](https://fleetdm.com/docs/deploying/configuration#osquery-policy-update-interval).
* Fix a bug in which edits to a query’s name, description, SQL did not appear until the user refreshed the Edit query page.
* Fix a bug in which the hosts count for a label returned 0 after modifying a label’s name or description.
* Improve error message when attempting to create or edit a user with an email that already exists.
* Add `fleetctl get software` command to list all software and the detected vulnerabilities. The Vulnerable software feature is currently in Beta. For information on how to configure the Vulnerable software feature and how exactly Fleet processes vulnerabilities, check out the [Vulnerability processing documentation](https://fleetdm.com/docs/using-fleet/vulnerability-processing).
* Add `disable_data_sync` vulnerabilities configuration option to avoid downloading the data streams. Documentation for this configuration option can be found [here on fleetdm.com/docs](https://fleetdm.com/docs/deploying/configuration#disable-data-sync).
* Only show observers the queries they have permissions to run on the **Queries** page. In, Fleet 4.0.0, the Admin, Maintainer, and Observer user roles were introduced. Documentation for the permissions associated with each role can be found [here on fleetdm.com/docs](https://fleetdm.com/docs/using-fleet/permissions).
* Add `connect_retry_attempts` Redis configuration option to retry failed connections. Documentation for this configuration option can be found [here on fleetdm.com/docs](https://fleetdm.com/docs/deploying/configuration#redis-connect-retry-attempts).
* Add `cluster_follow_redirections` Redis configuration option to follow cluster redirections. Documentation for this configuration option can be found [here on fleetdm.com/docs](https://fleetdm.com/docs/deploying/configuration#redis-cluster-follow-redirections).
* Add `max_jitter_percent` osquery configuration option to prevent all hosts from returning data at roughly the same time. Note that this improves the Fleet server performance, but it will now take longer for new labels to populate. Documentation for this configuration option can be found [here on fleetdm.com/docs](https://fleetdm.com/docs/deploying/configuration#osquery-max-jitter-percent).
* Add Policies feature for detecting device compliance with organizational policies.
* Run/edit query experience has been completely redesigned.
* Add support for MySQL read replicas. This allows the Fleet server to scale to more hosts.
* Add configurable webhook to notify when a specified percentage of hosts have been offline for over the specified amount of days.
* Add `fleetctl package` command for building Orbit packages.
* Add enroll secret dialog on host dashboard.
* Expose free disk space in gigs and percentage for hosts.
* Add 15-minute interval option on Schedule page.
* Clean up advanced options UI.
* 404 and 500 page now include buttons for Osquery community Slack and to file an issue
* Update all empty and error states for cleaner UI.
* Add warning banners in Fleet UI and `fleetctl` for license expiration.
* Render query performance information on host vitals page pack section.
* Improved performance for app loading.
* Make team schedule names more user friendly and hide the stats for global and team schedules when showing host pack stats.
* Display `query_name` in when referencing scheduled queries for more consistent UI/UX.
* Query action added for observers on host vitals page.
* Add `server_settings.debug_host_ids` to gather more detailed information about what the specified hosts are sending to fleet.
* Allow deeper linking into the Fleet application by saving filters in URL parameters.
* Rename Basic Tier to Premium Tier, and Core Tier to Free Tier.
* Improve vulnerability detection compatibility with database configurations.
* MariaDB compatibility fixes: add explicit foreign key constraint and on cascade delete for host_software to allow for hosts with software to be deleted.
* Fix migration that was incompatible with MySQL primary key requirements (default on DigitalOcean MySQL 5.8).
* Add 30 second SMTP timeout for mail configuration.
* Fix display of platform Labels on manage hosts page
* Fix a bug recording scheduled query statistics.
* When a label is removed, ignore query executions for that label.
* Add fleet serve config to change the redis connection timeout and keep alive interval.
* Remove hardcoded limits in label searches when targeting queries.
* Allow host users to be readded.
* Move email template images from github to fleetdm.com.
* Fix bug rendering CPU in host vitals.
* Update the schema for host_users to allow for bulk inserts without locking, and allow for users without unique uid.
* When using dynamic vulnerability processing node, try to create the vulnerability.databases-path.
* Fix `fleetctl get host <hostname>` to properly output JSON when the command line flag is supplied i.e `fleetctl get host --json foobar`
* Fix a bug in which live queries would fail for deployments that use Redis Cluster.
* Fix a bug in which some new Fleet deployments don't include the default global agent options. Documentation for global and team agent options can be found [here](https://fleetdm.com/docs/using-fleet/configuration-files#agent-options).
* Improve how a host's `users` are stored in MySQL to prevent deadlocks. This information is available in the "Users" table on each host's **Host details** page and in the `GET /api/v1/fleet/hosts/{id}` API route.
* Add a new built in label "All Linux" to target all hosts that run any linux flavor.
* Allow finer grained configuration of the vulnerability processing capabilities.
* Fix performance issues when updating pack contents.
* Fix a build issue that caused external network access to panic in certain Linux distros (Ubuntu).
* Fix rendering of checkboxes in UI when modals appear.
* Orbit: sync critical file writes to disk.
* Add "-o" flag to fleetctl convert command to ensure consistent output rather than relying on shell redirection (this was causing issues with file encodings).
* Fix table column wrapping for manage queries page.
* Fix wrapping in Label pills.
* Side panels in UI have a fresher look, Teams/Roles UI greyed out conditionally.
* Improved sorting in UI tables.
* Improve detection of CentOS in label membership.
* When a connection from a live query websocket is closed, Fleet now timeouts the receive and handles the different cases correctly to not hold the connection to Redis.
* Read live query results from Redis in a thread safe manner.
* Allow observers and maintainers to refetch a host in a team they belong to.
* Add ability to simultaneously filter hosts by status (`online`, `offline`, `new`, `mia`) and by label on the **Hosts** page.
* Add ability to filter hosts by team in the Fleet UI, fleetctl CLI tool, and Fleet API. *Available for Fleet Basic customers*.
* Add ability to create a Team schedule in Fleet. The Schedule feature was released in Fleet 4.1.0. For more information on the new Schedule feature, check out the [Fleet 4.1.0 release blog post](https://blog.fleetdm.com/fleet-4-1-0-57dfa25e89c1). *Available for Fleet Basic customers*.
* Add Beta Vulnerable software feature which surfaces vulnerable software on the **Host details** page and the `GET /api/v1/fleet/hosts/{id}` API route. For information on how to configure the Vulnerable software feature and how exactly Fleet processes vulnerabilities, check out the [Vulnerability processing documentation](https://github.com/fleetdm/fleet/blob/main/docs/01-Using-Fleet/13-Vulnerability-Processing.md#vulnerability-processing).
* Add ability to see which logging destination is configured for Fleet in the Fleet UI. To see this information, head to the **Schedule** page and then select "Schedule a query." Configured logging destination information is also available in the `GET api/v1/fleet/config` API route.
* Improve the `fleetctl preview` experience by downloading Fleet's standard query library and loading the queries into the Fleet UI.
* Improve the user interface for the **Packs** page and **Queries** page in the Fleet UI.
* Add ability to modify scheduled queries in your Schedule in Fleet. The Schedule feature was released in Fleet 4.1.0. For more information on the new Schedule feature, check out the [Fleet 4.1.0 release blog post](https://blog.fleetdm.com/fleet-4-1-0-57dfa25e89c1).
* Add ability to disable the Users feature in Fleet by setting the new `enable_host_users` key to `true` in the `config` yaml, configuration file. For documentation on using configuration files in yaml syntax, check out the [Using yaml files in Fleet](https://github.com/fleetdm/fleet/tree/main/docs/01-Using-Fleet/configuration-files#using-yaml-files-in-fleet) documentation.
* Improve performance of the Software inventory feature. Software inventory is currently under a feature flag. To enable this feature flag, check out the [feature flag documentation](./docs/02-Deploying/03-Configuration.md#feature-flags).
* Improve performance of inserting `pack_stats` in the database. The `pack_stats` information is used to display "Frequency" and "Last run" information for a specific host's scheduled queries. You can find this information on the **Host details** page.
* Improve Fleet server logging so that it is more uniform.
* Fix a bug in which a user with the Observer role was unable to run a live query.
* Fix a bug that prevented the new **Home** page from being displayed in some Fleet instances.
* Fix a bug that prevented accurate sorting issues across multiple pages on the **Hosts** page.
The primary additions in Fleet 4.1.0 are the new Schedule and Activity feed features.
Schedule lets you add queries which are executed on your devices at regular intervals without having to understand or configure osquery query packs. For experienced Fleet and osquery users, the ability to create new, and modify existing, query packs is still available in the Fleet UI and fleetctl command-line tool. To reach the **Packs** page in the Fleet UI, head to **Schedule > Advanced**.
Activity feed adds the ability to observe when, and by whom, queries are changes, packs are created, live queries are run, and more. The Activity feed feature is located on the new Home page in the Fleet UI. Select the logo in the top right corner of the Fleet UI to navigate to the new **Home** page.
### New features breakdown
* Add ability to create teams and update their respective agent options and enroll secrets using the new `teams` yaml document and fleetctl. Available in Fleet Basic.
* Adds a new **Home** page to the Fleet UI. The **Home** page presents a breakdown of the enrolled hosts by operating system.
* Add a "Users" table on the **Host details** page. The `username` information displayed in the "Users" table, as well as the `uid`, `type`, and `groupname` are available in the Fleet REST API via the `/api/v1/fleet/hosts/{id}` API route.
* Add ability to create a user without an invitation. You can now create a new user by heading to **Settings > Users**, selecting "Create user," and then choosing the "Create user" option.
* Add ability to search and sort installed software items in the "Software" table on the **Host details** page.
* Add ability to delete a user from Fleet using a new `fleetctl user delete` command.
* Add ability to retrieve hosts' `status`, `display_text`, and `labels` using the `fleetctl get hosts` command.
* Add a new `user_roles` yaml document that allows users to manage user roles via fleetctl. Available in Fleet Basic.
* Change default ordering of the "Hosts" table in the Fleet UI to ascending order (A-Z).
* Improve performance of the Software inventory feature by reducing the amount of inserts and deletes are done in the database when updating each host's
software inventory.
* Remove YUM and APT sources from Software inventory.
* Fix an issue in which disabling SSO at the organization level would not disable SSO for all users.
* Fix an issue with data migrations in which enroll secrets are duplicated after the `name` column was removed from the `enroll_secrets` table.
* Fix an issue in which it was not possible to clear host settings by applying the `config` yaml document. This allows users to successfully remove the `additional_queries` property after adding it.
* Fix printing of failed record count in AWS Kinesis/Firehose logging plugins.
* Fix compatibility with GCP Memorystore Redis due to missing CLUSTER command.
The primary additions in Fleet 4.0.0 are the new Role-based access control (RBAC) and Teams features.
RBAC adds the ability to define a user's access to features in Fleet. This way, more individuals in an organization can utilize Fleet with appropriate levels of access.
* Check out the [permissions documentation](https://github.com/fleetdm/fleet/blob/2f42c281f98e39a72ab4a5125ecd26d303a16a6b/docs/1-Using-Fleet/9-Permissions.md) for a breakdown of the new user roles.
Teams adds the ability to separate hosts into exclusive groups. This way, users can easily act on consistent groups of hosts.
* Read more about the Teams feature in [the documentation here](https://github.com/fleetdm/fleet/blob/2f42c281f98e39a72ab4a5125ecd26d303a16a6b/docs/1-Using-Fleet/10-Teams.md).
### New features breakdown
* Add ability to define a user's access to features in Fleet by introducing the Admin, Maintainer, and Observer roles. Available in Fleet Core.
* Add ability to separate hosts into exclusive groups with the Teams feature. The Teams feature is available for Fleet Basic customers. Check out the list below for the new functionality included with Teams:
* Teams: Add ability to enroll hosts to one team using team specific enroll secrets.
* Teams: Add ability to manually transfer hosts to a different team in the Fleet UI.
* Teams: Add ability to apply unique agent options to each team. Note that "osquery options" have been renamed to "agent options."
* Teams: Add ability to grant users access to one or more teams. This allows you to define a user's access to specific groups of hosts in Fleet.
* Add ability to create an API-only user. API-only users cannot access the Fleet UI. These users can access all Fleet API endpoints and `fleetctl` features. Available in Fleet Core.
* Add Redis cluster support. Available in Fleet Core.
* Fix a bug that prevented the columns chosen for the "Hosts" table from persisting after logging out of Fleet.
### Upgrade plan
Fleet 4.0.0 is a major release and introduces several breaking changes and database migrations. The following sections call out changes to consider when upgrading to Fleet 4.0.0:
* The structure of Fleet's`.tar.gz` and`.zip` release archives have changed slightly. Deployments that use the binary artifacts may need to update scripts or tooling. The `fleetdm/fleet` Docker container maintains the same API.
* Use strictly `fleet` in Fleet's configuration, API routes, and environment variables. Users must update all usage of `kolide` in these items (deprecated since Fleet 3.8.0).
* Change your SAML SSO URI to use fleet instead of kolide. This is due to the changes to Fleet's API routes outlined in the section above.
* Change configuration option `server_tlsprofile` to `server_tls_compatibility`. This options previously had an inconsistent key name.
* Replace the use of the `api/v1/fleet/spec/osquery/options` with `api/v1/fleet/config`. In Fleet 4.0.0, "osquery options" are now called "agent options." The new agent options are moved to the Fleet application config spec file and the `api/v1/fleet/config` API endpoint.
* Enroll secrets no longer have "names" and are now either global or for a specific team. Hosts no longer store the “name” of the enroll secret that was used. Users that want to be able to segment hosts (for configuration, queries, etc.) based on the enrollment secret should use the Teams feature in Fleet Basic.
* JWT encoding is no longer used for session keys. Sessions now default to expiring in 4 hours of inactivity. `auth_jwt_key` and `auth_jwt_key_file` are no longer accepted as configuration.
* The `username` artifact has been removed in favor of the more recognizable `name` (Full name). As a result the `email` artifact is now used for uniqueness in Fleet. Upon upgrading to Fleet 4.0.0, existing users will have the `name` field populated with `username`. SAML users may need to update their username mapping to match user emails.
* As of Fleet 4.0.0, Fleet Device Management Inc. periodically collects anonymous information about your instance. Sending usage statistics is turned off by default for users upgrading from a previous version of Fleet. Read more about the exact information collected [here](https://github.com/fleetdm/fleet/blob/2f42c281f98e39a72ab4a5125ecd26d303a16a6b/docs/1-Using-Fleet/11-Usage-statistics.md).
The primary additions in Fleet 4.0.0 are the new Role-based access control (RBAC) and Teams features.
RBAC adds the ability to define a user's access to features in Fleet. This way, more individuals in an organization can utilize Fleet with appropriate levels of access.
* Check out the [permissions documentation](https://github.com/fleetdm/fleet/blob/5e40afa8ba28fc5cdee813dfca53b84ee0ee65cd/docs/1-Using-Fleet/8-Permissions.md) for a breakdown of the new user roles.
Teams adds the ability to separate hosts into exclusive groups. This way, users can easily act on consistent groups of hosts.
* Read more about the Teams feature in [the documentation here](https://github.com/fleetdm/fleet/blob/5e40afa8ba28fc5cdee813dfca53b84ee0ee65cd/docs/1-Using-Fleet/9-Teams.md).
### New features breakdown
* Add ability to define a user's access to features in Fleet by introducing the Admin, Maintainer, and Observer roles. Available in Fleet Core.
* Add ability to separate hosts into exclusive groups with the Teams feature. The Teams feature is available for Fleet Basic customers. Check out the list below for the new functionality included with Teams:
* Teams: Add ability to enroll hosts to one team using team specific enroll secrets.
* Teams: Add ability to manually transfer hosts to a different team in the Fleet UI.
* Teams: Add ability to apply unique agent options to each team. Note that "osquery options" have been renamed to "agent options."
* Teams: Add ability to grant users access to one or more teams. This allows you to define a user's access to specific groups of hosts in Fleet.
* Add ability to create an API-only user. API-only users cannot access the Fleet UI. These users can access all Fleet API endpoints and `fleetctl` features. Available in Fleet Core.
* Add Redis cluster support. Available in Fleet Core.
* Fix a bug that prevented the columns chosen for the "Hosts" table from persisting after logging out of Fleet.
### Upgrade plan
Fleet 4.0.0 is a major release and introduces several breaking changes and database migrations.
* Use strictly `fleet` in Fleet's configuration, API routes, and environment variables. Users must update all usage of `kolide` in these items (deprecated since Fleet 3.8.0).
* Change configuration option `server_tlsprofile` to `server_tls_compatability`. This option previously had an inconsistent key name.
* Replace the use of the `api/v1/fleet/spec/osquery/options` with `api/v1/fleet/config`. In Fleet 4.0.0, "osquery options" are now called "agent options." The new agent options are moved to the Fleet application config spec file and the `api/v1/fleet/config` API endpoint.
* Enroll secrets no longer have "names" and are now either global or for a specific team. Hosts no longer store the “name” of the enroll secret that was used. Users that want to be able to segment hosts (for configuration, queries, etc.) based on the enrollment secret should use the Teams feature in Fleet Basic.
*`auth_jwt_key` and `auth_jwt_key_file` are no longer accepted as configuration.
* JWT encoding is no longer used for session keys. Sessions now default to expiring in 4 hours of inactivity.
### Known issues
There are currently no known issues in this release. However, we recommend only upgrading to Fleet 4.0.0-rc2 for testing purposes. Please file a GitHub issue for any issues discovered when testing Fleet 4.0.0!
RBAC adds the ability to define a user's access to information and features in Fleet. This way, more individuals in an organization can utilize Fleet with appropriate levels of access. Check out the [permissions documentation](https://github.com/fleetdm/fleet/blob/main/docs/01-Using-Fleet/09-Permissions.md) for a breakdown of the new user roles and their respective capabilities.
Teams adds the ability to separate hosts into exclusive groups. This way, users can easily observe and apply operations to consistent groups of hosts. Read more about the Teams feature in [the documentation here](https://github.com/fleetdm/fleet/blob/main/docs/01-Using-Fleet/10-Teams.md).
There are several known issues that will be fixed for the stable release of Fleet 4.0.0. Therefore, we recommend only upgrading to Fleet 4.0.0 RC1 for testing purposes. Please file a GitHub issue for any issues discovered when testing Fleet 4.0.0!
### New features breakdown
* Add ability to define a user's access to information and features in Fleet by introducing the Admin, Maintainer, and Observer roles.
* Add ability to separate hosts into exclusive groups with the Teams feature. The Teams feature is available for Fleet Basic customers. Check out the list below for the new functionality included with Teams:
* Add ability to enroll hosts to one team using team specific enroll secrets.
* Add ability to manually transfer hosts to a different team in the Fleet UI.
* Add ability to apply unique agent options to each team. Note that "osquery options" have been renamed to "agent options."
* Add ability to grant users access to one or more teams. This allows you to define a user's access to specific groups of hosts in Fleet.
### Upgrade plan
Fleet 4.0.0 is a major release and introduces several breaking changes and database migrations.
* Use strictly `fleet` in Fleet's configuration, API routes, and environment variables. This means that you must update all usage of `kolide` in these items. The backwards compatibility introduced in Fleet 3.8.0 is no longer valid in Fleet 4.0.0.
* Change configuration option `server_tlsprofile` to `server_tls_compatability`. This options previously had an inconsistent key name.
* Replace the use of the `api/v1/fleet/spec/osquery/options` with `api/v1/fleet/config`. In Fleet 4.0.0, "osquery options" are now called "agent options." The new agent options are moved to the Fleet application config spec file and the `api/v1/fleet/config` API endpoint.
* Enroll secrets no longer have "names" and are now either global or for a specific team. Hosts no longer store the “name” of the enroll secret that was used. Users that want to be able to segment hosts (for configuration, queries, etc.) based on the enrollment secret should use the Teams feature in Fleet Basic.
*`auth_jwt_key` and `auth_jwt_key_file` are no longer accepted as configuration.
* JWT encoding is no longer used for session keys. Sessions now default to expiring in 4 hours of inactivity.
* Improve performance of the `additional_queries` feature by moving `additional` query results into a separate table in the MySQL database. Please note that the `/api/v1/fleet/hosts` API endpoint now return only the requested `additional` columns. See documentation on the changes to the hosts API endpoint [here](https://github.com/fleetdm/fleet/blob/06b2e564e657492bfbc647e07eb49fd4efca5a03/docs/1-Using-Fleet/3-REST-API.md#list-hosts).
* Fix a bug in which running a live query in the Fleet UI would return no results and the query would seem "hung" on a small number of devices.
* Improve viewing live query errors in the Fleet UI by including the “Errors” table in the full screen view.
* Improve `fleetctl preview` experience by adding the `fleetctl preview reset` and `fleetctl preview stop` commands to reset and stop simulated hosts running in Docker.
* Add several improvements to the Fleet UI including additional contrast on checkboxes and dropdown pills.
* Add scheduled queries to the _Host details_ page. Surface the "Name", "Description", "Frequency", and "Last run" information for each query in a pack that apply to a specific host.
* Improve the freshness of host vitals by adding the ability to "refetch" the data on the _Host details_ page.
* Add ability to copy log fields into Google Cloud Pub/Sub attributes. This allows users to use these values for subscription filters.
* Add ability to duplicate live query results in Redis. When the `redis_duplicate_results` configuration option is set to `true`, all live query results will be copied to an additional Redis Pub/Sub channel named LQDuplicate.
* Add ability to controls the server-side HTTP keepalive property. Turning off keepalives has helped reduce outstanding TCP connections in some deployments.
* Fix an issue on the _Packs_ page in which Fleet would incorrectly handle the configured `server_url_prefix`.
* Improve Fleet performance by batch updating host seen time instead of updating synchronously. This improvement reduces MySQL CPU usage by ~33% with 4,000 simulated hosts and MySQL running in Docker.
* Add support for software inventory, introducing a list of installed software items on each host's respective _Host details_ page. This feature is flagged off by default (for now). Check out [the feature flag documentation for instructions on how to turn this feature on](./docs/02-Deploying/03-Configuration.md#software-inventory).
* Add Windows support for `fleetctl` agent autoupdates. The `fleetctl updates` command provides the ability to self-manage an agent update server. Available for Fleet Basic customers.
* Make runningcommon queries more convenient by adding the ability to select a saved query directly from a host's respective _Host details_ page.
* Fix an issue on the _Query_ page in which Fleet would override the CMD + L browser hotkey.
* Fix an issue in which a host would display an unreasonable time in the "Last fetched" column.
* Fix a frontend bug that prevented the "Pack" page and "Edit pack" page from rendering in the Fleet UI. This issue occurred when the `platform` key, in the requested pack's configuration, was set to any value other than `darwin`, `linux`, `windows`, or `all`.
* Add `fleetctl` agent auto-updates beta which introduces the ability to self-manage an agent update server. Available for Fleet Basic customers.
* Add option for Identity Provider-Initiated (IdP-initiated) Single Sign-On (SSO).
* Improve logging. All errors are logged regardless of log level, some non-errors are logged regardless of log level (agent enrollments, runs of live queries etc.), and all other non-errors are logged on debug level.
* Improve login resilience by adding rate-limiting to login and password reset attempts and preventing user enumeration.
* Add Fleet version and Go version in the My Account page of the Fleet UI.
* Improvements to `fleetctl preview` that ensure the latest version of Fleet is fired up on every run. In addition, the Fleet UI is now accessible without having to click through browser security warning messages.
* Add configurable host identifier to help with duplicate host enrollment scenarios. By default, Fleet's behavior does not change (it uses the identifier configured in osquery's `--host_identifier` flag), but for users with overlapping host UUIDs changing `--osquery_host_identifier` to `instance` may be helpful.
* Make cool-down period for host enrollment configurable to control load on the database in scenarios in which hosts are using the same identifier. By default, the cooldown is off, reverting to the behavior of Fleet <=3.4.0. The cooldown can be enabled with `--osquery_enroll_cooldown`.
* Refresh the Fleet UI with a new layout and horizontal navigation bar.
* Trim down the size of Fleet binaries.
* Improve handling of config_refresh values from osquery clients.
* Fix an issue with IP addresses and host additional info dropping.
* Add search, sort, and column selection in the hosts dashboard.
* Add AWS Lambda logging plugin.
* Improve messaging about number of hosts responding to live query.
* Update host listing API endpoints to support search.
* Fixes to the `fleetctl preview` experience.
* Fix `denylist` parameter in scheduled queries.
* Fix an issue with errors table rendering on live query page.
* Deprecate `KOLIDE_` environment variable prefixes in favor of `FLEET_` prefixes. Deprecated prefixes continue to work and the Fleet server will log warnings if the deprecated variable names are used.
* Deprecate `/api/v1/kolide` routes in favor of `/api/v1/fleet`. Deprecated routes continue to work and the Fleet server will log warnings if the deprecated routes are used.
* Change the default `--server_tls_compatibility` to `intermediate`. The new settings caused TLS connectivity issues for users in some environments. This new default is a more appropriate balance of security and compatibility, as recommended by Mozilla.
* **Security**: Fixed a vulnerability in which a malicious actor with a valid node key can send a badly formatted request that causes the Fleet server to exit, resulting in denial of service. See https://github.com/fleetdm/fleet/security/advisories/GHSA-xwh8-9p3f-3x45 and the linked content within that advisory.
* Add new Host details page which includes a rich view of a specific host’s attributes.
* Reveal live query errors in the Fleet UI and `fleetctl` to help target and diagnose hosts that fail.
* Add Helm chart to make it easier for users to deploy to Kubernetes.
* Add support for `denylist` parameter in scheduled queries.
* Add debug flag to `fleetctl` that enables logging of HTTP requests and responses to stderr.
* Improvements to the `fleetctl preview` experience that include adding containerized osquery agents, displaying login information, creating a default directory, and checking for Docker daemon status.
* Add improved error handling in host enrollment to make debugging issues with the enrollment process easier.
* Upgrade TLS compatibility settings to match Mozilla.
* Add comments in generated flagfile to add clarity to different features being configured.
* Fix a bug in Fleet UI that allowed user to edit a scheduled query after it had been deleted from a pack.
* **Security**: Introduce XML validation library to mitigate Go stdlib XML parsing vulnerability effecting SSO login. See https://github.com/fleetdm/fleet/security/advisories/GHSA-w3wf-cfx3-6gcx and the linked content within that advisory.
Follow up: Rotate `--auth_jwt_key` to invalidate existing sessions. Audit for suspicious activity in the Fleet server.
* **Security**: Prevent new queries from using the SQLite `ATTACH` command. This is a mitigation for the osquery vulnerability https://github.com/osquery/osquery/security/advisories/GHSA-4g56-2482-x7q8.
Follow up: Audit existing saved queries and logs of live query executions for possible malicious use of `ATTACH`. Upgrade osquery to 4.6.0 to prevent `ATTACH` queries from executing.
* Update icons and fix hosts dashboard for wide screen sizes.
* Backend performance overhaul. The Fleet server can now handle hundreds of thousands of connected hosts.
* Pagination implemented in the web UI. This makes the UI usable for any host count supported by the backend.
* Add capability to collect "additional" information from hosts. Additional queries can be set to be updated along with the host detail queries. This additional information is returned by the API.
* Removed extraneous network interface information to optimize server performance. Users that require this information can use the additional queries functionality to retrieve it.
* Add "manual" labels implementation. Static labels can be set by providing a list of hostnames with `fleetctl`.
* Add JSON output for `fleetctl get` commands.
* Add `fleetctl get host` to retrieve details for a single host.
* Update table schema for osquery 4.4.0.
* Add support for multiple enroll secrets.
* Logging verbosity reduced by default. Logs are now much less noisy.
* Fix import of github.com/kolide/fleet Go packages for consumers outside of this repository.
* Add `--server_url_prefix` flag to configure a URL prefix to prepend on all Fleet URLs. This can be useful to run fleet behind a reverse-proxy on a hostname shared with other services.
* Add option to automatically expire hosts that have not checked in within a certain number of days. Configure this in the "Advanced Options" of "App Settings" in the browser UI.
* Add ability to search for hosts by UUID when targeting queries.
* Allow SAML IdP name to be as short as 4 characters.
* Security: Upgrade Go to 1.12.8 to fix CVE-2019-9512, CVE-2019-9514, and CVE-2019-14809.
* Add capability to export packs, labels, and queries as yaml in `fleetctl get` with the `--yaml` flag. Include queries with a pack using `--with-queries`.
* Modify email templates to load image assets from Github CDN rather than Fleet server (fixes broken images in emails when Fleet server is not accessible from email clients).
* Add warning in query UI when Redis is not functioning.
* Fix minor bugs in frontend handling of scheduled queries.
* Add GCP PubSub logging plugin. Thanks to Michael Samuel for adding this capability.
* Improved escaping for target search in live query interface. It is now easier to target hosts with + and - characters in the name.
* Server and browser performance improvements by reduced loading of hosts in frontend. Host status will only update on page load when over 100 hosts are present.
* Utilize details sent by osquery in enrollment request to more quickly display details of new hosts. Also fixes a bug in which hosts could not complete enrollment if certain platform-dependent options were used.
* Fix a bug in which the default query runs after targets are edited.
* Add capability to log osquery status and results to AWS Firehose. Note that this deprecated some existing logging configuration (`--osquery_status_log_file` and `--osquery_result_log_file`). Existing configurations will continue to work, but will be removed at some point.
* Automatically clean up "incoming hosts" that do not complete enrollment.
* Fix bug with SSO requests that caused issues with some IdPs.
* Hide built-in platform labels that have no hosts.
* Fix references to Fleet documentation in emails.
* Minor improvements to UI in places where editing objects is disabled.
* Fix a bug where duplicate queries were being created in the same pack but only one was ever delivered to osquery. A migration was added to delete duplicate queries in packs created by the UI.
* It is possible to schedule the same query with different options in one pack, but only via the CLI.
* If you thought you were relying on this functionality via the UI, note that duplicate queries will be deleted when you run migrations as apart of a cleanup fix. Please check your configurations and make sure to create any double-scheduled queries via the CLI moving forward.
* Fix a bug in which packs created in UI could not be loaded by fleetctl.
* Fix a bug where deleting a query would not delete it from the packs that the query was scheduled in.
The primary new addition in Fleet 2 is the new `fleetctl` CLI and file-format, which dramatically increases the flexibility and control that administrators have over their osquery deployment. The CLI and the file format are documented [in the Fleet documentation](https://github.com/fleetdm/fleet/blob/main/docs/01-Using-Fleet/02-fleetctl-CLI.md).
* New `fleetctl` CLI for managing your entire osquery workflow via CLI, API, and source controlled files!
* You can use `fleetctl` to manage osquery packs, queries, labels, and configuration.
* In addition to the CLI, Fleet 2.0.0 introduces a new file format for articulating labels, queries, packs, options, etc. This format is designed for composability, enabling more effective sharing and re-use of intelligence.
```yaml
apiVersion: v1
kind: query
spec:
name: pending_updates
query: >
select value
from plist
where
path = "/Library/Preferences/ManagedInstalls.plist" and
key = "PendingUpdateCount" and
value > "0";
```
* Run live osquery queries against arbitrary subsets of your infrastructure via the `fleetctl query` command.
* Use `fleetctl setup`, `fleetctl login`, and `fleetctl logout` to manage the authentication life-cycle via the CLI.
* Use `fleetctl get`, `fleetctl apply`, and `fleetctl delete` to manage the state of your Fleet data.
* Manage any osquery option you want and set platform-specific overrides with the `fleetctl` CLI and file format.
* Managing osquery options via the UI has been removed in favor of the more flexible solution provided by the CLI. If you have customized your osquery options with Fleet, there is [a database migration](./server/datastore/mysql/migrations/data/20171212182458_MigrateOsqueryOptions.go) which will port your existing data into the new format when you run `fleet prepare db`. To download your osquery options after migrating your database, run `fleetctl get options > options.yaml`. Further modifications to your options should occur in this file and it should be applied with `fleetctl apply -f ./options.yaml`.
* Added feature that allows users to import existing Osquery configuration files using the [configimporter](https://github.com/kolide/configimporter) utility.
The Kolide server now tracks the `distributed_interval` and `config_tls_refresh` values for each individual host (these can be different if they are set via flagfile and not through Kolide), to ensure that online status is represented as accurately as possible.
* Log rotation is no longer the default setting for Osquery status and results logs. To enable log rotation use the `--osquery_enable_log_rotation` flag.
* Add a debug endpoint for collecting performance statistics and profiles.
When `kolide serve --debug` is used, additional handlers will be started to provide access to profiling tools. These endpoints are authenticated with a randomly generated token that is printed to the Kolide logs at startup. These profiling tools are not intended for general use, but they may be useful when providing performance-related bug reports to the Kolide developers.
osquery 2.3.2 incorrectly reports an empty value for `platform` on CentOS6 hosts. We added a workaround to properly detect platform in Kolide, and also [submitted a fix](https://github.com/facebook/osquery/pull/3071) to upstream osquery.
Previously this item was visible to non-admin users and if selected, a blank options page would be displayed since server side authorization constraints prevent regular users from viewing or changing options.
In an effort to provide a more resilient web server, timeouts are more strictly enforced by the Kolide HTTP server (regardless of whether or not you're using the built-in TLS termination).
For customers using Kolide's built-in TLS server (if the `server.tls` configuration is `true`), the server was hardened to only accept modern cipher suites as recommended by [Mozilla](https://wiki.mozilla.org/Security/Server_Side_TLS#Modern_compatibility).
* Improve the mechanism used to calculate whether or not hosts are online.
Previously, hosts were categorized as "online" if they had been seen within the past 30 minutes. To make the "online" status more representative of reality, hosts are marked "online" if the Kolide server has heard from them within two times the lowest polling interval as described by the Kolide-managed osquery configuration. For example, if you've configured osqueryd to check-in with Kolide every 10 seconds, only hosts that Kolide has heard from within the last 20 seconds will be marked "online".
Customers running Kolide behind a web balancer lacking support for websockets were unable to use the distributed query feature. Also, in certain circumstances, Safari users with a self-signed cert for Kolide would receive an error. This release add a fallback mechanism from websockets using SockJS for improved compatibility.
Previously Kolide was determining platform based on the OS of the system osquery was built on instead of the OS it was running on. Please note: Offline hosts may continue to report an erroneous platform until they check-in with Kolide.
* Support MySQL client certificate authentication. More details can be found in the [Configuring the Fleet binary docs](./docs/infrastructure/configuring-the-fleet-binary.md).
