#11939
- This PR fixes typos in three CIS Windows queries (the queries were
failing with `invalid SQL syntax`).
- Also adds tooling to perform similar testing that we ran for macOS
(using `fleetd_tables` as an extension).
changelog for the version bump can be found here:
https://github.com/macadmins/osquery-extension/releases/tag/v0.0.15
related to #13158 as this fixes a bug in the extension causing `profiles
show --type enrollment` to be run almost every time we queried something
from the `mdm` table.
I couldn't find any new tables. Other than that, some dependencies were
updated on their repo as well, most notably `osquery/osquery-go`
relates to #12986
This is a fix for a case where the `os_versions` endpoint was returning
a 404 when searching for `no teams`.
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Manual QA for all new/changed functionality
related to #12483, we have found out that in distributed scenarios, the
URL of the Puppet server used for the request is appended to the
identifier, and it can be different between `/preassign` and `/match`
calls.
to account for this, we're only grabbing the first 36 characters of the
identifier.
Prior to 4.35.0, some rows in the scheduled_query table might have a
`NULL` value due to a race condition with database replicas and the way
`ds.EnsureGlobalPack` and `ApplyPackSpecs` work together.
This is no longer the case, but some databases are left in weird states,
which were not accounted by this migration.
Chaning the migration in-place because that's the approach we took in
previous migrations with similar problems.
relates to #12907
implement adding an enroll secret when creating a new team with
`fleetctl apply` if none is provided.
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Added/updated tests
- [x] Manual QA for all new/changed functionality
This relates to #12600
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
- [X] Manual QA for all new/changed functionality
## Addresses #12999
<img width="1282" alt="Screenshot 2023-07-27 at 11 59 01 AM"
src="https://github.com/fleetdm/fleet/assets/61553566/b60d3b41-3d7b-4550-ba7c-8615bae085a6">
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Jacob Shandling <jacob@fleetdm.com>
## Addresses #12646
### See issue for list of completed work
### Also see PR #12713 **notes for review** on that PR for help manually
testing this work in lieu of the completed API.
- [x] Changes file added for user-visible changes in `changes/`
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Jacob Shandling <jacob@fleetdm.com>
## Addresses #12636
### See issue for list work done
### Notes for review:
- Because other work is based on this branch, TODOs / fixes are noted
here until the team comes to a strategy for merging all of the work:
- Add missing space in the Performance impact column "Undetermined"
tooltip text
- I'm having trouble confirming that the inherited queries table is
working right with the mock hard-coded data, though I did see it working
correctly previously. There's an issue with the page reverting to "All
teams" when trying to show the inherited table, though it does show the
table before re-rendering.
- This work is organized clearly by commit, so that might be a
manageable way to go through this code.
- Since the updated API for this work is not yet complete, this work can
be manually tested by either:
- Using mock API infrastructure, or
- in `ManageQueriesPage.tsx`, comment out the two `useQuery` calls and
add appropriate mock data. You can then modify any fields of interest to
test their related UI functionality. For example, lines 119 -242 might
read:
```
// const {
// data: curTeamEnhancedQueries,
// error: curTeamQueriesError,
// isFetching: isFetchingCurTeamQueries,
// refetch: refetchCurTeamQueries,
// } = useQuery<IListQueriesResponse, Error, IEnhancedQuery[]>(
// [{ scope: "queries", teamId: teamIdForApi }],
// () => queriesAPI.loadAll(teamIdForApi),
// {
// refetchOnWindowFocus: false,
// enabled: isRouteOk,
// select: (data) => data.queries.map(enhanceQuery),
// }
// );
// // If a team is selected, fetch inherited global queries as well
// const {
// data: globalEnhancedQueries,
// error: globalQueriesError,
// isFetching: isFetchingGlobalQueries,
// refetch: refetchGlobalQueries,
// } = useQuery<IListQueriesResponse, Error, IEnhancedQuery[]>(
// [{ scope: "queries", teamId: -1 }],
// () => queriesAPI.loadAll(),
// {
// refetchOnWindowFocus: false,
// enabled: isRouteOk && isAnyTeamSelected,
// select: (data) => data.queries.map(enhanceQuery),
// }
// );
const [
curTeamEnhancedQueries,
curTeamQueriesError,
isFetchingCurTeamQueries,
refetchCurTeamQueries,
] = useMemo(() => {
return [
[
{
created_at: "2023-06-08T15:31:35Z",
updated_at: "2023-06-08T15:31:35Z",
id: 2,
name: "test",
description: "",
query: "SELECT * FROM osquery_info;",
team_id: 43,
platform: "darwin",
min_osquery_version: "",
automations_enabled: true,
logging: "snapshot",
saved: true,
// interval: 300,
interval: 0,
observer_can_run: false,
author_id: 1,
author_name: "Jacob",
author_email: "jacob@fleetdm.com",
packs: [],
stats: {
// system_time_p50: 1,
// system_time_p95: null,
// user_time_p50: 1,
// user_time_p95: null,
// total_executions: 1,
},
performance: "Undetermined",
platforms: ["darwin"],
},
] as IEnhancedQuery[],
undefined,
false,
() => {
console.log("got the new queries");
},
];
}, []);
const [
globalEnhancedQueries,
globalQueriesError,
isFetchingGlobalQueries,
refetchGlobalQueries,
] = useMemo(() => {
return [
[
{
created_at: "2023-06-08T15:31:35Z",
updated_at: "2023-06-08T15:31:35Z",
id: 200,
name: "test",
description: "",
query: "SELECT * FROM osquery_info;",
team_id: null,
platform: "darwin",
min_osquery_version: "",
automations_enabled: true,
logging: "snapshot",
saved: true,
// interval: 300,
interval: 0,
observer_can_run: false,
author_id: 1,
author_name: "Jacob",
author_email: "jacob@fleetdm.com",
packs: [],
stats: {
// system_time_p50: 1,
// system_time_p95: null,
// user_time_p50: 1,
// user_time_p95: null,
// total_executions: 1,
},
performance: "Undetermined",
platforms: ["darwin"],
},
] as IEnhancedQuery[],
undefined,
false,
() => {
console.log("got the new inherited queries");
},
];
}, []);
```
- [x] Changes file added for user-visible changes in `changes/`
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Jacob Shandling <jacob@fleetdm.com>
Related to #12608, this automatically sets the
`DeferForceAtUserLoginMaxBypassAttempts` property to `1` on the
FileVault profile that's generated by Fleet.
This changeset also includes a migration to modify old FileVault
profiles that already exist in the database, and by virtue of that a
`InstallProfile` command will be issued to hosts that already have FV
enabled. During testing we found:
1. This doesn't affect users with FV already installed, they silently
get the profile updated without any changes.
2. Since the profile needs to be re-delivered, it'll go through the full
"pending" -> "verifying" -> "verified" cycle.
This is related #12701
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
- [X] Manual QA for all new/changed functionality
This relates to #12263
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [X] Added/updated tests
---------
Co-authored-by: Roberto Dip <me@roperzh.com>
#10292, #12554
When scanning tens of thousands of files for permissions, using the
`find` command exposed as a fleetd table is more performant than trying
to use the `file` table. This change caused the watchdog to *stop*
killing osquery because of exceeding memory or CPU limit.
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [X] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
relates to #12289
Implements enable and disable Windows mdm activity UI
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
- [x] Manual QA for all new/changed functionality
## Addresses #11355, pt.1 (pt. 2 already fixed)
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Jacob Shandling <jacob@fleetdm.com>
#10292
The query was processing *every* file under `/Applications/`, which
makes it super expensive both in CPU usage and Memory footprint. This
query was the main culprit of triggering worker process kills by the
watchdog.
On some runs it triggered CPU usage alerts:
```
7716:W0623 15:38:05.402959 221732864 watcher.cpp:415] osqueryd worker (72976) stopping:
Maximum sustainable CPU utilization limit 1200ms exceeded for 12 seconds
```
And on other runs it triggered memory usage alerts:
```
4431 W0626 07:28:50.868021 147312640 watcher.cpp:424] osqueryd worker (21453) stopping:
Memory limits exceeded: 214020096 bytes (limit is 200MB)
```
For the above logs I used a custom osqueryd branch to be able to print
more information: https://github.com/osquery/osquery/pull/8070
The metrics for the old query were CPU usage: ~4521 ms
```
435:level=warn ts=2023-06-26T09:58:29.665712Z query=fleet_policy_query_1233 queryTime=4521 memory=12226560 msg="distributed query performance is excessive" hostID=308 platform=darwin
```
With the new query, CPU usage: ~210 ms.
```
23893:level=debug ts=2023-06-26T18:06:08.242456Z query=fleet_policy_query_1233 queryTime=210 msg=stats memory=0 hostID=308 platform=darwin
```
Basically a ~20x improvement.
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- ~[ ] Added/updated tests~
- [X] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
## Addresses #11037
### Implement the `privacy_preferences` table for the Fleetd Chrome
extension. Columns correspond to the available properties of
[`chrome.privacy`](https://developer.chrome.com/docs/extensions/reference/privacy/).
Chrome on mac:
<img width="816" alt="Screenshot 2023-06-23 at 11 55 21 AM"
src="https://github.com/fleetdm/fleet/assets/61553566/a4700749-6325-442e-acf2-c14b1c9adf8f">
Chromebook with enterprise access (actual use case):
* Chromebook w/o enterprise access: as you can see, sometimes certain
APIs are not available - this error occurs because the expected API
object that would have a `get` method is actually `undefined` TODO – How
to handle this case given that we want to let errors bubble up to the
level at which Fleet can catch them? Maybe it would be nice to catch
such errors and send them up to the Fleet layer, and still allow the
loop to continue to populate the columns whose APIs _are_ available.
_Decision: catch API errors here to preserve functionality of the
remaining columns_
- [x] Changes file
- [x] Manual QA
---------
Co-authored-by: Jacob Shandling <jacob@fleetdm.com>
This PR requires the Windows MDM configuration changes - This will be
updated next week
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- [x] Documented any permissions changes
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
relates to #11932
This improves the UI error messaging for AMB 400 errors
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
Issue reported by @jarodreyes.
The user was not informed that `fleetctl login` was using the PASSWORD
environment variable:
`main`:
```sh
export PASSWORD=wrong
fleetctl login
Log in using the standard Fleet credentials.
Email: a@b.c
Error: Login failed: login received status 401 Authentication failed: Authentication failed
```
And with the changes on this PR:
```
export PASSWORD=wrong
fleetctl login
Log in using the standard Fleet credentials.
Email: a@b.c
Using value of environment variable $PASSWORD as password.
Error: Login failed: login received status 401 Authentication failed: Authentication failed
```
## Addresses
[confidential/2940](https://github.com/fleetdm/confidential/issues/2940)
Patched a potential security issue in UI
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Jacob Shandling <jacob@fleetdm.com>
#11266
PS: I first attempted a serialization trick by introducing a new
`appConfigResponse` and implementing `json.Marshal` to exclude these
fields but it was too hacky and hard to maintain moving forward, so I'm
bitting the bullet now. Happy to hear other ideas.
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
## Addresses #11828
- [x] Add ChromeOS platform filter
- [x] Increase dropdown widths to 180px for screen >1100px of:
- [x] platform dropdown
- [x] Labels dropdown (for consistency)
- [x] Add new null empty cell value “Not supported” for Chromebooks
- [x] Apply to the following columns:
- [x] Disk space available
- [x] MDM status
- [x] MDM server URL
- [x] Last restarted
- [x] Fix a misaligned icon in the labels dropdown, other small fixes
<img width="1184" alt="Screenshot 2023-06-06 at 7 09 26 PM"
src="https://github.com/fleetdm/fleet/assets/61553566/ee6fb72c-f66e-44a4-a321-8eaefff3b70b">
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Jacob Shandling <jacob@fleetdm.com>
for #11257, h/t to @mna for the idea of resetting `token_update_tally`.
this is to cover scenarios where a host might be re-enrolling (eg: the
device has been wiped) but we don't know about it.
since `TokenUpdate` might be called multiple times during the lifecycle
of an MDM enrollment, we add a check on the value of
`nano_enrollments.token_update_tally`. For the scenarios described
above, the tally is still `> 0` even thought the host is enrolling for
the first time.
to mitigate this, we reset its value to 0 when we receive an
`Authenticate` message (which only happens only per enrollment)
I set the value to `0` because it's incremented to `current_value+1` by
nanomdm before calling our handler.
## Addresses #11825
- [x] Add ChromeOS to Dashboard page: <img width="1365" alt="Screenshot
2023-06-02 at 4 01 12 AM"
src="https://github.com/fleetdm/fleet/assets/61553566/e846c4b6-5fcb-4847-af05-67b2237ada39">
- [x] Add to platforms dropdown, confirm order of platform options, add
route
- [x] Hosts summary card
- [x] Add responsiveness for <980px <img width="952" alt="Screenshot
2023-06-02 at 4 02 44 AM"
src="https://github.com/fleetdm/fleet/assets/61553566/93662957-c590-40e0-876d-6ce4adabad2b">
- [x] TODO: Confirm label number of chrome hosts label - ask Juan on
[this issue](https://github.com/fleetdm/fleet/issues/11829) - needed to
call an API to get this id
- [x] Missing hosts card (didn’t need any changes)
- [x] Low disk space hosts card (Not supported)
- [x] Operating systems card
**Note for reviewers:** There is an API call happening from the
HostsSummary component to get the id for the ChromeOS label needed for
the URL to the filtered manage hosts page. This feature working properly
depends on the response from that endpoint, which is WIP. UPDATE 6/5 -
the endpoint is now working and being called correctly, though the id
being returned is WIP (backend). No need to replace anything to test.
## Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Jacob Shandling <jacob@fleetdm.com>
## Addresses #11856
Improve performance of the rendering of live query results by:
- rendering the table on a set interval instead of with each new result
- preventing redundant rerenders of various sorts
Partial run, with memory leak:
<img width="2552" alt="partial run with memory leak, after smaller
optimizations, before debouncing queryResults"
src="https://github.com/fleetdm/fleet/assets/61553566/5288bffb-6940-43da-9083-59adb4a25916">
Full run after debounce, no memory leak (10x improvement of max JS heap
size):
<img width="2559" alt="full run after debounce, no memory leak"
src="https://github.com/fleetdm/fleet/assets/61553566/be056610-e7a5-4289-a433-1070cf016e83">
**NOTE** - there are further optimizations to try on this page, and the
debounce interval can potentially be shortened to improve UX. In
experimenting with that, it's not immediately clear what a good balance
of UX / performance is. Since the customer seems keen to solve this, I
think we should merge as-is and send them a demo build to confirm this
fixes their problem, then iterate once they've confirmed it does.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Lucas Rodriguez <lucas@fleetdm.com>
Co-authored-by: Jacob Shandling <jacob@fleetdm.com>
Co-authored-by: Sarah Gillespie <sarah@fleetdm.com>
for #10605, this modifies the cron used to ping the list/sync devices
API from ABM to account for the "deleted" and "modified" operation
types.
We know that:
1. Sometimes, Apple sends a "modified" operation type when a device's
MDM server is reassigned in ABM, up until now, we were ignoring these
devices.
2. Devices that are no longer assigned to Fleet in ABM can't be
migrated.
relates to #11238
This implements the Verified status for the profile statute on the macOS
settings pages and the Host Details and My Device pages.
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
- [x] Manual QA for all new/changed functionality
- Add section for viewing recent MDM commands using fleetctl get mdm-commands
- Add example output for fleetctl get mdm-commands and fleetctl get mdm-command-results
This could help future users to detect this issue: #10957
It also adds an error log in Fleet that prints the actual error.
The error is displayed if I kill Redis during a live session or if I set
`client-output-buffer-limit` to something real low like `CONFIG SET
"client-output-buffer-limit" "pubsub 100kb 50kb 60"`:
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- [X] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- ~[ ] Added/updated tests~
- [X] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
