fleet/server/service
Lucas Manuel Rodriguez 6415f062c6
Reduce size of DistributedQueryResult to improve live query performance (#11882)
This was found while working on #10957.

When running a live query, a lot of unused host data is stored in Redis
and sent on every live query result message via websockets. The frontend
and fleetctl just need `id`, `hostname` and `display_name`. (This
becomes worse every time we add new fields to the `Host` struct.)

Sample of one websocket message result when running `SELECT * from
osquery_info;`:

size in `main`: 2234 bytes
```
a["{\"type\":\"result\",\"data\":{\"distributed_query_execution_id\":57,\"host\":
{\"created_at\":\"2023-05-22T12:14:11Z\",\"updated_at\":\"2023-05-23T12:31:51Z\",
\"software_updated_at\":\"0001-01-01T00:00:00Z\",\"id\":106,\"detail_updated_at\":\"2023-05-23T11:50:04Z\",
\"label_updated_at\":\"2023-05-23T11:50:04Z\",\"policy_updated_at\":\"1970-01-02T00:00:00Z\",
\"last_enrolled_at\":\"2023-05-22T12:14:12Z\",
\"seen_time\":\"2023-05-23T09:52:23.876311-03:00\",\"refetch_requested\":false,
\"hostname\":\"lucass-macbook-pro.local\",\"uuid\":\"BD4DFA10-E334-41D9-8136-D2163A8FE588\",\"platform\":\"darwin\",\"osquery_version\":\"5.8.2\",\"os_version\":\"macOS 13.3.1\",\"build\":\"22E261\",\"platform_like\":\"darwin\",\"code_name\":\"\",
\"uptime\":91125000000000,\"memory\":34359738368,\"cpu_type\":\"x86_64h\",\"cpu_subtype\":\"Intel x86-64h Haswell\",\"cpu_brand\":\"Intel(R) Core(TM) i7-1068NG7 CPU @ 2.30GHz\",\"cpu_physical_cores\":4,\"cpu_logical_cores\":8,\"hardware_vendor\":\"Apple Inc.\",\"hardware_model\":\"MacBookPro16,2\",\"hardware_version\":\"1.0\",
\"hardware_serial\":\"0DPQR4HMD1FZ\",
\"computer_name\":\"Lucas’s MacBook Pro\",\"public_ip\":\"\",
\"primary_ip\":\"192.168.0.230\",\"primary_mac\":\"68:2f:67:8e:b6:1f\",
\"distributed_interval\":1,\"config_tls_refresh\":60,\"logger_tls_period\":10,\"team_id\":null,
\"pack_stats\":null,\"team_name\":null,
\"gigs_disk_space_available\":386.23,\"percent_disk_space_available\":40,
\"issues\":{\"total_issues_count\":0,\"failing_policies_count\":0},
\"mdm\":{\"enrollment_status\":null,\"server_url\":null,\"name\":\"\",\"encryption_key_available\":false},
\"status\":\"online\",\"display_text\":\"lucass-macbook-pro.local\",\"display_name\":\"Lucas’s MacBook Pro\"},
\"rows\":[{\"build_distro\":\"10.14\",\"build_platform\":\"darwin\",
\"config_hash\":\"b7ee9363a7c686e76e99ffb122e9c5241a791e69\",\"config_valid\":\"1\",
\"extensions\":\"active\",\"host_display_name\":\"Lucas’s MacBook Pro\",
\"host_hostname\":\"lucass-macbook-pro.local\",\"instance_id\":\"cde5de81-344b-4c76-b1c5-dae964fdd4f2\",\"pid\":\"8370\",\"platform_mask\":\"21\",\"start_time\":\"1684757652\",
\"uuid\":\"BD4DFA10-E334-41D9-8136-D2163A8FE588\",
\"version\":\"5.8.2\",\"watcher\":\"8364\"}],\"error\":null}}"]
```

vs. size of the message result on this branch: 675 bytes
```
a["{\"type\":\"result\",\"data\":{\"distributed_query_execution_id\":59,
\"host\":{\"id\":106,\"hostname\":\"lucass-macbook-pro.local\",
\"display_name\":\"Lucas’s MacBook Pro\"},
\"rows\":[{\"build_distro\":\"10.14\",\"build_platform\":\"darwin\",
\"config_hash\":\"f80dee827635db39077a458243379b3ad63311fd\",
\"config_valid\":\"1\",\"extensions\":\"active\",\"host_display_name\":\"Lucas’s MacBook Pro\",
\"host_hostname\":\"lucass-macbook-pro.local\",
\"instance_id\":\"cde5de81-344b-4c76-b1c5-dae964fdd4f2\",\"pid\":\"8370\",\"platform_mask\":\"21\",
\"start_time\":\"1684757652\",\"uuid\":\"BD4DFA10-E334-41D9-8136-D2163A8FE588\",\"version\":\"5.8.2\",
\"watcher\":\"8364\"}]}}"]
```

Manual tests included running with an old fleetctl running with a new
fleet server, and vice-versa, a new fleetctl running against an old
fleet server.

- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [X] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- ~[ ] Documented any permissions changes~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
  - ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
2023-05-25 08:11:53 -03:00
..
async Ingest pending MDM hosts (#9065) 2022-12-26 15:32:39 -06:00
externalsvc remove unused logic specific to DEP+Okta+ROP (#11261) 2023-04-21 14:57:52 -03:00
middleware remove feature flags to enable MDM (#10746) 2023-03-27 16:30:29 -03:00
mock update mockimpl version to allow concurrent mock calls (#9989) 2023-02-21 16:36:06 -03:00
osquery_utils Fixed query used when aggregating WinOS stats (#11840) 2023-05-24 22:11:43 -04:00
redis_policy_set Migrate team-related endpoints to new pattern (#3740) 2022-01-19 10:52:14 -05:00
schedule Run cleanup of cron_stats outside of the schedule package to prevent outages from breaking cron jobs (#10439) 2023-03-13 16:15:30 -03:00
testdata Add mTLS support to fleetd (#11319) 2023-04-27 08:44:39 -03:00
activities_test.go New observer_plus role (#10675) 2023-04-05 15:23:49 -03:00
activities.go Allow setting user roles during JIT provisioning (#10193) 2023-03-01 20:18:40 -03:00
appconfig_test.go Implement worker jobs that update/re-assign setup assistants on changes (#11630) 2023-05-15 14:06:09 -04:00
appconfig.go Add device-authenticated endpoint to trigger MDM migration webhook request (#11724) 2023-05-17 09:16:26 -05:00
apple_mdm_test.go pre-populate username/fullname during account creation (#11557) 2023-05-18 12:50:00 -03:00
apple_mdm.go Modify GET /mdm/apple/profiles API endpoint to return empty array instead of null when results set is empty (#11904) 2023-05-24 18:56:39 -05:00
base_client_errors.go Finalize MDM commands part 2: implement fleetctl mdm run-command (#10866) 2023-04-03 14:25:49 -04:00
base_client_test.go Add mTLS support to fleetd (#11319) 2023-04-27 08:44:39 -03:00
base_client.go Add mTLS support to fleetd (#11319) 2023-04-27 08:44:39 -03:00
campaigns_test.go Make activities typed and auto-generate docs for each type (#9069) 2022-12-23 13:05:16 -03:00
campaigns.go Remove attach check on queries and return proper bad request error (#10427) 2023-03-13 11:42:26 -03:00
carves_test.go report errors that can occur during file carving (#8972) 2022-12-09 13:21:30 -03:00
carves.go Add UUID to Fleet errors and clean up error msgs (#10411) 2023-03-13 13:44:06 -03:00
client_appconfig.go Add macos custom profiles support via fleetctl apply (#9824) 2023-02-15 13:01:44 -05:00
client_apple_mdm.go Improve error handling for fleetctl mdm run-comand (#11588) 2023-05-09 13:43:15 -05:00
client_carves.go Introduce API version 2022-04, deprecate use of /global in paths (#4731) 2022-04-05 11:35:53 -04:00
client_debug.go don't clear errors automatically when reading them from Redis (#5623) 2022-05-10 11:44:43 -03:00
client_hosts.go Add display_name to more results (#8204) 2022-10-14 11:14:18 -03:00
client_labels.go Introduce API version 2022-04, deprecate use of /global in paths (#4731) 2022-04-05 11:35:53 -04:00
client_live_query_test.go Reduce size of DistributedQueryResult to improve live query performance (#11882) 2023-05-25 08:11:53 -03:00
client_live_query.go Send custom headers for live queries (#8991) 2022-12-12 07:56:35 -08:00
client_mdm.go cli: try to infer the bootstrap package name from the URL too (#11571) 2023-05-11 10:36:28 -03:00
client_packs.go Introduce API version 2022-04, deprecate use of /global in paths (#4731) 2022-04-05 11:35:53 -04:00
client_policies.go Introduce API version 2022-04, deprecate use of /global in paths (#4731) 2022-04-05 11:35:53 -04:00
client_profiles.go Add mdm loadtest script (#11696) 2023-05-16 09:41:07 -03:00
client_queries.go Filter out non-observer_can_run queries for observers in fleetctl get queries command to match the UI. (#11251) 2023-04-26 11:38:20 -03:00
client_sessions.go Introduce API version 2022-04, deprecate use of /global in paths (#4731) 2022-04-05 11:35:53 -04:00
client_setup.go fleetctl preview to use v1 setup path, to support previous versions of fleet (#5331) 2022-04-25 10:02:21 -03:00
client_software.go add upgrade tests (#6596) 2022-07-19 15:11:51 -06:00
client_targets.go Add display_name to more results (#8204) 2022-10-14 11:14:18 -03:00
client_teams.go Add mdm loadtest script (#11696) 2023-05-16 09:41:07 -03:00
client_test.go cli: try to infer the bootstrap package name from the URL too (#11571) 2023-05-11 10:36:28 -03:00
client_trigger.go Add fleetctl trigger command (#8877) 2022-12-06 09:56:54 -06:00
client_users.go Filter out non-observer_can_run queries for observers in fleetctl get queries command to match the UI. (#11251) 2023-04-26 11:38:20 -03:00
client.go Implement worker jobs that update/re-assign setup assistants on changes (#11630) 2023-05-15 14:06:09 -04:00
cron_schedules_test.go Add instanceID to schedule logging (#10413) 2023-03-13 15:37:03 -03:00
cron_schedules.go Implement schedule triggers (#8747) 2022-11-28 13:28:06 -06:00
debug_handler_test.go Add fleetctl debug migrations command to get missing migrations (#2967) 2021-11-22 14:06:12 -03:00
debug_handler.go Enable errcheck linter for golangci-lint (#8899) 2022-12-05 16:50:49 -06:00
device_client_test.go base logic to show/hide the new Migrate to Fleet FD menu (#11679) 2023-05-15 17:00:52 -03:00
device_client.go add migration support to FD and orbit (#11741) 2023-05-18 14:21:54 -03:00
devices_test.go Track host DEP assignments in new table (#11875) 2023-05-23 13:01:04 -05:00
devices.go Track host DEP assignments in new table (#11875) 2023-05-23 13:01:04 -05:00
endpoint_campaigns.go override the websocket upgrader checkorigin func to return true (#10841) 2023-03-30 08:09:47 -04:00
endpoint_middleware_test.go Add UUID to Fleet errors and clean up error msgs (#10411) 2023-03-13 13:44:06 -03:00
endpoint_middleware.go Add UUID to Fleet errors and clean up error msgs (#10411) 2023-03-13 13:44:06 -03:00
endpoint_setup.go Add read replica testing helpers and fix non-sso login bug (#4908) 2022-04-04 16:52:05 -07:00
endpoint_utils_test.go Set interface for response types (#9121) 2022-12-27 11:26:59 -03:00
endpoint_utils.go Add UUID to Fleet errors and clean up error msgs (#10411) 2023-03-13 13:44:06 -03:00
frontend.go add browser-related security headers to HTML responses (#8180) 2022-10-12 10:19:21 -03:00
global_policies_test.go Make activities typed and auto-generate docs for each type (#9069) 2022-12-23 13:05:16 -03:00
global_policies.go New gitops role (#10850) 2023-04-12 16:11:04 -03:00
global_schedule_test.go Refactor license so it is stored in the context (#8544) 2022-11-15 09:08:05 -05:00
global_schedule.go Set interface for response types (#9121) 2022-12-27 11:26:59 -03:00
handler_test.go remove feature flags to enable MDM (#10746) 2023-03-27 16:30:29 -03:00
handler.go Add device-authenticated endpoint to trigger MDM migration webhook request (#11724) 2023-05-17 09:16:26 -05:00
hosts_test.go Implement worker jobs that update/re-assign setup assistants on changes (#11630) 2023-05-15 14:06:09 -04:00
hosts.go Implement worker jobs that update/re-assign setup assistants on changes (#11630) 2023-05-15 14:06:09 -04:00
http_auth_test.go Add UUID to Fleet errors and clean up error msgs (#10411) 2023-03-13 13:44:06 -03:00
http_auth.go Remove unused request-id in context (#3632) 2022-01-13 14:12:56 -08:00
http_publicip.go Log all successful logins as activity and all attempts with ip in stderr. (#9095) 2022-12-21 18:29:51 +01:00
installer_test.go Add UUID to Fleet errors and clean up error msgs (#10411) 2023-03-13 13:44:06 -03:00
installer.go Add UUID to Fleet errors and clean up error msgs (#10411) 2023-03-13 13:44:06 -03:00
integration_core_test.go Track host DEP assignments in new table (#11875) 2023-05-23 13:01:04 -05:00
integration_desktop_test.go Track host DEP assignments in new table (#11875) 2023-05-23 13:01:04 -05:00
integration_ds_only_test.go Support async saving of hosts' last seen time (#5640) 2022-05-10 11:29:17 -04:00
integration_enterprise_test.go Track host DEP assignments in new table (#11875) 2023-05-23 13:01:04 -05:00
integration_live_queries_test.go Remove attach check on queries and return proper bad request error (#10427) 2023-03-13 11:42:26 -03:00
integration_logger_test.go Add UUID to Fleet errors and clean up error msgs (#10411) 2023-03-13 13:44:06 -03:00
integration_mdm_test.go Modify GET /mdm/apple/profiles API endpoint to return empty array instead of null when results set is empty (#11904) 2023-05-24 18:56:39 -05:00
integration_sandbox_test.go Bug 10767: Don't return 500s if enroll secret not found (#11121) 2023-04-13 16:16:40 -04:00
integration_sso_test.go Add enable_jit_role_sync config (#10478) 2023-03-14 17:17:08 -03:00
invites_test.go Add UUID to Fleet errors and clean up error msgs (#10411) 2023-03-13 13:44:06 -03:00
invites.go Fix SMTP e-mail send when SMTP server has credentials (#10758) 2023-03-28 15:23:15 -03:00
jitter_test.go Migrate special-case endpoints to new pattern (#4511) 2022-03-08 11:27:38 -05:00
jitter.go Migrate special-case endpoints to new pattern (#4511) 2022-03-08 11:27:38 -05:00
labels_test.go Refactor license so it is stored in the context (#8544) 2022-11-15 09:08:05 -05:00
labels.go /api/_version_/fleet/hosts to return bad request instead of server error when passing invalid mdm_enrollment_status (#11242) 2023-04-19 18:43:15 -03:00
live_queries.go Set interface for response types (#9121) 2022-12-27 11:26:59 -03:00
mail_test.go Fix SMTP e-mail send when SMTP server has credentials (#10758) 2023-03-28 15:23:15 -03:00
mdm_test.go New observer_plus role (#10675) 2023-04-05 15:23:49 -03:00
mdm.go add support for displaying EULA during ADE/DEP (#11532) 2023-05-05 14:36:13 -03:00
mem_failing_policies_set_test.go Fix failing policy sets for redis cluster mode (#3725) 2022-01-17 15:16:54 -05:00
metrics_appconfig.go Fix SMTP e-mail send when SMTP server has credentials (#10758) 2023-03-28 15:23:15 -03:00
metrics_change_email.go Update go-kit to 0.4.0 (#1411) 2017-03-15 08:55:30 -07:00
metrics_invites.go Add v4 suffix in go.mod (#1224) 2021-06-25 21:46:51 -07:00
metrics_labels.go Add v4 suffix in go.mod (#1224) 2021-06-25 21:46:51 -07:00
metrics_sessions.go add back-end implementation for SSO JIT provisioning (#7182) 2022-08-15 14:42:33 -03:00
metrics_users.go Add v4 suffix in go.mod (#1224) 2021-06-25 21:46:51 -07:00
metrics.go Issue 1362 fleetctl user roles (#1397) 2021-07-16 15:28:13 -03:00
orbit_client.go Add mTLS support to fleetd (#11319) 2023-04-27 08:44:39 -03:00
orbit.go Track host DEP assignments in new table (#11875) 2023-05-23 13:01:04 -05:00
osquery_test.go Reduce size of DistributedQueryResult to improve live query performance (#11882) 2023-05-25 08:11:53 -03:00
osquery.go Reduce size of DistributedQueryResult to improve live query performance (#11882) 2023-05-25 08:11:53 -03:00
packs_test.go Fix permissions on GitOps user for searching hosts or count targets (#11448) 2023-05-01 12:57:28 -03:00
packs.go Fix permissions on GitOps user for searching hosts or count targets (#11448) 2023-05-01 12:57:28 -03:00
queries_test.go Remove attach check on queries and return proper bad request error (#10427) 2023-03-13 11:42:26 -03:00
queries.go New gitops role (#10850) 2023-04-12 16:11:04 -03:00
scheduled_queries_test.go Refactor license so it is stored in the context (#8544) 2022-11-15 09:08:05 -05:00
scheduled_queries.go Set interface for response types (#9121) 2022-12-27 11:26:59 -03:00
service_appconfig_test.go New gitops role (#10850) 2023-04-12 16:11:04 -03:00
service_appconfig.go New gitops role (#10850) 2023-04-12 16:11:04 -03:00
service_campaign_test.go add configuration to disable origin header check (#10779) 2023-03-28 01:23:29 -04:00
service_campaigns.go Enable errcheck linter for golangci-lint (#8899) 2022-12-05 16:50:49 -06:00
service_errors.go Add UUID to Fleet errors and clean up error msgs (#10411) 2023-03-13 13:44:06 -03:00
service_users.go New gitops role (#10850) 2023-04-12 16:11:04 -03:00
service.go Finalize MDM commands part 3: add the fleetctl get mdm-command-results command (#10964) 2023-04-05 10:50:36 -04:00
sessions_test.go Add enable_jit_role_sync config (#10478) 2023-03-14 17:17:08 -03:00
sessions.go gate DEP enrollment behind SSO when configured (#11309) 2023-04-27 09:43:20 -03:00
software_test.go Refactor license so it is stored in the context (#8544) 2022-11-15 09:08:05 -05:00
software.go Update aggregated_stats to support "no team" in addition to "all teams" (#10466) 2023-03-14 17:01:16 -04:00
status.go Set interface for response types (#9121) 2022-12-27 11:26:59 -03:00
targets_test.go Refactor license so it is stored in the context (#8544) 2022-11-15 09:08:05 -05:00
targets.go Set interface for response types (#9121) 2022-12-27 11:26:59 -03:00
team_policies_test.go Make activities typed and auto-generate docs for each type (#9069) 2022-12-23 13:05:16 -03:00
team_policies.go New gitops role (#10850) 2023-04-12 16:11:04 -03:00
team_schedule_test.go Refactor license so it is stored in the context (#8544) 2022-11-15 09:08:05 -05:00
team_schedule.go Set interface for response types (#9121) 2022-12-27 11:26:59 -03:00
teams_test.go New gitops role (#10850) 2023-04-12 16:11:04 -03:00
teams.go Add disk_encryption option to config and team YAML (#10185) 2023-02-28 15:34:46 -05:00
testing_client.go Feature: Store installed file path when ingesting software (#11214) 2023-05-17 14:49:09 -04:00
testing_utils.go Add device-authenticated endpoint to trigger MDM migration webhook request (#11724) 2023-05-17 09:16:26 -05:00
translator.go Set interface for response types (#9121) 2022-12-27 11:26:59 -03:00
transport_error_test.go Add UUID to Fleet errors and clean up error msgs (#10411) 2023-03-13 13:44:06 -03:00
transport_error.go Add UUID to Fleet errors and clean up error msgs (#10411) 2023-03-13 13:44:06 -03:00
transport_setup.go Update go-kit to 0.4.0 (#1411) 2017-03-15 08:55:30 -07:00
transport_test.go For requests with invalid list options, return 400 instead of 500 (#11632) 2023-05-17 13:41:30 -07:00
transport.go For requests with invalid list options, return 400 instead of 500 (#11632) 2023-05-17 13:41:30 -07:00
trigger.go Set interface for response types (#9121) 2022-12-27 11:26:59 -03:00
user_roles.go Set interface for response types (#9121) 2022-12-27 11:26:59 -03:00
users_test.go Add UUID to Fleet errors and clean up error msgs (#10411) 2023-03-13 13:44:06 -03:00
users.go New gitops role (#10850) 2023-04-12 16:11:04 -03:00
validation_setup.go Use new error handling approach in other packages (#2954) 2021-11-22 09:13:26 -05:00