Commit Graph

583 Commits

Author SHA1 Message Date
Robert Henry
81a6c76bed
fix typo in help message in serve.go (#14037) 2023-09-26 13:06:17 -03:00
Jacob Shandling
1993c02b3a
Move code for --local-wix-dir into ee directory (#14104) 2023-09-25 10:38:03 -07:00
Jacob Shandling
b6f072775f
Add ability for fleetctl package to use local WiX v3 binaries when generating installer .msi (#14033) 2023-09-22 08:49:01 -07:00
Tim Lee
e80ea50240
Flock vulnerability false positive (#13827) 2023-09-14 10:38:36 -06:00
Roberto Dip
541cc638e4
Add an activity log when an script execution is requested (#13655)
For #9553
2023-09-05 20:09:29 -03:00
gillespi314
37fb4b0dab
Add fleetctl run-script command (#13622) 2023-09-05 14:14:09 -05:00
Martin Angers
cbc3f32e9d
Adjust response payload, messages and validations for /scripts/run/* endpoints. (#13607) 2023-08-31 09:08:50 -05:00
Lucas Manuel Rodriguez
9142c5de79
Prevent thundering herd when applying large number of policies on large number of hosts (#13552)
#13527

(Adding @mna to double check the changes in the async implementation of
policy result storage)

This PR also adds the osquery-perf changes needed to define the count of
macOS and Windows hosts.

- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)~
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [X] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
  - ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~

Test with 80k hosts: 70k simulated macOS, 10k simulated Windows.
Apply Windows policies first, then apply macOS policies:
```
fleetctl apply -f ee/cis/win-10/cis-policy-queries.yml

# Leave running for some time

fleetctl apply -f ee/cis/macos-13/cis-policy-queries.yml
```

After applying CIS policies previous to these changes:
![Screenshot 2023-08-23 at 11 36
18](https://github.com/fleetdm/fleet/assets/2073526/72c1dc7d-e601-4248-be35-93c85b749f5d)

After applying these changes and applying the same policies:
![Screenshot 2023-08-28 at 15 42
57](https://github.com/fleetdm/fleet/assets/2073526/6b6d76b8-6acb-4893-a913-bf603a68f1a4)
2023-08-31 10:58:50 -03:00
Martin Angers
090b142c49
Implement script execution on the fleetd agent (disabled by default) (#13569) 2023-08-30 14:02:44 -04:00
Roberto Dip
b50e1939db
Allow to configure fleetd for script execution (#13564)
Related to #13310 and #13304 this adds two ways to enable script
execution in `fleetd` (the orbit component)

- By building a package with `--enable-scripts`
- By providing a setting via a configuration profile (macOS only)

Due to how the profile assignment works, this change automatically
updates the `com.fleetdm.fleetd.config` for hosts that already have the
profile installed.

> [!NOTE]
> Documentation is in
[#13577](https://github.com/fleetdm/fleet/pull/13577) to decouple
reviews.
2023-08-30 10:18:34 -03:00
Martin Angers
edf4a4d02f
Add script execution simulation to osquery-perf in preparation for load testing (part 3 of ticket) (#13456) 2023-08-23 18:31:47 -04:00
Tim Lee
3b61adf7a4
Add validation for policy specs (#13294) 2023-08-21 10:22:07 -06:00
Jacob Shandling
a7f1501054
Update fleetctl convert for schedulable queries (#13125) 2023-08-08 10:39:20 -07:00
Martin Angers
37ba43d404
Add fleetctl upgrade-packs command to migrate 2017 packs to queries (#13078) 2023-08-08 08:21:57 -04:00
Martin Angers
554e024f7b
Fix gitops access when using --dry-run with fleetctl apply (#13178) 2023-08-07 13:51:11 -04:00
Juan Fernandez
63c7c8a3ae
If no global queries, show proper message at the end (#13006)
If no global queries, show proper message at the end
2023-07-27 16:55:07 -04:00
Juan Fernandez
1dde10d5c3
Print inherited message if no queries found (#12998)
If the team has no queries, then fleetctl output should also show the number of inherited queries.
2023-07-27 15:08:40 -04:00
Juan Fernandez
e4cc0c3098
Fixed format issues with fleetctl get queries (#12983)
Output from `fleetctl get queries` should include the team the query is in and also the scheduling information.
2023-07-27 09:29:09 -04:00
Martin Angers
6f77911ffe
Fix performance regression found in load testing (#12981) 2023-07-26 17:13:27 -04:00
Lucas Manuel Rodriguez
2afbd24021
Combine Schedules and Queries: API changes (#12778)
Combining schedules and queries API changes.
2023-07-24 20:17:20 -04:00
Juan Fernandez
6b664a2a82
Merge branch 'main' into 7765-combined-schedules-and-queries 2023-07-21 13:53:13 -04:00
Lucas Manuel Rodriguez
b803bbe5bf
osquery-perf changes needed for load testing with simulated Windows hosts (#12754)
Changes in osquery-perf to allow for testing of Windows hosts in
loadtest environments.
2023-07-14 13:06:34 -03:00
Gabriel Hernandez
9aa7c0c714
add dark and light background logo colors and show them on mdm migrat… (#12681) 2023-07-13 19:35:25 +01:00
Roberto Dip
666ae8d787
ensure ds.TeamByName returns a 4xx response if no team is found (#12620)
this helps consumer of the datastore method handle the not found
scenario better and ensures we always return a 4xx code by default if we
can't find a matching team.

seems like calls to this method were special-cased everywhere except in
the apply user roles endpoint, where we returned a `500` status code if
we couldn't find a team.
2023-07-13 11:55:05 -03:00
Juan Fernandez
390e0565d0
Updated delete method on the DB layer 2023-07-07 07:31:36 -04:00
Juan Fernandez
807b2e35d3
Updated QueryByName DB access method 2023-07-06 19:37:08 -04:00
Marcos Oviedo
96449dd47b
Adding support for RequestSecurityToken messages - Windows MDM enroll endpoint (#12555)
This relates to #12263 

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [X] Added/updated tests

---------

Co-authored-by: Roberto Dip <me@roperzh.com>
2023-07-05 10:06:37 -03:00
gillespi314
410cbc3972
Add certificate management for Microsoft MDM (WSTEP) (#12543)
Issue #12261

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [ ] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- [ ] Documented any permissions changes
- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [ ] Added/updated tests
- [ ] Manual QA for all new/changed functionality
  - For Orbit and Fleet Desktop changes:
- [ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
2023-06-29 19:31:53 -03:00
Marcos Oviedo
821f6b064f
Adding support for GetPolicies message (#12477)
This relates to #12262 

- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
2023-06-27 12:59:33 -03:00
Marcos Oviedo
22bb16bf2e
Pushing initial support for MS-MDE2 Discovery message (#12387)
This PR requires the Windows MDM configuration changes - This will be
updated next week

- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- [x] Documented any permissions changes
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
  - For Orbit and Fleet Desktop changes:
2023-06-22 17:31:17 -03:00
Juan Fernandez
d08cc18111
Don't analyze non-windows OS using the MSRC scanner (#12421)
Don't analyze non-windows OS using the MSRC scanner
2023-06-21 14:19:55 -04:00
Martin Angers
1c249b60da
Add support to configure and enable Windows MDM, notify elegible hosts (#12340) 2023-06-20 14:06:45 -04:00
Lucas Manuel Rodriguez
bac309c978
Provide feedback to user in fleetctl login when using env vars (#12371)
Issue reported by @jarodreyes.

The user was not informed that `fleetctl login` was using the PASSWORD
environment variable:

`main`:
```sh
export PASSWORD=wrong

fleetctl login
Log in using the standard Fleet credentials.
Email: a@b.c
Error: Login failed: login received status 401 Authentication failed: Authentication failed
```
And with the changes on this PR:
```
export PASSWORD=wrong

fleetctl login
Log in using the standard Fleet credentials.
Email: a@b.c
Using value of environment variable $PASSWORD as password.
Error: Login failed: login received status 401 Authentication failed: Authentication failed
```
2023-06-16 11:42:04 -03:00
Juan Fernandez
7226b7f087
Warnings in fleetctl should use Stderr not Stdout (#12316)
Fixed issue were the expired license banner was being sent to Stdout instead of Stderr
2023-06-15 13:13:41 -04:00
Martin Angers
68fa60c54d
Add a transferred_hosts activity when hosts are transferred to a new team (#12287) 2023-06-14 08:15:05 -04:00
Lucas Manuel Rodriguez
2a532ede94
Do not return empty SSO and SMTP settings for non-global-admins (#12180)
#11266

PS: I first attempted a serialization trick by introducing a new
`appConfigResponse` and implementing `json.Marshal` to exclude these
fields but it was too hacky and hard to maintain moving forward, so I'm
bitting the bullet now. Happy to hear other ideas.

- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
  - ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
2023-06-07 16:06:36 -03:00
Martin Angers
f390055847
Fix table formatting of fleetctl get mdm-command-results output (#12197) 2023-06-07 14:35:48 -04:00
Martin Angers
68ddaafac0
Fix bug preventing gitops role from fleetctl applying macos setup assistant (and bootstrap package) (#12193) 2023-06-07 13:29:36 -04:00
Martin Angers
f27fcddd55
Prevent clearing macos updates settings when applying/modifying a team without those settings (#12160) 2023-06-06 14:31:33 -04:00
Martin Angers
48774876ea
Move post-DEP-enrollment processing to a worker job (#12017) 2023-06-05 11:58:23 -04:00
Lucas Manuel Rodriguez
2d21b2be29
Provide more feedback to the user when there's a Redis connection issue when running live queries (#11947)
This could help future users to detect this issue: #10957
It also adds an error log in Fleet that prints the actual error.

The error is displayed if I kill Redis during a live session or if I set
`client-output-buffer-limit` to something real low like `CONFIG SET
"client-output-buffer-limit" "pubsub 100kb 50kb 60"`:
![Screenshot 2023-05-25 at 09 08
08](https://github.com/fleetdm/fleet/assets/2073526/f021a77a-3a22-4b48-8073-bae9c6e21a11)

- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- [X] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- ~[ ] Added/updated tests~
- [X] Manual QA for all new/changed functionality
  - ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
2023-06-01 16:11:55 -03:00
Martin Angers
4322a28f5a
Implement preassign endpoint as first step to match profiles and hosts to teams (#12046) 2023-05-31 09:24:22 -04:00
Lucas Manuel Rodriguez
f0dbb981a3
Fix small fleetctl query bug when running with --exit flag (#11894)
Bug found while working on #10957.

Can be reproduced in our dogfood environment:
```
fleetctl query --context dogfood --query "SELECT * from osquery_info;" --hosts dogfood-centos-box --exit
⠋ %
```

With the changes in this PR:
```
fleetctl query --context dogfood --query "SELECT * from osquery_info;" --hosts dogfood-centos-box --exit
{"host":"dogfood-centos-box","rows":[{"build_distro":"centos7","build_platform":"linux","config_hash":"e3832343af2f8dc3e5ab62e709c78d3c3ef32b86","config_valid":"1","extensions":"active","host_display_name":"dogfood-centos-box","host_hostname":"dogfood-centos-box","instance_id":"9f0f6433-fbcf-4f15-8f1b-4dedc669ee2d","pid":"2760450","platform_mask":"9","start_time":"1684821735","uuid":"911CBDBA-7B3A-4B96-88F7-B28CECBEF400","version":"5.8.2","watcher":"2760447"}]}
⠦ 0% responded (0% online) | 0/1 targeted hosts (0/1 online) %
```
2023-05-25 08:12:45 -03:00
Lucas Manuel Rodriguez
3f05208192
Improve osquery-perf support for live queries (simulate no results and/or query failures) (#11883)
These osquery-perf changes were useful to test/troubleshoot #10957.
It basically allows osquery-perf to simulate no results and/or failures
when running live queries on the devices.

- ~[ ] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.~
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- [X] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- ~[ ] Added/updated tests~
- [X] Manual QA for all new/changed functionality
  - ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
2023-05-25 08:12:10 -03:00
Lucas Manuel Rodriguez
6415f062c6
Reduce size of DistributedQueryResult to improve live query performance (#11882)
This was found while working on #10957.

When running a live query, a lot of unused host data is stored in Redis
and sent on every live query result message via websockets. The frontend
and fleetctl just need `id`, `hostname` and `display_name`. (This
becomes worse every time we add new fields to the `Host` struct.)

Sample of one websocket message result when running `SELECT * from
osquery_info;`:

size in `main`: 2234 bytes
```
a["{\"type\":\"result\",\"data\":{\"distributed_query_execution_id\":57,\"host\":
{\"created_at\":\"2023-05-22T12:14:11Z\",\"updated_at\":\"2023-05-23T12:31:51Z\",
\"software_updated_at\":\"0001-01-01T00:00:00Z\",\"id\":106,\"detail_updated_at\":\"2023-05-23T11:50:04Z\",
\"label_updated_at\":\"2023-05-23T11:50:04Z\",\"policy_updated_at\":\"1970-01-02T00:00:00Z\",
\"last_enrolled_at\":\"2023-05-22T12:14:12Z\",
\"seen_time\":\"2023-05-23T09:52:23.876311-03:00\",\"refetch_requested\":false,
\"hostname\":\"lucass-macbook-pro.local\",\"uuid\":\"BD4DFA10-E334-41D9-8136-D2163A8FE588\",\"platform\":\"darwin\",\"osquery_version\":\"5.8.2\",\"os_version\":\"macOS 13.3.1\",\"build\":\"22E261\",\"platform_like\":\"darwin\",\"code_name\":\"\",
\"uptime\":91125000000000,\"memory\":34359738368,\"cpu_type\":\"x86_64h\",\"cpu_subtype\":\"Intel x86-64h Haswell\",\"cpu_brand\":\"Intel(R) Core(TM) i7-1068NG7 CPU @ 2.30GHz\",\"cpu_physical_cores\":4,\"cpu_logical_cores\":8,\"hardware_vendor\":\"Apple Inc.\",\"hardware_model\":\"MacBookPro16,2\",\"hardware_version\":\"1.0\",
\"hardware_serial\":\"0DPQR4HMD1FZ\",
\"computer_name\":\"Lucas’s MacBook Pro\",\"public_ip\":\"\",
\"primary_ip\":\"192.168.0.230\",\"primary_mac\":\"68:2f:67:8e:b6:1f\",
\"distributed_interval\":1,\"config_tls_refresh\":60,\"logger_tls_period\":10,\"team_id\":null,
\"pack_stats\":null,\"team_name\":null,
\"gigs_disk_space_available\":386.23,\"percent_disk_space_available\":40,
\"issues\":{\"total_issues_count\":0,\"failing_policies_count\":0},
\"mdm\":{\"enrollment_status\":null,\"server_url\":null,\"name\":\"\",\"encryption_key_available\":false},
\"status\":\"online\",\"display_text\":\"lucass-macbook-pro.local\",\"display_name\":\"Lucas’s MacBook Pro\"},
\"rows\":[{\"build_distro\":\"10.14\",\"build_platform\":\"darwin\",
\"config_hash\":\"b7ee9363a7c686e76e99ffb122e9c5241a791e69\",\"config_valid\":\"1\",
\"extensions\":\"active\",\"host_display_name\":\"Lucas’s MacBook Pro\",
\"host_hostname\":\"lucass-macbook-pro.local\",\"instance_id\":\"cde5de81-344b-4c76-b1c5-dae964fdd4f2\",\"pid\":\"8370\",\"platform_mask\":\"21\",\"start_time\":\"1684757652\",
\"uuid\":\"BD4DFA10-E334-41D9-8136-D2163A8FE588\",
\"version\":\"5.8.2\",\"watcher\":\"8364\"}],\"error\":null}}"]
```

vs. size of the message result on this branch: 675 bytes
```
a["{\"type\":\"result\",\"data\":{\"distributed_query_execution_id\":59,
\"host\":{\"id\":106,\"hostname\":\"lucass-macbook-pro.local\",
\"display_name\":\"Lucas’s MacBook Pro\"},
\"rows\":[{\"build_distro\":\"10.14\",\"build_platform\":\"darwin\",
\"config_hash\":\"f80dee827635db39077a458243379b3ad63311fd\",
\"config_valid\":\"1\",\"extensions\":\"active\",\"host_display_name\":\"Lucas’s MacBook Pro\",
\"host_hostname\":\"lucass-macbook-pro.local\",
\"instance_id\":\"cde5de81-344b-4c76-b1c5-dae964fdd4f2\",\"pid\":\"8370\",\"platform_mask\":\"21\",
\"start_time\":\"1684757652\",\"uuid\":\"BD4DFA10-E334-41D9-8136-D2163A8FE588\",\"version\":\"5.8.2\",
\"watcher\":\"8364\"}]}}"]
```

Manual tests included running with an old fleetctl running with a new
fleet server, and vice-versa, a new fleetctl running against an old
fleet server.

- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [X] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- ~[ ] Documented any permissions changes~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
  - ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
2023-05-25 08:11:53 -03:00
Roberto Dip
d0658868dc
adjust the UI to show a refetching spinner on critical queries (#11868)
A possible solution for https://github.com/fleetdm/fleet/issues/11860,
this updates the "My Device" page to show the spinner rotating if
critical queries are being fetched:



https://github.com/fleetdm/fleet/assets/4419992/0a84508a-4df2-4da3-b9bc-b3e75bab5e82


# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Manual QA for all new/changed functionality
2023-05-23 12:43:50 -03:00
Roberto Dip
dcc22f436b
allow to set await_device_configured in DEP profile (#11789)
#10744
2023-05-19 07:08:35 -03:00
Juan Fernandez
009a87d33e
Feature 10196: Add filepath to end-points and third party integrations (#11285)
Adds the software installed path property to the proper end-points and third party integrations (webhook, Zendesk and Jira).
2023-05-17 16:53:15 -04:00
Juan Fernandez
7f83135aa1
Feature: Store installed file path when ingesting software (#11214)
Store software installed paths into the host_software_installed_paths table when ingesting osquery software data.
2023-05-17 14:49:09 -04:00
Roberto Dip
4dd127d577
base logic to show/hide the new Migrate to Fleet FD menu (#11679)
Related to #11670
2023-05-15 17:00:52 -03:00