Commit Graph

709 Commits

Author SHA1 Message Date
Victor Lyuboslavsky
a5482f6e6a
Team host expiry. (#16188)
Teams can configure their own host expiry setting. If global host expiry
is enabled, teams cannot disable host expiry, but they can set a longer
(or shorter) expiry time (in days).
- Added `host_expiry_settings` to team spec, which can be used via
fleetctl apply.
- Added `host_expiry_settings` to PATH /fleet/teams/:id endpoint.


#15609 (parent)
#15966 (subtask)

TODO: Create PR for API docs change: added parameter to `PATCH
/fleet/teams/:id` endpoint

# Checklist for submitter

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
2024-01-22 10:00:02 -06:00
Roberto Dip
5fe65d414c
allow to configure a custom configuration_web_url (#16175)
for #15633
2024-01-17 19:47:04 -03:00
Victor Lyuboslavsky
f0154608fd
For software/titles and software/versions endpoints, the browser property is no longer included in the response when empty. (#16126)
For `software/titles` and `software/versions` endpoints, the `browser`
property is no longer included in the response when empty.
#16056

# Checklist for submitter
<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
2024-01-16 10:33:03 -06:00
Victor Lyuboslavsky
ab4c505f4b
Enroll secret in macOS keychain and Windows Credential Manager (#16068)
#13832

For macOS hosts, fleetd now stores and retrieves enroll secret from
macOS keychain.
- this feature must use the official signed and notarized version of
fleetd
- for contributors, this feature can disabled with either:
  - fleetctl package flag: --disable-keystore
  - fleetd runtime flag: --disable-keystore

This feature does not cover the MDM usecase where enroll secret is
stored in the MDM profile. This usecase will hopefully be worked on next
sprint with the MDM team.

For Windows hosts, fleetd now stores and retrieves enroll secret from
Windows Credential Manager.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
  - For Orbit and Fleet Desktop changes:
- [x] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [x] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
2024-01-16 06:51:37 -06:00
Martin Angers
fc3304c902
Move nanomdm dependency in monorepo (#16015)
#15557 

Following the precedent that Lucas used for other similar PRs, the best
way to review is probably by commits.

* The first one simply copies over the files from the fork to the
monorepo
* Second one adjusts all import paths
* Third one tidies up the `go.mod` files
* Last one fixes the linter issues in the nanomdm package

# Checklist for submitter

- ~~Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.~~ (not a user-visible change)
- [x] Manual QA for all new/changed functionality (ran test suite,
re-generated mocks)

I also verified that our Go test suite did run the newly moved `nanomdm`
package steps:

```
ok  	github.com/fleetdm/fleet/v4/server/mdm/nanomdm/cryptoutil	0.003s	coverage: 0.0% of statements in github.com/fleetdm/fleet/v4/...
ok  	github.com/fleetdm/fleet/v4/server/mdm/nanomdm/mdm	0.005s	coverage: 46.2% of statements in github.com/fleetdm/fleet/v4/...
ok  	github.com/fleetdm/fleet/v4/server/mdm/nanomdm/service/certauth	1.320s	coverage: 20.7% of statements in github.com/fleetdm/fleet/v4/...
ok  	github.com/fleetdm/fleet/v4/server/mdm/nanomdm/storage/file	0.007s	coverage: 24.1% of statements in github.com/fleetdm/fleet/v4/...
```
2024-01-11 23:28:48 -03:00
Sarah Gillespie
2bbef8c56e
Bugfix: Only return host timeout error message when script exit code is nil (#15967) 2024-01-10 15:54:41 -06:00
Lucas Manuel Rodriguez
4627a92447
Move external dependency osquery-in-a-box to monorepo (#15871)
#15563 

- [X] Manual QA for all new/changed functionality

Tested by running the following:

If the changes haven't been merged to `main`:
```sh
fleetctl preview --preview-config 15563-move-external-dep-osquery-in-a-box-to-monorepo
fleetctl preview stop
fleetctl preview reset
```
If the changes were already merged to `main`:
```sh
fleetctl preview 
fleetctl preview stop
fleetctl preview reset
```
2024-01-10 11:45:52 -03:00
Jahziel Villasana-Espinoza
71b7b8b46c
feat: disable script endpoints and add error to cli (#15941)
> 📜 Relevant PR: #14500

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
2024-01-08 16:28:45 -05:00
Tim Lee
9f7cf607bb
Bugfix: Missing Software Aggregation in vuln processing command (#15954) 2024-01-08 11:22:07 -07:00
Sarah Gillespie
e262b4a64f
Update osquery-perf template for simulated Windows host (#15943) 2024-01-08 10:57:37 -06:00
Sarah Gillespie
caac8fbbe2
Allow context and debug flags after fleetctl generate subcommands (#15934) 2024-01-05 12:08:58 -06:00
Tim Lee
d690867a6a
Cleanup Query Results Cron (#15917) 2024-01-05 06:05:01 -07:00
Roberto Dip
3b00b70786
fix unreleased bugs for the increased script timeout (#15897)
for https://github.com/fleetdm/fleet/issues/15196. The main problem was
that we have two timeouts:

1. The timeout used by the host to kill the script execution
2. The timeout used by the server to wait for the script results

Before the changes in https://github.com/fleetdm/fleet/pull/15779, the
server timeout was longer than the host timeout, but we inadvertently
set both values to 5 minutes, which breaks the logic we have to handle
both kinds of timeouts.
2024-01-04 10:26:13 -03:00
Martin Angers
d943fbbf8e
Extend script execution timeout (#15779)
#15196 This is the work of @ghernandez345 except for adding the
`ResponseController` thing in Go to override the server timeout for that
specific sync endpoint so that the calls don't timeout waiting for a
script response (the default HTTP server timeout was 90s for our
server).

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
  - For Orbit and Fleet Desktop changes:
- [x] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [x] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).

---------

Co-authored-by: Gabriel Hernandez <ghernandez345@gmail.com>
Co-authored-by: Roberto Dip <me@roperzh.com>
Co-authored-by: Roberto Dip <dip.jesusr@gmail.com>
2024-01-03 16:39:53 -03:00
Roberto Dip
a8a8080b3b
Fix a bug that didn't allow to enable team disk encryption if macOS MDM is not configured (#15886)
for #15817
2024-01-03 16:15:09 -03:00
Jahziel Villasana-Espinoza
2833c80cfe
feat: add option to disable scripts (#15815)
> 📜 Related issue: https://github.com/fleetdm/fleet/issues/14500

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
2024-01-03 10:42:08 -05:00
Lucas Manuel Rodriguez
417f45fc61
Move external dependency fleetdm/kolide-kit to monorepo (#15861)
#15556

We will need to pay attention when releasing fleet (the github actions
were modified to use the local file now).

Should be reviewed by commits (first commit is the actual adding of the
`version.go` file)

- [X] Manual QA for all new/changed functionality

Manually tested the following:
- `Settings -> My account` on the UI and checked the `/version` endpoint
response. (Or also visiting https://localhost:8080/version on a
browser).
- Ran `make fleetctl fleet`, `./build/fleetctl --version` and
`./build/fleet version`.
2024-01-02 18:22:52 -03:00
Victor Lyuboslavsky
ce8eb74897
Added gigs_total_disk_space to host endpoint responses. (#15756)
Added `gigs_total_disk_space` to host endpoint responses.
- Now user will know host disk size even when disk is full
(percent_disk_space_available==0)
#15058 

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
2023-12-22 12:46:33 -06:00
Martin Angers
2e8da551d0
Custom email device-mapping: implement the CLI (fleetd + fleetctl) changes (#15763)
Co-authored-by: Sarah Gillespie <73313222+gillespi314@users.noreply.github.com>
2023-12-21 11:22:59 -06:00
Victor Lyuboslavsky
bb30ffb21f
Added live query stats to activity. (#15682)
📺 Loom:
https://www.loom.com/share/474ac220dd054c389ebed6ba13f943b1?sid=70c2d94e-9fd9-40f5-af73-13e6d2e4c90a
Final piece for #467 -- Added live query stats to activity feed.
# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
2023-12-18 09:31:07 -06:00
Victor Lyuboslavsky
0e040cc7b0
fleetctl now runs saved queries (#15667)
📺 Looom:
https://www.loom.com/share/1aec4616fa4449e7abac579084aef0ba?sid=0884f742-feb3-48bb-82dc-b7834bc9a6e1

Fixed fleetctl issue where it was creating a new query when running a
query by name, as opposed to using the existing saved query.
#15630

API change will be in a separate PR:
https://github.com/fleetdm/fleet/pull/15673

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
2023-12-15 12:55:39 -06:00
Lucas Manuel Rodriguez
024a20ac11
Allow enrolling fleetd using osquery's instance identifier (#15570)
#14879

- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
  - For Orbit and Fleet Desktop changes:
- [x] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [x] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
2023-12-15 15:26:32 -03:00
Victor Lyuboslavsky
0df5160629
Fixing live query data race in test. (#15647)
Seems like a pain to fix data races in an async test. Maybe there is an
easier way.
2023-12-14 14:14:43 -06:00
Victor Lyuboslavsky
835eedae42
Performance stats for live queries (#15440)
📺 Loom explaining the PR:
https://www.loom.com/share/8c6ec0c362014910931f183d68307525?sid=7de51e6f-a59d-4d8f-b06b-f703f1692f17

#467 
Live Queries now collect stats. Stats are collected for saved queries
present in the Queries tab.
- After running a live query, user will see updated stats in Queries
tab.
- Query stats on Host Details page will no longer be cleared after
host/agent reboots.
- Query stats are now deleted when query is deleted.

# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
2023-12-13 14:46:59 -06:00
Roberto Dip
fbaa479d01
pre-calculate host counts for software titles (#15601)
found during the load test, this addresses a performance issue with
software titles.

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

<!-- Note that API documentation changes are now addressed by the
product design team. -->

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
2023-12-13 12:48:57 -03:00
Gabriel Hernandez
9d65a2dc8c
Implement UI for new software pages (#15579) 2023-12-12 15:03:33 -06:00
Roberto Dip
2ed3026819
Add pagination meta to software versions endpoint (#15550) 2023-12-12 12:24:20 -06:00
Sarah Gillespie
0e468b4981
Update software titles cron to include browser field (#15491) 2023-12-07 17:43:37 -06:00
Martin Angers
8b7d1a7e47
Remove Windows MDM feature flag (#15167)
For #14959 

---------

Co-authored-by: Gabriel Hernandez <ghernandez345@gmail.com>
2023-12-07 14:24:32 -03:00
Martin Angers
e3d225ade7
Update fleetctl get software to list titles and versions. (#15444) 2023-12-06 16:07:03 -05:00
Sarah Gillespie
f19dc8abe0
Add GET software/versions and GET software/versions/:id endpoints (#15450) 2023-12-06 08:30:49 -06:00
Sarah Gillespie
0db59a7a25
Add cron for software titles feature (#15422) 2023-12-04 10:09:23 -06:00
Martin Angers
f5f66ebe63
Unify profiles database identifier to profile_uuid for macOS and Windows profiles. (#15297) 2023-12-04 10:04:06 -05:00
Victor Lyuboslavsky
a7ae828581
Adding extension_id and browser columns to software table. (#15285)
https://github.com/fleetdm/confidential/issues/2118
API doc changes are at: https://github.com/fleetdm/fleet/pull/15092

This change affects Chrome-based and Firefox browsers. I tried Safari,
but it appears that latest Safari extensions simply show up as regular
apps on macOS.

# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
  - Tested detail query changes on macOS, Windows, Ubuntu, and ChromeOS
  - Tested Chrome, Edge, Brave, and Firefox browsers

Testing notes:
1. Install some browser extensions.
2. Update fleet to this change.
3. Refresh hosts, and trigger vulnerabilities job (which updates host
software).
4. Do something like `GET
https://localhost:8080/api/v1/fleet/software?query=Password%20Manager`
to see `extension_id` and `browser` for the hosts that have updated.
2023-11-30 19:06:17 -06:00
Roberto Dip
b07fbdc1d7
implement profile verification for windows (#15348)
for #14426.

In order to prevent import cycles and be able to use some type
definitions/constants I followed the same pattern we did for macOS by
creating a new package named `syncml`. This makes the changelog look
bigger than it actually is, so I split it into two commits to make it
easier to review:

-
[d7c233d](d7c233d54c)
moves the relevant bits to this new package
-
[7531a07](7531a0742b)
implements profile verification
2023-11-30 09:17:07 -03:00
Martin Angers
2f927df4f0
Implement Windows OS Updates (feature branch). (#15359) 2023-11-29 11:07:24 -05:00
Martin Angers
0b5eedb801
Fix edge case of AppConfig changes getting lost in cached mysql. (#15352) 2023-11-29 10:09:37 -05:00
Sarah Gillespie
1535b72305
Add hardware serial to host by identifier search (#15353) 2023-11-29 08:39:35 -06:00
Roberto Dip
f439432fae
feature: windows profiles (#15349) 2023-11-29 09:32:42 -05:00
Roberto Dip
9c7b740b03
disable windows profiles feature in the back-end (#15278)
per discussion in slack, reverting relevant user-facing changes for
#13281
2023-11-27 16:08:52 -03:00
Tim Lee
c5ea7eac4f
Cache policy counts (#15244) 2023-11-21 12:52:06 -07:00
Lucas Manuel Rodriguez
4194c44131
Use NVD API 2.0 to download CVE information (#15102)
#14888

@getvictor This is ready for review, but keeping as draft as there are
probably many tests that need amending.

I used the new version of the `./tools/nvd/nvdvuln/nvdvuln.go` to
compare the current vulnerabilities found in our dogfood environment
with the vulnerabilities found by the code in this PR and both results
match:
```
go run -race -tags fts5 ./tools/nvd/nvdvuln/nvdvuln.go --debug --db_dir ./local --software_from_url <dogfood URL> --software_from_api_token <API_TOKEN> --sync 2>&1 | tee out.txt
[...]
CVEs found and expected matched!
```

- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Added/updated tests
- [X] Manual QA for all new/changed functionality

---------

Co-authored-by: Victor Lyuboslavsky <victor@fleetdm.com>
Co-authored-by: Victor Lyuboslavsky <victor.lyuboslavsky@gmail.com>
2023-11-21 12:30:07 -06:00
Victor Lyuboslavsky
eada583ff1
Updating CPE generator to use new NVD API. (#15018)
Loom explaining changes (hit 5 min limit):
https://www.loom.com/share/e59b63bf638e4d9cad7984ef589b878d?sid=111fff75-115a-4a44-ae4f-6f25fede0d51

#14887

- [x] Need to merge fleetdm/nvd PR
https://github.com/fleetdm/nvd/pull/25 before this one.

# Checklist for submitter

- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
- Manually tested (with corresponding fleetdm/fleet changes) in my
personal fork: https://github.com/getvictor/nvd/releases

# QA Plan (must be done before merging this PR, and after merging the
nvd PR)
- [ ] Fork https://github.com/fleetdm/nvd and point `generate.yml` to
this branch.
[example](9d8e54930b/.github/workflows/generate.yml (L26))
- [ ] Add NVD_API_KEY to nvd secrets, and run the the nvd generate
GitHub action. Get key:
https://nvd.nist.gov/developers/request-an-api-key
- [ ] Compare the generated `cpe-###.sqlite.gz` to the previous one. One
way is to open it up with sqlite3 and `select * from cpe_2 order by
cpe23;` and dump results to a CSV file. Known differences are:
   - New file has ~2,500 more records
- Backslashes are handled differently for `Backpack\CRUD` and `Philips
In.Sight B120\37` products -- not a new issue since we do not support
those products right now
- `cpe:2.3🅰️moodle:moodle:4.2.0:*:*:*:*:*:*:*` -- this appears OK.
Also, it is a PHP plugin, and we don't support these currently.
- [ ] Record the existing vulnerabilities of current hosts.
- [ ] Stop any running fleet server. Delete `/tmp/vulndbs/cpe.sqlite`.
Can also delete other files there, or not delete this file -- it should
be overwritten by the new file. Also delete all rows in software_cpe and
software_cve DB tables. (Or can just spin up a fresh fleet server with
fresh DB, and re-enroll hosts (after setting the new env variable
below))
- [ ] Find the path to the generated `cpe-###.sqlite.gz` file
- [ ] Set `FLEET_VULNERABILITIES_CPE_DATABASE_URL` environment variable
to the above path, and start fleet server.
- [ ] After server's vulnerabilities cron job runs, the new
vulnerabilities should match the previous vulnerabilities
2023-11-20 16:10:00 -06:00
Sarah Gillespie
57df2f250c
Add Windows MDM profiles to host details API response (#15210) 2023-11-20 14:34:57 -06:00
Roberto Dip
420dfe1cd0
batch set profiles as pending on profile-related actions (#15179)
final step of #14364
2023-11-20 11:16:02 -03:00
Sharon Katz
c67d9eb3a5
Hackathon add metrics from Fleet servers. (#14216)
Adding a few stats to the metrics Fleet Servers send to Heroku.
2023-11-16 10:25:19 -05:00
Jahziel Villasana-Espinoza
1fca8b1e38
fix: sort order for Last restarted (#14878)
# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Documented any API changes (docs/REST API/rest-api.md or
docs/Contributing/API-for-contributors.md)
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality

---------

Co-authored-by: Rachael Shaw <r@rachael.wtf>
2023-11-15 16:42:57 -05:00
Roberto Dip
f82cfa307d
allow fleetctl to configure windows mdm profiles for teams and "no team" (#15161)
final step of #14361 , this integrates the work of the other PRs.

currently branched off `14361-windows-custom-settings-configs`

---------

Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
2023-11-15 18:04:24 -03:00
Roberto Dip
319b64f368
allow to set mdm.windows_settings.custom_settings in configs (#15145)
for #14361 this adds the bits related to saving a slice of strings with
paths to configuration profiles.

---------

Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
Co-authored-by: Marko Lisica <83164494+marko-lisica@users.noreply.github.com>
2023-11-15 13:58:46 -03:00
Roberto Dip
8478171256
add a cron job to deliver windows MDM profiles (#15065)
for #14364 , this implements the delivery mechanism for windows hosts.

I will follow up in another PR with logic to update the profile status
when we get responses from the device.
2023-11-10 11:05:10 -03:00
Martin Angers
b2ab4553cc
Validate that WSTEP is configured before enabling Windows MDM (#14858)
#14446 

~~Note that the fix requires a frontend change too, so this should not
be merged before the frontend is also ready.~~ Frontend
[PR](https://github.com/fleetdm/fleet/pull/14971) is ready.
2023-11-09 10:08:54 -03:00
Roberto Dip
33db665d63
show full formatted results for windows commands in fleetctl (#14922)
for #14912 this adds the full results to the "RESULTS" column of
`fleetctl get mdm-command-results`.

Additionally I included formatting of the XML output to improve
readability.
2023-11-03 12:01:43 -03:00
Roberto Dip
b04b20fcf0
fix fleetctl to set different CmdUID for Windows (#14895)
for #14891
2023-11-02 15:06:37 -03:00
Roberto Dip
9cf30a9131
Feat windows msmdm (#14837)
for #13069

---------

Co-authored-by: Marcos Oviedo <marcos@fleetdm.com>
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
Co-authored-by: gillespi314 <73313222+gillespi314@users.noreply.github.com>
2023-11-01 11:13:12 -03:00
guangwu
33858d7301
chore: remove refs to deprecated io/ioutil (#14485)
# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [ ] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- [ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)
- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [ ] Added/updated tests
- [ ] Manual QA for all new/changed functionality
  - For Orbit and Fleet Desktop changes:
- [ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).

Signed-off-by: guoguangwu <guoguangwu@magic-shield.com>
2023-10-27 15:28:54 -03:00
Jahziel Villasana-Espinoza
c570ef1e21
feat: make analytics always true for premium customers (#14607)
# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
2023-10-27 11:41:18 -04:00
Roberto Dip
5dfc72b2a1
allow to use --context in fleetctl mdm run-command (#14665)
for #13712
2023-10-26 19:09:31 -03:00
Lucas Manuel Rodriguez
fd99fd4121
Add optimizations for query reports at scale (#14710)
#7766

- ~[ ] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.~
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
  - ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~

---------

Co-authored-by: Jahziel Villasana-Espinoza <jahziel@fleetdm.com>
2023-10-25 19:20:27 -03:00
Lucas Manuel Rodriguez
5d7ee58a87
Do not leak ticks and fix race condition (#14667)
This PR introduces the following fixes:
- Fixes a race condition by protecting the `scheduledQueries` and
`scheduledQueryData` fields with a `sync.Mutex` (*).
- Adds some more information about request counts for /log, /config and
/distributed/read requests (and uptime).
- Fixes the resource leaks around creating time.Ticks on every for loop
iteration.

(*) Sample of the race condition when running with `-race`:
```
==================
WARNING: DATA RACE
Read at 0x00c000604800 by goroutine 20:
  main.(*agent).runLoop()
      /Users/luk/fleetdm/git/fleet/cmd/osquery-perf/agent.go:525 +0x71b
  main.main.func2()
      /Users/luk/fleetdm/git/fleet/cmd/osquery-perf/agent.go:1737 +0x4e

Previous write at 0x00c000604800 by goroutine 40:
  main.(*agent).config()
      /Users/luk/fleetdm/git/fleet/cmd/osquery-perf/agent.go:915 +0xb30
  main.(*agent).runLoop.func2()
      /Users/luk/fleetdm/git/fleet/cmd/osquery-perf/agent.go:512 +0x37
```
2023-10-20 10:29:59 -03:00
Lucas Manuel Rodriguez
22bba274b7
Run logger, distributed and config in separate goroutines (#14584) 2023-10-17 11:30:59 -03:00
Lucas Manuel Rodriguez
8c6cefc4f4
Fix panic in osquery-perf (#14534)
This is safe to merge during freeze as this is an osquery-perf only
change.
```
panic: runtime error: index out of range [1] with length 1

goroutine 14 [running]:
main.(*agent).config(0xc0001e5b80)
        /Users/luk/fleetdm/git/fleet/cmd/osquery-perf/agent.go:847 +0xb74
main.(*agent).runLoop(0xc0001e5b80, 0x0?, 0x0?)
        /Users/luk/fleetdm/git/fleet/cmd/osquery-perf/agent.go:463 +0x4d2
created by main.main in goroutine 1
        /Users/luk/fleetdm/git/fleet/cmd/osquery-perf/agent.go:1666 +0x1285
```
2023-10-13 14:26:46 -03:00
Jahziel Villasana-Espinoza
f92695fda5
osquery-perf: can send scheduled query results (#14502) 2023-10-12 20:41:04 -06:00
Jacob Shandling
70e0182e4d Merge branch 'main' into 7766-main-merge 2023-10-11 12:56:52 -07:00
Roberto Dip
9ffa11c25d
Feat: saved scripts (#14409)
For #9537
2023-10-10 19:00:45 -03:00
Jacob Shandling
fb494ca3cb Merge branch '7766-frontend' into 7766-backend-frontend 2023-10-10 13:21:19 -07:00
Jahziel Villasana-Espinoza
0435431ae9
feat: can delete via cron job if global flag set (#14398)
# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] Added/updated tests

---------

Co-authored-by: Lucas Rodriguez <lucas@fleetdm.com>
2023-10-10 10:06:54 -03:00
Tim Lee
fa55d72118
Save Query Results to DB (#14335)
# Checklist for submitter

#13486 

If some of the following don't apply, delete the relevant line.

- [ ] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- [ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality

---------

Co-authored-by: RachelElysia <71795832+RachelElysia@users.noreply.github.com>
Co-authored-by: Eric <eashaw@sailsjs.com>
Co-authored-by: Sampfluger88 <108141731+Sampfluger88@users.noreply.github.com>
Co-authored-by: Mike Thomas <78363703+mike-j-thomas@users.noreply.github.com>
Co-authored-by: Roberto Dip <me@roperzh.com>
Co-authored-by: Tyler Diderich <15862572+tdiderich@users.noreply.github.com>
Co-authored-by: Dave Herder <27025660+dherder@users.noreply.github.com>
Co-authored-by: Rachael Shaw <r@rachael.wtf>
Co-authored-by: Lucas Manuel Rodriguez <lucas@fleetdm.com>
Co-authored-by: Sharon Katz <121527325+sharon-fdm@users.noreply.github.com>
Co-authored-by: Mike McNeil <mikermcneil@users.noreply.github.com>
Co-authored-by: Nathanael Holliday <100959072+hollidayn@users.noreply.github.com>
Co-authored-by: Katheryn Satterlee <me@ksatter.com>
Co-authored-by: Mo Zhu <mo@fleetdm.com>
Co-authored-by: Mo Zhu <mozhu888@gmail.com>
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
Co-authored-by: Robert Fairburn <8029478+rfairburn@users.noreply.github.com>
Co-authored-by: Sabrina Coy <13890648+sabrinabuckets@users.noreply.github.com>
Co-authored-by: Isabell Reedy <113355639+ireedy@users.noreply.github.com>
Co-authored-by: Noah Talerman <47070608+noahtalerman@users.noreply.github.com>
Co-authored-by: Alex Mitchell <105945793+alexmitchelliii@users.noreply.github.com>
2023-10-10 09:44:03 -03:00
Jahziel Villasana-Espinoza
5c868c9d3d
feat: deletion for query results (#14302)
# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
2023-10-09 17:43:17 -04:00
Jacob Shandling
eb327faabf
maintenance merge of main into feature branch (#14393)
maintenance merge of `main` into feature branch
2023-10-09 14:28:35 -07:00
Marcos Oviedo
f0d77ab3db
Merging Bitlocker feature branch (#14350)
This relates to #12577

---------

Co-authored-by: gillespi314 <73313222+gillespi314@users.noreply.github.com>
Co-authored-by: Roberto Dip <dip.jesusr@gmail.com>
2023-10-06 19:04:33 -03:00
Tim Lee
ccd6746633
7766 backmerge main (#14324) 2023-10-05 11:39:48 -06:00
Tim Lee
4b50cf811b
Revert Backmerge from Main 2023-10-04 20:56:17 -06:00
Jacob Shandling
ab50f0f59d
Maintenance merge of main into feature branch (#14308) 2023-10-04 15:59:36 -07:00
Tim Lee
38af9678f4
Backmerge from main (#14298) 2023-10-04 14:24:40 -06:00
Noah Talerman
9c7079f50c
Windows MDM docs: turn on Windows MDM (#13943)
- Add new "Windows setup" page to "Device Management" section of docs
- Rename "MDM setup"  page to "macOS setup." Update links and add redirect
- Rename existing "macOS setup" page to "macOS setup experience." Update links. Did not add redirect because of conflict with "macOS setup" page
- Remove "MDM" from all MDM doc page titles
2023-10-04 15:39:09 -04:00
Lucas Manuel Rodriguez
71a237042a
Add continues to not crash with panics on connection errors (#14223)
I found a few panics in osquery-perf while working in load testing
#13287.

These panic happened due to e.g. network errors.
2023-10-04 09:10:16 -03:00
Robert Henry
81a6c76bed
fix typo in help message in serve.go (#14037) 2023-09-26 13:06:17 -03:00
Jacob Shandling
1993c02b3a
Move code for --local-wix-dir into ee directory (#14104) 2023-09-25 10:38:03 -07:00
Jacob Shandling
b6f072775f
Add ability for fleetctl package to use local WiX v3 binaries when generating installer .msi (#14033) 2023-09-22 08:49:01 -07:00
Tim Lee
e80ea50240
Flock vulnerability false positive (#13827) 2023-09-14 10:38:36 -06:00
Roberto Dip
541cc638e4
Add an activity log when an script execution is requested (#13655)
For #9553
2023-09-05 20:09:29 -03:00
gillespi314
37fb4b0dab
Add fleetctl run-script command (#13622) 2023-09-05 14:14:09 -05:00
Martin Angers
cbc3f32e9d
Adjust response payload, messages and validations for /scripts/run/* endpoints. (#13607) 2023-08-31 09:08:50 -05:00
Lucas Manuel Rodriguez
9142c5de79
Prevent thundering herd when applying large number of policies on large number of hosts (#13552)
#13527

(Adding @mna to double check the changes in the async implementation of
policy result storage)

This PR also adds the osquery-perf changes needed to define the count of
macOS and Windows hosts.

- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)~
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [X] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
  - ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~

Test with 80k hosts: 70k simulated macOS, 10k simulated Windows.
Apply Windows policies first, then apply macOS policies:
```
fleetctl apply -f ee/cis/win-10/cis-policy-queries.yml

# Leave running for some time

fleetctl apply -f ee/cis/macos-13/cis-policy-queries.yml
```

After applying CIS policies previous to these changes:
![Screenshot 2023-08-23 at 11 36
18](https://github.com/fleetdm/fleet/assets/2073526/72c1dc7d-e601-4248-be35-93c85b749f5d)

After applying these changes and applying the same policies:
![Screenshot 2023-08-28 at 15 42
57](https://github.com/fleetdm/fleet/assets/2073526/6b6d76b8-6acb-4893-a913-bf603a68f1a4)
2023-08-31 10:58:50 -03:00
Martin Angers
090b142c49
Implement script execution on the fleetd agent (disabled by default) (#13569) 2023-08-30 14:02:44 -04:00
Roberto Dip
b50e1939db
Allow to configure fleetd for script execution (#13564)
Related to #13310 and #13304 this adds two ways to enable script
execution in `fleetd` (the orbit component)

- By building a package with `--enable-scripts`
- By providing a setting via a configuration profile (macOS only)

Due to how the profile assignment works, this change automatically
updates the `com.fleetdm.fleetd.config` for hosts that already have the
profile installed.

> [!NOTE]
> Documentation is in
[#13577](https://github.com/fleetdm/fleet/pull/13577) to decouple
reviews.
2023-08-30 10:18:34 -03:00
Martin Angers
edf4a4d02f
Add script execution simulation to osquery-perf in preparation for load testing (part 3 of ticket) (#13456) 2023-08-23 18:31:47 -04:00
Tim Lee
3b61adf7a4
Add validation for policy specs (#13294) 2023-08-21 10:22:07 -06:00
Jacob Shandling
a7f1501054
Update fleetctl convert for schedulable queries (#13125) 2023-08-08 10:39:20 -07:00
Martin Angers
37ba43d404
Add fleetctl upgrade-packs command to migrate 2017 packs to queries (#13078) 2023-08-08 08:21:57 -04:00
Martin Angers
554e024f7b
Fix gitops access when using --dry-run with fleetctl apply (#13178) 2023-08-07 13:51:11 -04:00
Juan Fernandez
63c7c8a3ae
If no global queries, show proper message at the end (#13006)
If no global queries, show proper message at the end
2023-07-27 16:55:07 -04:00
Juan Fernandez
1dde10d5c3
Print inherited message if no queries found (#12998)
If the team has no queries, then fleetctl output should also show the number of inherited queries.
2023-07-27 15:08:40 -04:00
Juan Fernandez
e4cc0c3098
Fixed format issues with fleetctl get queries (#12983)
Output from `fleetctl get queries` should include the team the query is in and also the scheduling information.
2023-07-27 09:29:09 -04:00
Martin Angers
6f77911ffe
Fix performance regression found in load testing (#12981) 2023-07-26 17:13:27 -04:00
Lucas Manuel Rodriguez
2afbd24021
Combine Schedules and Queries: API changes (#12778)
Combining schedules and queries API changes.
2023-07-24 20:17:20 -04:00
Juan Fernandez
6b664a2a82
Merge branch 'main' into 7765-combined-schedules-and-queries 2023-07-21 13:53:13 -04:00
Lucas Manuel Rodriguez
b803bbe5bf
osquery-perf changes needed for load testing with simulated Windows hosts (#12754)
Changes in osquery-perf to allow for testing of Windows hosts in
loadtest environments.
2023-07-14 13:06:34 -03:00
Gabriel Hernandez
9aa7c0c714
add dark and light background logo colors and show them on mdm migrat… (#12681) 2023-07-13 19:35:25 +01:00
Roberto Dip
666ae8d787
ensure ds.TeamByName returns a 4xx response if no team is found (#12620)
this helps consumer of the datastore method handle the not found
scenario better and ensures we always return a 4xx code by default if we
can't find a matching team.

seems like calls to this method were special-cased everywhere except in
the apply user roles endpoint, where we returned a `500` status code if
we couldn't find a team.
2023-07-13 11:55:05 -03:00