for https://github.com/fleetdm/fleet/issues/15196. The main problem was
that we have two timeouts:
1. The timeout used by the host to kill the script execution
2. The timeout used by the server to wait for the script results
Before the changes in https://github.com/fleetdm/fleet/pull/15779, the
server timeout was longer than the host timeout, but we inadvertently
set both values to 5 minutes, which breaks the logic we have to handle
both kinds of timeouts.
#15196 This is the work of @ghernandez345 except for adding the
`ResponseController` thing in Go to override the server timeout for that
specific sync endpoint so that the calls don't timeout waiting for a
script response (the default HTTP server timeout was 90s for our
server).
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [x] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [x] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
---------
Co-authored-by: Gabriel Hernandez <ghernandez345@gmail.com>
Co-authored-by: Roberto Dip <me@roperzh.com>
Co-authored-by: Roberto Dip <dip.jesusr@gmail.com>
> 📜 Related issue: https://github.com/fleetdm/fleet/issues/14500
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
#15556
We will need to pay attention when releasing fleet (the github actions
were modified to use the local file now).
Should be reviewed by commits (first commit is the actual adding of the
`version.go` file)
- [X] Manual QA for all new/changed functionality
Manually tested the following:
- `Settings -> My account` on the UI and checked the `/version` endpoint
response. (Or also visiting https://localhost:8080/version on a
browser).
- Ran `make fleetctl fleet`, `./build/fleetctl --version` and
`./build/fleet version`.
Added `gigs_total_disk_space` to host endpoint responses.
- Now user will know host disk size even when disk is full
(percent_disk_space_available==0)
#15058
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
📺 Looom:
https://www.loom.com/share/1aec4616fa4449e7abac579084aef0ba?sid=0884f742-feb3-48bb-82dc-b7834bc9a6e1
Fixed fleetctl issue where it was creating a new query when running a
query by name, as opposed to using the existing saved query.
#15630
API change will be in a separate PR:
https://github.com/fleetdm/fleet/pull/15673
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
#14879
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [x] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [x] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
📺 Loom explaining the PR:
https://www.loom.com/share/8c6ec0c362014910931f183d68307525?sid=7de51e6f-a59d-4d8f-b06b-f703f1692f17#467
Live Queries now collect stats. Stats are collected for saved queries
present in the Queries tab.
- After running a live query, user will see updated stats in Queries
tab.
- Query stats on Host Details page will no longer be cleared after
host/agent reboots.
- Query stats are now deleted when query is deleted.
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
found during the load test, this addresses a performance issue with
software titles.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
https://github.com/fleetdm/confidential/issues/2118
API doc changes are at: https://github.com/fleetdm/fleet/pull/15092
This change affects Chrome-based and Firefox browsers. I tried Safari,
but it appears that latest Safari extensions simply show up as regular
apps on macOS.
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
- Tested detail query changes on macOS, Windows, Ubuntu, and ChromeOS
- Tested Chrome, Edge, Brave, and Firefox browsers
Testing notes:
1. Install some browser extensions.
2. Update fleet to this change.
3. Refresh hosts, and trigger vulnerabilities job (which updates host
software).
4. Do something like `GET
https://localhost:8080/api/v1/fleet/software?query=Password%20Manager`
to see `extension_id` and `browser` for the hosts that have updated.
for #14426.
In order to prevent import cycles and be able to use some type
definitions/constants I followed the same pattern we did for macOS by
creating a new package named `syncml`. This makes the changelog look
bigger than it actually is, so I split it into two commits to make it
easier to review:
-
[d7c233d](d7c233d54c)
moves the relevant bits to this new package
-
[7531a07](7531a0742b)
implements profile verification
#14888
@getvictor This is ready for review, but keeping as draft as there are
probably many tests that need amending.
I used the new version of the `./tools/nvd/nvdvuln/nvdvuln.go` to
compare the current vulnerabilities found in our dogfood environment
with the vulnerabilities found by the code in this PR and both results
match:
```
go run -race -tags fts5 ./tools/nvd/nvdvuln/nvdvuln.go --debug --db_dir ./local --software_from_url <dogfood URL> --software_from_api_token <API_TOKEN> --sync 2>&1 | tee out.txt
[...]
CVEs found and expected matched!
```
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Added/updated tests
- [X] Manual QA for all new/changed functionality
---------
Co-authored-by: Victor Lyuboslavsky <victor@fleetdm.com>
Co-authored-by: Victor Lyuboslavsky <victor.lyuboslavsky@gmail.com>
Loom explaining changes (hit 5 min limit):
https://www.loom.com/share/e59b63bf638e4d9cad7984ef589b878d?sid=111fff75-115a-4a44-ae4f-6f25fede0d51#14887
- [x] Need to merge fleetdm/nvd PR
https://github.com/fleetdm/nvd/pull/25 before this one.
# Checklist for submitter
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
- Manually tested (with corresponding fleetdm/fleet changes) in my
personal fork: https://github.com/getvictor/nvd/releases
# QA Plan (must be done before merging this PR, and after merging the
nvd PR)
- [ ] Fork https://github.com/fleetdm/nvd and point `generate.yml` to
this branch.
[example](9d8e54930b/.github/workflows/generate.yml (L26))
- [ ] Add NVD_API_KEY to nvd secrets, and run the the nvd generate
GitHub action. Get key:
https://nvd.nist.gov/developers/request-an-api-key
- [ ] Compare the generated `cpe-###.sqlite.gz` to the previous one. One
way is to open it up with sqlite3 and `select * from cpe_2 order by
cpe23;` and dump results to a CSV file. Known differences are:
- New file has ~2,500 more records
- Backslashes are handled differently for `Backpack\CRUD` and `Philips
In.Sight B120\37` products -- not a new issue since we do not support
those products right now
- `cpe:2.3🅰️moodle:moodle:4.2.0:*:*:*:*:*:*:*` -- this appears OK.
Also, it is a PHP plugin, and we don't support these currently.
- [ ] Record the existing vulnerabilities of current hosts.
- [ ] Stop any running fleet server. Delete `/tmp/vulndbs/cpe.sqlite`.
Can also delete other files there, or not delete this file -- it should
be overwritten by the new file. Also delete all rows in software_cpe and
software_cve DB tables. (Or can just spin up a fresh fleet server with
fresh DB, and re-enroll hosts (after setting the new env variable
below))
- [ ] Find the path to the generated `cpe-###.sqlite.gz` file
- [ ] Set `FLEET_VULNERABILITIES_CPE_DATABASE_URL` environment variable
to the above path, and start fleet server.
- [ ] After server's vulnerabilities cron job runs, the new
vulnerabilities should match the previous vulnerabilities
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Documented any API changes (docs/REST API/rest-api.md or
docs/Contributing/API-for-contributors.md)
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Rachael Shaw <r@rachael.wtf>
final step of #14361 , this integrates the work of the other PRs.
currently branched off `14361-windows-custom-settings-configs`
---------
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
for #14361 this adds the bits related to saving a slice of strings with
paths to configuration profiles.
---------
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
Co-authored-by: Marko Lisica <83164494+marko-lisica@users.noreply.github.com>
for #14364 , this implements the delivery mechanism for windows hosts.
I will follow up in another PR with logic to update the profile status
when we get responses from the device.
for #14912 this adds the full results to the "RESULTS" column of
`fleetctl get mdm-command-results`.
Additionally I included formatting of the XML output to improve
readability.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [ ] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- [ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)
- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [ ] Added/updated tests
- [ ] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
Signed-off-by: guoguangwu <guoguangwu@magic-shield.com>
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
#7766
- ~[ ] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.~
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
---------
Co-authored-by: Jahziel Villasana-Espinoza <jahziel@fleetdm.com>
This PR introduces the following fixes:
- Fixes a race condition by protecting the `scheduledQueries` and
`scheduledQueryData` fields with a `sync.Mutex` (*).
- Adds some more information about request counts for /log, /config and
/distributed/read requests (and uptime).
- Fixes the resource leaks around creating time.Ticks on every for loop
iteration.
(*) Sample of the race condition when running with `-race`:
```
==================
WARNING: DATA RACE
Read at 0x00c000604800 by goroutine 20:
main.(*agent).runLoop()
/Users/luk/fleetdm/git/fleet/cmd/osquery-perf/agent.go:525 +0x71b
main.main.func2()
/Users/luk/fleetdm/git/fleet/cmd/osquery-perf/agent.go:1737 +0x4e
Previous write at 0x00c000604800 by goroutine 40:
main.(*agent).config()
/Users/luk/fleetdm/git/fleet/cmd/osquery-perf/agent.go:915 +0xb30
main.(*agent).runLoop.func2()
/Users/luk/fleetdm/git/fleet/cmd/osquery-perf/agent.go:512 +0x37
```
This is safe to merge during freeze as this is an osquery-perf only
change.
```
panic: runtime error: index out of range [1] with length 1
goroutine 14 [running]:
main.(*agent).config(0xc0001e5b80)
/Users/luk/fleetdm/git/fleet/cmd/osquery-perf/agent.go:847 +0xb74
main.(*agent).runLoop(0xc0001e5b80, 0x0?, 0x0?)
/Users/luk/fleetdm/git/fleet/cmd/osquery-perf/agent.go:463 +0x4d2
created by main.main in goroutine 1
/Users/luk/fleetdm/git/fleet/cmd/osquery-perf/agent.go:1666 +0x1285
```
