> Related issue: #16411
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] If database migrations are included, checked table schema to
confirm autoupdate
- For database migrations:
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Manual QA for all new/changed functionality
This is the feature branch for the [queued
scripts](https://github.com/fleetdm/fleet/issues/15529) story.
---------
Co-authored-by: Jahziel Villasana-Espinoza <jahziel@fleetdm.com>
Co-authored-by: Gabriel Hernandez <ghernandez345@gmail.com>
Co-authored-by: Sarah Gillespie <73313222+gillespi314@users.noreply.github.com>
Co-authored-by: Roberto Dip <dip.jesusr@gmail.com>
for #14715
---------
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
Co-authored-by: Sarah Gillespie <73313222+gillespi314@users.noreply.github.com>
Co-authored-by: Jahziel Villasana-Espinoza <jahziel@fleetdm.com>
Teams can configure their own host expiry setting. If global host expiry
is enabled, teams cannot disable host expiry, but they can set a longer
(or shorter) expiry time (in days).
- Added `host_expiry_settings` to team spec, which can be used via
fleetctl apply.
- Added `host_expiry_settings` to PATH /fleet/teams/:id endpoint.
#15609 (parent)
#15966 (subtask)
TODO: Create PR for API docs change: added parameter to `PATCH
/fleet/teams/:id` endpoint
# Checklist for submitter
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
On My device, Host details pages, and `GET /api/v1/fleet/hosts/:id` API
endpoint, failing policies are now listed first.
#10379
REST API change to be documented in a separate PR.
# Checklist for submitter
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
#15557
Following the precedent that Lucas used for other similar PRs, the best
way to review is probably by commits.
* The first one simply copies over the files from the fork to the
monorepo
* Second one adjusts all import paths
* Third one tidies up the `go.mod` files
* Last one fixes the linter issues in the nanomdm package
# Checklist for submitter
- ~~Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.~~ (not a user-visible change)
- [x] Manual QA for all new/changed functionality (ran test suite,
re-generated mocks)
I also verified that our Go test suite did run the newly moved `nanomdm`
package steps:
```
ok github.com/fleetdm/fleet/v4/server/mdm/nanomdm/cryptoutil 0.003s coverage: 0.0% of statements in github.com/fleetdm/fleet/v4/...
ok github.com/fleetdm/fleet/v4/server/mdm/nanomdm/mdm 0.005s coverage: 46.2% of statements in github.com/fleetdm/fleet/v4/...
ok github.com/fleetdm/fleet/v4/server/mdm/nanomdm/service/certauth 1.320s coverage: 20.7% of statements in github.com/fleetdm/fleet/v4/...
ok github.com/fleetdm/fleet/v4/server/mdm/nanomdm/storage/file 0.007s coverage: 24.1% of statements in github.com/fleetdm/fleet/v4/...
```
#15236
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Manual QA for all new/changed functionality
> 📜 Related issue: https://github.com/fleetdm/fleet/issues/14500
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
Added a new synchronous live query endpoint: POST
/api/v1/fleet/queries/:id/run
#14800
All relevant integration tests have been updated to work with the old
endpoint and new endpoint.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
#15556
We will need to pay attention when releasing fleet (the github actions
were modified to use the local file now).
Should be reviewed by commits (first commit is the actual adding of the
`version.go` file)
- [X] Manual QA for all new/changed functionality
Manually tested the following:
- `Settings -> My account` on the UI and checked the `/version` endpoint
response. (Or also visiting https://localhost:8080/version on a
browser).
- Ran `make fleetctl fleet`, `./build/fleetctl --version` and
`./build/fleet version`.
#15555
Probably the best way to review this is commit by commit:
- First commit does the actual moving.
- Second commit fixes golangci-lint issues (in the least effort way to
avoid refactoring or rearrangement of some of the code).
- Third commit moves a printf to before the migration step is executed.
In the past some customers hitting migration issues (like migration
steps hanging or taking long to execute) and wanted to know which one
was it. The only way to know was to look at the repository and looking
for the next migration after the last one logged.
Checks:
- [X] Manual QA for all new/changed functionality
Manual tests:
- `make fleet && make db-reset`.
- Adding a new migration via `make migration name=Foobar` and then
running `./build/fleet prepare db`.
- Enrolling a new device to Fleet (smoke test).
> 📜 Related issue: https://github.com/fleetdm/fleet/issues/15522
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
Fixed missing 'Last restarted' values when filtering hosts by label.
#15822
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
Fixed badly formatted error messages in /api/fleet/orbit/device_token
endpoint and others.
In /api/fleet/orbit/device_token:
- Added token validation -- empty token not allowed
- Replaced 500 error with 409 when token conflicts with another host
#15832
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA
Added `host_count_updated_at` to policy API responses.
#15323
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
When deleting queries, associated query stats and aggregated stats are
deleted with separate SQL statements. When testing, we need to wait in a
for loop for both of these to be deleted before continuing with the
test.
Added `gigs_total_disk_space` to host endpoint responses.
- Now user will know host disk size even when disk is full
(percent_disk_space_available==0)
#15058
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
# Checklist for submitter
To test, go through the repro steps in the issue; you should see that
the field is true instead of false!
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
Live query stats are cleared when query SQL is modified.
Also, when deleting one or more queries, the associated aggregated stats
are now deleted as well.
#15709
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
📺 Looom:
https://www.loom.com/share/1aec4616fa4449e7abac579084aef0ba?sid=0884f742-feb3-48bb-82dc-b7834bc9a6e1
Fixed fleetctl issue where it was creating a new query when running a
query by name, as opposed to using the existing saved query.
#15630
API change will be in a separate PR:
https://github.com/fleetdm/fleet/pull/15673
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
#14879
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [x] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [x] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
📺 Loom explaining the PR:
https://www.loom.com/share/8c6ec0c362014910931f183d68307525?sid=7de51e6f-a59d-4d8f-b06b-f703f1692f17#467
Live Queries now collect stats. Stats are collected for saved queries
present in the Queries tab.
- After running a live query, user will see updated stats in Queries
tab.
- Query stats on Host Details page will no longer be cleared after
host/agent reboots.
- Query stats are now deleted when query is deleted.
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
found during the load test, this addresses a performance issue with
software titles.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
#15472
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
Issue #13034
TODO: Frontend requirements will be covered in a separate PR.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
- [x] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
This fixes the deadlock reported in #14779.
We found a deadlock in software ingestion during load tests performed in
October:
```
2023-10-26T17:20:41.719627Z 0 [Note] [MY-012468] [InnoDB] Transactions deadlock detected, dumping detailed information. (lock0lock.cc:6482)
2023-10-26T17:20:41.719661Z 0 [Note] [MY-012469] [InnoDB] *** (1) TRANSACTION: (lock0lock.cc:6496)
TRANSACTION 3069866646, ACTIVE 0 sec starting index read
mysql tables in use 2, locked 2
LOCK WAIT 8 lock struct(s), heap size 1136, 18 row lock(s), undo log entries 10
MySQL thread id 95, OS thread handle 70431326097136, query id 340045 10.12.3.105 fleet executing
DELETE FROM software WHERE id IN (165, 79, 344, 47, 212, 21, 60, 127, 173, 145) AND
NOT EXISTS (
SELECT 1 FROM host_software hsw WHERE hsw.software_id = software.id
)
2023-10-26T17:20:41.719700Z 0 [Note] [MY-012469] [InnoDB] *** (1) HOLDS THE LOCK(S): (lock0lock.cc:6496)
RECORD LOCKS space id 932 page no 8 n bits 256 index PRIMARY of table `fleet`.`software` trx id 3069866646 lock_mode X locks rec but not gap
Record lock, heap no 22 PHYSICAL RECORD: n_fields 11; compact format; info bits 0
0: len 8; hex 0000000000000015; asc ;;
1: len 6; hex 0000a74c4a7c; asc LJ|;;
2: len 7; hex 82000000d00264; asc d;;
3: len 26; hex 616e74692d76697275735f666f725f736f70686f735f686f6d65; asc anti-virus_for_sophos_home;;
4: len 5; hex 322e322e36; asc 2.2.6;;
5: len 4; hex 61707073; asc apps;;
6: len 0; hex ; asc ;;
7: len 0; hex ; asc ;;
8: len 0; hex ; asc ;;
9: len 0; hex ; asc ;;
10: len 0; hex ; asc ;;
Record lock, heap no 48 PHYSICAL RECORD: n_fields 11; compact format; info bits 0
0: len 8; hex 000000000000002f; asc /;;
1: len 6; hex 0000a74c4aad; asc LJ ;;
2: len 7; hex 81000000e30220; asc ;;
3: len 10; hex 7265616c706c61796572; asc realplayer;;
4: len 11; hex 31322e302e312e31373338; asc 12.0.1.1738;;
5: len 4; hex 61707073; asc apps;;
6: len 0; hex ; asc ;;
7: len 0; hex ; asc ;;
8: len 0; hex ; asc ;;
9: len 0; hex ; asc ;;
10: len 0; hex ; asc ;;
Record lock, heap no 61 PHYSICAL RECORD: n_fields 11; compact format; info bits 0
0: len 8; hex 000000000000003c; asc <;;
1: len 6; hex 0000a74c4afb; asc LJ ;;
2: len 7; hex 820000017501ba; asc u ;;
3: len 7; hex 636f6e6e656374; asc connect;;
4: len 5; hex 332e322e37; asc 3.2.7;;
5: len 4; hex 61707073; asc apps;;
6: len 0; hex ; asc ;;
7: len 0; hex ; asc ;;
8: len 0; hex ; asc ;;
9: len 0; hex ; asc ;;
10: len 0; hex ; asc ;;
Record lock, heap no 80 PHYSICAL RECORD: n_fields 11; compact format; info bits 0
0: len 8; hex 000000000000004f; asc O;;
1: len 6; hex 0000a74c4b32; asc LK2;;
2: len 7; hex 820000008a01cb; asc ;;
3: len 7; hex 68697063686174; asc hipchat;;
4: len 4; hex 342e3330; asc 4.30;;
5: len 4; hex 61707073; asc apps;;
6: len 0; hex ; asc ;;
7: len 0; hex ; asc ;;
8: len 0; hex ; asc ;;
9: len 0; hex ; asc ;;
10: len 0; hex ; asc ;;
2023-10-26T17:20:41.720564Z 0 [Note] [MY-012469] [InnoDB] *** (1) WAITING FOR THIS LOCK TO BE GRANTED: (lock0lock.cc:6496)
RECORD LOCKS space id 695 page no 5994 n bits 1000 index host_software_software_id_fk of table `fleet`.`host_software` trx id 3069866646 lock mode S waiting
Record lock, heap no 31 PHYSICAL RECORD: n_fields 2; compact format; info bits 32
0: len 8; hex 000000000000004f; asc O;;
1: len 4; hex 0000000c; asc ;;
2023-10-26T17:20:41.720650Z 0 [Note] [MY-012469] [InnoDB] *** (2) TRANSACTION: (lock0lock.cc:6496)
TRANSACTION 3069866680, ACTIVE 0 sec starting index read
mysql tables in use 2, locked 2
LOCK WAIT 7 lock struct(s), heap size 1136, 12 row lock(s), undo log entries 8
MySQL thread id 98, OS thread handle 70375801900784, query id 340524 10.12.3.9 fleet executing
DELETE FROM software WHERE id IN (49, 113, 183, 187, 223, 79, 81, 116) AND
NOT EXISTS (
SELECT 1 FROM host_software hsw WHERE hsw.software_id = software.id
)
2023-10-26T17:20:41.720682Z 0 [Note] [MY-012469] [InnoDB] *** (2) HOLDS THE LOCK(S): (lock0lock.cc:6496)
RECORD LOCKS space id 695 page no 5994 n bits 1000 index host_software_software_id_fk of table `fleet`.`host_software` trx id 3069866680 lock_mode X locks rec but not gap
Record lock, heap no 31 PHYSICAL RECORD: n_fields 2; compact format; info bits 32
0: len 8; hex 000000000000004f; asc O;;
1: len 4; hex 0000000c; asc ;;
2023-10-26T17:20:41.720760Z 0 [Note] [MY-012469] [InnoDB] *** (2) WAITING FOR THIS LOCK TO BE GRANTED: (lock0lock.cc:6496)
RECORD LOCKS space id 932 page no 8 n bits 256 index PRIMARY of table `fleet`.`software` trx id 3069866680 lock_mode X locks rec but not gap waiting
Record lock, heap no 80 PHYSICAL RECORD: n_fields 11; compact format; info bits 0
0: len 8; hex 000000000000004f; asc O;;
1: len 6; hex 0000a74c4b32; asc LK2;;
2: len 7; hex 820000008a01cb; asc ;;
3: len 7; hex 68697063686174; asc hipchat;;
4: len 4; hex 342e3330; asc 4.30;;
5: len 4; hex 61707073; asc apps;;
6: len 0; hex ; asc ;;
7: len 0; hex ; asc ;;
8: len 0; hex ; asc ;;
9: len 0; hex ; asc ;;
10: len 0; hex ; asc ;;
2023-10-26T17:20:41.720984Z 0 [Note] [MY-012469] [InnoDB] *** WE ROLL BACK TRANSACTION (2) (lock0lock.cc:6496)
```
I was able to reproduce this issue on `main` with the added test. The
solution is to remove the deletion (cleanup) of `software` to a separate
transaction after the main transaction is done.
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
> #14920
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Documented any permissions changes (docs/Using
Fleet/manage-access.md)
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
#14779
This PR fixes the deadlock when upserting to `host_batteries`.
Which probably happens because InnoDB uses row-locking.
I was able to reproduce in main with the new test
`TestHosts/ReplaceHostBatteriesDeadlock`.
I refactored `ds.ReplaceHostBatteries` to use the same upsert pattern as
`ds.ReplaceHostDeviceMapping` (given `battery` is assumed to return just
a few rows per host). With such pattern the tests does not fail with
deadlock errors anymore.
Here are some of the techniques MySQL recommends:
https://dev.mysql.com/doc/refman/5.7/en/innodb-deadlocks-handling.html
Basically by changing the upsert pattern the deadlock goes away (It's
hard to know exactly why the original code deadlocks).
Here's the deadlock trace from load test performed in October:
```
2023-10-26T17:19:17.244707Z 0 [Note] [MY-012468] [InnoDB] Transactions deadlock detected, dumping detailed information. (lock0lock.cc:6482)
2023-10-26T17:19:17.244756Z 0 [Note] [MY-012469] [InnoDB] *** (1) TRANSACTION: (lock0lock.cc:6496)
TRANSACTION 3069771944, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
LOCK WAIT 7 lock struct(s), heap size 1136, 5 row lock(s), undo log entries 1
MySQL thread id 75, OS thread handle 70369297350384, query id 658 10.12.3.201 fleet update
INSERT INTO
host_batteries (
host_id,
serial_number,
cycle_count,
health
)
VALUES
(27472, '0000', 505, 'Good'),(27472, '0001', 730, 'Good')
ON DUPLICATE KEY UPDATE
cycle_count = VALUES(cycle_count),
health = VALUES(health),
updated_at = CURRENT_TIMESTAMP
2023-10-26T17:19:17.244800Z 0 [Note] [MY-012469] [InnoDB] *** (1) HOLDS THE LOCK(S): (lock0lock.cc:6496)
RECORD LOCKS space id 867 page no 320 n bits 280 index PRIMARY of table `fleet`.`host_batteries` trx id 3069771944 lock_mode X locks gap before rec
Record lock, heap no 205 PHYSICAL RECORD: n_fields 9; compact format; info bits 0
0: len 4; hex 00526996; asc Ri ;;
1: len 6; hex 0000b6f900d0; asc ;;
2: len 7; hex 82000033370110; asc 37 ;;
3: len 4; hex 0000d829; asc );;
4: len 4; hex 30303030; asc 0000;;
5: len 4; hex 8000065b; asc [;;
6: len 4; hex 506f6f72; asc Poor;;
7: len 4; hex 653a9f95; asc e: ;;
8: len 4; hex 653a9f95; asc e: ;;
2023-10-26T17:19:17.245027Z 0 [Note] [MY-012469] [InnoDB] *** (1) WAITING FOR THIS LOCK TO BE GRANTED: (lock0lock.cc:6496)
RECORD LOCKS space id 867 page no 320 n bits 280 index PRIMARY of table `fleet`.`host_batteries` trx id 3069771944 lock_mode X locks gap before rec insert intention waiting
Record lock, heap no 205 PHYSICAL RECORD: n_fields 9; compact format; info bits 0
0: len 4; hex 00526996; asc Ri ;;
1: len 6; hex 0000b6f900d0; asc ;;
2: len 7; hex 82000033370110; asc 37 ;;
3: len 4; hex 0000d829; asc );;
4: len 4; hex 30303030; asc 0000;;
5: len 4; hex 8000065b; asc [;;
6: len 4; hex 506f6f72; asc Poor;;
7: len 4; hex 653a9f95; asc e: ;;
2023-10-26T17:19:17.245239Z 0 [Note] [MY-012469] [InnoDB] *** (2) TRANSACTION: (lock0lock.cc:6496)
TRANSACTION 3069771958, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
LOCK WAIT 7 lock struct(s), heap size 1136, 5 row lock(s), undo log entries 1
MySQL thread id 9, OS thread handle 70369296809712, query id 708 10.12.2.156 fleet update
INSERT INTO
host_batteries (
host_id,
serial_number,
cycle_count,
health
)
VALUES
(59161, '0000', 1384, 'Fair'),(59161, '0001', 396, 'Good')
ON DUPLICATE KEY UPDATE
cycle_count = VALUES(cycle_count),
health = VALUES(health),
updated_at = CURRENT_TIMESTAMP
2023-10-26T17:19:17.245272Z 0 [Note] [MY-012469] [InnoDB] *** (2) HOLDS THE LOCK(S): (lock0lock.cc:6496)
RECORD LOCKS space id 867 page no 320 n bits 280 index PRIMARY of table `fleet`.`host_batteries` trx id 3069771958 lock_mode X locks gap before rec
Record lock, heap no 205 PHYSICAL RECORD: n_fields 9; compact format; info bits 0
0: len 4; hex 00526996; asc Ri ;;
1: len 6; hex 0000b6f900d0; asc ;;
2: len 7; hex 82000033370110; asc 37 ;;
3: len 4; hex 0000d829; asc );;
4: len 4; hex 30303030; asc 0000;;
5: len 4; hex 8000065b; asc [;;
6: len 4; hex 506f6f72; asc Poor;;
7: len 4; hex 653a9f95; asc e: ;;
8: len 4; hex 653a9f95; asc e: ;;
2023-10-26T17:19:17.245504Z 0 [Note] [MY-012469] [InnoDB] *** (2) WAITING FOR THIS LOCK TO BE GRANTED: (lock0lock.cc:6496)
RECORD LOCKS space id 867 page no 320 n bits 280 index PRIMARY of table `fleet`.`host_batteries` trx id 3069771958 lock_mode X locks gap before rec insert intention waiting
Record lock, heap no 205 PHYSICAL RECORD: n_fields 9; compact format; info bits 0
0: len 4; hex 00526996; asc Ri ;;
1: len 6; hex 0000b6f900d0; asc ;;
2: len 7; hex 82000033370110; asc 37 ;;
3: len 4; hex 0000d829; asc );;
4: len 4; hex 30303030; asc 0000;;
5: len 4; hex 8000065b; asc [;;
6: len 4; hex 506f6f72; asc Poor;;
7: len 4; hex 653a9f95; asc e: ;;
8: len 4; hex 653a9f95; asc e: ;;
2023-10-26T17:19:17.245730Z 0 [Note] [MY-012469] [InnoDB] *** WE ROLL BACK TRANSACTION (2) (lock0lock.cc:6496)
```
- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
https://github.com/fleetdm/confidential/issues/2118
API doc changes are at: https://github.com/fleetdm/fleet/pull/15092
This change affects Chrome-based and Firefox browsers. I tried Safari,
but it appears that latest Safari extensions simply show up as regular
apps on macOS.
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
- Tested detail query changes on macOS, Windows, Ubuntu, and ChromeOS
- Tested Chrome, Edge, Brave, and Firefox browsers
Testing notes:
1. Install some browser extensions.
2. Update fleet to this change.
3. Refresh hosts, and trigger vulnerabilities job (which updates host
software).
4. Do something like `GET
https://localhost:8080/api/v1/fleet/software?query=Password%20Manager`
to see `extension_id` and `browser` for the hosts that have updated.
for #14426.
In order to prevent import cycles and be able to use some type
definitions/constants I followed the same pattern we did for macOS by
creating a new package named `syncml`. This makes the changelog look
bigger than it actually is, so I split it into two commits to make it
easier to review:
-
[d7c233d](d7c233d54c)
moves the relevant bits to this new package
-
[7531a07](7531a0742b)
implements profile verification
for #15068
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
#15037
For endpoint fleet/hosts, filtering hosts with invalid team_id now
returns 400 error.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
When Munki is deleted and reinstalled on the host, Fleet will show Munki
info again.
#12409
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
Query report now shows the host display name instead of hostname.
#14763
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Documented any API changes (docs/REST API/rest-api.md or
docs/Contributing/API-for-contributors.md)
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Rachael Shaw <r@rachael.wtf>
for #14361 this adds the bits related to saving a slice of strings with
paths to configuration profiles.
---------
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
Co-authored-by: Marko Lisica <83164494+marko-lisica@users.noreply.github.com>
migrations are static and not meant to change, so these tests have
successfully accomplished their purpose.
the main motivation is to prevent CI from timming out, as each migration
test runs all prior migrations from scratch, making these very time
expensive.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
<!-- Note that API documentation changes are now addressed by the
product design team. -->
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
for #14364 , this implements the delivery mechanism for windows hosts.
I will follow up in another PR with logic to update the profile status
when we get responses from the device.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
Loom explaining changes:
https://www.loom.com/share/f05f241a77304c19bc6ba1d0702c7bd8?sid=ea86b282-0bda-4ba4-a6cf-4520f0db610d#14571
Enabled support and validation of 'after' parameter for the following
endpoints:
- GET /api/v1/fleet/carves
Setting 'after' parameter no longer returns SQL syntax error for the
following endpoints:
- GET /api/v1/fleet/carves
- GET /api/v1/fleet/invites
- GET /api/v1/fleet/labels
- GET /api/v1/fleet/packs
- GET /api/v1/fleet/global/policies
- GET /api/v1/fleet/teams/{id}/policies
- GET /api/v1/fleet/queries
- GET /api/v1/fleet/packs/{id}/scheduled
- GET /api/v1/fleet/teams
- GET /api/v1/fleet/users
API doc changes PR: https://github.com/fleetdm/fleet/pull/15061
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
---------
Co-authored-by: Rachael Shaw <r@rachael.wtf>
for #14912 this adds the full results to the "RESULTS" column of
`fleetctl get mdm-command-results`.
Additionally I included formatting of the XML output to improve
readability.
#14554
For the following endpoints:
/api/v1/fleet/software
/api/v1/fleet/software/count
- added validation on `page`, `per_page`, `order_key`, `order_direction`
-- invalid values will now return 400 HTTP status code
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [ ] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- [ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)
- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.
- [ ] Added/updated tests
- [ ] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- [ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).
Signed-off-by: guoguangwu <guoguangwu@magic-shield.com>
#7766
- ~[ ] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.~
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
---------
Co-authored-by: Jahziel Villasana-Espinoza <jahziel@fleetdm.com>
for #13703 and #13992, this updates the logic used by the functions that
gather hosts that need DEP profile updates to use hosts directly from
`host_dep_assignments`, regardless of their MDM status.
#14551
- ~[ ] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.~
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [x] Manual QA for all new/changed functionality
- ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] Added/updated tests
---------
Co-authored-by: Lucas Rodriguez <lucas@fleetdm.com>
# Checklist for submitter
#13486
If some of the following don't apply, delete the relevant line.
- [ ] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- [ ] Documented any permissions changes (docs/Using
Fleet/manage-access.md)
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
---------
Co-authored-by: RachelElysia <71795832+RachelElysia@users.noreply.github.com>
Co-authored-by: Eric <eashaw@sailsjs.com>
Co-authored-by: Sampfluger88 <108141731+Sampfluger88@users.noreply.github.com>
Co-authored-by: Mike Thomas <78363703+mike-j-thomas@users.noreply.github.com>
Co-authored-by: Roberto Dip <me@roperzh.com>
Co-authored-by: Tyler Diderich <15862572+tdiderich@users.noreply.github.com>
Co-authored-by: Dave Herder <27025660+dherder@users.noreply.github.com>
Co-authored-by: Rachael Shaw <r@rachael.wtf>
Co-authored-by: Lucas Manuel Rodriguez <lucas@fleetdm.com>
Co-authored-by: Sharon Katz <121527325+sharon-fdm@users.noreply.github.com>
Co-authored-by: Mike McNeil <mikermcneil@users.noreply.github.com>
Co-authored-by: Nathanael Holliday <100959072+hollidayn@users.noreply.github.com>
Co-authored-by: Katheryn Satterlee <me@ksatter.com>
Co-authored-by: Mo Zhu <mo@fleetdm.com>
Co-authored-by: Mo Zhu <mozhu888@gmail.com>
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
Co-authored-by: Robert Fairburn <8029478+rfairburn@users.noreply.github.com>
Co-authored-by: Sabrina Coy <13890648+sabrinabuckets@users.noreply.github.com>
Co-authored-by: Isabell Reedy <113355639+ireedy@users.noreply.github.com>
Co-authored-by: Noah Talerman <47070608+noahtalerman@users.noreply.github.com>
Co-authored-by: Alex Mitchell <105945793+alexmitchelliii@users.noreply.github.com>
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
# Checklist for submitter
The implementation #13488 was wrong; we needed to have query results
discarded by default, so customers shouldn't notice anything when they
upgrade.
If some of the following don't apply, delete the relevant line.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- fix: use function that returns params
- fix: missing integration test
fixes: #13809
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
