empayre/fleet
14
0
Fork 0
mirror of https://github.com/empayre/fleet.git synced 2024-11-06 08:55:24 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
608038a1bb
fleet/changes
History
Lucas Manuel Rodriguez 608038a1bb
Fix deadlock when deleting software during data ingestion (#15459) 
This fixes the deadlock reported in #14779.

We found a deadlock in software ingestion during load tests performed in
October:
```
2023-10-26T17:20:41.719627Z 0 [Note] [MY-012468] [InnoDB] Transactions deadlock detected, dumping detailed information. (lock0lock.cc:6482)
2023-10-26T17:20:41.719661Z 0 [Note] [MY-012469] [InnoDB]  *** (1) TRANSACTION:  (lock0lock.cc:6496)
TRANSACTION 3069866646, ACTIVE 0 sec starting index read
mysql tables in use 2, locked 2
LOCK WAIT 8 lock struct(s), heap size 1136, 18 row lock(s), undo log entries 10
MySQL thread id 95, OS thread handle 70431326097136, query id 340045 10.12.3.105 fleet executing
DELETE FROM software WHERE id IN (165, 79, 344, 47, 212, 21, 60, 127, 173, 145) AND
        NOT EXISTS (
                SELECT 1 FROM host_software hsw WHERE hsw.software_id = software.id
        )
2023-10-26T17:20:41.719700Z 0 [Note] [MY-012469] [InnoDB]  *** (1) HOLDS THE LOCK(S):  (lock0lock.cc:6496)
RECORD LOCKS space id 932 page no 8 n bits 256 index PRIMARY of table `fleet`.`software` trx id 3069866646 lock_mode X locks rec but not gap
Record lock, heap no 22 PHYSICAL RECORD: n_fields 11; compact format; info bits 0
 0: len 8; hex 0000000000000015; asc         ;;
 1: len 6; hex 0000a74c4a7c; asc    LJ|;;
 2: len 7; hex 82000000d00264; asc       d;;
 3: len 26; hex 616e74692d76697275735f666f725f736f70686f735f686f6d65; asc anti-virus_for_sophos_home;;
 4: len 5; hex 322e322e36; asc 2.2.6;;
 5: len 4; hex 61707073; asc apps;;
 6: len 0; hex ; asc ;;
 7: len 0; hex ; asc ;;
 8: len 0; hex ; asc ;;
 9: len 0; hex ; asc ;;
 10: len 0; hex ; asc ;;

Record lock, heap no 48 PHYSICAL RECORD: n_fields 11; compact format; info bits 0
 0: len 8; hex 000000000000002f; asc        /;;
 1: len 6; hex 0000a74c4aad; asc    LJ ;;
 2: len 7; hex 81000000e30220; asc        ;;
 3: len 10; hex 7265616c706c61796572; asc realplayer;;
 4: len 11; hex 31322e302e312e31373338; asc 12.0.1.1738;;
 5: len 4; hex 61707073; asc apps;;
6: len 0; hex ; asc ;;
 7: len 0; hex ; asc ;;
 8: len 0; hex ; asc ;;
 9: len 0; hex ; asc ;;
 10: len 0; hex ; asc ;;

Record lock, heap no 61 PHYSICAL RECORD: n_fields 11; compact format; info bits 0
 0: len 8; hex 000000000000003c; asc        <;;
 1: len 6; hex 0000a74c4afb; asc    LJ ;;
 2: len 7; hex 820000017501ba; asc     u  ;;
 3: len 7; hex 636f6e6e656374; asc connect;;
 4: len 5; hex 332e322e37; asc 3.2.7;;
 5: len 4; hex 61707073; asc apps;;
 6: len 0; hex ; asc ;;
 7: len 0; hex ; asc ;;
 8: len 0; hex ; asc ;;
 9: len 0; hex ; asc ;;
 10: len 0; hex ; asc ;;

Record lock, heap no 80 PHYSICAL RECORD: n_fields 11; compact format; info bits 0
 0: len 8; hex 000000000000004f; asc        O;;
 1: len 6; hex 0000a74c4b32; asc    LK2;;
 2: len 7; hex 820000008a01cb; asc        ;;
 3: len 7; hex 68697063686174; asc hipchat;;
 4: len 4; hex 342e3330; asc 4.30;;
 5: len 4; hex 61707073; asc apps;;
 6: len 0; hex ; asc ;;
 7: len 0; hex ; asc ;;
 8: len 0; hex ; asc ;;
 9: len 0; hex ; asc ;;
 10: len 0; hex ; asc ;;

2023-10-26T17:20:41.720564Z 0 [Note] [MY-012469] [InnoDB]  *** (1) WAITING FOR THIS LOCK TO BE GRANTED:  (lock0lock.cc:6496)
RECORD LOCKS space id 695 page no 5994 n bits 1000 index host_software_software_id_fk of table `fleet`.`host_software` trx id 3069866646 lock mode S waiting
Record lock, heap no 31 PHYSICAL RECORD: n_fields 2; compact format; info bits 32
 0: len 8; hex 000000000000004f; asc        O;;
 1: len 4; hex 0000000c; asc     ;;

2023-10-26T17:20:41.720650Z 0 [Note] [MY-012469] [InnoDB]  *** (2) TRANSACTION:  (lock0lock.cc:6496)
TRANSACTION 3069866680, ACTIVE 0 sec starting index read
mysql tables in use 2, locked 2
LOCK WAIT 7 lock struct(s), heap size 1136, 12 row lock(s), undo log entries 8
MySQL thread id 98, OS thread handle 70375801900784, query id 340524 10.12.3.9 fleet executing
DELETE FROM software WHERE id IN (49, 113, 183, 187, 223, 79, 81, 116) AND
        NOT EXISTS (
                SELECT 1 FROM host_software hsw WHERE hsw.software_id = software.id
        )
2023-10-26T17:20:41.720682Z 0 [Note] [MY-012469] [InnoDB]  *** (2) HOLDS THE LOCK(S):  (lock0lock.cc:6496)
RECORD LOCKS space id 695 page no 5994 n bits 1000 index host_software_software_id_fk of table `fleet`.`host_software` trx id 3069866680 lock_mode X locks rec but not gap
Record lock, heap no 31 PHYSICAL RECORD: n_fields 2; compact format; info bits 32
 0: len 8; hex 000000000000004f; asc        O;;
 1: len 4; hex 0000000c; asc     ;;

2023-10-26T17:20:41.720760Z 0 [Note] [MY-012469] [InnoDB]  *** (2) WAITING FOR THIS LOCK TO BE GRANTED:  (lock0lock.cc:6496)
RECORD LOCKS space id 932 page no 8 n bits 256 index PRIMARY of table `fleet`.`software` trx id 3069866680 lock_mode X locks rec but not gap waiting
Record lock, heap no 80 PHYSICAL RECORD: n_fields 11; compact format; info bits 0
 0: len 8; hex 000000000000004f; asc        O;;
 1: len 6; hex 0000a74c4b32; asc    LK2;;
 2: len 7; hex 820000008a01cb; asc        ;;
 3: len 7; hex 68697063686174; asc hipchat;;
 4: len 4; hex 342e3330; asc 4.30;;
 5: len 4; hex 61707073; asc apps;;
 6: len 0; hex ; asc ;;
 7: len 0; hex ; asc ;;
 8: len 0; hex ; asc ;;
 9: len 0; hex ; asc ;;
 10: len 0; hex ; asc ;;

2023-10-26T17:20:41.720984Z 0 [Note] [MY-012469] [InnoDB] *** WE ROLL BACK TRANSACTION (2)  (lock0lock.cc:6496)
```

I was able to reproduce this issue on `main` with the added test. The
solution is to remove the deletion (cleanup) of `software` to a separate
transaction after the main transaction is done.

- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
2023-12-07 09:34:53 -03:00
..
.keep Issue 1009 calculate diff software (#1305) 2021-07-08 13:57:43 -03:00
2118-browser-extension_id Adding extension_id and browser columns to software table. (#15285) 2023-11-30 19:06:17 -06:00
12037-fix-dep-reenroll-activity Fix installed_from_dep value in activity for DEP re-enrolling (#15419) 2023-12-04 17:14:16 -03:00
14484-whitespace-in-policies UI – Show line breaks in Policy details modals (#15354) 2023-12-01 09:11:36 -08:00
14748-host-by-identifier-serial-number Add hardware serial to host by identifier search (#15353) 2023-11-29 08:39:35 -06:00
14779-fix-host_batteries-deadlock Fix deadlock when replacing (upserting) host_batteries (#15447) 2023-12-05 18:24:58 -03:00
14779-fix-software-deadlock Fix deadlock when deleting software during data ingestion (#15459) 2023-12-07 09:34:53 -03:00
14851-table-reload fix: prevent table from rerendering when opening modals (#15375) 2023-11-30 17:21:35 -05:00
14920-device-health feat: device health endpoint (#15432) 2023-12-06 14:42:29 -05:00
14970-line-break-in-example-command UI – Fix bug with line breaks in example command (#15331) 2023-12-01 09:11:56 -08:00
15133-screen-mdm-profiles-reserved-names Add validations to disallow custom MDM profiles that contain names reserved by Fleet (#15373) 2023-11-30 17:19:18 -06:00
15168-scheduled-outside-fleet Fix osquery result logging when queries are configured outside of Fleet (#15393) 2023-12-04 11:18:49 -03:00
15229-list-software-versions Add GET software/versions and GET software/versions/:id endpoints (#15450) 2023-12-06 08:30:49 -06:00
15362-windows-mdm-query improve mdm_windows query to account for multiple registry entries (#15391) 2023-12-04 11:03:05 -03:00
issue-13281-backend-changes feature: windows profiles (#15349) 2023-11-29 09:32:42 -05:00
issue-14027-implement-windows-os-updates Implement Windows OS Updates (feature branch). (#15359) 2023-11-29 11:07:24 -05:00
issue-14028-support-windows-os-updates Implement Windows OS Updates (feature branch). (#15359) 2023-11-29 11:07:24 -05:00
issue-14029-apply-windows-os-updates Implement Windows OS Updates (feature branch). (#15359) 2023-11-29 11:07:24 -05:00
issue-14045-add-windows-update-activites Implement Windows OS Updates (feature branch). (#15359) 2023-11-29 11:07:24 -05:00
issue-14090-add-copied-message add copied message to copy button on input (#15366) 2023-11-29 17:27:01 +00:00
issue-14359-windows-profiles feature: windows profiles (#15349) 2023-11-29 09:32:42 -05:00
issue-14441-add-truncation-query-docs add truncation to dropdown options for the query table documentation sidebar (#15368) 2023-11-29 18:02:51 +00:00
issue-14714-fix-cached-appconfig-race Fix edge case of AppConfig changes getting lost in cached mysql. (#15352) 2023-11-29 10:09:37 -05:00
issue-14979-improve-safety-checks-for-cached-items Require a custom Clone for cacheable items, add tooling and CI check to help catch issues (#15458) 2023-12-06 14:34:22 -05:00
issue-15274-unify-profiles-uuid Unify profiles database identifier to profile_uuid for macOS and Windows profiles. (#15297) 2023-12-04 10:04:06 -05:00
issue-15345-filter-hosts-by-software Allow filtering hosts by software_version_id and software_title_id. (#15433) 2023-12-06 14:59:00 -05:00
issue-15406-fleetctl-get-software Update fleetctl get software to list titles and versions. (#15444) 2023-12-06 16:07:03 -05:00