608038a1bb
This fixes the deadlock reported in #14779. We found a deadlock in software ingestion during load tests performed in October: ``` 2023-10-26T17:20:41.719627Z 0 [Note] [MY-012468] [InnoDB] Transactions deadlock detected, dumping detailed information. (lock0lock.cc:6482) 2023-10-26T17:20:41.719661Z 0 [Note] [MY-012469] [InnoDB] *** (1) TRANSACTION: (lock0lock.cc:6496) TRANSACTION 3069866646, ACTIVE 0 sec starting index read mysql tables in use 2, locked 2 LOCK WAIT 8 lock struct(s), heap size 1136, 18 row lock(s), undo log entries 10 MySQL thread id 95, OS thread handle 70431326097136, query id 340045 10.12.3.105 fleet executing DELETE FROM software WHERE id IN (165, 79, 344, 47, 212, 21, 60, 127, 173, 145) AND NOT EXISTS ( SELECT 1 FROM host_software hsw WHERE hsw.software_id = software.id ) 2023-10-26T17:20:41.719700Z 0 [Note] [MY-012469] [InnoDB] *** (1) HOLDS THE LOCK(S): (lock0lock.cc:6496) RECORD LOCKS space id 932 page no 8 n bits 256 index PRIMARY of table `fleet`.`software` trx id 3069866646 lock_mode X locks rec but not gap Record lock, heap no 22 PHYSICAL RECORD: n_fields 11; compact format; info bits 0 0: len 8; hex 0000000000000015; asc ;; 1: len 6; hex 0000a74c4a7c; asc LJ|;; 2: len 7; hex 82000000d00264; asc d;; 3: len 26; hex 616e74692d76697275735f666f725f736f70686f735f686f6d65; asc anti-virus_for_sophos_home;; 4: len 5; hex 322e322e36; asc 2.2.6;; 5: len 4; hex 61707073; asc apps;; 6: len 0; hex ; asc ;; 7: len 0; hex ; asc ;; 8: len 0; hex ; asc ;; 9: len 0; hex ; asc ;; 10: len 0; hex ; asc ;; Record lock, heap no 48 PHYSICAL RECORD: n_fields 11; compact format; info bits 0 0: len 8; hex 000000000000002f; asc /;; 1: len 6; hex 0000a74c4aad; asc LJ ;; 2: len 7; hex 81000000e30220; asc ;; 3: len 10; hex 7265616c706c61796572; asc realplayer;; 4: len 11; hex 31322e302e312e31373338; asc 12.0.1.1738;; 5: len 4; hex 61707073; asc apps;; 6: len 0; hex ; asc ;; 7: len 0; hex ; asc ;; 8: len 0; hex ; asc ;; 9: len 0; hex ; asc ;; 10: len 0; hex ; asc ;; Record lock, heap no 61 PHYSICAL RECORD: n_fields 11; compact format; info bits 0 0: len 8; hex 000000000000003c; asc <;; 1: len 6; hex 0000a74c4afb; asc LJ ;; 2: len 7; hex 820000017501ba; asc u ;; 3: len 7; hex 636f6e6e656374; asc connect;; 4: len 5; hex 332e322e37; asc 3.2.7;; 5: len 4; hex 61707073; asc apps;; 6: len 0; hex ; asc ;; 7: len 0; hex ; asc ;; 8: len 0; hex ; asc ;; 9: len 0; hex ; asc ;; 10: len 0; hex ; asc ;; Record lock, heap no 80 PHYSICAL RECORD: n_fields 11; compact format; info bits 0 0: len 8; hex 000000000000004f; asc O;; 1: len 6; hex 0000a74c4b32; asc LK2;; 2: len 7; hex 820000008a01cb; asc ;; 3: len 7; hex 68697063686174; asc hipchat;; 4: len 4; hex 342e3330; asc 4.30;; 5: len 4; hex 61707073; asc apps;; 6: len 0; hex ; asc ;; 7: len 0; hex ; asc ;; 8: len 0; hex ; asc ;; 9: len 0; hex ; asc ;; 10: len 0; hex ; asc ;; 2023-10-26T17:20:41.720564Z 0 [Note] [MY-012469] [InnoDB] *** (1) WAITING FOR THIS LOCK TO BE GRANTED: (lock0lock.cc:6496) RECORD LOCKS space id 695 page no 5994 n bits 1000 index host_software_software_id_fk of table `fleet`.`host_software` trx id 3069866646 lock mode S waiting Record lock, heap no 31 PHYSICAL RECORD: n_fields 2; compact format; info bits 32 0: len 8; hex 000000000000004f; asc O;; 1: len 4; hex 0000000c; asc ;; 2023-10-26T17:20:41.720650Z 0 [Note] [MY-012469] [InnoDB] *** (2) TRANSACTION: (lock0lock.cc:6496) TRANSACTION 3069866680, ACTIVE 0 sec starting index read mysql tables in use 2, locked 2 LOCK WAIT 7 lock struct(s), heap size 1136, 12 row lock(s), undo log entries 8 MySQL thread id 98, OS thread handle 70375801900784, query id 340524 10.12.3.9 fleet executing DELETE FROM software WHERE id IN (49, 113, 183, 187, 223, 79, 81, 116) AND NOT EXISTS ( SELECT 1 FROM host_software hsw WHERE hsw.software_id = software.id ) 2023-10-26T17:20:41.720682Z 0 [Note] [MY-012469] [InnoDB] *** (2) HOLDS THE LOCK(S): (lock0lock.cc:6496) RECORD LOCKS space id 695 page no 5994 n bits 1000 index host_software_software_id_fk of table `fleet`.`host_software` trx id 3069866680 lock_mode X locks rec but not gap Record lock, heap no 31 PHYSICAL RECORD: n_fields 2; compact format; info bits 32 0: len 8; hex 000000000000004f; asc O;; 1: len 4; hex 0000000c; asc ;; 2023-10-26T17:20:41.720760Z 0 [Note] [MY-012469] [InnoDB] *** (2) WAITING FOR THIS LOCK TO BE GRANTED: (lock0lock.cc:6496) RECORD LOCKS space id 932 page no 8 n bits 256 index PRIMARY of table `fleet`.`software` trx id 3069866680 lock_mode X locks rec but not gap waiting Record lock, heap no 80 PHYSICAL RECORD: n_fields 11; compact format; info bits 0 0: len 8; hex 000000000000004f; asc O;; 1: len 6; hex 0000a74c4b32; asc LK2;; 2: len 7; hex 820000008a01cb; asc ;; 3: len 7; hex 68697063686174; asc hipchat;; 4: len 4; hex 342e3330; asc 4.30;; 5: len 4; hex 61707073; asc apps;; 6: len 0; hex ; asc ;; 7: len 0; hex ; asc ;; 8: len 0; hex ; asc ;; 9: len 0; hex ; asc ;; 10: len 0; hex ; asc ;; 2023-10-26T17:20:41.720984Z 0 [Note] [MY-012469] [InnoDB] *** WE ROLL BACK TRANSACTION (2) (lock0lock.cc:6496) ``` I was able to reproduce this issue on `main` with the added test. The solution is to remove the deletion (cleanup) of `software` to a separate transaction after the main transaction is done. - [X] Changes file added for user-visible changes in `changes/` or `orbit/changes/`. See [Changes files](https://fleetdm.com/docs/contributing/committing-changes#changes-files) for more information. - [X] Added/updated tests - [X] Manual QA for all new/changed functionality |
||
|---|---|---|
| .github | ||
| .storybook | ||
| .vscode | ||
| articles | ||
| assets | ||
| changes | ||
| charts | ||
| cmd | ||
| docs | ||
| ee | ||
| frontend | ||
| git-hooks | ||
| handbook | ||
| infrastructure | ||
| mdm_profiles | ||
| orbit | ||
| pkg | ||
| proposals | ||
| schema | ||
| scripts | ||
| server | ||
| terraform | ||
| test/upgrade | ||
| tools | ||
| website | ||
| .dockerignore | ||
| .eslintrc.js | ||
| .gitattributes | ||
| .gitignore | ||
| .gitpod.yml | ||
| .golangci.yml | ||
| .goreleaser-snapshot.yml | ||
| .goreleaser.yml | ||
| .npmignore | ||
| .prettierignore | ||
| .prettierrc.json | ||
| .trivyignore | ||
| babel.config.json | ||
| CHANGELOG.md | ||
| CODE_OF_CONDUCT.md | ||
| codecov.yml | ||
| CODEOWNERS | ||
| docker-compose-redis-cluster.yml | ||
| docker-compose.yml | ||
| Dockerfile | ||
| Dockerfile-desktop-linux | ||
| Dockerfile.osquery-perf | ||
| go.mod | ||
| go.sum | ||
| LICENSE | ||
| Makefile | ||
| manifest.yml.cloudgov.example | ||
| package.json | ||
| postcss.config.js | ||
| README.md | ||
| SECURITY.md | ||
| tools.go | ||
| tsconfig.json | ||
| webpack.config.js | ||
| yarn.lock | ||
News · Report a bug · Handbook · Why open source? · Art
Open-source platform for IT and security teams with thousands of computers. Designed for APIs, GitOps, webhooks, YAML, and humans.
What's it for?
Organizations like Fastly and Gusto use Fleet for vulnerability reporting, detection engineering, device management (MDM), device health monitoring, posture-based access control, managing unused software licenses, and more.
Explore data
To see what kind of data you can use Fleet to gather, check out the table reference documentation.
Out-of-the-box policies
Fleet includes out-of-the box support for all CIS benchmarks for macOS and Windows, as well as many simpler queries.
Take as much or as little as you need for your organization.
Supported platforms
Here are the platforms Fleet currently supports:
- Linux (all distros)
- macOS
- Windows
- Chromebooks
- Amazon Web Services (AWS)
- Google Cloud (GCP)
- Azure (Microsoft cloud)
- Data centers
- Containers (kube, etc)
- Linux-based IoT devices
Lighter than air
Fleet is lightweight and modular. You can use it for security without using it for MDM, and vice versa. You can turn off features you are not using.
Openness
Fleet is dedicated to flexibility, accessibility, and clarity. We think everyone can contribute and that tools should be as easy as possible for everyone to understand.
Good neighbors
Fleet has no ambition to replace all of your other tools. (Though it might replace some, if you want it to.) Ready-to-use, enterprise-friendly integrations exist for Snowflake, Splunk, GitHub Actions, Vanta, Elastic Jira, Zendesk, and more.
Fleet plays well with Munki, Chef, Puppet, and Ansible, as well as with security tools like Crowdstrike and SentinelOne. For example, you can use the free version of Fleet to quickly report on what hosts are actually running your EDR agent.
While most folks prefer to use one or the other, Fleet can also coexist peacefully with Rapid7 and other agent-based vulnerability scanners. This can be useful during migrations.
Free as in free
The free version of Fleet will always be free. Fleet is independently backed and actively maintained with the help of many amazing contributors.
Longevity
The company behind Fleet is founded (and majority-owned) by true believers in open source. The company's business model is influenced by GitLab (NYSE: GTLB), with great investors, happy customers, and the capacity to become profitable at any time.
In keeping with Fleet's value of openness, Fleet Device Management's company handbook is public and open source. You can read about the history of Fleet and osquery and our commitment to improving the product.
Is it any good?
Fleet is used in production by IT and security teams with thousands of laptops and servers. Many deployments support tens of thousands of hosts, and a few large organizations manage deployments as large as 400,000+ hosts.
Chat
Please join us in MacAdmins Slack or in osquery Slack.
The Fleet community is full of kind and helpful people. Whether or not you are a paying customer, if you need help, just ask.
Contributing 
The landscape of cybersecurity and IT is too complex. Let's open it up.
Contributions are welcome, whether you answer questions on Slack / GitHub / StackOverflow / LinkedIn / Twitter, improve the documentation or website, write a tutorial, give a talk at a conference or local meetup, give an interview on a podcast, troubleshoot reported issues, or submit a patch. The Fleet code of conduct is on GitHub.
What's next?
To see what Fleet can do, head over to fleetdm.com and try it out for yourself, grab time with one of the maintainers to discuss, or visit the docs and roll it out to your organization.
Production deployment
Fleet is simple enough to spin up for yourself. Or you can have us host it for you. Premium features are available either way.
Documentation
Complete documentation for Fleet can be found at https://fleetdm.com/docs.
License
The free version of Fleet is available under the MIT license. The commercial license is also designed to allow contributions to paid features for users whose employment agreements allow them to contribute to open source projects. (See LICENSE.md for details.)
Fleet is built on osquery, nanoMDM, Nudge, and swiftDialog.