Configuration and fixes for the Fleet server and frontend to add support
for https://github.com/Uptycs/kubequery.
Co-authored-by: Michal Nicpon <michal@fleetdm.com>
* Bump go to 1.19.1
* Bump remaining go-version to the 1.19.1
* Add extra paths for test-go
* Oops, putting the right path in the right place
* gofmt file
* gofmt ALL THE THINGS
* Moar changes
* Actually, go.mod doesn't like minor versions
* Updated testing.md
Updated the name of testing.md to testing-and-local-development.md based off https://github.com/fleetdm/fleet/issues/4706
* add redirect for /docs/contributing/testing
Co-authored-by: Eric <eashaw@sailsjs.com>
This improves the installerstore CLI tool with:
- The ability to create tests buckets for local development (otherwise you have to interact with another CLI or the MinIO UI)
- Improved error handling and messaging.
Related to #6365 this adds a new tool to upload pre-built Orbit installers to a storage blob. It uses the same file conventions that the Fleet server expects, making it useful for local testing and infra envs alike.\
Usage and details in the README
Related to #6364 and #6363, this:
- Adds a new Docker image, `fleetdm/fleetctl` equipped with all necessary dependencies to build Fleet-osquery binaries for all platforms
- Modifies the package generation logic to special case this scenario via an environment variable `FLEETCTL_NATIVE_TOOLING`
- Adds a new GitHub workflow to test this
There are more details in the README, but part of the special-casing logic is in place to output the binaries to a folder named `build` when they are run with `FLEETCTL_NATIVE_TOOLING`, this is so we can persist the binary generated by the docker container via a bind mount:
```bash
docker run -v "$(pwd):/build" fleetdm/fleetctl package --type=msi
```
To test this changeset, I have generated packages for all platforms, both via the new Docker image and via the classic `fleetctl package`.
* fix old root dir in orbit
* add changes
* Add automation for orbit shell (with TUF)
* Fix workflow syntax
* Add logging to latest fleetctl preview action
* Add changes to fix workflow
* Use macOS host for TUF server and package generation
* Remove copy/paste if clause
* Fix orbit logs on macOS, Ubuntu
* Simplify TUF and generation of packages
* Set enroll secret instead of getting it
* Increase timeouts
* Add step id
* Fixes to the upload/download of artifacts
* Rearrange steps to not lose the downloads
* Fix copy/paste
* Add fleetctl login step
* Add missing config set
* Fix quotes on Windows
* Increase timeout
* Fix job termination
* Disable FLEET_DESKTOP for now
* Checkout repository on macOS
* Fix logs path
* Enable fleet desktop
* Use cancel, nitpick
Co-authored-by: Michal Nicpon <michal@fleetdm.com>
* Prepare TUF scripts for CI and support different dev setups
* Allow only generating selected platforms for pkgs
* Force linux/amd64 on Dockerfile to support M1
* Add (beta) support for Fleet Desktop to linux
* Add dependency for linux desktop
* Amend makefile uname check
* Clarify env vars used for linux in execuser
* Add final set of fixes
* Remove -it from docker run
* Add desktop to the update runner for Linux
* Re-arrange tag.gz and fix upgrade check for linux desktop
* remove outdated config file description
`example_config.json` was removed in f11da7b05b but the documentation
was still there, this removes the description from the README as well.
* add a note about how to use scripts with premium features
* Reorganized infrastructure, updated for frontend's loadtesting
* Add changes suggested by @chiiph
* Moved files per suggestion by Ben
* Update docs with new links
* Add config for multi account assume role
* Make Orbit update interval configurable
- Also increase default interval from 10s to 15m
* Add update-interval configuration to fleetctl package (#5050)
Co-authored-by: Lucas Manuel Rodriguez <lucas@fleetdm.com>
not set on the INSERT.
- OUT: Only sets the ID on the passed session and returns it. (`CreatedAt`, `AccessedAt`, are not set.)
New version:
```go
func (ds *Datastore) NewSession(ctx context.Context, userID uint, sessionKey string) (*fleet.Session, error) {
sqlStatement := `
INSERT INTO sessions (
user_id,
` + "`key`" + `
)
VALUES(?,?)
`
result, err := ds.writer.ExecContext(ctx, sqlStatement, userID, sessionKey)
if err != nil {
return nil, ctxerr.Wrap(ctx, err, "inserting session")
}
id, _ := result.LastInsertId() // cannot fail with the mysql driver
return ds.sessionByID(ctx, ds.writer, uint(id))
}
```
- IN: Define arguments that are truly used when creating a session.
- OUT: Load and return the fleet.Session struct with all values set (using the `ds.writer` to support read replicas correctly).
PS: The new `NewSession` version mimics what we already do with other entities, like policies (`Datastore.NewGlobalPolicy`).
* Orbit: Add Fleet Desktop support to Windows
* Rename workflow, fix linux build
* Do not compile systray on linux
* nolint on unused
* Fix lint properly
* nolint both checkers
* Fix monitor logic in desktopRunner
* Fix interrupt and execute order
- Update CHANGELOG
- Bump versioning
- Tweak documentation
- Default `session_duration` to `5d`
- Add extra `#` to "Team policies" section so it doesn't show up in top level nav for docs
* WIP
* WIP2
* Fix orbit and fleetctl tests
* Amend macos-app default
* Add some fixes
* Use fleetctl updates roots command
* Add more fixes to Updater
* Fixes to app publishing and downloading
* Add more changes to support fleetctl cross generation
* Amend comment
* Add pkg generation to ease testing
* Make more fixes
* Add changes entry
* Add legacy targets (until our TUF system exposes the new app)
* Fix fleetctl preview
* Fix bool flag
* Fix orbit logic for disabled-updates and dev-mode
* Fix TestPreview
* Remove constant and fix zip-slip attack (codeql)
* Return unknown error
* Fix updater's checkExec
* Add support for executable signing in init_tuf.sh
* Try only signing orbit
* Fix init_tuf.sh targets, macos-app only for osqueryd
* Specify GOARCH to support M1s
* Add workflow to generate osqueryd.app.tar.gz
* Use 5.2.2 on init_tuf.sh
* Add unit test for tar.gz target
* Use artifacts instead of releases
* Remove copy paste residue
* Fleet Desktop Packaging WIP
* Ignore gosec warning
* Trigger on PR too
* Install Go in workflow
* Pass url parameter to desktop app
* Fix fleetctl package
* Final set of changes for v1 of Fleet Desktop
* Add changes
* PR fixes
* Fix CI build
* add larger menu bar icon
* Add transparency item
* Delete host_device_auth entry on host deletion
* Add SetTargetChannel
* Update white logo and add desktop to update runner
* Add fleet-desktop monitoring to orbit
* Define fleet-desktop app exec name
* Fix update runner creation
* Add API test before enabling the My device menu item
Co-authored-by: Zach Wasserman <zach@fleetdm.com>
* WIP
* WIP2
* Fix orbit and fleetctl tests
* Amend macos-app default
* Add some fixes
* Use fleetctl updates roots command
* Add more fixes to Updater
* Fixes to app publishing and downloading
* Add more changes to support fleetctl cross generation
* Amend comment
* Add pkg generation to ease testing
* Make more fixes
* Add changes entry
* Add legacy targets (until our TUF system exposes the new app)
* Fix fleetctl preview
* Fix bool flag
* Fix orbit logic for disabled-updates and dev-mode
* Fix TestPreview
* Remove constant and fix zip-slip attack (codeql)
* Return unknown error
* Fix updater's checkExec
* Add support for executable signing in init_tuf.sh
* Try only signing orbit
* Fix init_tuf.sh targets, macos-app only for osqueryd
* Specify GOARCH to support M1s
* Add workflow to generate osqueryd.app.tar.gz
* Use 5.2.2 on init_tuf.sh
* Add unit test for tar.gz target
* Use artifacts instead of releases
* Remove copy paste residue
* Trigger workflow on PR
* Fixes to ease handling of artifact
* Fix, do not use target name as dir
* Remove workaround
* Renaming files and a lot of find and replace
* pageRank meta tags, sorting by page rank
* reranking
* removing numbers
* revert changing links that are locked to a commit
* update metatag name, uncomment github contributers
* Update basic-documentation.page.js
* revert link change
* more explicit errors, change pageOrderInSection numbers, updated sort
* Update build-static-content.js
* update comment
* update handbook link
* handbook entry
* update sort
* update changelog doc links to use fleetdm.com
* move standard query library back to old location, update links/references to location
* revert unintentional link changes
* Update handbook/community.md
Co-authored-by: Mike Thomas <78363703+mike-j-thomas@users.noreply.github.com>
Co-authored-by: Mike Thomas <78363703+mike-j-thomas@users.noreply.github.com>
Co-authored-by: Mike McNeil <mikermcneil@users.noreply.github.com>
* Wait for osquery extension manager socket to be ready
* PR review feedback
* Add WIP Github Action to test orbit
* Set fleet address on fleetctl
* Add logging for troubleshooting
* Add prepare db statement
* Use tls-skip-verify on fleetctl
* Move steps around
* Fix addresses
* Fix fleetctl path
* Add certs.pem to orbit root dir
* Add orbit logs
* Increase timeout
* Add proper orbit log paths
* Fix tmp path orbit
* Add get hosts command to troubleshoot
* Fix orbit job termination
* Add comments to workflows
* Wait for server to go down
* Add orbit Windows job
* Use bash on windws-latest
* Fix missing quote
* Run orbit on Ubuntu
* Bump host count
* Increase timeout for extensions and not terminate on err
* Add comment to clarify high timeout value
* Revert change, we do want to exit in case of err
* Add terraform for the loadtesting environment
* Add some checkov stuff and minor updates
* Remove defunct commented stuff
* Add separate cloudwatch namespace for migrations
* Remove defunct code
* checkin
* Better jitter (#3716) (#3744)
* Better jitter (#3716)
* Better jitter
* Fix lint
* Use milliseconds
* Make duration milliseconds
* Update based on Lucas' suggestion
* Add changes file
* Panic on error
* Fix compilation error
* Use time.after instead of time.tick to not leak
* Include browser extensions in software inventory (#3733) (#3757)
Use appropriate JOINs against users table to include all results.
For #3557
* Build Docker images on push to any branch (#3756) (#3758)
Allows for testing release branches and any others.
* Optimize users detail query (#3754) (#3770)
@Smjert reported instances of Windows Domain Controllers having massive
resource utilization and being killed by the watchdog when running this
query. In his test environment, this new query performs much better.
* Refactor async host processing to avoid redis SCAN keys (for policies) (#3657) (#3755)
* Issue 3707 clean targets on delete (#3739) (#3776)
* wip
* Delete targets when deleting teams, hosts, and labels
* Add changes file
* Fix error message
* Remove unused teamsTable
* Cleanup new pack
* Clean new packs at end of test
* Update operating system policy (#3779)
- Update policy's query to check for operating system versions greater than or equal to
* Improve loading of manage policies page (#3695) (#3781)
Co-authored-by: gillespi314 <73313222+gillespi314@users.noreply.github.com>
* Fix logo (#3765)
* fixed duplicate query modal not closing (#3787)
* fixed style for connection line on activity feed (#3789)
* Improve loading state for query platform compatibility (#3752) (#3783)
Co-authored-by: gillespi314 <73313222+gillespi314@users.noreply.github.com>
* Disable pagination for teams page (#3794)
* Shorten "simple" query API period to 25s (#3775) (#3804)
This helps the period stay under the default request timeouts for most
load balancers.
Some default timeouts:
* AWS ALB - 60s
* Nginx - 60s
* GCP LB - 30s
* Clear user checkbox selections after successfully removing selected row from table UI #3798 (#3801)
* Alias fleetctl sandbox to fleetctl preview (#3803) (#3806)
Preparing for some work the product team is planning to rebrand preview
to sandbox.
* Remove flash of spinner to no spinner (#3799)
* Revert "Better jitter (#3716) (#3744)" (#3807)
This reverts commit c857b52d2b373ed9e302d643e6d2212af7353408.
* Fix teams dropdown sizing (#3759) (#3810)
Co-authored-by: gillespi314 <73313222+gillespi314@users.noreply.github.com>
* fixed link for expiration message (#3788)
* Fixed empty string for compatible platforms (#3812)
* fixed empty string for compatible platforms
* default checked for platforms on policy query
* Block autocomplete for username and password in SMTP settings (#3732) (#3816)
* No hosts for software/policy renders better message (#3701) (#3821)
Co-authored-by: Martavis Parker <47053705+martavis@users.noreply.github.com>
* Get failing policies webhook data on page load (#3824)
* Fix reset password flow (#3826)
* Prepare for 4.9.0 (#3820)
- Add CHANGELOG
- Bump versioning
- Remove changes files
- Spelling fix for operating system policy's description
* Remove merge artifact
Co-authored-by: Zach Wasserman <zach@fleetdm.com>
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
Co-authored-by: Noah Talerman <47070608+noahtalerman@users.noreply.github.com>
Co-authored-by: Luke Heath <luke@fleetdm.com>
Co-authored-by: gillespi314 <73313222+gillespi314@users.noreply.github.com>
Co-authored-by: RachelElysia <71795832+RachelElysia@users.noreply.github.com>
Co-authored-by: Martavis Parker <47053705+martavis@users.noreply.github.com>
Added reference architectures using https://docs.gitlab.com/ee/administration/reference_architectures/ as inspiration.
- updated terraform based on some feedback of usage
- pinned fleet docker version in terraform as to no get unexpected upgrades when applying
- updated some documentation around apply migration tasks
The low default ulimit `nofiles` value (`4096`) in Fargate was observed
to cause errors when running with a large number of hosts and a small
number of servers. Each server should be able to server more than 4096
simultaneous clients.
- Emphasize generating an "osquery installer" (also referred as "Orbit") using the `fleetctl package` command
- Add instructions for adding multiple hosts and automatically adding hosts to a team
- Remove instructions for generating an enrollment package with the tooling in `tools/mac/`
- Remove the `tools/mac/` directory from the repository because it is no longer used or referenced in any documentation
- Update "Automatically adding hosts to a team" section of "Teams" documentation to point to "Adding hosts" documentation
- Add instructions for migrating from plain osquery to Fleet's osquery installers
* refactor to allow bootstrapping* move monitoring into its own package, update readme
* add variable for license_key
* replication lag alarm less sensitive
