Configuration and fixes for the Fleet server and frontend to add support
for https://github.com/Uptycs/kubequery.
Co-authored-by: Michal Nicpon <michal@fleetdm.com>
* Bump go to 1.19.1
* Bump remaining go-version to the 1.19.1
* Add extra paths for test-go
* Oops, putting the right path in the right place
* gofmt file
* gofmt ALL THE THINGS
* Moar changes
* Actually, go.mod doesn't like minor versions
* Updated testing.md
Updated the name of testing.md to testing-and-local-development.md based off https://github.com/fleetdm/fleet/issues/4706
* add redirect for /docs/contributing/testing
Co-authored-by: Eric <eashaw@sailsjs.com>
This improves the installerstore CLI tool with:
- The ability to create tests buckets for local development (otherwise you have to interact with another CLI or the MinIO UI)
- Improved error handling and messaging.
Related to #6365 this adds a new tool to upload pre-built Orbit installers to a storage blob. It uses the same file conventions that the Fleet server expects, making it useful for local testing and infra envs alike.\
Usage and details in the README
Related to #6364 and #6363, this:
- Adds a new Docker image, `fleetdm/fleetctl` equipped with all necessary dependencies to build Fleet-osquery binaries for all platforms
- Modifies the package generation logic to special case this scenario via an environment variable `FLEETCTL_NATIVE_TOOLING`
- Adds a new GitHub workflow to test this
There are more details in the README, but part of the special-casing logic is in place to output the binaries to a folder named `build` when they are run with `FLEETCTL_NATIVE_TOOLING`, this is so we can persist the binary generated by the docker container via a bind mount:
```bash
docker run -v "$(pwd):/build" fleetdm/fleetctl package --type=msi
```
To test this changeset, I have generated packages for all platforms, both via the new Docker image and via the classic `fleetctl package`.
* fix old root dir in orbit
* add changes
* Add automation for orbit shell (with TUF)
* Fix workflow syntax
* Add logging to latest fleetctl preview action
* Add changes to fix workflow
* Use macOS host for TUF server and package generation
* Remove copy/paste if clause
* Fix orbit logs on macOS, Ubuntu
* Simplify TUF and generation of packages
* Set enroll secret instead of getting it
* Increase timeouts
* Add step id
* Fixes to the upload/download of artifacts
* Rearrange steps to not lose the downloads
* Fix copy/paste
* Add fleetctl login step
* Add missing config set
* Fix quotes on Windows
* Increase timeout
* Fix job termination
* Disable FLEET_DESKTOP for now
* Checkout repository on macOS
* Fix logs path
* Enable fleet desktop
* Use cancel, nitpick
Co-authored-by: Michal Nicpon <michal@fleetdm.com>
* Prepare TUF scripts for CI and support different dev setups
* Allow only generating selected platforms for pkgs
* Force linux/amd64 on Dockerfile to support M1
* Add (beta) support for Fleet Desktop to linux
* Add dependency for linux desktop
* Amend makefile uname check
* Clarify env vars used for linux in execuser
* Add final set of fixes
* Remove -it from docker run
* Add desktop to the update runner for Linux
* Re-arrange tag.gz and fix upgrade check for linux desktop
* remove outdated config file description
`example_config.json` was removed in f11da7b05b but the documentation
was still there, this removes the description from the README as well.
* add a note about how to use scripts with premium features
* Reorganized infrastructure, updated for frontend's loadtesting
* Add changes suggested by @chiiph
* Moved files per suggestion by Ben
* Update docs with new links
* Add config for multi account assume role
* Make Orbit update interval configurable
- Also increase default interval from 10s to 15m
* Add update-interval configuration to fleetctl package (#5050)
Co-authored-by: Lucas Manuel Rodriguez <lucas@fleetdm.com>
not set on the INSERT.
- OUT: Only sets the ID on the passed session and returns it. (`CreatedAt`, `AccessedAt`, are not set.)
New version:
```go
func (ds *Datastore) NewSession(ctx context.Context, userID uint, sessionKey string) (*fleet.Session, error) {
sqlStatement := `
INSERT INTO sessions (
user_id,
` + "`key`" + `
)
VALUES(?,?)
`
result, err := ds.writer.ExecContext(ctx, sqlStatement, userID, sessionKey)
if err != nil {
return nil, ctxerr.Wrap(ctx, err, "inserting session")
}
id, _ := result.LastInsertId() // cannot fail with the mysql driver
return ds.sessionByID(ctx, ds.writer, uint(id))
}
```
- IN: Define arguments that are truly used when creating a session.
- OUT: Load and return the fleet.Session struct with all values set (using the `ds.writer` to support read replicas correctly).
PS: The new `NewSession` version mimics what we already do with other entities, like policies (`Datastore.NewGlobalPolicy`).
* Orbit: Add Fleet Desktop support to Windows
* Rename workflow, fix linux build
* Do not compile systray on linux
* nolint on unused
* Fix lint properly
* nolint both checkers
* Fix monitor logic in desktopRunner
* Fix interrupt and execute order
- Update CHANGELOG
- Bump versioning
- Tweak documentation
- Default `session_duration` to `5d`
- Add extra `#` to "Team policies" section so it doesn't show up in top level nav for docs
* WIP
* WIP2
* Fix orbit and fleetctl tests
* Amend macos-app default
* Add some fixes
* Use fleetctl updates roots command
* Add more fixes to Updater
* Fixes to app publishing and downloading
* Add more changes to support fleetctl cross generation
* Amend comment
* Add pkg generation to ease testing
* Make more fixes
* Add changes entry
* Add legacy targets (until our TUF system exposes the new app)
* Fix fleetctl preview
* Fix bool flag
* Fix orbit logic for disabled-updates and dev-mode
* Fix TestPreview
* Remove constant and fix zip-slip attack (codeql)
* Return unknown error
* Fix updater's checkExec
* Add support for executable signing in init_tuf.sh
* Try only signing orbit
* Fix init_tuf.sh targets, macos-app only for osqueryd
* Specify GOARCH to support M1s
* Add workflow to generate osqueryd.app.tar.gz
* Use 5.2.2 on init_tuf.sh
* Add unit test for tar.gz target
* Use artifacts instead of releases
* Remove copy paste residue
* Fleet Desktop Packaging WIP
* Ignore gosec warning
* Trigger on PR too
* Install Go in workflow
* Pass url parameter to desktop app
* Fix fleetctl package
* Final set of changes for v1 of Fleet Desktop
* Add changes
* PR fixes
* Fix CI build
* add larger menu bar icon
* Add transparency item
* Delete host_device_auth entry on host deletion
* Add SetTargetChannel
* Update white logo and add desktop to update runner
* Add fleet-desktop monitoring to orbit
* Define fleet-desktop app exec name
* Fix update runner creation
* Add API test before enabling the My device menu item
Co-authored-by: Zach Wasserman <zach@fleetdm.com>
* WIP
* WIP2
* Fix orbit and fleetctl tests
* Amend macos-app default
* Add some fixes
* Use fleetctl updates roots command
* Add more fixes to Updater
* Fixes to app publishing and downloading
* Add more changes to support fleetctl cross generation
* Amend comment
* Add pkg generation to ease testing
* Make more fixes
* Add changes entry
* Add legacy targets (until our TUF system exposes the new app)
* Fix fleetctl preview
* Fix bool flag
* Fix orbit logic for disabled-updates and dev-mode
* Fix TestPreview
* Remove constant and fix zip-slip attack (codeql)
* Return unknown error
* Fix updater's checkExec
* Add support for executable signing in init_tuf.sh
* Try only signing orbit
* Fix init_tuf.sh targets, macos-app only for osqueryd
* Specify GOARCH to support M1s
* Add workflow to generate osqueryd.app.tar.gz
* Use 5.2.2 on init_tuf.sh
* Add unit test for tar.gz target
* Use artifacts instead of releases
* Remove copy paste residue
* Trigger workflow on PR
* Fixes to ease handling of artifact
* Fix, do not use target name as dir
* Remove workaround
* Renaming files and a lot of find and replace
* pageRank meta tags, sorting by page rank
* reranking
* removing numbers
* revert changing links that are locked to a commit
* update metatag name, uncomment github contributers
* Update basic-documentation.page.js
* revert link change
* more explicit errors, change pageOrderInSection numbers, updated sort
* Update build-static-content.js
* update comment
* update handbook link
* handbook entry
* update sort
* update changelog doc links to use fleetdm.com
* move standard query library back to old location, update links/references to location
* revert unintentional link changes
* Update handbook/community.md
Co-authored-by: Mike Thomas <78363703+mike-j-thomas@users.noreply.github.com>
Co-authored-by: Mike Thomas <78363703+mike-j-thomas@users.noreply.github.com>
Co-authored-by: Mike McNeil <mikermcneil@users.noreply.github.com>
* Wait for osquery extension manager socket to be ready
* PR review feedback
* Add WIP Github Action to test orbit
* Set fleet address on fleetctl
* Add logging for troubleshooting
* Add prepare db statement
* Use tls-skip-verify on fleetctl
* Move steps around
* Fix addresses
* Fix fleetctl path
* Add certs.pem to orbit root dir
* Add orbit logs
* Increase timeout
* Add proper orbit log paths
* Fix tmp path orbit
* Add get hosts command to troubleshoot
* Fix orbit job termination
* Add comments to workflows
* Wait for server to go down
* Add orbit Windows job
* Use bash on windws-latest
* Fix missing quote
* Run orbit on Ubuntu
* Bump host count
* Increase timeout for extensions and not terminate on err
* Add comment to clarify high timeout value
* Revert change, we do want to exit in case of err
* Add terraform for the loadtesting environment
* Add some checkov stuff and minor updates
* Remove defunct commented stuff
* Add separate cloudwatch namespace for migrations
* Remove defunct code
* checkin
* Better jitter (#3716) (#3744)
* Better jitter (#3716)
* Better jitter
* Fix lint
* Use milliseconds
* Make duration milliseconds
* Update based on Lucas' suggestion
* Add changes file
* Panic on error
* Fix compilation error
* Use time.after instead of time.tick to not leak
* Include browser extensions in software inventory (#3733) (#3757)
Use appropriate JOINs against users table to include all results.
For #3557
* Build Docker images on push to any branch (#3756) (#3758)
Allows for testing release branches and any others.
* Optimize users detail query (#3754) (#3770)
@Smjert reported instances of Windows Domain Controllers having massive
resource utilization and being killed by the watchdog when running this
query. In his test environment, this new query performs much better.
* Refactor async host processing to avoid redis SCAN keys (for policies) (#3657) (#3755)
* Issue 3707 clean targets on delete (#3739) (#3776)
* wip
* Delete targets when deleting teams, hosts, and labels
* Add changes file
* Fix error message
* Remove unused teamsTable
* Cleanup new pack
* Clean new packs at end of test
* Update operating system policy (#3779)
- Update policy's query to check for operating system versions greater than or equal to
* Improve loading of manage policies page (#3695) (#3781)
Co-authored-by: gillespi314 <73313222+gillespi314@users.noreply.github.com>
* Fix logo (#3765)
* fixed duplicate query modal not closing (#3787)
* fixed style for connection line on activity feed (#3789)
* Improve loading state for query platform compatibility (#3752) (#3783)
Co-authored-by: gillespi314 <73313222+gillespi314@users.noreply.github.com>
* Disable pagination for teams page (#3794)
* Shorten "simple" query API period to 25s (#3775) (#3804)
This helps the period stay under the default request timeouts for most
load balancers.
Some default timeouts:
* AWS ALB - 60s
* Nginx - 60s
* GCP LB - 30s
* Clear user checkbox selections after successfully removing selected row from table UI #3798 (#3801)
* Alias fleetctl sandbox to fleetctl preview (#3803) (#3806)
Preparing for some work the product team is planning to rebrand preview
to sandbox.
* Remove flash of spinner to no spinner (#3799)
* Revert "Better jitter (#3716) (#3744)" (#3807)
This reverts commit c857b52d2b373ed9e302d643e6d2212af7353408.
* Fix teams dropdown sizing (#3759) (#3810)
Co-authored-by: gillespi314 <73313222+gillespi314@users.noreply.github.com>
* fixed link for expiration message (#3788)
* Fixed empty string for compatible platforms (#3812)
* fixed empty string for compatible platforms
* default checked for platforms on policy query
* Block autocomplete for username and password in SMTP settings (#3732) (#3816)
* No hosts for software/policy renders better message (#3701) (#3821)
Co-authored-by: Martavis Parker <47053705+martavis@users.noreply.github.com>
* Get failing policies webhook data on page load (#3824)
* Fix reset password flow (#3826)
* Prepare for 4.9.0 (#3820)
- Add CHANGELOG
- Bump versioning
- Remove changes files
- Spelling fix for operating system policy's description
* Remove merge artifact
Co-authored-by: Zach Wasserman <zach@fleetdm.com>
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
Co-authored-by: Noah Talerman <47070608+noahtalerman@users.noreply.github.com>
Co-authored-by: Luke Heath <luke@fleetdm.com>
Co-authored-by: gillespi314 <73313222+gillespi314@users.noreply.github.com>
Co-authored-by: RachelElysia <71795832+RachelElysia@users.noreply.github.com>
Co-authored-by: Martavis Parker <47053705+martavis@users.noreply.github.com>
Added reference architectures using https://docs.gitlab.com/ee/administration/reference_architectures/ as inspiration.
- updated terraform based on some feedback of usage
- pinned fleet docker version in terraform as to no get unexpected upgrades when applying
- updated some documentation around apply migration tasks
The low default ulimit `nofiles` value (`4096`) in Fargate was observed
to cause errors when running with a large number of hosts and a small
number of servers. Each server should be able to server more than 4096
simultaneous clients.
- Emphasize generating an "osquery installer" (also referred as "Orbit") using the `fleetctl package` command
- Add instructions for adding multiple hosts and automatically adding hosts to a team
- Remove instructions for generating an enrollment package with the tooling in `tools/mac/`
- Remove the `tools/mac/` directory from the repository because it is no longer used or referenced in any documentation
- Update "Automatically adding hosts to a team" section of "Teams" documentation to point to "Adding hosts" documentation
- Add instructions for migrating from plain osquery to Fleet's osquery installers
* refactor to allow bootstrapping* move monitoring into its own package, update readme
* add variable for license_key
* replication lag alarm less sensitive
- Fix Windows MSI generation by changing permissions (#2655).
- Refactor temp directory initialization.
- Use root user for Wine in WiX Docker container.
- Support .pkg packaging on Linux without dependencies (besides Docker)
* Add infra for loadtest
* Move loadtest stuff to a new file and parametrize fleet min/max capacity
* wip
* wip
* wip
* wip
* wip
* wip
* wip
* Update to be ready for review
* Update link and other variables needed
* Address review comments and update links
* Rename core->free and basic->premium
* Fix lint js
* Comment out portion of test that seems to timeout
* Rename tier to premium if basic is still loaded
* Add global policies
* Update documentation and add extra parameter to config
* Fix failing tests
* Store historic policy records
* Address review comments
And also remove other inmem references I saw by chance
* Add documentation for get by id request
* Add parameter doc
* Move schema generation to a cmd instead of a test
Otherwise it messes up running all tests sometimes depending on how parallel it does
* Remove brain dump for another task
* Make migration tests a separate beast
* Make schema generation idempotent and move dbutils cmd to tools
* Allow all filters and add counts to Policy
* Add test for Policy
- Use goreleaser to automate release process.
- Add new dockerfiles for fleet (with fleetctl) and fleetctl (only).
- Add GitHub Action Workflow to run goreleaser on new tag.
- Update NPM to match new archive naming.
* Remove username from UI code
* Remove username from tests
* Remove username from database
* Modify server endpoints for removing username
* Implement backend aspects of removing username
* Update API docs
* Add name to fleetctl
- Update names/roles of users in `make e2e-setup`.
- Update test SSO user info.
- Add Cypress commands for seeding users/Teams.
- Stub Cypress tests for team/tier matrix.
- In tests and documentation, replace `@fleetdm.com` with `@example.com`
- In documentation, replace `hello@fleetdm.com` with `fleetdm.com/contact`
- In documentation, replace `security@fleetdm.com` with `fleetdm.com/contact`
- In Dockerfiles, replace `engineering@fleetdm.com` with `hello@fleetdm.com`. These two files are the only remaining files with a `@fleetdm.com` email.
- Include only hosts that the user has access to in search targets API.
- Add parameter to specify whether `observer` hosts should be included.
- Generate counts based on which hosts user can access.
- Update API doc.
- Add new `2-Orbit-osquery/` directory to the top-level `docs/` directory.
- Rename `2-Deployment/` -> `3-Deployment/` to accommodate new Orbit directory.
- Rename `3-Contribution/` -> `4-Contribution/` to accommodate new Orbit directory.
- Add FAQ section to Orbit documentation.
Ubuntu 20.04 uses an older version of Node by default that does not
support the rmSync method we were using previously. This should extend
compatibility.
We found in some environments users run `sudo npm install -g fleetctl`
to install via NPM, and NPM does not execute postinstall scripts when
run as root.
This new approach should be compatible with more environments as the
download of fleetctl is done as necessary when the script is invoked.
- Support both /api/v1/fleet and /api/v1/kolide routes in server.
- Add logging for use of deprecated routes.
- Rename routes in frontend JS.
- Rename routes and add notes in documentation.
Providing a Helm chart makes it easier for users to deploy to Kubernetes.
I think this is good to go, and I've tested deploying to Kind and GKE using the GKE ingress, sql proxy, and managed certificate, but I haven't tested anything AWS related.
Somewhere around osquery 4.4.0 these messages were added to query
responses. We can now expose them to the API clients rather than using
the placeholder text.
Required for #192
- Add endpoints for osquery to register and continue a carve.
- Implement client functionality for retrieving carve details and contents in fleetctl.
- Add documentation on using file carving with Fleet.
Addresses kolide/fleet#1714
* Perform migration to delete any entries with `deleted` set, and
subsequently drop columns `deleted` and `deleted_at`.
* Remove `deleted` and `deleted_at` references.
Closes#2146
* Use YAML anchors to avoid repeating config blocks
* Use docker volumes to persist data for mysql
* Allow setting `FLEET_SERVER` (fixes#2127) when using the docker-compose file to spin up multiple osquery clients
Almost two years ago, we began referring to the project as Fleet, but there are
many occurences of the term "Kolide" throughout the UI and documentation. This
PR attempts to clear up those uses where it is easily achievable.
The term "Kolide" is used throughout the code as well, but modifying this would
be more likely to introduce bugs.
Add a gRPC server that will interact with osquery through Launcher. This endpoint will expose the osquery configuration suitable for use via the Launcher plugin, and collect log and query results.
- Add SSH configuration to allow checkout of Kolide private repos in CI
- Add kolide/agent-api repo to glide.yaml
- Update testify version to fix broken test build
Closes#1545
This PR partially addresses #1456, providing SSO SAML support. The flow of the code is as follows.
A Kolide user attempts to access a protected resource and is directed to log in.
If SSO identity providers (IDP) have been configured by an admin, the user is presented with SSO log in.
The user selects SSO, which invokes a call the InitiateSSO passing the URL of the protected resource that the user was originally trying access. Kolide server loads the IDP metadata and caches it along with the URL. We then build an auth request URL for the IDP which is returned to the front end.
The IDP calls the server, invoking CallbackSSO with the auth response.
We extract the original request id from the response and use it to fetch the cached metadata and the URL. We check the signature of the response, and validate the timestamps. If everything passes we get the user id from the IDP response and use it to create a login session. We then build a page which executes some javascript that will write the token to web local storage, and redirect to the original URL.
I've created a test web page in tools/app/authtest.html that can be used to test and debug new IDP's which also illustrates how a front end would interact with the IDP and the server. This page can be loaded by starting Kolide with the environment variable KOLIDE_TEST_PAGE_PATH to the full path of the page and then accessed at https://localhost:8080/test
If server is started without a JWT key, a message like the following is printed:
```
################################################################################
# ERROR:
# A value must be supplied for --auth_jwt_key. This value is used to create
# session tokens for users.
#
# Consider using the following randomly generated key:
# om3w95gMA2drT5xAdLd2Q5oE8fLw+Miz
################################################################################
```
Closes#1480.
On CentOS6 there is a bug in which osquery incorrectly reports an empty string
for platform. This PR fixes our detection of centos in this case.
Fixes#1339
the scripts in tools/pkgrepos can be used to update apt and yum repositories as well as create zip archives for a release.
The dl.kolide.co repository is archived as a static bucket on GCP.
Adds a `make lint-license` command that will crawl through the Golang and
Javascript dependencies, trying to automatically determine the license of each
dependency. If any dependencies have incompatible licensing (or if the
automatic detection fails), the tool will print an error and exit with a
non-zero status code. After a successful run, the
docs/third-party/dependencies.md file will be updated with attribution
information as determined by the crawl.
The configuration file has been bootstrapped with the manual analysis needed for
all of the existing dependencies.
This PR includes:
- `make demo-dump` which dumps all tables from your running development Kolide instance into `./tools/app/demo.sql`
```
$> make demo-dump
mysqldump --extended-insert=FALSE --skip-dump-date \
-u kolide -p \
-h 192.168.99.100 kolide \
> ./tools/app/demo.sql
Enter password: ******
$>
```
- `make demo-restore` which restores `./tools/app/demo.sql` into your MySQL database, to be demo'd and extended by your running Kolide instance
```
$> make demo-restore
mysql --binary-mode -u kolide -p \
-h 192.168.99.100 kolide \
< ./tools/app/demo.sql
Enter password: ******
```
This included demo dump contains the following content:
## Queries
- 34 queries shared as ~70 scheduled queries
## Packs
- 8 Packs
- 13 Scheduled Queries in the "Intrusion Detection" pack
- 7 Scheduled Queries in the "Osquery Monitoring" pack
- 4 Scheduled Queries in the "Asset Management" pack
- 12 Scheduled Queries in the "Hardware Monitoring" pack
- 8 Scheduled Queries in the "Incident Response" pack
- 8 Scheduled Queries in the "Compliance" pack
- 6 Scheduled Queries in the "Vulnerability Management" pack
- 10 Scheduled Queries in the "Systems Monitoring" pack
## Labels
- 2 Labels
## Hosts
There are intentionally no hosts. Live hosts should be fabricated using Docker and attached to a running demo instance, to better look and behave like a real infrastructure.
