#9253
- ~[ ] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.~
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
- For Orbit and Fleet Desktop changes:
- [X] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
---------
Co-authored-by: Sharon Katz <121527325+sharon-fdm@users.noreply.github.com>
This pull request relies on the `mdm-gen-cert` command from
https://github.com/fleetdm/fleet/pull/8884.
Closes: https://github.com/fleetdm/fleet/issues/8223
Changes:
- Updated the deploy Fleet website workflow to:
- Add Go as a dependency
- Build the mdm-gen-cert binary in `/website/.tools/`
- add the `/.tools/` folder to the Heroku app
- Added `deliver-apple-csr.js` - an API that:
- can be called by making a `POST` request to
`/api/v1/deliver-apple-csr`
- accepts `csr` as an input
- runs the `mdm-gen-cert` command with the `csr` set as an environment
variable
- returns an `invalidEmailDomain` response if the user's email domain is
in the array of banned email domains.
- saves the users organization and email address to the website's
database
- Sends an email to the requesting user's email address with the signed
CSR attached as a text file named `apple-apns-request.txt`
- Posts a message to a channel in the Fleet Slack.
- Added a new model: `CertificateSigningRequests` that contains two
required attributes: `emailAddress` and `organization`
- Added a new email template `email-signed-csr-for-apns`
- Updated routes, policies, eslintrc, and rebuilt cloud-sdk
Before this can be merged, we will need to:
- [x] Add new config variables in Heroku
- [x] `sails.config.custom.mdmVendorCertPem`
- [x] `sails.config.custom.mdmVendorKeyPem`
- [x] `sails.config.custom.mdmVendorKeyPassphrase`
- [x] `sails.config.custom.slackWebhookUrlForMDMSignups`
- [x] Add the `CertificateSigningRequests` model to the website's
database
This PR will change our github issues template from the previous
**new-feature-work** to **Story** and **sub-task**.
It can only be tested after merging to main.
# Testing required:
After merging this PR to main, make sure that when creating a new issue,
these templates are available instead of the old one.
# Checklist for submitter
- [ ] Manual QA for all new/changed functionality
.
Co-authored-by: Mike McNeil <mikermcneil@users.noreply.github.com>
Changes:
- Updated the `build-static-content` script to use a GitHub API token
for requests if one is provided e.g., `sails run build-static-content
--githubAccessToken="foo"`
- Updated the `build-for-prod` npm script to run the
`build-static-content` script with a variable named `BUILD_SCRIPT_ARGS`.
- Updated the "Deploy Fleet website" and "Test Fleet website" workflows
to run the `build-for-prod` script with a GitHub API token
. .
Co-authored-by: Mike McNeil <mikermcneil@users.noreply.github.com>
* WIP
* Add more logging
* Check rate limit at end of action
* Add github client in more places
* Add new published firefox 93 vulnerabilities to tests
* Remove fmt printfs
* Restore CI check settings
* Readd newline
A customer had a seccomp profile that seemed to be incompatible with the
changes in glibc in ubuntu-22.04. Setting the builder back to 20.04
explicitly to resolve this issue.
* Add quay push to the snapshot pusher to start
* Tags need to be just the tag part in this one
* Put the tag in a variable
* Fix typos
* Switch up how we define registry to see if it finds the image like this
* Add quay push everywhere else
* Update go to 1.19.4
* Comment out failing package test
* Comment out ALL the packaging tests for windows for the moment
* Update go to 1.19.4
* Comment out failing package test
* Comment out ALL the packaging tests for windows for the moment
* Update changelog
* Bump versions
* Update changelog to reflect this being a security release
* Fix confusion with tags on dogfood deploy workflow
* Update .github/workflows/dogfood-deploy.yml
Co-authored-by: Michal Nicpon <39177923+michalnicp@users.noreply.github.com>
Co-authored-by: Michal Nicpon <39177923+michalnicp@users.noreply.github.com>
We don't have a `matrix.go-version` defined, so the task was using
`go1.17.13` as the default.
This explicitly sets the version to `go1.19.1`, I didn't use a `matrix`
definition because at least for now, we only want to run this test using a
single Go version.
Since I was there, I also updated test-native-tooling-packaging.yml to
use `go1.19.1` too.
* Create .trivyignore
Adding original trivy ignore file. Working to resolve/document more of the findings, especially around go.mod. Will add a github action as well.
* Adding default trivy scan for testing
* Update trivy_scan.yml
Making it manual + daily for now
* Update trivy_scan.yml
updating name
* Renamed + configured Trivy scan
* Adding a new synchronization mechanism between fleet-desktop app and Orbit service. Improved windows service teardown to ensure that fleet-desktop does not get force killed without getting signaled. Improved windows process enumeration to avoid unnecessary delays during windows service start and windows service teardown. Updating windows service to reflect service teardown extra time due to synchronization.
this modifies the migration order CI check to only check for added files
by:
1. Escaping the blob we give to git, so bash doesn't perform expansion,
this lets git handle the blob matching, which for reasons I don't
fully understand allows to find file renames.
2. Applying `--diff-filter=A`, which makes git only list file additions.
Related to #6142, this adds a CI check for the order of migrations.
As I noted in a comment on the workflow file, it's important to keep in mind that some migrations might still go unnoticed even with this check, example:
1. PR1 adds a migration, CI check pass
2. PR2 adds a migration, CI pass, gets merged
3. PR1 can still be merged because the CI checks aren't run again
The check will fail in `main` however, so if we find the current script to be reliable, we could setup a Slack ping or something similar, to make sure somebody takes a look
* Bump go to 1.19.1
* Bump remaining go-version to the 1.19.1
* Add extra paths for test-go
* Oops, putting the right path in the right place
* gofmt file
* gofmt ALL THE THINGS
* Moar changes
* Actually, go.mod doesn't like minor versions
In https://github.com/fleetdm/fleet/pull/7399 we accidentally:
1. added a condition to `.github/workflows/fleet-and-orbit.yml` that always evaluates to `false` making this workflow always fail
2. modified the condition of `.github/workflows/fleetctl-preview-latest.yml`
This reverts those changes.
This adds a new workflow to CI in order to test that the PR doesn't contain uncommited schema changes, which are the source of many merge conflicts and developer frustration.
- Update "Learn how to use Fleet" docs page to walk a Fleet Sandbox user through adding their device and running a query
- Add a "Get operating system information" query to standard query library for the "Learn how to use Fleet" walkthrough
- Update Fleet's top level README to point users who want to try Fleet to Fleet Sandbox
- Update "How to install osquery..." (macOS, Windows, Linux) blog posts to point users who want to try Fleet to Fleet Sandbox
- Move `fleetctl preview` questions to "Contributing" FAQ section in docs. This is because `fleetctl preview` is now a testing tool for Fleet contributors
- Update "Deploying" docs to point users who want to try Fleet to Fleet Sandbox
The macOS runners installing Docker are having problems initializing the new Docker version (4.11.0) which effectively blocks PRs with Go code.
This locks the Docker version we install to 4.10.0, which works until we figure out a solution or a new Docker version goes out.
Related to #6365, this extends the datastore/s3 package to retrieve installers from S3 according to the conventions listed in the parent issue. This also includes:
- A minor refactor to decouple Carves-related functionality from the core S3 functionality
- Set-up to run tests using minio (only enabled via the FILE_STORAGE_TEST env flag)
* add tf vars for cloudwatch log retention & rds snapshot backup retention, update github workflow to deploy new dogfood configurations for new tf vars
* typo and tf fmt
Related to #6364 and #6363, this:
- Adds a new Docker image, `fleetdm/fleetctl` equipped with all necessary dependencies to build Fleet-osquery binaries for all platforms
- Modifies the package generation logic to special case this scenario via an environment variable `FLEETCTL_NATIVE_TOOLING`
- Adds a new GitHub workflow to test this
There are more details in the README, but part of the special-casing logic is in place to output the binaries to a folder named `build` when they are run with `FLEETCTL_NATIVE_TOOLING`, this is so we can persist the binary generated by the docker container via a bind mount:
```bash
docker run -v "$(pwd):/build" fleetdm/fleetctl package --type=msi
```
To test this changeset, I have generated packages for all platforms, both via the new Docker image and via the classic `fleetctl package`.
* Do not use golangci action for better reproducibility
* Add fix to trigger build
* Fix all reported issues
* fix more lint errors
* Add missing import
* Remove unused method
* Remove change not necessary
* fix old root dir in orbit
* add changes
* Add automation for orbit shell (with TUF)
* Fix workflow syntax
* Add logging to latest fleetctl preview action
* Add changes to fix workflow
* Use macOS host for TUF server and package generation
* Remove copy/paste if clause
* Fix orbit logs on macOS, Ubuntu
* Simplify TUF and generation of packages
* Set enroll secret instead of getting it
* Increase timeouts
* Add step id
* Fixes to the upload/download of artifacts
* Rearrange steps to not lose the downloads
* Fix copy/paste
* Add fleetctl login step
* Add missing config set
* Fix quotes on Windows
* Increase timeout
* Fix job termination
* Disable FLEET_DESKTOP for now
* Checkout repository on macOS
* Fix logs path
* Enable fleet desktop
* Use cancel, nitpick
Co-authored-by: Michal Nicpon <michal@fleetdm.com>
* remove unused iam poilcy attributes and remove github action on pull request, only workflow dispatch will be required
* update github.tf, commenting out all resources, but leaving in place in case someone else wants to use ODIC providers & Github actions
* Kickoff documentation for Orbit releasing
* Fixes to the github action
* Missing follow redirect on curl
* Run osqueryd --version to verify before uploading artifacts
* Use cmd as shell for windows-latest runner
* Final set of changes to the guide
* Add (beta) support for Fleet Desktop to linux
* Add dependency for linux desktop
* Amend makefile uname check
* Clarify env vars used for linux in execuser
* Add final set of fixes
* Remove -it from docker run
* Add desktop to the update runner for Linux
* Re-arrange tag.gz and fix upgrade check for linux desktop
* Add logs to troubleshoot orbit
* Run journalctl on a different step
* Add legacy orbit support to opt version of fleetctl
* Fix macos logs permission error
* Checkout repository
* Compile fleetctl from branch
* Add fleetctl preview automation for latest changes
* Fix pwd invocation and remove slack notification
* Just run on ubuntu-latest and macos-latest
* Fix path
* Adding docker to dependabot
Adding the docker ecosystem to Dependabot, so PRs get auto-created when necessary.
* Update dependabot.yml
Changed comment on top
Practically, we were scanning enough previously (at least once a day, plus any commit that changed source files), but this will help check the box in CII Best Practices.
* Orbit: Add Fleet Desktop support to Windows
* Rename workflow, fix linux build
* Do not compile systray on linux
* nolint on unused
* Fix lint properly
* nolint both checkers
* Fix monitor logic in desktopRunner
* Fix interrupt and execute order
* Added explicit read permissions + tweaked permissions
As a part of #4698 - this should fix the remaining warnings we get from the OSSF scorecard in relation to github workflows. They now all have explicit read permissions with more granular permissions granted in jobs.
* Update tfsec.yml
New workflow that I had not fixed in this PR.
* Adding permissions to docs.yml and integration.yml
* Update codeql-analysis.yml
Adding top level read permissions to codeql workflow
* Update codeql-analysis.yml
Adding manual dispatch to codeql - to be able to test it easier
* Update deploy-fleet-website.yml
Adding top level read permission + write in the job so it can push the website
* Update test-website.yml
test-website should only need read permissions on content.
* Update fleet-and-orbit.yml
Testing Fleet and Orbit should be fine with top level read access
* Update fleetctl-preview.yml
fleetctl-preview should be fine with just read access at top level
* Update push-osquery-perf-to-ecr.yml
ECR is out of github so read permissions should be enough
* Update semgrep-analysis.yml
semgrep should only need read
* Update test-packaging.yml
Should only need read permission - setting on top
* Update test.yml
Should not need any write access - setting to READ on top.
* Update deploy-fleet-website.yml
Removing git write permission - since this pushes to Heroku not GitHub
* Tweaked as per Zach's comments
Removed some useless restrictions (contents none on a public repo for example)
* Removed meaningless permissions
contents: none - this does not have any security advantage on a public repo
* WIP
* WIP2
* Fix orbit and fleetctl tests
* Amend macos-app default
* Add some fixes
* Use fleetctl updates roots command
* Add more fixes to Updater
* Fixes to app publishing and downloading
* Add more changes to support fleetctl cross generation
* Amend comment
* Add pkg generation to ease testing
* Make more fixes
* Add changes entry
* Add legacy targets (until our TUF system exposes the new app)
* Fix fleetctl preview
* Fix bool flag
* Fix orbit logic for disabled-updates and dev-mode
* Fix TestPreview
* Remove constant and fix zip-slip attack (codeql)
* Return unknown error
* Fix updater's checkExec
* Add support for executable signing in init_tuf.sh
* Try only signing orbit
* Fix init_tuf.sh targets, macos-app only for osqueryd
* Specify GOARCH to support M1s
* Add workflow to generate osqueryd.app.tar.gz
* Use 5.2.2 on init_tuf.sh
* Add unit test for tar.gz target
* Use artifacts instead of releases
* Remove copy paste residue
* Fleet Desktop Packaging WIP
* Ignore gosec warning
* Trigger on PR too
* Install Go in workflow
* Pass url parameter to desktop app
* Fix fleetctl package
* Final set of changes for v1 of Fleet Desktop
* Add changes
* PR fixes
* Fix CI build
* add larger menu bar icon
* Add transparency item
* Delete host_device_auth entry on host deletion
* Add SetTargetChannel
* Update white logo and add desktop to update runner
* Add fleet-desktop monitoring to orbit
* Define fleet-desktop app exec name
* Fix update runner creation
* Add API test before enabling the My device menu item
Co-authored-by: Zach Wasserman <zach@fleetdm.com>
* WIP
* WIP2
* Fix orbit and fleetctl tests
* Amend macos-app default
* Add some fixes
* Use fleetctl updates roots command
* Add more fixes to Updater
* Fixes to app publishing and downloading
* Add more changes to support fleetctl cross generation
* Amend comment
* Add pkg generation to ease testing
* Make more fixes
* Add changes entry
* Add legacy targets (until our TUF system exposes the new app)
* Fix fleetctl preview
* Fix bool flag
* Fix orbit logic for disabled-updates and dev-mode
* Fix TestPreview
* Remove constant and fix zip-slip attack (codeql)
* Return unknown error
* Fix updater's checkExec
* Add support for executable signing in init_tuf.sh
* Try only signing orbit
* Fix init_tuf.sh targets, macos-app only for osqueryd
* Specify GOARCH to support M1s
* Add workflow to generate osqueryd.app.tar.gz
* Use 5.2.2 on init_tuf.sh
* Add unit test for tar.gz target
* Use artifacts instead of releases
* Remove copy paste residue
* Trigger workflow on PR
* Fixes to ease handling of artifact
* Fix, do not use target name as dir
* Remove workaround
* Renaming files and a lot of find and replace
* pageRank meta tags, sorting by page rank
* reranking
* removing numbers
* revert changing links that are locked to a commit
* update metatag name, uncomment github contributers
* Update basic-documentation.page.js
* revert link change
* more explicit errors, change pageOrderInSection numbers, updated sort
* Update build-static-content.js
* update comment
* update handbook link
* handbook entry
* update sort
* update changelog doc links to use fleetdm.com
* move standard query library back to old location, update links/references to location
* revert unintentional link changes
* Update handbook/community.md
Co-authored-by: Mike Thomas <78363703+mike-j-thomas@users.noreply.github.com>
Co-authored-by: Mike Thomas <78363703+mike-j-thomas@users.noreply.github.com>
Co-authored-by: Mike McNeil <mikermcneil@users.noreply.github.com>
Update scorecard analysis version from 1.0.2 to 1.0.4 - has updates on how env variables are handled, need to update and see if it fixes the issue we have with the cron job.
* Wait for osquery extension manager socket to be ready
* PR review feedback
* Add WIP Github Action to test orbit
* Set fleet address on fleetctl
* Add logging for troubleshooting
* Add prepare db statement
* Use tls-skip-verify on fleetctl
* Move steps around
* Fix addresses
* Fix fleetctl path
* Add certs.pem to orbit root dir
* Add orbit logs
* Increase timeout
* Add proper orbit log paths
* Fix tmp path orbit
* Add get hosts command to troubleshoot
* Fix orbit job termination
* Add comments to workflows
* Wait for server to go down
* Add orbit Windows job
* Use bash on windws-latest
* Fix missing quote
* Run orbit on Ubuntu
* Bump host count
* Increase timeout for extensions and not terminate on err
* Add comment to clarify high timeout value
* Revert change, we do want to exit in case of err
* Run go tests with race at night
* Add missing run on step
* Fix boolean logic
* Allow manual run for test-go.yaml
* Add slack notification step
* Try global environment variable for cron schedule
* Fix indentation
* Try number 2
* Try setting cron as usual
* Remove global env
* Only send notification in case of failure when running schedule
* Run with race enable to test
* Add more fixes
* Fix github event variable name
* Set timeouts
* Fix slack notification link
* Re-enable if clause
* Last try on Github Actions
* Re-enable the if clause
* Add fixes for running tests with mysql:8
* Add getServer function
* Test github matrix
* Add changes file for the user facing fix
* Remove unused mysql8 docker-compose
Add nightly and manual job to run `fleetctl preview` and check for enrolled hosts.
Works on macOS and Linux. Windows cannot be tested on GitHub Actions due to lack of Docker Linux support.
* Add software count API
* Fix makefile
* Fine no mock generating at this point
* Actually, one last try
* Use go install instead
* Fix go sum/mod
* Improve documentation
* Try setting node to 14
- Fix Windows MSI generation by changing permissions (#2655).
- Refactor temp directory initialization.
- Use root user for Wine in WiX Docker container.
- Support .pkg packaging on Linux without dependencies (besides Docker)
