* Add rate limits for device authed endpoints
* Fix lint
* Add missing test
* Fix test
* Increase the quota for desktop endpoints
* Add comment about quota
* Add server support for Fleet Sandbox demo login
This adds an endpoint `/api/latest/fleet/demologin` that provides a
redirect for the fleetdm.com portion of Fleet Sandbox to automatically
log in a user. The username and password must be provided as form
values. The endpoint is only enabled if `FLEET_DEMO=1` is set in the
server environment.
This was tested locally with the following HTML served by `python3 -m
http.server`, and the Fleet server running with `FLEET_DEMO=1
./build/fleet serve --dev`:
```
<!DOCTYPE html>
<body>
<form
method="post"
action="https://localhost:8080/api/latest/fleet/demologin"
id="demologin"
>
<input type="hidden" name="email" value="admin@example.com" />
<input type="hidden" name="password" value="admin123123#" />
<input type="submit"/>
</form>
<script type="text/javascript">
document.forms["demologin"].submit();
</script>
</body>
</html>
```
For Fleet sandbox purposes, the `action` should be set to the correct
hostname for the sandbox instance, while the `email` and `password`
should be set to the same credentials that were provided when creating
the instance.
* lucas comments
* Add integration tests
* Fix status codes and add comments
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
Related to #5898, this reports an anonymized summary of errors stored in Redis into the analytics payload.
For each error stored, this includes:
- A `count` attribute with the number of occurrences of the error
- A `loc` attribute with the 3 topmost lines in the stack trace. Note that stack traces only contain package name + line number (example: github.com/fleetdm/fleet/server.go:12
This also includes a minor refactor around error types.
This commit replaces `ioutil.TempDir` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.
Prior to this commit, temporary directory created using `ioutil.TempDir`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}
is also tedious, but `t.TempDir` handles this for us nicely.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* Do not use golangci action for better reproducibility
* Add fix to trigger build
* Fix all reported issues
* fix more lint errors
* Add missing import
* Remove unused method
* Remove change not necessary
Related to #6063, this adds a new device API to get an object with boolean values that we can use as feature flags to manage backwards compatibility in Fleet Desktop.
Feature: Improve our capability to detect vulnerable software on Ubuntu hosts
To improve the capability of detecting vulnerable software on Ubuntu, we are now using OVAL definitions to detect vulnerable software on Ubuntu hosts. If data sync is enabled (disable_data_sync=false) OVAL definitions are automatically kept up to date (they are 'refreshed' once per day) - there's also the option to manually download the OVAL definitions using the 'fleetctl vulnerability-data-stream' command. Downloaded definitions are then parsed into an intermediary format and then used to identify vulnerable software on Ubuntu hosts. Finally, any 'recent' detected vulnerabilities are sent to any third-party integrations.
This removes policy information from `GET /api/_version_/fleet/device/{token}` from non-premium Fleet instances.
Starting the server with `./build/fleet serve --dev --dev_license`
```bash
$ curl -s https://localhost:8080/api/latest/fleet/device/1804e808-171f-4dda-9bec-f695b2f2371a | jq '.host.policies // "not present"'
[
{
"id": 3,
"name": "Antivirus healthy (Linux)",
"query": "SELECT score FROM (SELECT case when COUNT(*) = 2 then 1 ELSE 0 END AS score FROM processes WHERE (name = 'clamd') OR (name = 'freshclam')) WHERE score == 1;",
"description": "Checks that both ClamAV's daemon and its updater service (freshclam) are running.",
"author_id": 1,
"author_name": "Roberto",
"author_email": "test@example.com",
"team_id": null,
"resolution": "Ensure ClamAV and Freshclam are installed and running.",
"platform": "darwin,linux",
"created_at": "2022-05-23T20:53:36Z",
"updated_at": "2022-06-03T13:17:42Z",
"response": ""
}
]
```
Starting the server with `./build/fleet serve --dev`
```bash
$ curl -s https://localhost:8080/api/latest/fleet/device/1804e808-171f-4dda-9bec-f695b2f2371a | jq '.host.policies // "not present"'
"not present"
```
This adds a new API client named DeviceClient to server/service, meant to consume device endpoints and be used from Fleet Desktop.
Some of the logic to make requests and parse responses was very repetitive, so I introduced a private baseClient type and moved some of the logic of the existent Client there.
Related to #5685 and #5697
This adds a new device authenticated endpoint, `/api/_version_/fleet/device/{token}/policies` to retrieve the device policies.
An example request / response looks like:
```bash
curl https://localhost:8080/api/latest/fleet/device/1804e808-171f-4dda-9bec-f695b2f2371a/policies
```
```json
{
"policies": [
{
"id": 3,
"name": "Antivirus healthy (Linux)",
"query": "SELECT score FROM (SELECT case when COUNT(*) = 2 then 1 ELSE 0 END AS score FROM processes WHERE (name = 'clamd') OR (name = 'freshclam')) WHERE score == 1;",
"description": "Checks that both ClamAV's daemon and its updater service (freshclam) are running.",
"author_id": 1,
"author_name": "Admin",
"author_email": "admin@example.com",
"team_id": null,
"resolution": "Ensure ClamAV and Freshclam are installed and running.",
"platform": "darwin,windows,linux",
"created_at": "2022-05-23T20:53:36Z",
"updated_at": "2022-05-23T20:53:36Z",
"response": "fail"
}
]
}
```
Related to [#5685](https://github.com/fleetdm/fleet/issues/5685), in another changeset I will be adding "client" endpoints so we can consume this endpoint from Fleet Desktop
Related to #5776, this ensures that when a host is re-enrolled on a different team we cleanup existing policy memberships in the same way we do when a host is assigned a team through the API.
This solves #5679 , and also implements #5515, #5509 and lays the ground for #5516
With the introduction of Wrap, Is and As in the standard library, we've now got built-in support for wrapping.
On top of that, a common pattern in the community is to define errors tailored to the context of each project while still conforming to the error and Unwrap interfaces (see Upspin, Chromium)
The output now includes stack traces and additional info
this change prevents errors from being automatically cleared once they are read. A new flag `-flush` is introduced to flush errors on read if necessary.
To add support for #400, we're using the macadmins/osquery-extension to gather Google Chrome profiles from hosts.
Under the hood, the extension looks and parses a json file in which Chrome stores a bunch of data. Given that emails are not required to create Chrome profiles, some of the profiles stored in this file and returned by the query we're using contain empty emails.
The idea after this change is to prevent empty emails from being ingested in the first place instead of filtering them after the fact. I have also included a migration to clean the rows with empty email columns.
Fixes#4780
this fixes#5377 by asserting that the error message starts with "tsl" or "x509" as either of the strings represent the same error depending on the mysql version.
we should refactor the code to avoid checking on the error messages and probably use different error types instead.
* Fix access control issues with users
* Fix access control issues with packs
* Fix access control issues with software
* Changes suggested by Martin
* All users can access the global schedule
* Restrict access to activities
* Add explicit test for team admin escalation vuln
* All global users should be able to read all software
* Handbook editor pass - Security - GitHub Security (#5108)
* Update security.md
All edits are recorded by line:
395 replaced “open-source” with “open source”
411 replaced “open-source” with “open source”
439 added “the” before “comment”; replaced “repositories,” with “repositories”
445 deleted “being” before “located”
458 added “and” after “PR”
489 replaced “on” with “in”
493 replaced “open-source” with “open source”; Replaced “privileges,” with “privileges”
* Update security.md
line 479
* Update security.md
added (static analysis tools used to identify problems in code) to line 479
* Fix UI
* Fix UI
* revert api v1 to latest in documentation (#5149)
* revert api v1 to latest in documentation
* Update fleetctl doc page
Co-authored-by: Noah Talerman <noahtal@umich.edu>
* Add team admin team policy automation; fix e2e
* Update to company page of the handbook (#5164)
Updated "Why do we use a wireframe-first approach?" section of company.md
* removed extra data on smaller screens (#5154)
* Update for team automations; e2e
* Jira Integration: Cypress e2e tests only (#5055)
* Update company.md (#5170)
This is to update the formatting under "empathy" and to fix the spelling of "help text."
This was done as per @mikermcneil .
This is related to #https://github.com/fleetdm/fleet/pull/4941 and https://github.com/fleetdm/fleet/issues/4902
* fix update updated_at for aggregated_stats (#5112)
Update the updated_at column when using ON DUPLICATE UPDATE so that
the counts_updated_at is up to date
* basic sql formatting in code ie whitespace around operators
* Fix e2e test
* Fix tests in server/authz
Co-authored-by: gillespi314 <73313222+gillespi314@users.noreply.github.com>
Co-authored-by: Desmi-Dizney <99777687+Desmi-Dizney@users.noreply.github.com>
Co-authored-by: Michal Nicpon <39177923+michalnicp@users.noreply.github.com>
Co-authored-by: Noah Talerman <noahtal@umich.edu>
Co-authored-by: Mike Thomas <78363703+mike-j-thomas@users.noreply.github.com>
Co-authored-by: Martavis Parker <47053705+martavis@users.noreply.github.com>
Co-authored-by: RachelElysia <71795832+RachelElysia@users.noreply.github.com>
Update the updated_at column when using ON DUPLICATE UPDATE so that
the counts_updated_at is up to date
* basic sql formatting in code ie whitespace around operators
To ensure the product works properly when using MySQL with `ANSI_QUOTES` mode enabled,
replace all uses of `""` for values inside SQL statements with `''`
Co-authored-by: Ricky Grassmuck <r.grassmuck@cpanel.net>
This optimizes the query in particular for Domain Controllers, where
there are a number of users with no local accounts and don't need to be
searched for software.
See #4261
not set on the INSERT.
- OUT: Only sets the ID on the passed session and returns it. (`CreatedAt`, `AccessedAt`, are not set.)
New version:
```go
func (ds *Datastore) NewSession(ctx context.Context, userID uint, sessionKey string) (*fleet.Session, error) {
sqlStatement := `
INSERT INTO sessions (
user_id,
` + "`key`" + `
)
VALUES(?,?)
`
result, err := ds.writer.ExecContext(ctx, sqlStatement, userID, sessionKey)
if err != nil {
return nil, ctxerr.Wrap(ctx, err, "inserting session")
}
id, _ := result.LastInsertId() // cannot fail with the mysql driver
return ds.sessionByID(ctx, ds.writer, uint(id))
}
```
- IN: Define arguments that are truly used when creating a session.
- OUT: Load and return the fleet.Session struct with all values set (using the `ds.writer` to support read replicas correctly).
PS: The new `NewSession` version mimics what we already do with other entities, like policies (`Datastore.NewGlobalPolicy`).
* Upgrade and replace kolide/osquery-go with osquery/osquery-go
* Upgrade macadmins/osquery-extension to v0.0.7
* Upgrade kolide/launcher to latest
* go mod tidy
* WIP
* WIP2
* Fix orbit and fleetctl tests
* Amend macos-app default
* Add some fixes
* Use fleetctl updates roots command
* Add more fixes to Updater
* Fixes to app publishing and downloading
* Add more changes to support fleetctl cross generation
* Amend comment
* Add pkg generation to ease testing
* Make more fixes
* Add changes entry
* Add legacy targets (until our TUF system exposes the new app)
* Fix fleetctl preview
* Fix bool flag
* Fix orbit logic for disabled-updates and dev-mode
* Fix TestPreview
* Remove constant and fix zip-slip attack (codeql)
* Return unknown error
* Fix updater's checkExec
* Add support for executable signing in init_tuf.sh
* Try only signing orbit
* Fix init_tuf.sh targets, macos-app only for osqueryd
* Specify GOARCH to support M1s
* Add workflow to generate osqueryd.app.tar.gz
* Use 5.2.2 on init_tuf.sh
* Add unit test for tar.gz target
* Use artifacts instead of releases
* Remove copy paste residue
* Fleet Desktop Packaging WIP
* Ignore gosec warning
* Trigger on PR too
* Install Go in workflow
* Pass url parameter to desktop app
* Fix fleetctl package
* Final set of changes for v1 of Fleet Desktop
* Add changes
* PR fixes
* Fix CI build
* add larger menu bar icon
* Add transparency item
* Delete host_device_auth entry on host deletion
* Add SetTargetChannel
* Update white logo and add desktop to update runner
* Add fleet-desktop monitoring to orbit
* Define fleet-desktop app exec name
* Fix update runner creation
* Add API test before enabling the My device menu item
Co-authored-by: Zach Wasserman <zach@fleetdm.com>
* geoip wip
* return nil if ip is empty string or if ParseIP returns nil
* add ui component to render geolocation if available, address PR feedback
* render public ip if available
* add changes file, document geoip in deployment guide
* update rest-api docs
* Error early if email already exists
* Update changes/issue-4361-mail-change-should-error
Co-authored-by: Michal Nicpon <39177923+michalnicp@users.noreply.github.com>
* Fix test
* Lint fixes
* Fix another test
* Also check for invites
* Improve error checks
* Update comment
* Update tests
* Fix test
Co-authored-by: Michal Nicpon <39177923+michalnicp@users.noreply.github.com>
* Add CentOS parsing and post-processing in fleet
* Add tests and amend SyncCPEDatabase
* Add test for centosPostProcessing
* Changes from PR comments
* Amend software test
* Fix sync test
* Add index to source and vendor
* Use os.MkdirTemp
* Rearrange migrations
* Regenerate test schema
* Add support for testing migrations (#4112)
* Add support for testing migrations
* Rename migration in tests
* Changes suggested in PR
* Go mod tidy
* Fix OSVersion to work properly when distribution does not follow symver
Certain distros such as ClearLinux or ArchLinux do not use symver or any
dotted versioning scheme for their releases. Archlinux uses the static
string "Rolling" and ClearLinux uses a single build number such as 35550
for their versions.
In Fleet console, this shows up as a string like "Archlinux 0.0.0.0"
which makes very little sense to the user. This change makes it so that
if OSQuery cannot generate a dotted version number, we should instead
use the build id as an opaque string.
* Add /changes
* Add tests for os_version
* fixup
* fixup
* Add platform filters for MDM/Munki/Chrome queries
This should help quiet warnings that users/customers have reported when
these queries try to run on platforms without the macadmins extension
tables.
For #4123
* Improve documentation
* add changes file
* revert doc formatting
* Update tests
* Yet another test fix
Co-authored-by: Tomas Touceda <chiiph@gmail.com>
* Split hosts in bucket of minutes
* New approach on jitter
* Use minutes to define the amount of buckets
* Add logging to jitter hash creation
* Clean up code and remove unused jitter
* Fix test
* Add docs and address review comments
* Address review comments
* Fix typo in doc
Co-authored-by: Tomas Touceda <chiiph@gmail.com>
* Allow to search with target_sw with a period
* Remove tests for a different thign
* Add a test for programs as a source as well
* Use MATCH again and add proper tests
* Add sentry
* Fix gosum
* More gosum fixes
* Add missing def for config
* Enrich sentry scope a bit
* Add changes file
* Add goroutine safe scope to errors
* Encapsulate sentry logic
* Add documentation for new flag
* Add sentry capturing to crons and other background tasks
* Only send to sentry when enabled
This helps the period stay under the default request timeouts for most
load balancers.
Some default timeouts:
* AWS ALB - 60s
* Nginx - 60s
* GCP LB - 30s
* wip
* Delete targets when deleting teams, hosts, and labels
* Add changes file
* Fix error message
* Remove unused teamsTable
* Cleanup new pack
* Clean new packs at end of test
@Smjert reported instances of Windows Domain Controllers having massive
resource utilization and being killed by the watchdog when running this
query. In his test environment, this new query performs much better.
* WIP
* Amend tests
* Do not load aggregated stats for packs
* Add option to host lite
* Fix remaining TODOs
* Fix osquery_utils tests
* Fix SQL
* Fix SQL (bis)
* Restore AuthenticateHost to load once
* Code improvements and re-add deferred host save
* More fixes to the PR
* Wrap users table update on tx
* Add caching to ListPacksForHost and ListScheduledQueriesInPack
* Remove SaveHostSoftware (replaced by UpdateHostSoftware)
* Add unit tests for new functionality
* Add changes file
* Fix scheduled queries test
* Remove cleanups and fk for host ids
* Readd missing things to the schema
* Remove unused
* Add changes file and fix some error messages
* Fix test
* Use tx instead of plain writer
* Other fixes
* More not found test fixes
* Go back to getcontext
* Add fixes for running tests with mysql:8
* Add getServer function
* Test github matrix
* Add changes file for the user facing fix
* Remove unused mysql8 docker-compose
* Add webhook to app config
* Add redis failing policies set and webhook
* Add basic webhook test
* Store hostname in redis
* Global policy deletion to remove policy ID from set and config
* Also process new passing policies
* Fix unit test
* Sort hosts
* Add more tests
* Add ListSets to the failing policies interface
* Fix server URL and garbage collect on the triggering side
* Do not use Redis SCAN
* Fix Redis operation order
* Add API changes to doc
* Add comments
* Add more tests
* Fix tests
* Add tests for config update upon deletion of policies
* Run make dump-test-schema
* Ignore policies that failed to run
* Add proper unit tests to trigger logic
* Fix comments
* WIP
* Add tests to service_osquerty_test.go
* Use SSCAN for listing hosts instead of SMEMBERS
* Add failing policies to docs/01-Using-Fleet/configuration-files/README.md
* Remove skip
* Fix PR comments
* Update return values to be null if the data is not available
* Return nil in the parent object if neither is available
* Improve readability of the code
* Finish first draft of API versions
* wip
* Finalize tests
* Revert change in handler
* Remove made up version
* Update versioning with aliases
* Add changes file
* Address review comments
* Revert overupdated routes
* Expand life time of deprecated APIs
* Fix test
* Comment out problematic part of test
* Revert bad path changes
* Draft for mdm and munki as direct ingest funcs
* Expose mdm/munki over API
* Add test for service and fix bug
* Update queries test
* Fix lint
* Address review comments
* Remove tx from SaveHost
Also change REPLACE for software for an insert.
* Remove bundle identifier from the get since it's not part of the key
* Revert unneeded change
* Add software count API
* Fix makefile
* Fine no mock generating at this point
* Actually, one last try
* Use go install instead
* Fix go sum/mod
* Improve documentation
* Try setting node to 14
* Do caching of app config per instance instead of across all of them in redis
* Add changes file
* Simplify code based on review comment
* Use go-cache instead of creating our own
* Dont export consts
* Copy app config before returning it
* Fix lint
* Update go sum
* Update go sum
* Add fleetctl command to get missing migrations
* Fix copy paste and lint
* Detect migrations applied out of order
* Add extra bullet to changes
* Trigger creation of migration status tables
* Fix unit tests
* PR fixes
* PR comment fixes
This is a temporary mitigation for the issue described in #3065.
The intent is to merge this, cut a 4.6.1 release, and then come up with
a more comprehensive solution for 4.7.0.
* Further optimize host select with policies
* Add missing row close
* Improve migration
* Skip migration if column already exists
* Add test for host with/without extras
