Related to #8167, from RFC5280:
> The serial number MUST be a positive integer assigned by the CA to
> each certificate. It MUST be unique for each certificate issued by a
> given CA (i.e., the issuer name and serial number identify a unique
> certificate).
Without this change, we're issuing two different certificates with the
same serial number (1, one) and issuer name:
1. One during `fleetctl generate mdm-apple`
1. One during the first SCEP enrollment
* Make activities type and auto-generate docs for each type
* Add pageOrderInSection to not break site
* Add do not edit note to generated file
* Add make generate-doc step
* Fix main merge
* Improve how we are logging failed queries and windows detail queries
* Update failing tests
* Remove unneeded table in this particular select
* Update tests now that we dont rely on release_id
* Update ingestion since we want to ingest failures for policies and live queries
* We also want labels with failures
* Fix test since we are logging errors slightly differently
* Improve logging and error handling
* Revert back test change
* Actually, we need to OR the booleans to maintain logic
This needed to be set specifically for the live query websocket dialer,
since that doesn't use the HTTP client that the rest of the API client
methods use.
Co-authored-by: Tomas Touceda <chiiph@gmail.com>
* add premium feature critical policies
* update documentation
* add test for premium-only field
* update old change-file
* test policies more comprehensively
* also test team policies
* PATCH returns wrong timestamp, updating test
rel: https://github.com/fleetdm/fleet/issues/8768 thanks to @gillespi314 for the idea, the test consistently fails with:
```
=== RUN TestHosts/TestHostOrder
hosts_test.go:5392:
Error Trace: /home/runner/work/fleet/fleet/server/datastore/mysql/hosts_test.go:5392
/home/runner/work/fleet/fleet/server/datastore/mysql/hosts_test.go:5416
/home/runner/work/fleet/fleet/server/datastore/mysql/hosts_test.go:139
Error: "[]" should have 3 item(s), but has 0
Test: TestHosts/TestHostOrder
```
and the relevant code around line `5416`:
```go
_, err = ds.writer.Exec(`UPDATE hosts SET created_at = created_at + id`)
require.NoError(t, err)
hosts, err = ds.ListHosts(ctx, fleet.TeamFilter{User: test.UserAdmin}, fleet.HostListOptions{
ListOptions: fleet.ListOptions{
OrderKey: "created_at",
After: "2010-10-22T20:22:03Z",
OrderDirection: fleet.OrderAscending,
},
})
require.NoError(t, err)
chk(hosts, "0001", "0004", "0003")
```
This PR changes it to be `UPDATE hosts SET created_at = DATE_ADD(created_at, INTERVAL id DAY)` instead, which seems to fix the issue (so far 3 runs without issues)
This test was failing in Dec 2022, because in this line:
```go
_, err := sut.GetFeed((now.AddDate(0, 1, 0)).Month(), now.Year())
```
`(now.AddDate(0, 1, 0)).Month()` returns `"January"` , and `now.Year()` returns `2022` , so we were sending a date in the past.
The value of this attribute is used by macOS in "System Preferences >
Profiles" to display a note saying "This Mac is supervised and managed
by <xyz>".
When <PayloadOrganization> is not set, it uses the URL of the MDM server
for the message, which looks unpolished.
The OVAL analyzer falsely assumes that any vulnerabilities detected on a
host only come from OVAL. However, it is possible that NVD detects
vulnerabilities on these hosts even though it excludes software from
deb_packages and rpm_packages. For example, a python package twisted
v22.20 has a vulnerability CVE-2022-39348 detected by NVD. The OVAL
analyzer would delete this vulnerability, and it would be re-inserted by
the NVD scanner on the next run. This creates a loop.
The fix is to only delete vulnerabilities that are actually detected
using OVAL. We already store this in the source column in the
software_cve table.
This improves osquery-perf with support for a more realistic orbit + fleet desktop simulation as described in #8212
This was based on the work done by @sharvilshah in his branch.
Previously, the primary ip was determined using the interface with the most traffic. This had a few issues and required explicitly filtering out certain interfaces that we don't want eg loopback, docker.
Instead, use the default route to determine the primary (private) ip. If there are multiple, use the one with the lowest metric and prefer ipv4 addresses.
Related to https://github.com/fleetdm/fleet/issues/8010 and https://github.com/fleetdm/fleet/issues/8013 this prevents a bug that happens when:
1. A team doesn't have a `config.features` key in the JSON stored in the table or `config` is `NULL`
2. The team is edited from the UI
All `config.features` will default to `false`, which can be a problem if your global settings are `true` for both (which is the default)
related to #8031, this adds the following headers to HTML responses:
- Strict-Transport-Security: informs browsers that the site should only
be accessed using HTTPS, and that any future attempts to access it
using HTTP should automatically be converted to HTTPS.
- X-Frames-Options: disallows embedding the UI in other sites via
<frame>, <iframe>, <embed> or <object>, which can prevent attacks like
clickjacking.
- X-Content-Type-Options: prevents browsers from trying to guess the MIME
type which can cause browsers to transform non-executable content into
executable content.
- Referrer-Policy: prevents leaking the origin of the referrer in the
Referer.
additionally, this ensures we set `X-Content-Type-Options` for CSV and
installer responses.
Related to #7664, this cleans up all policy memberships for a host when its re-enrolled, afterwards only the relevant policy memberships for the host will be created.
* WIP
* Adding DEP functionality to Fleet
* Better organize additional MDM code
* Add cmdr.py and amend API paths
* Fix lint
* Add demo file
* Fix demo.md
* go mod tidy
* Add munki setup to Fleet
* Add diagram to demo.md
* Add fixes
* Update TODOs and demo.md
* Fix cmdr.py and add TODO
* Add endpoints to demo.md
* Add more Munki PoC/demo stuff
* WIP
* Remove proposals from PoC
* Replace prepare commands with fleetctl commands
* Update demo.md with current state
* Remove config field
* Amend demo
* Remove Munki setup from MVP-Dogfood
* Update demo.md
* Add apple mdm commands (#7769)
* fleetctl enqueue mdm command
* fix deps
* Fix build
Co-authored-by: Lucas Rodriguez <lucas@fleetdm.com>
* Add command to upload installers
* go mod tidy
* fix subcommands help
There is a bug in urfave/cli where help text is not generated properly when subcommands
are nested too deep.
* Add support for installing apps
* Add a way to list enrolled devices
* Add dep listing
* Rearrange endpoints
* Move DEP routine to schedule
* Define paths globally
* Add a way to list enrollments and installers
* Parse device-ids as comma-separated string
* Remove unused types
* Add simple commands and nest under enqueue-command
* Fix simple commands
* Add help to enqueue-command
* merge apple_mdm database
* Fix commands
* update nanomdm
* Split nanomdm and nanodep schemas
* Set 512 MB in memory for upload
* Remove empty file
* Amend profile
* Add sample commands
* Add delete installers and fix bug in DEP profile assigning
* Add dogfood.md deployment guide
* Update schema.sql
* Dump schema with MySQL 5
* Set default value for authenticate_at
* add tokens to enrollment profiles
When a device downloads an MDM enrollment profile, verify the token passed
as a query parameter. This ensures untrusted devices don't enroll with
our MDM server.
- Rename enrollments to enrollment profiles. Enrollments is used by nano
to refer to devices that are enrolled with MDM
- Rename endpoint /api/<version>/fleet/mdm/apple/enrollments to ../enrollmentprofiles
- Generate a token for authentication when creating an enrollment profile
- Return unauthorized if token is invalid when downloading an enrollment profile from /api/mdm/apple/enroll?token=
* remove mdm apple server url
* update docs
* make dump-test-schema
* Update nanomdm with missing prefix table
* Add docs and simplify changes
* Add changes file
* Add method docs
* Fix compile and revert prepare.go changes
* Revert migration status check change
* Amend comments
* Add more docs
* Clarify storage of installers
* Remove TODO
* Remove unused
* update dogfood.md
* remove cmdr.py
* Add authorization tests
* Add TODO comment
* use kitlog for nano logging
* Add yaml tags
* Remove unused flag
* Remove changes file
* Only run DEP routine if MDM is enabled
* Add docs to all new exported types
* Add docs
* more nano logging changes
* Fix unintentional removal
* more nano logging changes
* Fix compile test
* Use string for configs and fix config test
* Add docs and amend changes
* revert changes to basicAuthHandler
* remove exported BasicAuthHandler
* rename rego authz type
* Add more information to dep list
* add db tag
* update deps
* Fix schema
* Remove unimplemented
Co-authored-by: Michal Nicpon <39177923+michalnicp@users.noreply.github.com>
Co-authored-by: Michal Nicpon <michal@fleetdm.com>
* Support environments with revoked enroll secrets
* Add instructions on how to fix Orbit enroll
* Rename to last_recorded_error
* Add alternative instructions
related to https://github.com/fleetdm/fleet/issues/7199, this adds email validation to the `verifyCreateShared` which is used for user creation in the server.
validation messages come directly from Go's `net/mail` package.
```
~/fleet $ curl 'https://localhost:8080/api/latest/fleet/users/admin' -X POST -H 'Authorization: Bearer $TOKEN' --data-raw '{"email":"asdf","name":"asdf@asd.com","password":"as;lkdfjasdlk;fja3234@","global_role":"observer","teams":[]}'
{
"message": "Validation Failed",
"errors": [
{
"name": "email",
"reason": "mail: missing '@' or angle-addr"
}
]
}
```
This adds a new mechanism to allow us to handle compatibility issues between Orbit, Fleet Server and Fleet Desktop.
The general idea is to _always_ send a custom header of the form:
```
fleet-capabilities-header = "X-Fleet-Capabilities:" capabilities
capabilities = capability * (,)
capability = string
```
Both from the server to the clients (Orbit, Fleet Desktop) and vice-versa. For an example, see: 8c0bbdd291
Also, the following applies:
- Backwards compat: if the header is not present, assume that orbit/fleet doesn't have the capability
- The current capabilities endpoint will be removed
### Motivation
This solution is trying to solve the following problems:
- We have three independent processes communicating with each other (Fleet Desktop, Orbit and Fleet Server). Each process can be updated independently, and therefore we need a way for each process to know what features are supported by its peers.
- We originally implemented a dedicated API endpoint in the server that returned a list of the capabilities (or "features") enabled, we found this, and any other server-only solution (like API versioning) to be insufficient because:
- There are cases in which the server also needs to know which features are supported by its clients
- Clients needed to poll for changes to detect if the capabilities supported by the server change, by sending the capabilities on each request we have a much cleaner way to handling different responses.
- We are also introducing an unauthenticated endpoint to get the server features, this gives us flexibility if we need to implement different authentication mechanisms, and was one of the pitfalls of the first implementation.
Related to https://github.com/fleetdm/fleet/issues/7929
Configuration and fixes for the Fleet server and frontend to add support
for https://github.com/Uptycs/kubequery.
Co-authored-by: Michal Nicpon <michal@fleetdm.com>
* Bump go to 1.19.1
* Bump remaining go-version to the 1.19.1
* Add extra paths for test-go
* Oops, putting the right path in the right place
* gofmt file
* gofmt ALL THE THINGS
* Moar changes
* Actually, go.mod doesn't like minor versions
- Add a new "Configuration for contributors" doc page. Move settings that are not recommended for production use
- Remove settings modified in the `config` YAML document from the deploying/configuration doc page
- Document all keys in `config` and `teams` YAML documents
- Add comments to several `.go` files and remove unused struct
This fixes an unreleased bug I introduced in eeefe2, as the config colum in the teams table is nulleable, it unmarshalls into nil and we can't dereference the variable
This adjusts functions related to applying team specs to make sure the `features` key works as expected, this is:
1. Create Team flow
1. if no features are provided, we copy whatever features are set in the global config
2. otherwise, we generate a new config merging whatever is provided with the defaults for new installs.
2. Edit Team flow: we always generate a new config merging whatever is provided with the defaults for new installs.
* Make Change software vendor width mig. more robust
If there are two or more entries in the software table with the same name, version, source, release and arch but different vendors then the migration used for increasing the vendor width will fail.
* Ingest installed Windows updates and store them in the windows_updates table.
* Added config option for enabling/disabling Windows update ingestion and Windows OS vuln. detection.
Related to https://github.com/fleetdm/fleet/issues/7312, the motivation behind these changes is to introduce a way to deprecate configurations in `AppConfig`, while still preserving backwards compatibility.
From the Epic:
> NOTE: `host_settings` is now replaced by `features`. We should still support `host_settings` as an alias to `features` for backwards compatibility, but in our communications, we should use and recommend features as the canonical way forward.
* Bug 7320: Fixed dulp. vulns. on software details.
The software details page was showing duplicated vulnerabilities if the software was being used by many hosts.
this adjusts the TestInstallerHeadCheck check to use `DoRaw` instead of `Do`, which sends a body with null when nil is passed for the params parameter.
Updated migration used to increate the width of 'software.vendor' to
address issue found when running migration in aurora: "ALGORITHM=INPLACE is not
supported. Reason: cannot silently convert NULL values, as required in
this SQL_MODE. Try ALGORITHM=COPY."
* improve error handling in vulnerabilities cron
* fix tests
* Use errHandler and go mod tidy
* Add dep that got removed by mod tidy
* add dsl to tools
* Add changes file
Co-authored-by: Michal Nicpon <michal@fleetdm.com>
Co-authored-by: Tomas Touceda <chiiph@gmail.com>
* add google analytics to sandbox instances
* Add serverType variable to frontend handler
* update version of html-webpack-plugin
Co-authored-by: Lucas Rodriguez <lucas@fleetdm.com>
Related to #6894, this entirely replaces FLEET_DEMO with the server config added in #6597
As part of this, I also implemented a small refactor to the integration test suite to allow setting a custom config when the server is initialized.
This commit replaces `os.Setenv` with `t.Setenv` in tests. The
environment variable is automatically restored to its original value
when the test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.Setenv
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
This improves the installerstore CLI tool with:
- The ability to create tests buckets for local development (otherwise you have to interact with another CLI or the MinIO UI)
- Improved error handling and messaging.
Related to #6365 this adds a new tool to upload pre-built Orbit installers to a storage blob. It uses the same file conventions that the Fleet server expects, making it useful for local testing and infra envs alike.\
Usage and details in the README
Related to #6365, this extends the datastore/s3 package to retrieve installers from S3 according to the conventions listed in the parent issue. This also includes:
- A minor refactor to decouple Carves-related functionality from the core S3 functionality
- Set-up to run tests using minio (only enabled via the FILE_STORAGE_TEST env flag)
* Add rate limits for device authed endpoints
* Fix lint
* Add missing test
* Fix test
* Increase the quota for desktop endpoints
* Add comment about quota
* Add server support for Fleet Sandbox demo login
This adds an endpoint `/api/latest/fleet/demologin` that provides a
redirect for the fleetdm.com portion of Fleet Sandbox to automatically
log in a user. The username and password must be provided as form
values. The endpoint is only enabled if `FLEET_DEMO=1` is set in the
server environment.
This was tested locally with the following HTML served by `python3 -m
http.server`, and the Fleet server running with `FLEET_DEMO=1
./build/fleet serve --dev`:
```
<!DOCTYPE html>
<body>
<form
method="post"
action="https://localhost:8080/api/latest/fleet/demologin"
id="demologin"
>
<input type="hidden" name="email" value="admin@example.com" />
<input type="hidden" name="password" value="admin123123#" />
<input type="submit"/>
</form>
<script type="text/javascript">
document.forms["demologin"].submit();
</script>
</body>
</html>
```
For Fleet sandbox purposes, the `action` should be set to the correct
hostname for the sandbox instance, while the `email` and `password`
should be set to the same credentials that were provided when creating
the instance.
* lucas comments
* Add integration tests
* Fix status codes and add comments
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
Related to #5898, this reports an anonymized summary of errors stored in Redis into the analytics payload.
For each error stored, this includes:
- A `count` attribute with the number of occurrences of the error
- A `loc` attribute with the 3 topmost lines in the stack trace. Note that stack traces only contain package name + line number (example: github.com/fleetdm/fleet/server.go:12
This also includes a minor refactor around error types.
This commit replaces `ioutil.TempDir` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.
Prior to this commit, temporary directory created using `ioutil.TempDir`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}
is also tedious, but `t.TempDir` handles this for us nicely.
Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
* Do not use golangci action for better reproducibility
* Add fix to trigger build
* Fix all reported issues
* fix more lint errors
* Add missing import
* Remove unused method
* Remove change not necessary
Related to #6063, this adds a new device API to get an object with boolean values that we can use as feature flags to manage backwards compatibility in Fleet Desktop.
Feature: Improve our capability to detect vulnerable software on Ubuntu hosts
To improve the capability of detecting vulnerable software on Ubuntu, we are now using OVAL definitions to detect vulnerable software on Ubuntu hosts. If data sync is enabled (disable_data_sync=false) OVAL definitions are automatically kept up to date (they are 'refreshed' once per day) - there's also the option to manually download the OVAL definitions using the 'fleetctl vulnerability-data-stream' command. Downloaded definitions are then parsed into an intermediary format and then used to identify vulnerable software on Ubuntu hosts. Finally, any 'recent' detected vulnerabilities are sent to any third-party integrations.
This removes policy information from `GET /api/_version_/fleet/device/{token}` from non-premium Fleet instances.
Starting the server with `./build/fleet serve --dev --dev_license`
```bash
$ curl -s https://localhost:8080/api/latest/fleet/device/1804e808-171f-4dda-9bec-f695b2f2371a | jq '.host.policies // "not present"'
[
{
"id": 3,
"name": "Antivirus healthy (Linux)",
"query": "SELECT score FROM (SELECT case when COUNT(*) = 2 then 1 ELSE 0 END AS score FROM processes WHERE (name = 'clamd') OR (name = 'freshclam')) WHERE score == 1;",
"description": "Checks that both ClamAV's daemon and its updater service (freshclam) are running.",
"author_id": 1,
"author_name": "Roberto",
"author_email": "test@example.com",
"team_id": null,
"resolution": "Ensure ClamAV and Freshclam are installed and running.",
"platform": "darwin,linux",
"created_at": "2022-05-23T20:53:36Z",
"updated_at": "2022-06-03T13:17:42Z",
"response": ""
}
]
```
Starting the server with `./build/fleet serve --dev`
```bash
$ curl -s https://localhost:8080/api/latest/fleet/device/1804e808-171f-4dda-9bec-f695b2f2371a | jq '.host.policies // "not present"'
"not present"
```
This adds a new API client named DeviceClient to server/service, meant to consume device endpoints and be used from Fleet Desktop.
Some of the logic to make requests and parse responses was very repetitive, so I introduced a private baseClient type and moved some of the logic of the existent Client there.
Related to #5685 and #5697
This adds a new device authenticated endpoint, `/api/_version_/fleet/device/{token}/policies` to retrieve the device policies.
An example request / response looks like:
```bash
curl https://localhost:8080/api/latest/fleet/device/1804e808-171f-4dda-9bec-f695b2f2371a/policies
```
```json
{
"policies": [
{
"id": 3,
"name": "Antivirus healthy (Linux)",
"query": "SELECT score FROM (SELECT case when COUNT(*) = 2 then 1 ELSE 0 END AS score FROM processes WHERE (name = 'clamd') OR (name = 'freshclam')) WHERE score == 1;",
"description": "Checks that both ClamAV's daemon and its updater service (freshclam) are running.",
"author_id": 1,
"author_name": "Admin",
"author_email": "admin@example.com",
"team_id": null,
"resolution": "Ensure ClamAV and Freshclam are installed and running.",
"platform": "darwin,windows,linux",
"created_at": "2022-05-23T20:53:36Z",
"updated_at": "2022-05-23T20:53:36Z",
"response": "fail"
}
]
}
```
Related to [#5685](https://github.com/fleetdm/fleet/issues/5685), in another changeset I will be adding "client" endpoints so we can consume this endpoint from Fleet Desktop
Related to #5776, this ensures that when a host is re-enrolled on a different team we cleanup existing policy memberships in the same way we do when a host is assigned a team through the API.
