Commit Graph

1630 Commits

Author SHA1 Message Date
Roberto Dip
5a09ac0bfc
refactor how we send Fleet initiated MDM commands (#9903)
https://github.com/fleetdm/fleet/issues/9590

- move the logic to send commands into its own service method that can
be used internally by cron jobs and other services.
- deprecate the use of `rawEnqueueCommand` as it's copyied from the
nanomdm codebase where it's used in other context as a general command
API handler
2023-02-17 16:26:51 -03:00
Lucas Manuel Rodriguez
1f97514200
Only set public IPs on host.public_ip and add docs (#9900)
#9857

The "Public IP address" field is sometimes set to a "Private IP" on the
following types of Fleet deployments:
- Local deployments.
- Deployments where Fleet is on a private network.
- Deployments where an agent connects to Fleet not via the public
internet.

This PR will prevent a private IP to be set on the `host.public_ip`
field.
And this PR also adds documentation on how Fleet deduces the public IPs
of the devices so that a user can make the changes to fix this.

- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- ~[ ] Added/updated tests~
- [X] Manual QA for all new/changed functionality
  - ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
2023-02-17 13:00:56 -03:00
gillespi314
086964726d
Add API endpoints for MDM config profiles (#9817)
Issue #9586

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- [ ] Documented any permissions changes
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
2023-02-17 09:28:28 -06:00
Benjamin Edwards
0db432f7e3
vuln processing distinct command (#9813)
closes https://github.com/fleetdm/fleet/issues/3723

Add new vuln processing command, configs, and documentation on how to utilize.
2023-02-17 10:00:57 -05:00
Lucas Manuel Rodriguez
b757e447bc
Fix private IP ingestion in network_interface_unix and network_interface_windows. (#9884)
#8924

This is reproduced in dogfood for `dogfood-centos-box` and
`dogfood-ubuntu-box` where their "Private IP" is also their "Public IP".
Given that these hosts have their "Primary IP" configured to be their
"Public IP" alongside their "Private IP", the `network_interface_unix`
and `network_interface_windows` queries are now changed to ingest only
private IPs for the "Private IP" field.

- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- ~[ ] Added/updated tests~
- [X] Manual QA for all new/changed functionality
  - ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
2023-02-16 17:16:40 -03:00
Martin Angers
4a1f3988f0
Add "edited macos profiles" activity when applying custom settings via fleetctl (#9862)
#9587 and #9639
2023-02-16 13:53:26 -03:00
Martin Angers
33f33163a9
Add macos custom profiles support via fleetctl apply (#9824) 2023-02-15 13:01:44 -05:00
gillespi314
8daead3ef5
Update route for hosts encryption key (#9821) 2023-02-14 12:17:31 -06:00
gillespi314
f1227d7303
Add authz and datastore methods for mdm config profiles (#9781) 2023-02-14 09:12:18 -06:00
Roberto Dip
0f5a35061e
don't filter DEP hosts by OS before ingesting and improve logs (#9815)
Related to https://github.com/fleetdm/fleet/issues/9653 I couldn't find
any documentation to back this up, but I have a strong suspicion that
the `os` field in the device sync response might come empty in some
scenarios (particularly, when a laptop is brand new, which is hard to
reproduce 😅)

My thoughts are:

1. For the recently purchased MacBooks,
`IngestMDMAppleDevicesFromDEPSync` didn't create an entry in the
database, BUT `nanodep.Assigner.ProcessDeviceResponse` correctly
assigned a DEP profile (the devices were able to enroll). Both methods
filter by `op_type` but only ours filters by `os`.
2. I think this is safe-ish to do, as you will normally assign a MDM
server per device type in ABM

![image](https://user-images.githubusercontent.com/4419992/218732609-0936e3a9-cadf-4485-9aa4-af2c9398cff9.png)
3. I have added extra logs to try to prove this hypothesis next time a
brand new device comes in, let's keep an eye on and re-evaluate this
approach.
2023-02-14 10:23:19 -03:00
Roberto Dip
119ada71da
fix XML formatting of RemoveProfile command (#9805)
potential fix for https://github.com/fleetdm/fleet/issues/9802, I still
can't reproduce locally but it might be that ngrok is doing some magic
encoding/decoding the bytes transferred.
2023-02-13 17:20:43 -03:00
gillespi314
bcdc4691bb
Enable installation and auto-updates of Nudge via Orbit (#9605)
Issue #9093 

Co-authored-by: Roberto Dip <me@roperzh.com>
Co-authored-by: Roberto Dip <dip.jesusr@gmail.com>
2023-02-10 17:03:43 -03:00
gillespi314
aca2449566
Add new data types and table for Apple MDM config profiles (#9758) 2023-02-08 18:36:20 -06:00
Roberto Dip
7cd581866a
add API endpoint to see disk encryption key (#9713)
https://github.com/fleetdm/fleet/issues/8708
2023-02-08 20:20:23 -03:00
Roberto Dip
046401d190
Ingest file vault recovery keys in macOS (#9712)
Related + details at https://github.com/fleetdm/fleet/issues/8708
2023-02-08 11:49:42 -03:00
Jacob Shandling
9630f64e40
Wire up UI and server for correct ABM credentials download (#9660)
Final alignment of UI and server for ABM credential downloads

- [x] Added/updated test inventory
- [x] Manual QA for all new/changed functionality

---------

Co-authored-by: Jacob Shandling <jacob@fleetdm.com>
Co-authored-by: Roberto Dip <dip.jesusr@gmail.com>
2023-02-03 11:02:50 -08:00
Lucas Manuel Rodriguez
dfe6b53963
Make fleetctl get teams --yaml output compatible with fleetctl apply -f (#9626)
#9535

- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
- ~[ ] Documented any permissions changes~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
  - ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
2023-02-02 14:46:50 -03:00
Roberto Dip
e06b00df11
Add readonly MDM.EnabledAndConfigured to app config and device responses (#9575)
Related to #9571, this adds a new value to both responses which is
calculated when the Fleet server is started, and only set to `true` if
the server is properly configured for MDM.

This helps the UI to determine wether or not we should show certain UI
elements that we only want to show to servers with MDM enabled.
2023-02-01 14:47:52 -03:00
Roberto Dip
ffed7f8ebe
return 422 status code if fleetdm.com returns any 4xx status for CSR (#9610)
Related to https://github.com/fleetdm/fleet/issues/9588, we now handle 4xx responses from the fleetdm.com server and forward those to the client.

At the time of this commit, the only 4xx response that wasn't already handled by the server is because of an invalid email domain, so we assume that, but we should look into establishing a pattern of error messages with the website instead.
2023-02-01 12:50:22 -03:00
Martin Angers
f47b7f538f
Mention that the min macos version activity is created when deadline is modified too (#9608)
Tiny follow-up to https://github.com/fleetdm/fleet/pull/9594
2023-02-01 09:10:29 -05:00
Martin Angers
c805ea2154
Record activity when the macos minimum version requirement is edited (#9594) 2023-01-31 16:36:18 -06:00
Roberto Dip
4c4c114e96
add mocks + tests and move things around (#9574)
#8948

- Add more go:generate commands for MDM mocks
- Add unit and integration tests for MDM code
- Move interfaces from their PoC location to match existing patterns
2023-01-31 11:46:01 -03:00
Martin Angers
8a137e2b5b
Move host details mdm properties to new mdm object (#9505) 2023-01-30 16:40:11 -06:00
Lucas Manuel Rodriguez
8163b7d8da
Update live query selector logic (OR -> AND) (#9559)
See requirements in #8682.

Two assumptions on the implementation (@zayhanlon please take a look):
- Hosts explicitly selected to run always run the live query (no matter
the values on the selectors).
- When selecting `All hosts`, selecting any other platform or label is
kind of a no-op. We should look into graying out all the selectors if
the user selects `All hosts`.

- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [X] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- ~[ ] Documented any permissions changes~
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
  - ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
2023-01-30 18:35:56 -03:00
Juan Fernandez
e376d53294
Bug 9386: Docs chrome extension has the wrong CPE. (#9524)
Related to #9386 - this should fix one of the three reported problems.

* Add the ability to add exclusion rules to cpe_translations.
* Added exclusion rule for Docs chrome extension.
2023-01-27 15:11:20 -04:00
Roberto Dip
b4f46bf91c
run go mod tidy and fix incorrectly imported dependency (#9529)
Two minor things are happening here:

1. A new direct dependency was added in #9489, this runs `go mod tidy`
to update `go.mod` and `go.sum`
2. `"go.uber.org/atomic"` was automatically imported instead of
`"sys/atomic"` in #9494
2023-01-27 14:51:27 -03:00
Roberto Dip
0c9a79b248
Adjust Nudge configuration to match the specification (#9519)
Related to #9013 this adjusts the Nudge configuration to match the spec.
2023-01-26 19:52:03 -03:00
Marcos Oviedo
86c2b9ada0
Setting restricted ACL for windows secret-orbit-node-file (#9457)
Setting restricted ACL for windows secret-orbit-node-file
2023-01-26 18:51:24 -03:00
Jan Christian Grünhage
e5790521ca
detect nixos as a linux platform (#9468)
# Checklist for submitter

- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
2023-01-25 18:03:24 -03:00
Roberto Dip
851545c21f
create and send Nudge configuration to hosts (#9491)
related to #9348
2023-01-25 17:03:40 -03:00
Martin Angers
d0e6891d10
Add endpoint to trigger CSR request for APNs on fleetdm.com (#9494) 2023-01-25 14:44:29 -05:00
Luke Heath
6d34bb8a97
Add note to docs indicating Fleet MDM specific endpoints. (#9473) 2023-01-24 10:57:22 -06:00
Roberto Dip
2d25a3f48d
add mdm root key and macos_updates to app and team configs (#9442)
Related to https://github.com/fleetdm/fleet/issues/9345,
https://github.com/fleetdm/fleet/issues/9358 and
https://github.com/fleetdm/fleet/issues/9346 this adds:

1. The ability to configure `mdm.macos_updates` via `PATCH /config` and
`PATCH /teams/{id}`
3. The ability to configure `mdm.macos_updates` by using `fleetctl apply
-f` for teams and global config.
2023-01-24 13:20:02 -03:00
gillespi314
1b4e8e692a
Add API endpoint to unenroll a host from Fleet's MDM (#9447) 2023-01-23 17:05:24 -06:00
Roberto Dip
c2f3d6916c
omit actor information from activities if not present (#9428)
This is a follow up from #9127 in order to completely omit those fields
from the response instead of returning `null` in order to be consistent
with how the API generally works.
2023-01-23 12:17:38 -03:00
Lucas Manuel Rodriguez
328004d679
Log failed login attempts as activities (#9430)
#9119

To test the SSO changes locally you can use:

https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Testing-and-local-development.md#testing-sso

@RachelElysia Please take a look at the UI changes (All I did was
copy/paste and amend the changes for the new activity type.)

IMO we shouldn't display an avatar because there's no "actual user"
involved in these failed login attempts activities (by "actual user" I
mean the user attributed to the activity):

<img width="446" alt="Screenshot 2023-01-19 at 10 41 05"
src="https://user-images.githubusercontent.com/2073526/213524771-b85901ce-eec0-4cf3-919c-73162285e20b.png">

- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [X] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- ~[ ] Documented any permissions changes~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [X] Added/updated tests
- [X] Manual QA for all new/changed functionality
  - ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
2023-01-20 12:43:22 -03:00
Lucas Manuel Rodriguez
1cdfe2d351
Add summary of hardcoded detail queries (#9425)
Adds a process to generate a summary of the hardcoded Fleet queries.
2023-01-19 17:44:03 -03:00
Lucas Manuel Rodriguez
0f436d26db
Run authorization check before processing policy spec (#9270)
I was getting a cryptic error and no logs in fleet when trying to apply
an invalid spec file:
```sh
fleetctl apply -f ./some-queries.yml
Error: applying policies: POST /api/latest/fleet/spec/policies received status 500 forbidden: forbidden
```

With the changes in this PR now I get a more descriptive error (bad
request):

```sh
fleetctl apply -f ./some-queries.yml
Error: applying policies: POST /api/latest/fleet/spec/policies received status 400 Bad request: policy spec payload verification: policy query cannot be empty
```

- [X] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- ~[ ] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)~
~- [ ] Documented any permissions changes~
- ~[ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)~
- ~[ ] Added support on fleet's osquery simulator `cmd/osquery-perf` for
new osquery data ingestion features.~
- [x] Added/updated tests
- [X] Manual QA for all new/changed functionality
  - ~For Orbit and Fleet Desktop changes:~
- ~[ ] Manual QA must be performed in the three main OSs, macOS, Windows
and Linux.~
- ~[ ] Auto-update manual QA, from released version of component to new
version (see [tools/tuf/test](../tools/tuf/test/README.md)).~
2023-01-18 12:26:32 -03:00
Gabriel Hernandez
7d4653baaa
add attribute to GET /activities endpoint with pagination metadata (#9279)
relates to https://github.com/fleetdm/fleet/issues/8928

This adds a new `meta` attribute to the "GET /activities" endpoint that
includes pagination metadata. This can allow clients to know if there
are additional items to request.


- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- [x] Added/updated tests
- [x] Manual QA for all new/changed functionality
2023-01-18 12:57:11 +00:00
Martin Angers
6c3738a0d2
Fix a flaky integration test (#9382) 2023-01-17 15:33:27 -05:00
Martin Angers
e89c45776a
Notify orbit via the GET config endpoint that the DEP profile needs to be renewed (#9373) 2023-01-17 13:19:48 -05:00
gillespi314
fba3607c4a
Add mdm status and server url to host endpoint responses (#9320) 2023-01-16 17:08:24 -06:00
Roberto Dip
f87c28e6df
automatically assign DEP enrolled devices to a team if set via config (#9135)
Related to https://github.com/fleetdm/fleet/issues/9068 and
https://github.com/fleetdm/fleet/issues/8733, this builds on the work
done in https://github.com/fleetdm/fleet/pull/9062 to tie the loose ends
and assign the configured team to the device that's enrolling
2023-01-16 17:38:51 -03:00
Roberto Dip
2447a371b0
use nanomdm's multi service + integration tests and fixes (#9269)
**Use nano's multi service**

This allows us to integrate more seamlessly with nano and to run our
custom MDM logic _after_ the request was handled by nano, which gives us
more flexibility (for example: now we can issue commands after a
TokenUpdate message)

From nano's code:

> MultiService executes multiple services for the same service calls.
> The first service returns values or errors to the caller. We give the
> first service a chance to alter any 'core' request data (say, the
> Enrollment ID) by waiting for it to finish then we run the remaining
> services' calls in parallel.

**Integration tests + fixes**

- Move some of the service logic from `cmd/` to `server/service`
- Add integration tests for the MDM enrollment flow, including SCEP
authentication.
- Fixed a bug that set `host_mdm.mdm_id = 0`  during MDM enrollment due
  to how MySQL reports the last insert id when `ON DUPLICATE KEY` is
  used.
- Completely remove the host row from `host_mdm` when a device is
  unenrolled from MDM to match the behavior of how we ingest MDM data
  from osquery

Related to https://github.com/fleetdm/fleet/issues/8708,
https://github.com/fleetdm/fleet/issues/9034
2023-01-16 17:06:30 -03:00
gillespi314
bae10e0ba2
Adjust max age for cron stats entries (#9190) 2023-01-16 10:23:52 -06:00
Martin Angers
e360013dc3
Implement API endpoint for MDM manual enrollment profile download (#9232) 2023-01-16 10:22:12 -05:00
Martin Angers
1abcb6e227
Add integration tests for setting apple BM default team (#9291) 2023-01-16 09:23:32 -05:00
Zach Wasserman
3016ac1136
Add CrOS as valid host platform (#9308)
This will help reduce spam in the logs from the unrecognized platform.
2023-01-13 14:10:38 -08:00
Tomas Touceda
65783d49c7
Add feature to change or disable Fleet's hardcoded detail queries (#9117)
#8984

- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Added/updated tests

Co-authored-by: Lucas Rodriguez <lucas@fleetdm.com>
2023-01-09 08:56:10 -03:00
Tomas Touceda
398bb38da7
Add software_updated_at field to host (#9116)
#9012

- [x] Changes file added for user-visible changes in `changes/` or
`orbit/changes/`.
See [Changes
files](https://fleetdm.com/docs/contributing/committing-changes#changes-files)
for more information.
- [x] Documented any API changes (docs/Using-Fleet/REST-API.md or
docs/Contributing/API-for-contributors.md)
- [x] Added/updated tests

Co-authored-by: Lucas Rodriguez <lucas@fleetdm.com>
2023-01-09 08:55:43 -03:00
Martin Angers
656e5bfc70
Flag when the Apple BM terms have expired (#9091)
#8862 

Co-authored-by: Roberto Dip <dip.jesusr@gmail.com>
Co-authored-by: gillespi314 <73313222+gillespi314@users.noreply.github.com>
2023-01-06 17:44:20 -03:00
gillespi314
f9f1a94b48
Update mdm enrollment status API response (#9189) 2023-01-05 15:52:46 -06:00
Lucas Manuel Rodriguez
ba6b6e12cc
Fix ingestion of MDM server for macOS and Windows hosts (#9133)
* Fix ingestion of MDM server for macOS and Windows hosts

* Return early if there are no mobile_device_management_solutions to clean up

* Run make dump-test-schema

* Fix TestHosts/DeleteHosts test

* Add test for host unenrolling from MDM

* Fix lint

* Fix typo

* Identify Fleet MDM

* Move logic to deduce MDM name out of Datastore

* Run make dump-test-schema

* Remove printfs from migration
2023-01-04 10:29:48 -03:00
Frank Sievertsen
91c90b4caa
Add new configuration option to set default team for Apple Business Manager (#9062) 2023-01-03 16:14:18 -06:00
gillespi314
4f1b97dc96
Add temporary host display name for MDM devices pending enrollment to Fleet (#9165) 2023-01-03 15:45:12 -06:00
Lucas Manuel Rodriguez
f64ba83b2e
Make test more stable (#9162) 2023-01-03 16:39:28 -03:00
Lucas Manuel Rodriguez
ac22aadc13
Fleet server and tooling to use NETWORK_TEST_GITHUB_TOKEN when environment variable is set. (#9143)
* WIP

* Add more logging

* Check rate limit at end of action

* Add github client in more places

* Add new published firefox 93 vulnerabilities to tests

* Remove fmt printfs

* Restore CI check settings

* Readd newline
2023-01-03 14:56:11 -03:00
gillespi314
b990121fdc
Add mdm enrollment pending filter to API endpoints (#9137) 2023-01-03 11:26:23 -06:00
Roberto Dip
1b47f9e700
add activities when a host is enrolled/unenrolled from MDM (#9127)
#8996
2022-12-28 16:41:18 -03:00
Lucas Manuel Rodriguez
34b62cc43e
Fix panic in hosts/{id}/mdm endpoint when host does not have MDM data (#9123) 2022-12-27 16:22:37 -03:00
Lucas Manuel Rodriguez
19ad7cc637
Set interface for response types (#9121)
* Set interface for response types

* Fix TestEndpointer test
2022-12-27 11:26:59 -03:00
gillespi314
94dd1c3745
Ingest pending MDM hosts (#9065)
Co-authored-by @roperzh
2022-12-26 15:32:39 -06:00
Lucas Manuel Rodriguez
c4733a4792
Fix race on TestTriggerMultipleInstances (#9122) 2022-12-26 12:32:37 -03:00
Lucas Manuel Rodriguez
e1bbcfcfda
Generate audit logs for activities (#9001)
* Generate audit logs for activities

* Fix config tests

* Fix TestGetConfig/IncludeServerConfig

* Fix use of AddAttributes in results only

* Stream activities asynchronously

* Fix index and add logging

* Revert change

* Documentation fixes
2022-12-23 19:04:13 -03:00
Roberto Dip
141fb5dbb0
issue SCEP certs using 2 as the initial SerialNumber (#9087)
Related to #8167, from RFC5280:

> The serial number MUST be a positive integer assigned by the CA to
> each certificate. It MUST be unique for each certificate issued by a
> given CA (i.e., the issuer name and serial number identify a unique
> certificate).

Without this change, we're issuing two different certificates with the
same serial number (1, one) and issuer name:

1. One during `fleetctl generate mdm-apple`
1. One during the first SCEP enrollment
2022-12-23 14:55:34 -03:00
Roberto Dip
e191f47aed
fix failed enrollments due to duplicate slashes on paths (#9089)
This ensures URLs in enrollment profiles are properly formatted,
preventing errors as described in #9088.
2022-12-23 14:55:17 -03:00
Lucas Manuel Rodriguez
0e500c8649
Orbit: log response body to ease troubleshooting (#9096) 2022-12-23 13:21:49 -03:00
Lucas Manuel Rodriguez
39f1029390
Make activities typed and auto-generate docs for each type (#9069)
* Make activities type and auto-generate docs for each type

* Add pageOrderInSection to not break site

* Add do not edit note to generated file

* Add make generate-doc step

* Fix main merge
2022-12-23 13:05:16 -03:00
Roberto Dip
80dc7e7338
add an activity for SSO login (#9110) 2022-12-22 20:06:33 -03:00
Tomas Touceda
af0842804d
Improve how we are logging failed queries and windows detail queries (#9059)
* Improve how we are logging failed queries and windows detail queries

* Update failing tests

* Remove unneeded table in this particular select

* Update tests now that we dont rely on release_id

* Update ingestion since we want to ingest failures for policies and live queries

* We also want labels with failures

* Fix test since we are logging errors slightly differently

* Improve logging and error handling

* Revert back test change

* Actually, we need to OR the booleans to maintain logic
2022-12-21 17:37:40 -03:00
Frank Sievertsen
8e65ec67b5
Add new activity that records create/edit/delete user roles (#9086) 2022-12-21 18:30:19 +01:00
Frank Sievertsen
e7d6ed0f3f
Log all successful logins as activity and all attempts with ip in stderr. (#9095) 2022-12-21 18:29:51 +01:00
Sharvil Shah
f1577de0ac
Orbit extensions autoupdate (#8906) 2022-12-21 01:00:55 +05:30
Lucas Manuel Rodriguez
e3a626a00d
Fix authorization check test (#9058) 2022-12-19 10:37:08 -03:00
Tomas Touceda
68bd8661e9
Simplify disk encryption query in linux and filter at ingestion (#9037)
* Simplify disk encryption query in linux and filter at ingestion

* Join with mounts to detect whatever is encrypting /
2022-12-19 10:01:59 -03:00
Gabriel Hernandez
3c136e3f47
Add MDM Card to Windows and All Dashboards and change mdm data source on host details page (#8960) 2022-12-16 15:12:11 -06:00
Frank Sievertsen
7ebb97cd70
Fire automations for hosts that failed before automation enabled (#9028) 2022-12-16 22:00:54 +01:00
Michal Nicpon
37f454c0be
Add dep key pair endpoint (#9007) 2022-12-16 19:39:36 +01:00
gillespi314
836553ba60
Fix cron trigger bug (#8950) 2022-12-16 12:00:42 -06:00
Stephan Miehe
b6a409fa3b
Support redis username (#9018) 2022-12-14 08:53:38 -05:00
Martin Angers
f18a320bd5
Add command get mdm-apple-bm to fleetctl (#8949) 2022-12-12 15:45:53 -05:00
Zach Wasserman
083b781c2c
Send custom headers for live queries (#8991)
This needed to be set specifically for the live query websocket dialer,
since that doesn't use the HTTP client that the rest of the API client
methods use.

Co-authored-by: Tomas Touceda <chiiph@gmail.com>
2022-12-12 07:56:35 -08:00
Frank Sievertsen
a228dcb170
Add critical policies as a premium feature (#8959)
* add premium feature critical policies

* update documentation

* add test for premium-only field

* update old change-file

* test policies more comprehensively

* also test team policies

* PATCH returns wrong timestamp, updating test
2022-12-09 19:23:08 +01:00
Roberto Dip
e68535d468
report errors that can occur during file carving (#8972)
related to https://github.com/fleetdm/fleet/issues/8117
2022-12-09 13:21:30 -03:00
Michal Nicpon
10b3179b63
Add fleetctl generate mdm-apple (#8812) 2022-12-07 18:24:42 +01:00
Martin Angers
d4a3730610
Add new configuration options for Apple Business Manager (#8886) 2022-12-06 15:50:56 -05:00
gillespi314
43b1241a0c
Add fleetctl trigger command (#8877) 2022-12-06 09:56:54 -06:00
Frank Sievertsen
e0766d21a3
Add ability to mark policy as critical. (#8896) 2022-12-06 15:59:20 +01:00
gillespi314
6fb3a87ae9
Enable errcheck linter for golangci-lint (#8899) 2022-12-05 16:50:49 -06:00
Martin Angers
866829dbfa
Implement fleetctl get mdm-apple (#8786) 2022-12-05 11:35:45 -05:00
Martin Angers
f5fc316ad3
Add options for configuring APNS and SCEP keys/certs in Fleet server (#8730) 2022-12-05 10:22:56 -05:00
Roberto Dip
f52e294718
try using DATE_ADD in test to ensure the timestamp is handled correctly (#8915)
rel:  https://github.com/fleetdm/fleet/issues/8768 thanks to @gillespi314 for the idea, the test consistently fails with:

```
 === RUN   TestHosts/TestHostOrder
    hosts_test.go:5392: 
        	Error Trace:	/home/runner/work/fleet/fleet/server/datastore/mysql/hosts_test.go:5392
        	            				/home/runner/work/fleet/fleet/server/datastore/mysql/hosts_test.go:5416
        	            				/home/runner/work/fleet/fleet/server/datastore/mysql/hosts_test.go:139
        	Error:      	"[]" should have 3 item(s), but has 0
        	Test:       	TestHosts/TestHostOrder
```

and the relevant code around line `5416`:

```go
	_, err = ds.writer.Exec(`UPDATE hosts SET created_at = created_at + id`)
	require.NoError(t, err)

	hosts, err = ds.ListHosts(ctx, fleet.TeamFilter{User: test.UserAdmin}, fleet.HostListOptions{
		ListOptions: fleet.ListOptions{
			OrderKey:       "created_at",
			After:          "2010-10-22T20:22:03Z",
			OrderDirection: fleet.OrderAscending,
		},
	})
	require.NoError(t, err)
	chk(hosts, "0001", "0004", "0003")
```

This PR changes it to be `UPDATE hosts SET created_at = DATE_ADD(created_at, INTERVAL id DAY)` instead, which seems to fix the issue (so far 3 runs without issues)
2022-12-02 17:25:12 -03:00
Roberto Dip
837fef4bc4
ensure we provide a future date in MSRC test (#8897)
This test was failing in Dec 2022, because in this line:

```go
_, err := sut.GetFeed((now.AddDate(0, 1, 0)).Month(), now.Year())
```

`(now.AddDate(0, 1, 0)).Month()` returns `"January"` , and `now.Year()` returns `2022` , so we were sending a date in the past.
2022-12-01 11:29:07 -03:00
gillespi314
49f7180a14
Use POST instead of GET for trigger endpoint (#8888) 2022-11-30 11:57:42 -06:00
Artemis Tosini
486590186a
orbit: Correctly handle errors in authenticatedRequest. Fixes #8472 (#8860) 2022-11-29 11:54:36 -05:00
gillespi314
d5c096fa02
Implement schedule triggers (#8747) 2022-11-28 13:28:06 -06:00
Frank Sievertsen
8fc32acf00
Store details (query_name and query_sql) about live query in activity feed (#8842) 2022-11-28 16:42:32 +01:00
gillespi314
4a73d4a887
Adjust flaky tests (#8811) 2022-11-25 12:09:55 -06:00
Tomas Touceda
fe1fa4d78c
Clone AppConfig and ScheduledQuery list by hand to improve CPU usage (#8794)
* Close AppConfig and ScheduledQuery list by hand to improve CPU usage

* Address review comments

* Update remaining mocks
2022-11-23 12:04:06 -03:00
Roberto Dip
d9bf269c60
add dev flag to aid UI development (#8789) 2022-11-22 14:26:36 -03:00
Martin Angers
dc6bfdec57
Add support for missing OS-specific osquery flags in agent options (#8743) 2022-11-21 10:32:17 -05:00
gillespi314
b99ce3865b
Adjust cron schedule tests (#8754) 2022-11-18 11:26:51 -06:00
Roberto Dip
b6f7efe164
add <PayloadOrganization> to MDM enrollment payload (#8752)
The value of this attribute is used by macOS in "System Preferences >
Profiles" to display a note saying "This Mac is supervised and managed
by <xyz>".

When <PayloadOrganization> is not set, it uses the URL of the MDM server
for the message, which looks unpolished.
2022-11-18 11:25:44 -03:00
gillespi314
267aaf0dbe
Add holdLock and releaseLock methods to schedule package (#8464) 2022-11-16 15:14:38 -06:00
Jan Christian Grünhage
8fee954dba
fix: check device mounted at filesystem root for encryption (#8719) 2022-11-15 10:50:58 -05:00
Tomas Touceda
9a801dec8d
detect void as a linux platform as well (#8717)
Co-authored-by: Jan Christian Grünhage <jan.christian@gruenhage.xyz>
2022-11-15 12:26:18 -03:00
Martin Angers
472c8bafb3
Refactor license so it is stored in the context (#8544) 2022-11-15 09:08:05 -05:00
Martin Angers
7f519e44d2
Add migration to drop gigs/percent disk space from hosts (#8646) 2022-11-15 08:27:40 -05:00
Martin Angers
54bb5448a0
Fix missing valid properties in validation of FIM and YARA agent options (#8707) 2022-11-14 15:50:41 -05:00
Michal Nicpon
9ad1721efd
fix issue with duplicate vulns detected using nvd (#8613)
The OVAL analyzer falsely assumes that any vulnerabilities detected on a
host only come from OVAL. However, it is possible that NVD detects
vulnerabilities on these hosts even though it excludes software from
deb_packages and rpm_packages. For example, a python package twisted
v22.20 has a vulnerability CVE-2022-39348 detected by NVD. The OVAL
analyzer would delete this vulnerability, and it would be re-inserted by
the NVD scanner on the next run. This creates a loop.

The fix is to only delete vulnerabilities that are actually detected
using OVAL. We already store this in the source column in the
software_cve table.
2022-11-10 10:28:00 -07:00
Frank Sievertsen
0dbbad8e10
Exclude windows servers from mdm aggregation and lists (#8585) 2022-11-08 10:29:40 +01:00
Frank Sievertsen
4743d31870
Support low_disk_space filter for endpoint /labels/{id}/hosts (#8598) 2022-11-08 09:19:34 +01:00
Tomas Touceda
5d3e27d169
Use tx instead of writer within withTx style code to prevent deadlocks (#8554) 2022-11-03 10:57:50 -03:00
Martin Angers
7fbe394b94
Ingest disk encryption flag for hosts, return in GET /hosts/{id} endpoint (#8526) 2022-11-02 15:44:02 -04:00
Frank Sievertsen
f9728b8435
allow created_at and updated_at as hosts filter (with "after") (#8535) 2022-11-02 15:51:31 +01:00
Mo Zhu
74b8e29cfe
add arch and linuxmint to list of linux distros (#8532)
* add arch and linuxmint to allowlist of linux distros

* fix comma in array

* refer to list of linux distros, not allowlist
2022-11-01 16:24:59 -07:00
Michal Nicpon
1571837d75
initialize empty slice for host counts usage statistics (#8510) 2022-11-01 15:49:02 -06:00
Martin Angers
8f21e026e3
Fix bug with fleetctl apply for teams, clear agent options only if key is present (#8508) 2022-11-01 15:22:45 -04:00
Frank Sievertsen
baa1ddc0f2
Add MDM detection for windows and mdm endpoints (#8479) 2022-11-01 18:22:07 +01:00
Roberto Dip
c51927e873
enhance support for orbit and fleet desktop in osquery-perf (#8217)
This improves osquery-perf with support for a more realistic orbit + fleet desktop simulation as described in #8212

This was based on the work done by @sharvilshah in his branch.
2022-10-28 14:27:21 -03:00
Juan Fernandez
53e112d264
Feature 7494: Use the MSRC security bulletin artifacts for detecting Win OS vulnerabilities (#7889)
Use the MSRC security bulletin artifacts for detecting Win OS vulnerabilities
2022-10-28 11:12:21 -04:00
Michal Nicpon
3801f1c6d1
improve primary ip host detail query (#8407)
Previously, the primary ip was determined using the interface with the most traffic. This had a few issues and required explicitly filtering out certain interfaces that we don't want eg loopback, docker.

Instead, use the default route to determine the primary (private) ip. If there are multiple, use the one with the lowest metric and prefer ipv4 addresses.
2022-10-27 16:34:49 -06:00
Lucas Manuel Rodriguez
1fa5475e6e
Log denylisted queries as errors (#8294) 2022-10-27 11:17:14 -05:00
Roberto Dip
fdcbbaac7e
Add method to fleetDesktopResponse to satisfy the errorer interface (#8457) 2022-10-26 16:17:11 -03:00
Martin Angers
49e910270b
Add vulnerability scores to Jira/Zendesk tickets for premium users (#8346) 2022-10-26 10:42:09 -04:00
gillespi314
51e6369653
Fix password reset request expiration (#8246) 2022-10-25 09:46:41 -05:00
Martin Angers
229b349118
Fix more hosts filters-related API docs (#8335) 2022-10-25 08:17:51 -04:00
Michal Nicpon
d11db25ba9
add subject to scep payload (#8267) 2022-10-24 14:01:20 -06:00
gillespi314
881194b3b4
Add orbit version and osquery version usage statistics (#8229) 2022-10-24 11:12:56 -05:00
Martin Angers
e19191161e
Validate team spec unknown keys (#8265) 2022-10-24 08:49:44 -04:00
Juan Fernandez
9444016571
Fixed bug with CPE matching, UTM.app was matching to the incorrect CPE (#8149)
Fixed bug with CPE matching, UTM.app was matching to the incorrect CPE.
2022-10-24 12:46:18 +00:00
Michal Nicpon
3f3c0b34c4
fix missing host when ordering by display name (#8376) 2022-10-20 14:56:58 -06:00
Martin Angers
a63c3ac475
Fix error returned when ordering by display_name (#8350) 2022-10-19 17:05:10 -04:00
Roberto Dip
d0cde1aaaa
Use the same HTTP server config as the production server in tests (#8254)
* Use the same HTTP server config as the production server in tests

This abstracts the default config we use to run the server into a
function so it can be used in tests to run an HTTP server using the same
configuration.

Additionally, this fixes a data race in tests, as an HTTP server
configuration can't be changed once you call `server.Start()`[1]

[1]:
https://cs.opensource.google/go/go/+/refs/tags/go1.19.2:src/net/http/httptest/server.go;l=40;drc=19309779ac5e2f5a2fd3cbb34421dafb2855ac21
2022-10-19 07:42:21 -03:00
Frank Sievertsen
7c3d9f007a
return http status 400 for bad json request body (#8287) 2022-10-18 14:43:16 +02:00
Martin Angers
323620bf39
Deep-clone the appconfig when getting from cache (#8194) 2022-10-17 15:03:49 -04:00
Frank Sievertsen
a2c47cf4f4
add host_display_name to live query rows (#8251) 2022-10-17 15:52:25 +02:00
Roberto Dip
e4cd25f4aa
fix a bug causing features to be false (#8240)
Related to https://github.com/fleetdm/fleet/issues/8010 and https://github.com/fleetdm/fleet/issues/8013 this prevents a bug that happens when:

1. A team doesn't have a `config.features` key in the JSON stored in the table or `config` is `NULL`
2. The team is edited from the UI

All `config.features` will default to `false`, which can be a problem if your global settings are `true` for both (which is the default)
2022-10-14 19:27:37 -03:00
gillespi314
59876d37ea
Add usage statistics to measure policy violations (#8199) 2022-10-14 13:55:37 -05:00
Lucas Manuel Rodriguez
c16ab5f823
Fix undetected missing migrations (#8228) 2022-10-14 13:26:04 -03:00
Lucas Manuel Rodriguez
03096ee546
Fix race in tests (modifying already started server config) (#8227) 2022-10-14 12:00:16 -03:00
Frank Sievertsen
e54f066266
Add display_name to more results (#8204)
* Make HostResponses with one dedicated function only
* more display_name for Hosts
2022-10-14 11:14:18 -03:00
Frank Sievertsen
c74161e8f0
new timestamp for HostDisplayName migration (#8224) 2022-10-14 15:59:58 +02:00
Martin Angers
fae8e4ca2c
Translate the AppConfig and Team validation messages to be more user-friendly (#8171) 2022-10-12 17:10:50 -04:00
Juan Fernandez
6939af045d
Fleet desktop should use lightweight endpoint for getting failing policies count (#8159)
Fleet desktop should use lightweight endpoint for getting failing policies count
2022-10-12 17:13:43 -03:00
Juan Fernandez
72cfdac634
Only ingest installed deb_packages (#8160)
Only ingest deb_packages with status = 'install ok installed'
2022-10-12 14:27:07 -04:00
Roberto Dip
4042f8d826
add browser-related security headers to HTML responses (#8180)
related to #8031, this adds the following headers to HTML responses:

- Strict-Transport-Security: informs browsers that the site should only
  be accessed using HTTPS, and that any future attempts to access it
  using HTTP should automatically be converted to HTTPS.
- X-Frames-Options: disallows embedding the UI in other sites via
  <frame>, <iframe>, <embed> or <object>, which can prevent attacks like
  clickjacking.
- X-Content-Type-Options: prevents browsers from trying to guess the MIME
  type which can cause browsers to transform non-executable content into
  executable content.
- Referrer-Policy: prevents leaking the origin of the referrer in the
  Referer.

additionally, this ensures we set `X-Content-Type-Options` for CSV and
installer responses.
2022-10-12 10:19:21 -03:00
Martin Angers
d321cfc68e
Add inherited policies to the team's list policies response payload (#8068) 2022-10-12 08:35:36 -04:00
Lucas Manuel Rodriguez
42c47a6fa7
Add missing return to ingestKubequeryInfo (#8178)
* Add missing return to ingestKubequeryInfo

* No need to log error twice
2022-10-12 09:00:49 -03:00
Frank Sievertsen
23199c3d83
Add missing display_name to getHostEndpoint (#8174) 2022-10-11 17:00:32 -04:00
Lucas Manuel Rodriguez
28744bf57e
Consistently log migrations (#8154)
* Consistently log migrations

* Fix name and update goose version
2022-10-11 15:20:12 -03:00
Tomas Touceda
d912376f02
Handle tcp read timeouts (#8163)
* Handle tcp read timeouts properly

* Add changes file

* Fix bad mini refactor after nailing the test

* Update based on review

* Update comment
2022-10-11 13:58:52 -03:00
Roberto Dip
e630fabf89
token rotation for fleet desktop (#7779)
This implements what's described in detail here https://github.com/fleetdm/fleet/blob/main/proposals/fleet-desktop-token-rotation.md
2022-10-10 17:15:35 -03:00
gillespi314
ca379e7459
Fix directIngestOSWindows error in query ingestion for non-Windows hosts (#8148) 2022-10-10 13:27:15 -05:00
Frank Sievertsen
1eacecf637
Add new missing value to existing status URL parameter at the GET /hosts endpoint (#7916) 2022-10-10 07:45:39 -04:00
Frank Sievertsen
b25e74b285
add missing_30_days_count to host_summary response (#7915) 2022-10-08 08:58:27 -04:00
Frank Sievertsen
e9f7066d87
7135 host display name (#7873) 2022-10-08 08:57:46 -04:00
Roberto Dip
bfe698d090
cleanup all policy memberships for a host on re-enrollment (#8120)
Related to #7664, this cleans up all policy memberships for a host when its re-enrolled, afterwards only the relevant policy memberships for the host will be created.
2022-10-07 11:36:17 -03:00
Juan Fernandez
582772fbee
Fixed bug with duplicated vulnerabilities (#8089)
Previous code assumed CVEs were unique per NVD file, which was   incorrect.
2022-10-05 19:48:37 -04:00
Lucas Manuel Rodriguez
9191f4ce66
Add Apple MDM functionality (#7940)
* WIP

* Adding DEP functionality to Fleet

* Better organize additional MDM code

* Add cmdr.py and amend API paths

* Fix lint

* Add demo file

* Fix demo.md

* go mod tidy

* Add munki setup to Fleet

* Add diagram to demo.md

* Add fixes

* Update TODOs and demo.md

* Fix cmdr.py and add TODO

* Add endpoints to demo.md

* Add more Munki PoC/demo stuff

* WIP

* Remove proposals from PoC

* Replace prepare commands with fleetctl commands

* Update demo.md with current state

* Remove config field

* Amend demo

* Remove Munki setup from MVP-Dogfood

* Update demo.md

* Add apple mdm commands (#7769)

* fleetctl enqueue mdm command

* fix deps

* Fix build

Co-authored-by: Lucas Rodriguez <lucas@fleetdm.com>

* Add command to upload installers

* go mod tidy

* fix subcommands help

There is a bug in urfave/cli where help text is not generated properly when subcommands
are nested too deep.

* Add support for installing apps

* Add a way to list enrolled devices

* Add dep listing

* Rearrange endpoints

* Move DEP routine to schedule

* Define paths globally

* Add a way to list enrollments and installers

* Parse device-ids as comma-separated string

* Remove unused types

* Add simple commands and nest under enqueue-command

* Fix simple commands

* Add help to enqueue-command

* merge apple_mdm database

* Fix commands

* update nanomdm

* Split nanomdm and nanodep schemas

* Set 512 MB in memory for upload

* Remove empty file

* Amend profile

* Add sample commands

* Add delete installers and fix bug in DEP profile assigning

* Add dogfood.md deployment guide

* Update schema.sql

* Dump schema with MySQL 5

* Set default value for authenticate_at

* add tokens to enrollment profiles

When a device downloads an MDM enrollment profile, verify the token passed
as a query parameter. This ensures untrusted devices don't enroll with
our MDM server.

- Rename enrollments to enrollment profiles. Enrollments is used by nano
  to refer to devices that are enrolled with MDM
- Rename endpoint /api/<version>/fleet/mdm/apple/enrollments to ../enrollmentprofiles
- Generate a token for authentication when creating an enrollment profile
- Return unauthorized if token is invalid when downloading an enrollment profile from /api/mdm/apple/enroll?token=

* remove mdm apple server url

* update docs

* make dump-test-schema

* Update nanomdm with missing prefix table

* Add docs and simplify changes

* Add changes file

* Add method docs

* Fix compile and revert prepare.go changes

* Revert migration status check change

* Amend comments

* Add more docs

* Clarify storage of installers

* Remove TODO

* Remove unused

* update dogfood.md

* remove cmdr.py

* Add authorization tests

* Add TODO comment

* use kitlog for nano logging

* Add yaml tags

* Remove unused flag

* Remove changes file

* Only run DEP routine if MDM is enabled

* Add docs to all new exported types

* Add docs

* more nano logging changes

* Fix unintentional removal

* more nano logging changes

* Fix compile test

* Use string for configs and fix config test

* Add docs and amend changes

* revert changes to basicAuthHandler

* remove exported BasicAuthHandler

* rename rego authz type

* Add more information to dep list

* add db tag

* update deps

* Fix schema

* Remove unimplemented

Co-authored-by: Michal Nicpon <39177923+michalnicp@users.noreply.github.com>
Co-authored-by: Michal Nicpon <michal@fleetdm.com>
2022-10-05 19:53:54 -03:00
RachelElysia
3054450065
Improve developer documentation: Update seed data documentation (#7904) 2022-10-05 12:42:45 -04:00
Martin Angers
c23a15394e
Keep created_at timestamp of existing enroll secrets (#8062) 2022-10-05 08:35:36 -04:00
Martin Angers
ec75fb10b2
Fix CI tests for Go (#8079) 2022-10-04 15:48:14 -04:00
gillespi314
e3590beaf7
Ingest ReleaseId from Windows registry (#8030) 2022-10-04 10:34:08 -05:00
Juan Fernandez
812d3c85de
Fixes various bugs with NVD vulnerability detection (#7963)
- Improved NVD CPE matching process.
- Fixed bug with the 'software/<id>' endpoint not showing the generated_cpe value.
2022-10-04 07:04:48 -04:00
Frank Sievertsen
a5912bedea
API Bugs: Team view (fleet/teams/{id}) #6970
Fix host_count and user_count being always returned as `0` in `teams/{id}` endpoint.
2022-10-04 09:26:04 +02:00
Martin Angers
7151e05c6d
Remove value validation of agent_options.{packs,schedule} (#8066) 2022-10-03 16:43:47 -04:00
Lucas Manuel Rodriguez
a4d7e81475
Orbit to support environments with revoked enroll secrets (#8056)
* Support environments with revoked enroll secrets

* Add instructions on how to fix Orbit enroll

* Rename to last_recorded_error

* Add alternative instructions
2022-10-03 17:28:19 -03:00
Roberto Dip
c0bb0cc4e7
avoid panic when storing a nil pointer in cached_mysql (#8020)
related to #7420, this improves the logic of the clone function in cached_mysql to properly handle nil and nil pointers.
2022-10-03 13:29:20 -03:00
Roberto Dip
cd2ab6b17c
add server side validation for emails (#8022)
related to https://github.com/fleetdm/fleet/issues/7199, this adds email validation to the `verifyCreateShared` which is used for user creation in the server.

validation messages come directly from Go's `net/mail` package.

```
~/fleet $ curl 'https://localhost:8080/api/latest/fleet/users/admin' -X POST -H 'Authorization: Bearer $TOKEN' --data-raw '{"email":"asdf","name":"asdf@asd.com","password":"as;lkdfjasdlk;fja3234@","global_role":"observer","teams":[]}'
{
  "message": "Validation Failed",
  "errors": [
    {
      "name": "email",
      "reason": "mail: missing '@' or angle-addr"
    }
  ]
}
```
2022-10-03 13:29:01 -03:00
Martin Angers
8eee0b35f3
Add validations for agent options' command-line flags section. (#7979) 2022-10-03 08:29:41 -04:00
gillespi314
2a6895b015
Log user email upon successful SSO login (#8012) 2022-09-29 09:25:45 -05:00
Martin Angers
8023193ea8
Fix integration logger test (#8014) 2022-09-28 16:35:54 -04:00
Sharvil Shah
b13683952d
add index to orbit_node_key (#7990)
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
2022-09-28 02:25:07 +05:30
Sharvil Shah
cfb3c7ae9f
switch orbit enroll logic to use osquery_host_id (#7984) 2022-09-28 01:29:25 +05:30
gillespi314
749ff9ec2b
Add logging to capture user email upon successful login (#7927)
* Log user email upon successful login

* Add user email to logger context

* Use logging.With Extras for login email
2022-09-27 16:32:46 -03:00
Martin Angers
142e298631
Update osquery validations from 5.4.0 to 5.5.1 (#7960) 2022-09-27 08:35:57 -04:00
Roberto Dip
b9927eade4
make orbit and device ping paths unversioned (#7933) 2022-09-26 14:39:56 -03:00
Roberto Dip
a509bdd0ac
add backwards compatiblity to orbit enrollment via the API (#7921) 2022-09-26 11:44:09 -03:00
Roberto Dip
2fcb27ed3f
add headers denoting capabilities between fleet server / desktop / orbit (#7833)
This adds a new mechanism to allow us to handle compatibility issues between Orbit, Fleet Server and Fleet Desktop.

The general idea is to _always_ send a custom header of the form:

```
fleet-capabilities-header = "X-Fleet-Capabilities:" capabilities
capabilities              = capability * (,)
capability                = string
```

Both from the server to the clients (Orbit, Fleet Desktop) and vice-versa. For an example, see: 8c0bbdd291

Also, the following applies:

- Backwards compat: if the header is not present, assume that orbit/fleet doesn't have the capability
- The current capabilities endpoint will be removed

### Motivation

This solution is trying to solve the following problems:

- We have three independent processes communicating with each other (Fleet Desktop, Orbit and Fleet Server). Each process can be updated independently, and therefore we need a way for each process to know what features are supported by its peers.
- We originally implemented a dedicated API endpoint in the server that returned a list of the capabilities (or "features") enabled, we found this, and any other server-only solution (like API versioning) to be insufficient because:
  - There are cases in which the server also needs to know which features are supported by its clients
  - Clients needed to poll for changes to detect if the capabilities supported by the server change, by sending the capabilities on each request we have a much cleaner way to handling different responses.
- We are also introducing an unauthenticated endpoint to get the server features, this gives us flexibility if we need to implement different authentication mechanisms, and was one of the pitfalls of the first implementation.

Related to https://github.com/fleetdm/fleet/issues/7929
2022-09-26 07:53:53 -03:00
Juan Fernandez
d7ca8fcd66
Reverted changes made to Fleet Desktop - should not use the lightweight 'desktop' endpoint (#7919)
Reverted changes made to Fleet Desktop. Desktop should not use the lightweight 'desktop' endpoint
2022-09-23 15:18:19 -04:00
Sharvil Shah
7d4e2e2b4b
Orbit remote management for flags (#7246)
Co-authored-by: Roberto Dip <dip.jesusr@gmail.com>
2022-09-24 00:30:23 +05:30
Martin Angers
5477acb688
Add low_disk_space_count property to GET /host_summary (#7872) 2022-09-21 15:56:17 -04:00
Martin Angers
db24cf32fa
Add new low_disk_space URL parameter to the GET /hosts endpoint (and GET /hosts/count) (#7853) 2022-09-21 15:16:31 -04:00
gillespi314
34688f531a
Refactor webhooks cron to new schedule package (#7840) 2022-09-20 14:26:36 -05:00
Martin Angers
478b4d3f69
Validate team and appconfig payloads, with dry-run and force modes (#7731) 2022-09-19 13:53:44 -04:00
gillespi314
6a3d9959fc
Refactor vulnerabilities cron to scheduler package (#7650) 2022-09-16 10:08:51 -05:00
Juan Fernandez
4ef883b311
Feature 6946: Fleet Desktop should use minimal api end-point for data (#7536)
Updated desktop client to use new EE desktop endpoint.
2022-09-15 12:12:50 -04:00
Zach Wasserman
486b67caca
Add initial support for kubequery (#6863)
Configuration and fixes for the Fleet server and frontend to add support
for https://github.com/Uptycs/kubequery.

Co-authored-by: Michal Nicpon <michal@fleetdm.com>
2022-09-15 10:02:17 -06:00
Juan Fernandez
765031d962
Bug 6717: Duplicate calls on vulnerability third party integrations. (#7698)
If the same vulnerability was present in more than one software, then duplicated third party integration calls were being issued.
2022-09-13 10:41:52 -04:00
Tomas Touceda
8457e55b53
Bump go to 1.19.1 (#7690)
* Bump go to 1.19.1

* Bump remaining go-version to the 1.19.1

* Add extra paths for test-go

* Oops, putting the right path in the right place

* gofmt file

* gofmt ALL THE THINGS

* Moar changes

* Actually, go.mod doesn't like minor versions
2022-09-12 20:32:43 -03:00
Juan Fernandez
b60d535d4a
Feature 7084: Add new EE endpoint for Fleet Desktop (#7530)
Added new EE endpoint, that is meant to be used by Fleet Desktop only. The new endpoint will return the number of failed policies.
2022-09-12 15:37:38 -04:00
Juan Fernandez
14d0499c4b
Bug 7682: Fixed issue with migration not running on MariaDB (#7701)
Update migration to account for different FK name under MariaDB.
2022-09-12 15:36:55 -04:00
Juan Fernandez
98e4d6e0bc
Feature 7394: Use MSRC parser to generate security bulletin artifacts (#7491)
Generate security artifacts using the MSRC parser.
2022-09-12 15:17:13 -04:00
Frank Sievertsen
dd8cf37a85
#7624 accept empty bearer authorization token. (#7685)
* Fix #7624 accept empty bearer authorization token.

* add changes file
2022-09-12 16:09:41 +02:00
Martin Angers
988f50fa34
Document all keys in config and team YAML documents (#7449)
- Add a new "Configuration for contributors" doc page. Move settings that are not recommended for production use
- Remove settings modified in the `config` YAML document from the deploying/configuration doc page
- Document all keys in `config` and `teams` YAML documents
- Add comments to several `.go` files and remove unused struct
2022-09-08 17:57:38 -04:00
Roberto Dip
740aafd817
Revert token rotation (#7628)
This reverts all changes related to token rotation.
2022-09-08 15:04:02 -03:00
gillespi314
abc131575b
Bring patch commits to main (#7647) 2022-09-08 12:20:07 -05:00
Roberto Dip
2deabc23d0
fix panic when reading team features from the database (#7633)
This fixes an unreleased bug I introduced in eeefe2, as the config colum in the teams table is nulleable, it unmarshalls into nil and we can't dereference the variable
2022-09-08 13:33:10 +00:00
Roberto Dip
9174c7c711
token rotation for Fleet Desktop (#7517)
See https://github.com/fleetdm/fleet/issues/6348 for a very detailed rundown

Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
2022-09-06 20:25:29 -03:00
Juan Fernandez
b6d084deec
Feature 7559: Include 3rd party severity scores in the vulnerability web-hook payload (#7581)
Premium users should receive vulnerability scores as part of the web-hook payload.
2022-09-06 15:12:17 -04:00
Juan Fernandez
e86aa9d99b
Bug 7465: Bump out of date migrations (#7478)
Bump out of date migrations
2022-09-06 10:58:10 -04:00
Martin Angers
aa0102d6b6
Remove aggregate host counts from GET /hosts responses (#7510) 2022-09-06 10:34:06 -04:00
gillespi314
26753ef62b
Add issues count to GET /labels/{id}/hosts response (#7381) 2022-09-02 09:13:53 -05:00
Michal Nicpon
0709d1bc5c
improve vuln cpe matching on macos (#6985)
* add cpe translations
* fix matching on target_sw
2022-09-01 10:02:07 -06:00
Roberto Dip
9dfd903c3a
set defaults for team spec configs related to features (#7476)
This adjusts functions related to applying team specs to make sure the `features` key works as expected, this is:

1. Create Team flow
    1. if no features are provided, we copy whatever features are set in the global config
    2. otherwise, we generate a new config merging whatever is provided with the defaults for new installs.
2. Edit Team flow: we always generate a new config merging whatever is provided with the defaults for new installs.
2022-08-31 17:30:46 -03:00
Juan Fernandez
2699c22143
Feature 7077: Add MSRC feed parser (#7424)
Added parser for MSRC
2022-08-30 16:39:50 -04:00
Juan Fernandez
87717df1da
Bug 7441: Upgrading to 4.19.0 might fail due to migration (#7446)
* Make Change software vendor width mig. more robust

If there are two or more entries in the software table with the same name, version, source, release and arch but different vendors then the migration used for increasing the vendor width will fail.
2022-08-30 11:20:48 -04:00
gillespi314
8157359f84
Ingest Windows host display version for hosts table (#7403) 2022-08-30 10:16:48 -05:00
Roberto Dip
eeefe2fab9
enable controlled rollout of features by teams (#7408) 2022-08-30 08:13:09 -03:00
Martin Angers
f602ea3446
Provide Munki issues for MacOS hosts (#7280) 2022-08-29 14:40:16 -04:00
Juan Fernandez
3048a07fd1
Feature 7076: Ingest installed windows updates (#7138)
* Ingest installed Windows updates and store them in the windows_updates table. 
* Added config option for enabling/disabling Windows update ingestion and Windows OS vuln. detection.
2022-08-26 14:55:03 -04:00
Roberto Dip
9a1d2066bb
deprecate host_settings in favor of features (#7358)
Related to #7312, this makes use of the changes introduced in #7353 to rename host_settings to features while keeping backwards compatibility.
2022-08-25 13:41:50 -03:00
Roberto Dip
c943e0665a
add a mechanism to deprecate AppConfig settings w/ backwards-compat (#7353)
Related to https://github.com/fleetdm/fleet/issues/7312, the motivation behind these changes is to introduce a way to deprecate configurations in `AppConfig`, while still preserving backwards compatibility.

From the Epic:

> NOTE: `host_settings` is now replaced by `features`. We should still support `host_settings` as an alias to `features` for backwards compatibility, but in our communications, we should use and recommend features as the canonical way forward.
2022-08-24 18:40:09 -03:00
Juan Fernandez
0fab2fe0eb
Feature 6538: Remove cpe_id from the software_cve table (#6587)
* Don't insert dummy values for non-match CPEs, exclude OVAL platforms when getting software iterator.
* Removed CPEID from SoftwareVulnerability type and software_cve.cpe_id column.
* Removed GeneratedCPEID property
2022-08-24 13:10:58 -04:00
Juan Fernandez
7b612aa030
Bug 7320: Fixed dulp. vulns. on software details. (#7342)
* Bug 7320: Fixed dulp. vulns. on software details.

The software details page was showing duplicated vulnerabilities if the software was being used by many hosts.
2022-08-24 09:45:20 -04:00
Martin Angers
dfd7d3f244
Add activity types for applied team spec and edited agent options (#7355) 2022-08-24 08:32:45 -04:00
Roberto Dip
fe9cdb7017
fix flaky TestIntegrationsSandbox/TestInstallerHeadCheck (#7345)
this adjusts the TestInstallerHeadCheck check to use `DoRaw` instead of `Do`, which sends a body with null when nil is passed for the params parameter.
2022-08-23 09:58:10 -03:00
gillespi314
7fb109e6b3
Handle errors for duplicate os records (#7294) 2022-08-22 14:34:00 -05:00
Juan Fernandez
e6aba8caaa
Bug with Vendor Width migration (#7339)
Updated migration used to increate the width of 'software.vendor' to
address issue found when running migration in aurora: "ALGORITHM=INPLACE is not
supported. Reason: cannot silently convert NULL values, as required in
this SQL_MODE. Try ALGORITHM=COPY."
2022-08-22 13:01:18 -04:00
Juan Fernandez
2932ea0899
Bug 7332: Include software without CPE entries (#7334)
When listing software for performing vuln. detection include software without CPE entries.
2022-08-22 11:45:17 -04:00
Juan Fernandez
4013cbbdfc
Feature 6975: Populate vendor column in software inventory (#7297)
- Populate 'software.vendor' when ingesting software from Windows hosts.
- Increate width of 'software.vendor'.
2022-08-18 18:02:56 -04:00
Zachary Winnerman
2103de275a
improve error handling in vulnerabilities cron (#7276)
* improve error handling in vulnerabilities cron

* fix tests

* Use errHandler and go mod tidy

* Add dep that got removed by mod tidy

* add dsl to tools

* Add changes file

Co-authored-by: Michal Nicpon <michal@fleetdm.com>
Co-authored-by: Tomas Touceda <chiiph@gmail.com>
2022-08-18 16:53:58 -03:00
Roberto Dip
8acf14ab43
adjust installers endpoint to avoid AJAX downloads (#7226)
Related to #7206, this delegates the handling of the download to the browser
2022-08-16 12:54:41 -03:00
Martin Angers
94d1ebb822
Set refetch_requested to true for newly enrolled hosts (#7225) 2022-08-16 08:33:15 -04:00
Roberto Dip
05ddeade90
add back-end implementation for SSO JIT provisioning (#7182)
Related to #7053, this uses the SSO config added in #7140 to enable JIT provisioning for premium instances.
2022-08-15 14:42:33 -03:00
Martin Angers
b891e0d7f7
Add mdm solution payload to GET /hosts response when filtering by mdm_id (#7198) 2022-08-15 12:57:25 -04:00
gillespi314
90f57f4849
Enhance API endpoints with host operating systems info (#7154) 2022-08-12 14:23:25 -05:00
Juan Fernandez
8dccc42027
Bug 5983: Performance issues when listing software (#6879)
Improve performance when listing software by using the software_host_counts aggregate table.
2022-08-10 17:43:22 -04:00
Martin Angers
c8cdddf0ea
Update /macadmins endpoints to include MDM name and aggregate count (#7137) 2022-08-10 15:15:01 -04:00
Roberto Dip
fc8c15c0d1
add application config setting to enable JIT provisioning (#7140)
As part of #7053, this adds a config setting to enable JIT provisioning.
2022-08-10 15:15:35 -03:00
gillespi314
e2194be61c
Add schedule package and refactor cron jobs for cleanups, aggregations, and usage statistics (#6618) 2022-08-10 11:00:56 -05:00
Martin Angers
9755eb2e27
Support async saving of scheduled query statistics (#7012) 2022-08-10 10:01:05 -04:00
gillespi314
3a88afaf98
Add new database tables and queries for host operating systems (#6920) 2022-08-09 13:34:41 -05:00
Lucas Manuel Rodriguez
6dcff28be0
Move specs parsing functionality to a new pkg/spec package (#7050) 2022-08-05 19:07:32 -03:00
Roberto Dip
2b8743e240
send enroll secret in query for installers (#7064)
This changes how the enroll secret is sent to the server, as they might contain /, which was causing problems with our router.
2022-08-04 18:39:38 -03:00
Juan Fernandez
966bfbf85e
Feature 6487: Deprecate cpe_id from software_cve table (#6562)
Part 2/3 of the removal of the cpe_id column from the software_cve table in favor of using the newly added software_id coumn.
2022-08-04 09:24:44 -04:00
Martin Angers
012869f971
Fleet Premium: de-anonymize usage stats (#7013) 2022-08-03 14:44:34 -04:00
Martin Angers
c1d38598e2
Prevent removing team enroll secrets when applying team specs without new secrets (#6890) 2022-08-02 09:51:03 -04:00
Gabriel Hernandez
f61a17bd9b
add google analytics to sandbox instances (#6941)
* add google analytics to sandbox instances

* Add serverType variable to frontend handler

* update version of html-webpack-plugin

Co-authored-by: Lucas Rodriguez <lucas@fleetdm.com>
2022-08-01 11:27:12 +01:00
Roberto Dip
90b723e45a
consolidate sandbox env flags (#6917)
Related to #6894, this entirely replaces FLEET_DEMO with the server config added in #6597

As part of this, I also implemented a small refactor to the integration test suite to allow setting a custom config when the server is initialized.
2022-07-27 16:47:39 -03:00
RachelElysia
52673b6ed2
Sentence case error (#6845) 2022-07-25 13:14:05 -04:00
Juan Fernandez
af80406b59
Bug 6777: Make the regexp used for detecting the oval platform major, min versions more permissive (#6800)
When getting the major/minor version for OVAL processing, handle the LTS post-fix for Ubuntu platforms
2022-07-21 12:55:07 -04:00
gillespi314
0312454f4c
Modify host battery health returned by GET /hosts/:id (#6782) 2022-07-20 21:16:03 -05:00
gillespi314
cc34585a47
Add statistics for number of hosts that have not submitted results for distributed queries (#6495) 2022-07-20 20:54:10 -05:00
Michal Nicpon
e3e17f3389
Fix double quote usage in sql query (#6707) 2022-07-20 10:10:03 -06:00
Michal Nicpon
d4be5ad2a1
add upgrade tests (#6596)
* add upgrade tests

* fix lint issues

go.mod

* remove req.cnf

* revert unrelated changes

* make version configurable in test

* fix golangci-lint ruleguard issue

Related to https://github.com/go-critic/go-critic/issues/1152
Need to have github.com/quasilyte/go-ruleguard/dsl

* fix lint issues

* fix

* clean up docker-compose.yml

* fix http request

* add readme

* fix lint issues

* address feedback

* fix

* add platform

* address feedback

* run go fmt
2022-07-19 15:11:51 -06:00
Zach Wasserman
710c304d94
Apply password requirements to admin-created users (#6667)
This was requested by a customer.
2022-07-19 09:47:25 -07:00
Lucas Manuel Rodriguez
de1717291d
Set authz checked when rate limiting device endpoints (#6702)
* Set authz checked when rate limiting device endpoints

* Unexport var and attempt to fix flaky test
2022-07-18 14:22:49 -03:00
Eng Zer Jun
1ab171faf3
test: use T.Setenv to set env vars in tests (#6714)
This commit replaces `os.Setenv` with `t.Setenv` in tests. The
environment variable is automatically restored to its original value
when the test and all its subtests complete.

Reference: https://pkg.go.dev/testing#T.Setenv
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-07-18 14:22:28 -03:00