Commit Graph

532 Commits

Author SHA1 Message Date
Zach Wasserman
2025afee71
Better handling of config_refresh values from clients (#388)
Since the original logic was implemented, there have been some changes
in the way that config refreshes are configured. This commit reflects
those changes and should be backwards compatible.

Closes #357
2021-03-05 08:25:46 -08:00
Zach Wasserman
f9312df174
Remove flaky log rotation test (#385)
This test has been flaky for years and it's time for it to go.
2021-03-04 16:29:50 -08:00
Zach Wasserman
de0b3324b1
Add AWS Lambda as logging plugin (#347)
This plugin invokes the provided function with each log line as the
payload.

Closes #342
2021-02-24 10:02:26 -08:00
Zach Wasserman
471f25e470
Return denylist parameter with pack configs (#339)
This was previously handled correctly within the Fleet server datastores
and API endpoints, but not returned to the actual osquery client.

Fixes #338
2021-02-21 09:58:36 -08:00
Lars Lehtonen
7e02b7abe3
server/datastore: fix swallowed test error (#337) 2021-02-19 08:19:43 -08:00
Zach Wasserman
c6c5e65a7d
Add support for filtering hosts in label (#335)
Support the same query syntax as the hosts endpoint, here also bounded
by the membership of the label.
2021-02-18 12:52:43 -08:00
Zach Wasserman
cca4802642
Add support for query parameter in list hosts endpoint (#327)
Uses a LIKE clause to search for hosts matching the query against
columns `host_name`, `uuid`, `hardware_serial`, and `primary_ip`.

Introduces the `searchLike` helper to add the appropriate filters to the
SQL query.
2021-02-16 16:53:42 -08:00
Zach Wasserman
6b5019d65e
Add aliases for order keys in API (#329)
Aliases `hostname` (`host_name`) and `memory` (`physical_memory`) when
used as keys for ordering in the API this allows for better consistency
on the frontend.

To be cleaned up further in #317
2021-02-16 15:25:34 -08:00
Zach Wasserman
06d919aa4e
Rename MySQL parameters for CircleCI (#328)
kolide -> fleet where appropriate
2021-02-16 14:40:38 -08:00
Zach Wasserman
b83fbf0ec9
Special case host_name on the backend to enable frontend consistency (#318)
See #317 for followup to this.
2021-02-15 11:58:05 -08:00
Zach Wasserman
d624e099fb
Deprecate environment variable prefix (#301)
- Support both `FLEET_` and `KOLIDE_` prefixes.
- Add logging about deprecated `KOLIDE_` prefix.
- Update documentation and sample configs.
2021-02-11 15:36:58 -08:00
Zach Wasserman
7a68e3de65
Deprecate /api/v1/kolide routes (#297)
- Support both /api/v1/fleet and /api/v1/kolide routes in server.
- Add logging for use of deprecated routes.
- Rename routes in frontend JS.
- Rename routes and add notes in documentation.
2021-02-10 12:13:11 -08:00
Zach Wasserman
e2e7e4707d
Update database migration tooling (#281)
- Use updated github.com/fleetdm/goose that generates templates
  requiring less editing.
- Add `make migration` to Makefile.
- Update documentation.
2021-02-05 09:48:06 -08:00
Zach Wasserman
ddb05cce94
Change default TLS compatibility to intermediate (#270)
In #212 these settings were updated and caused connectivity issues for
users in common environment configurations. The new changes are
aggressive (modern enforces TLS 1.3) and Mozilla indicates that
intermediate is an appropriate default. This will ensure better
compatibility for common deployments while still allowing the option to
use the strictest settings.

Document unintentional mismatched yaml key.

Fixes #269
2021-02-03 11:48:48 -08:00
Zach Wasserman
f68f4238e8
Merge pull request from GHSA-xwh8-9p3f-3x45
- Fix the specific case that caused panic.
- Add panic handler around entire live query results handler. This will
  prevent similar issues from causing crashes in the future.

Note that other endpoints already have panic handling but this one is
special due to the use of websockets.
2021-02-03 08:47:43 -08:00
Zach Wasserman
22d9205d35
Add debug flag to fleetctl (#266)
This flag enables logging of HTTP requests and responses to stderr.

Closes #187
2021-02-02 18:55:16 -08:00
Zach Wasserman
a9c0c472fc
Update Kolide to Fleet as appropriate in documentation (#263) 2021-02-02 12:16:59 -08:00
Zach Wasserman
c3a18ea7d1
Rename prefixed system queries (#253)
Renames live query names that the system uses from `kolide_` to
`fleet_`.
2021-02-01 11:39:53 -08:00
Zach Wasserman
ac9a85ab88
Expose more information for migration status errors (#240)
We saw these errors live today and it would be helpful to have the
additional error context for debugging.
2021-01-28 09:49:06 -08:00
Zach Wasserman
bb124dc2b9
Improvements to fleetctl preview experience (#234)
- Make the preview directory in the default .fleet directory.
- Check for Docker daemon installed but not running.
- Add message for Chrome users on self-signed certs.
- Display login information on later invocations of command.
- Remove "Kolide" from error messages.

Closes #190

Part of #197
2021-01-28 07:57:32 -08:00
Zach Wasserman
87330a9753
Add support for denylist parameter in scheduled queries (#209)
Closes #202
2021-01-25 16:26:14 -08:00
Zach Wasserman
6215acdd1b
Add host details in API responses (#223)
Add label and pack information for the returned hosts in the single-host
API endpoints.

Example:

```
curl -k 'https://localhost:8080/api/v1/kolide/hosts/7' -H 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzZXNzaW9uX2tleSI6Ii9oNEZ4MUpEVmlvQWhtMC8wNUJKbzZpdldsUDZpMDhjQVBuZXRLeFIvWjNOUGgvMW9VdCsxQnFlNU1CVDVsMlU3ckVGMm5Sb1VxS3ZSUllzSmJJR2lBPT0ifQ.GQQsJgBU3JA1H1o4Y8fPjyfF78F_VY4c9AbrP5k0sCg'
{
  "host": {
    "created_at": "2021-01-16T00:22:33Z",
    "updated_at": "2021-01-16T00:22:51Z",
    "id": 7,
    "detail_updated_at": "1970-01-02T00:00:00Z",
    "label_updated_at": "1970-01-02T00:00:00Z",
    "last_enrolled_at": "2021-01-16T00:22:33Z",
    "seen_time": "2021-01-16T00:22:51Z",
    "hostname": "55d91fc9c303",
    "uuid": "853a4588-0000-0000-a061-7d494d04e9c4",
    "platform": "ubuntu",
    "osquery_version": "4.6.0",
    "os_version": "Ubuntu 20.04.0",
    "build": "",
    "platform_like": "debian",
    "code_name": "",
    "uptime": 0,
    "memory": 16794206208,
    "cpu_type": "x86_64",
    "cpu_subtype": "158",
    "cpu_brand": "Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz\u0000\u0000\u0000\u0000\u0000\u0000\u0000",
    "cpu_physical_cores": 8,
    "cpu_logical_cores": 8,
    "hardware_vendor": "",
    "hardware_model": "",
    "hardware_version": "",
    "hardware_serial": "",
    "computer_name": "55d91fc9c303",
    "primary_ip": "",
    "primary_mac": "",
    "distributed_interval": 10,
    "config_tls_refresh": 0,
    "logger_tls_period": 10,
    "enroll_secret_name": "default",
    "labels": [
      {
        "created_at": "2020-12-22T01:22:47Z",
        "updated_at": "2020-12-22T01:22:47Z",
        "id": 6,
        "name": "All Hosts",
        "description": "All hosts which have enrolled in Fleet",
        "query": "select 1;",
        "label_type": "builtin",
        "label_membership_type": "dynamic"
      }
    ],
    "packs": [
      {
        "created_at": "2021-01-20T16:36:42Z",
        "updated_at": "2021-01-20T16:36:42Z",
        "id": 2,
        "name": "test"
      }
    ],
    "status": "offline",
    "display_text": "55d91fc9c303"
  }
}
```
2021-01-25 13:05:02 -08:00
Zach Wasserman
deaf8880f3
Expose live query error messages via API (#205)
Somewhere around osquery 4.4.0 these messages were added to query
responses. We can now expose them to the API clients rather than using
the placeholder text.

Required for #192
2021-01-19 14:52:29 -08:00
Zach Wasserman
f254a9a343
Make name and secret required for enroll secrets (#207)
Adds a check to prevent users from unintentionally setting empty
secrets.

Fixes #188
2021-01-19 14:49:53 -08:00
Zach Wasserman
44be83eefd
Improved error handling in host enrollment (#203)
This will make it easier to debug issues with the enrollment process,
such as #200.
2021-01-19 14:45:58 -08:00
Zach Wasserman
d13aadb5f1
Fix empty enroll secret before setup. (#182)
On new installations we unintentionally set the enroll secret to empty
string during database migrations. The enroll secret would be reset
during the setup process. This fixes the migration to not create any
enroll secret until the setup process.
2021-01-07 14:12:51 -08:00
CptOfEvilMinions
626429c38e
Added support to read jwt and mysql password from a file (#141)
The current implementation of FleetDM doesn't support Docker secrets for supplying the MySQL password and JWT key. This PR provides the ability for a file path to read in secrets. The goal of this PR is to avoid storing secrets in a static config or in an environment variable. 

Example config for Docker:
```yaml
mysql:
  address: mysql:3306
  database: fleet
  username: fleet
  password_path: /run/secrets/mysql-fleetdm-password
redis:
  address: redis:6379
server:
  address: 0.0.0.0:8080
  cert: /run/secrets/fleetdm-tls-cert
  key: /run/secrets/fleetdm-tls-key
auth:
  jwt_key_path: /run/secrets/fleetdm-jwt-key
filesystem:
  status_log_file: /var/log/osquery/status.log
  result_log_file: /var/log/osquery/result.log
  enable_log_rotation: true
logging:
  json: true
```
2021-01-04 07:58:43 -08:00
Matteo Piano
c89cd370d5
Add AWS S3 as file carving backend (#126)
This adds the option to set up an S3 bucket as the storage backend for file carving (partially solving #111).

It works by using the multipart upload capabilities of S3 to maintain compatibility with the "upload in blocks" protocol that osquery uses. It does this basically replacing the carve_blocks table while still maintaining the metadata in the original place (it would probably be possible to rely completely on S3 by using object tagging at the cost of listing performance). To make this pluggable, I created a new field in the service struct dedicated to the CarveStore which, if no configuration for S3 is set up will be just a reference to the standard datastore, otherwise it will point to the S3 one (effectively this separation will allow in the future to add more backends).
2020-12-16 09:16:55 -08:00
Zach Wasserman
55a2aa2c95
Fix logging for query creation on error (#132) 2020-12-14 18:45:03 -08:00
Zach Wasserman
57812a532e
Merge pull request from GHSA-w3wf-cfx3-6gcx
* Update github.com/russellhaering/goxmldsig

* Update signature validation to include Mattermost XML validator
2020-12-14 18:14:14 -08:00
Zach Wasserman
5b432cccee
Prevent SQLite ATTACH from being used in queries (#129)
Mitigate
[CVE-2020-26273](https://github.com/osquery/osquery/security/advisories/GHSA-4g56-2482-x7q8)
by attempting to prevent users from executing or saving queries that use
the SQLite `ATTACH` command.

Users must still update to osquery 4.6.0 to ensure the functionality is
fully disabled in osquery.
2020-12-14 18:13:34 -08:00
Zach Wasserman
6f6464f206
Migrate remaining URLs to fleetdm.com (#116) 2020-12-10 11:26:00 -08:00
Zach Wasserman
9ff16a370c
Improve MySQL transaction error logging (#113)
Clarify what exactly happened when logging these errors.
2020-12-10 11:05:15 -08:00
Zach Wasserman
beb7e8b965
Add host enrollment cooldown period (#112)
This addresses an issue some users experienced in which performance
problems were encountered when hosts were "competing" for enrollment
using the same osquery host identifier. The issue is addressed by adding
a cooldown period for host enrollment, preventing the same (as judged by
osquery host identifier) host from enrolling more than once per minute.

When users end up in the problematic scenario, they will see quite a bit
of error logs due to this issue. For now that's probably a good thing as
users need to be aware of the lack of visibility. We can explore rate
limiting the logging if that becomes an issue for someone.

Fixes #102
2020-12-10 11:04:58 -08:00
Zach Wasserman
bb921bc973
Allow scheduling same query more than once in UI (#96)
Fixes #88
2020-12-04 09:50:39 -08:00
noahtalerman
51cfb07f64
Implement global colors. Add italic font. (#64)
The goal of this PR was to insert new colors, remove legacy colors, and update only the styles accordingly. The Nunito Sans Italic font was also added.

Later PRs will include layout, copy, and style change to individual components. These later changes will more exactly resemble the current mockups.
2020-11-24 08:59:03 -08:00
noahtalerman
8df16949f5
Add fleet assets including logos, 404 image, and 500 image. (#61)
This is the second PR as part of the Fleet UI Refresh #38.

The goal of this PR was to insert all updated Fleet assets and remove all old assets. More style changes, including the exact sizing and placement of the new images, will occur in a future PR.
2020-11-23 11:22:42 -08:00
noahtalerman
23ce98ec51
Add Nunito Sans font and update global styles. (#59)
This is the first PR as a part of the Fleet UI Refresh #38.

Changes include:

- Add Nunito Sans font files and modify global styles to reflect the font change.
- Modify global font variables to reflect new sizing and weight naming conventions.
-- New sizing and naming conventions:
--- SIZE: xx-small: 12px, x-small: 14px, small: 16px, medium: 20px, large: 24px, x-large: 28px
--- WEIGHT: regular: 400, bold: 700
- Remove the old Oxygen font files.

Changes to other style sheets reflect the changes to the new font sizing and weight naming conventions for global variables. The changes don't necessarily use the correct size (as illustrated by mockups). Those "up to spec" sizing changes are to come.
2020-11-19 16:51:22 -08:00
Zach Wasserman
eea6aa5c64
Fix unexpected print of migration status warning (#50)
Replace the now-deleted migration
server/datastore/mysql/migrations/data/20181119180000_DeleteSoftDeletedEntities.go
with a new migration containing the same timestamp. This allows Fleet to
see the appropriate migration state for users upgrading from previous
versions without actually modifying the DB.

Fixes #48
2020-11-18 11:44:06 -08:00
Zach Wasserman
e7912c2d31
Minor code/comment cleanups (#47)
- Fixes an initialization error panic to a fatal log
2020-11-18 11:10:55 -08:00
Zach Wasserman
0bb9d69ece
Add debug endpoints and associated fleetctl commands (#45)
Adds endpoints and fleetctl commands to retrieve various debug profiles
from the Fleet server.

The best summary is from the help text:

```
fleetctl debug
NAME:
   fleetctl debug - Tools for debugging Fleet

USAGE:
   fleetctl debug command [command options] [arguments...]

COMMANDS:
   profile    Record a CPU profile from the Fleet server.
   cmdline    Get the command line used to invoke the Fleet server.
   heap       Report the allocated memory in the Fleet server.
   goroutine  Get stack traces of all goroutines (threads) in the Fleet server.
   trace      Record an execution trace on the Fleet server.
   archive    Create an archive with the entire suite of debug profiles.

OPTIONS:
   --config value   Path to the Fleet config file (default: "/Users/zwass/.fleet/config") [$CONFIG]
   --context value  Name of Fleet config context to use (default: "default") [$CONTEXT]
   --help, -h       show help
```
2020-11-17 17:12:37 -08:00
Zach Wasserman
ab94d94da0
Fix fleetctl Windows issues (#40)
- Properly set the path for the config file on Windows.
- Check for appropriate settings for TLS config.

Fixes #39
2020-11-17 16:02:14 -08:00
billcobbler
618ba56484
Add host additional info filters (#28)
This change adds the ability to filter additional host info via the list hosts endpoint; a continuation from [here](https://github.com/kolide/fleet/pull/2330), but now filtering is accomplished via SQL. 

Additional object without filter:
```
curl 'https://localhost:8080/api/v1/kolide/hosts'
...
"additional": {
        "macs": [
          {
            "mac": "00:00:00:00:00:00"
          },
          {
            "mac": "02:42:c0:a8:10:05"
          }
        ],
        "time": [
          {
            "day": "13",
            "hour": "3",
            "year": "2020",
            "month": "10",
            "minutes": "43",
            "seconds": "11",
            "weekday": "Tuesday",
            "datetime": "2020-10-13T03:43:11Z",
            "iso_8601": "2020-10-13T03:43:11Z",
            "timezone": "GMT",
            "timestamp": "Tue Oct 13 03:43:11 2020 UTC",
            "unix_time": "1602560591",
            "local_time": "1602560591",
            "local_timezone": "UTC"
          }
},
...
```

Additional object with filter:
```
curl 'https://localhost:8080/api/v1/kolide/hosts?additional_info_filters=macs,notreal'
...
"additional": {
        "macs": [
          {
            "mac": "00:00:00:00:00:00"
          },
          {
            "mac": "02:42:c0:a8:10:05"
          }
        ],
        "notreal": null
},
...
```
2020-11-13 16:33:25 -08:00
Zach Wasserman
8f435a95ba
Allow @ character in usernames (#37)
PR #9 unintentionally exposed the validation that prevented the @
character in usernames. We have decided there is no reason to block this
character.

Fixes #36
2020-11-13 15:52:20 -08:00
Zach Wasserman
6cbd10965c
Add dev infrastructure and docs for Prometheus monitoring (#33)
- Set up a simple example of Prometheus monitoring in the development
  docker-compose.yml.
- Add documentation for configuring Prometheus.
2020-11-12 19:06:56 -08:00
Mike Arpaia
af96e52a00
Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
noahtalerman
ecacbf212b
Changes to additional links for transition to fleetdm (#21) 2020-11-04 21:48:39 -08:00
Zachary Wasserman
a1720db58e
Fix regression creating packs (#17)
This is another error introduced in
https://github.com/kolide/fleet/pull/2327 we did not catch previously
due to insufficient unit test coverage. Test is now added.
2020-11-04 21:18:02 -08:00
Zachary Wasserman
e452cc6a8a
Add file carving support (#15)
- Add endpoints for osquery to register and continue a carve.
- Implement client functionality for retrieving carve details and contents in fleetctl.
- Add documentation on using file carving with Fleet.

Addresses kolide/fleet#1714
2020-11-04 20:45:16 -08:00
Zachary Wasserman
7c923d9d19
Implement fleetctl user create (#9)
- Allow user creation via `fleetctl user create`
- Cleanup and rename existing methods for clarity

Fixes https://github.com/kolide/fleet/issues/2306
2020-11-04 17:06:55 -08:00