Commit Graph

68 Commits

Author SHA1 Message Date
Lucas Manuel Rodriguez
c641e39ef4
Allow disabling auto updates in fleetctl package and orbit (#4296)
* Add disable-updates flag to fleetctl and orbit

* Fix ruleguard execution error on make lint-go

* Introduce dev-mode for ease of development of orbit

* Add changes file
2022-02-18 15:42:39 -03:00
Zachary Winnerman
c5c72ed713
Add apm for testing apm (#4053)
* Add apm for testing apm

* Testing opentracing

* testing

* Testing

* go fmt

* Add config switch for tracing.

* fixup

* Update cmd/fleet/serve.go

Co-authored-by: Tomas Touceda <chiiph@gmail.com>

* Add support for both elasticapm and opentelemetry

* Fix driver stuff and config options

* Fixup

* fixup

* Add changes file

* Add config for sql driver

* fixup

* Add doc to exported field

* testing

* fixup

* fixup

* Testing again

* fixup

* testing

* Undo

Co-authored-by: Tomas Touceda <chiiph@gmail.com>
2022-02-15 12:42:22 -05:00
Lucas Manuel Rodriguez
aa9e257a2a
Upgrade fleetdm/goose version to latest (#4195) 2022-02-14 17:33:37 -03:00
Lucas Manuel Rodriguez
be72dc356c
Add CentOS parsing+post-processing to reduce false positives in vulnerability processing (#4037)
* Add CentOS parsing and post-processing in fleet

* Add tests and amend SyncCPEDatabase

* Add test for centosPostProcessing

* Changes from PR comments

* Amend software test

* Fix sync test

* Add index to source and vendor

* Use os.MkdirTemp

* Rearrange migrations

* Regenerate test schema

* Add support for testing migrations (#4112)

* Add support for testing migrations

* Rename migration in tests

* Changes suggested in PR

* Go mod tidy
2022-02-14 15:13:44 -03:00
Lucas Manuel Rodriguez
9c25ea1641
Prepare LoadHostByNodeKey query once (#4128)
* Prepare LoadHostByNodeKey query once

* Use a protected map for storing statements

* Add proposed test
2022-02-14 12:13:38 -03:00
Zach Wasserman
1b1d3f8a61
Update go-tuf dependency (#3837)
Resolves the warning described in #3699 by updating to the latest
version of the dependency with the warning fixed.

The warning should go away on all clients after new metadata is 
generated with these changes.
2022-02-10 08:16:36 -08:00
Lucas Manuel Rodriguez
81672ee50e
Add dev sql interceptor for logging of queries (#3815)
* Add dev sql interceptor for logging of queries

* Remove extra spaces and tabs from queries

* Make regex global
2022-01-21 14:28:21 -03:00
Tomas Touceda
9d572309ae
Add sentry (#3669)
* Add sentry

* Fix gosum

* More gosum fixes

* Add missing def for config

* Enrich sentry scope a bit

* Add changes file

* Add goroutine safe scope to errors

* Encapsulate sentry logic

* Add documentation for new flag

* Add sentry capturing to crons and other background tasks

* Only send to sentry when enabled
2022-01-20 16:41:02 -03:00
Martin Angers
e988d16eb3
Update the prometheus go client library (#3140) 2021-12-20 09:20:58 -05:00
Zach Wasserman
4e72ebb68f
Build Orbit releases in GitHub Actions (#3413)
Complete build including macOS signing+notarization.
2021-12-19 17:06:17 -08:00
Tomas Touceda
a0404b6736
Add software count API (#3105)
* Add software count API

* Fix makefile

* Fine no mock generating at this point

* Actually, one last try

* Use go install instead

* Fix go sum/mod

* Improve documentation

* Try setting node to 14
2021-12-03 10:54:17 -03:00
Tomas Touceda
a36e029d95
App config caching per instance (#3071)
* Do caching of app config per instance instead of across all of them in redis

* Add changes file

* Simplify code based on review comment

* Use go-cache instead of creating our own

* Dont export consts

* Copy app config before returning it

* Fix lint

* Update go sum

* Update go sum
2021-11-29 12:51:57 -03:00
Martin Angers
c997f853e5
Make creation of http.Client uniform across the codebase (#3097) 2021-11-24 15:56:54 -05:00
Victor Vrantchan
a760cdc26d
Add MacAdmins tables in Orbit extension (#2140)
Uses the extension tables from https://github.com/macadmins/osquery-extension.

Co-authored-by: Zach Wasserman <zach@fleetdm.com>
2021-11-15 18:40:53 -08:00
Lucas Manuel Rodriguez
5c1edaf527
Kill existing instances of orbit and osqueryd in fleetctl preview (#2921)
* Kill existing instances of orbit and osqueryd in fleetctl preview

* Add changes file
2021-11-15 18:12:25 -03:00
Zach Wasserman
e279dc1682
Implement fleetctl updates rotate (#2831)
Add support for updating keys used in the TUF update system.
2021-11-15 10:01:48 -08:00
Martin Angers
348841c90b
Add explanation output to fleetctl package (#2864) 2021-11-15 08:40:58 -05:00
Lucas Manuel Rodriguez
837bfed167
Try goqu as sql builder on listSoftwareDB (#2773)
* Try goqu as sql builder on listSoftwareDB

* Fix case insensitive search and add test
2021-11-04 15:21:39 -03:00
Tomas Touceda
2902da76ca
Issue 2027 better error visibility (#2069) 2021-11-02 13:35:57 -04:00
Lucas Manuel Rodriguez
43d1a8c9a0
Fix races in go tests and run with -race on CI (#2712)
* Fix races in go tests and run with -race on CI

* Fix race in pubsub

* Increase timeout to 15m for go tests

* CI takes forever, try disabling race

* Remove timeout from go tests
2021-11-01 15:31:01 -03:00
Tomas Touceda
825939e3dc
Improve osquery perf (#2503)
* Improve osquery perf

* Update dependencies

* Embed template so this can be run from the root of the repo

* Fix lint

* Address review comments
2021-10-14 10:09:58 -03:00
Tomas Touceda
8600d71d35
Add osquery perf (#2190)
* Add osquery perf

* Update dockerfile and gh action
2021-09-22 17:18:55 -03:00
Martin Angers
0ac4fc4c1a
Update redisc and replace EachRedisNode implementation (#1949) 2021-09-07 13:25:36 -04:00
Tomas Touceda
d6d53acda5
Issue 1600 fleetctl license expiration (#1800)
* Show banner in fleet and fleetctl if license expired

* Ignore if license is nil or tier is core

* Address review comments
2021-08-26 10:28:53 -03:00
Tomas Touceda
c6c63ab12a
Refactor app config (POC, for now) (#1685) 2021-08-20 12:27:41 -03:00
Tomas Touceda
96c0244c04
Fix redis locking receive (#1655)
* Receive redis data with timeout to not hold the connection

* Address review comments
2021-08-13 11:16:11 -03:00
Tomas Touceda
989e638cc0
Make creating dirs and files more secure by checking permissions (#1566)
* Add safe mkdirall and open

* Use secure as much as possible and merge gomodules for orbit to fleet

* Improve openfile and mkdirall to check for permissiveness instead of equality

* Don't shift

* Fix links

* Address review comments
2021-08-11 11:02:22 -03:00
Zach Wasserman
9db49a1128
Update JWT library to github.com/golang-jwt/jwt (#1568)
- The former repo has been migrated (see
  https://github.com/dgrijalva/jwt-go/issues/462).

- Dependabot alerted on a security issue in the old
  version (https://github.com/advisories/GHSA-w73w-5m7g-f7qc). This issue
  should not effect Fleet due to no use of `aud` claims.
2021-08-06 08:49:38 -07:00
Tomas Touceda
f8b7a83cc6
Process stored CPEs and store found CVEs (#1533)
* WIP

* WIP

* Make path optional and fix tests

* Add first generate

* Move to nvd package

* remove replace

* Re-add replace

* It's path, not file name

* Change how db path is set and use etag

* Fix typos

* Make db generation faster

* Remove quotes

* Doesn't like comments

* Samitize etag and save to file

* Refactor some things and improve writing of etagenv

* Compress file and truncate amount of items for faster testing

* Remove quotes

* Try to improve performance

* Ignore truncate error if not exists

* Minor cleanup and make sqlite have cpe prefix

* Simplify code and test sync

* Add VCR for sync test

* Check for nvdRelease nil

* Add test for the actual translation

* Address review comments

* Rename generate command because we'll have a cve one too

* Move to its own dir

* Add first cve db generation

* WIP but with final strategy, preparring to merge main

* Fix merge conflicts

* WIP

* wip

* Insert CVEs to the db

* Remove unused code

* Use wg instead of counting

* Call cancelFunc to avoid ctx leak

* Fix logs for better readability

* Point code to fleetdm instead of my repo
2021-08-04 18:01:39 -03:00
Tomas Touceda
46b0b7765b
Issue 1435 software to cpe (#1488)
* WIP

* WIP

* Make path optional and fix tests

* Add first generate

* Move to nvd package

* remove replace

* Re-add replace

* It's path, not file name

* Change how db path is set and use etag

* Fix typos

* Make db generation faster

* Remove quotes

* Doesn't like comments

* Samitize etag and save to file

* Refactor some things and improve writing of etagenv

* Compress file and truncate amount of items for faster testing

* Remove quotes

* Try to improve performance

* Ignore truncate error if not exists

* Minor cleanup and make sqlite have cpe prefix

* Simplify code and test sync

* Add VCR for sync test

* Check for nvdRelease nil

* Add test for the actual translation

* Address review comments

* Rename generate command because we'll have a cve one too

* Move to its own dir

* Address review comments
2021-07-29 13:10:34 -03:00
Tomas Touceda
d5e40f329e
Issue 1324 add activity feed (#1343)
* Add activities generation

* Add activities endpoint

* Fix merge error

* Fix indentation issue

* Add changes file

* Address PR review comments

* Add mock activity func

* Address codacy warings

* Set foreign key but on delete set null

* Make user_id set to null if deleted
2021-07-13 16:54:22 -03:00
Zach Wasserman
c5280c0517
Add v4 suffix in go.mod (#1224) 2021-06-25 21:46:51 -07:00
Zach Wasserman
07b89c4cbd
Implement release automation (#1215)
- Use goreleaser to automate release process.
- Add new dockerfiles for fleet (with fleetctl) and fleetctl (only).
- Add GitHub Action Workflow to run goreleaser on new tag.
- Update NPM to match new archive naming.
2021-06-25 18:40:26 -07:00
Zach Wasserman
2ad557e3b3 Merge branch 'main' into teams 2021-06-18 09:42:20 -07:00
dsbaha
47b423ee29
Add Redis cluster support (#1045)
This should support Redis in both cluster and non-cluster modes.

Updates were made separately to github.com/throttled/throttled to support the slight changes in types.

Co-authored-by: Joseph Macaulay <joseph.macaulay@uber.com>
Co-authored-by: Zach Wasserman <zach@fleetdm.com>
2021-06-18 08:51:47 -07:00
Zach Wasserman
9876dbe6b6
Implement license loading and validation (#908)
- Load license from ES256 signed JWT key.
- Parse license claims into LicenseInfo struct.
- Update contribution documentation with sample license key.

Closes #816.
2021-05-31 11:35:15 -07:00
Zach Wasserman
69f73b56f8
Upgrade dependencies and remove replace directive (#802)
Now that https://github.com/throttled/throttled/pull/87 is merged, we
can remove the stopgap `replace` directive in `go.mod`.
2021-05-18 17:00:20 -07:00
Zach Wasserman
1638d5e2da
Update Redigo version and timeouts (#793)
- Use newer version of Redigo that properly handles TLS mismatch.
- Add timeouts for Redis TCP connection.

Fixes #792
2021-05-18 15:04:14 -07:00
WangXiang
468754f2b9
Format and clean code (#774)
1. use [staticcheck](https://staticcheck.io/) to check the code, and fix some issues.
2. use `go fmt` to format the code.
3. use `go mod tidy` clean the go mod.
2021-05-17 10:29:50 -07:00
Zach Wasserman
e8669818eb
Initial backend software inventory implementation (#678)
- Maintain software inventory with detail queries.
- Associated database migrations.
- Feature flagged off by default (see documentation for details to turn on).
- Documentation.
- New test helper for slice element comparisons skipping ID.
2021-04-26 08:44:22 -07:00
Zach Wasserman
9064ef9b59
Clean up Go dependencies (#619)
Removes unused libraries and switches a use of bmizerany/assert to
stretchr/testify/assert.
2021-04-08 16:45:04 -07:00
Zach Wasserman
a79ad7d312
Improvements to built-in labels (#562)
- Fix issue with built-in labels showing multiple platforms when hosts
  are reinstalled with new platform.
- Add Red Hat Linux built-in label.
- Display more labels by default in target selector.

Fixes #546, #553
2021-03-30 15:37:33 -07:00
Zach Wasserman
2203bd81a2
Add support for IdP-initiated SSO login (#556)
This feature is off by default due to minor potential security concerns
with IdP-initiated SSO (see
https://auth0.com/docs/protocols/saml-protocol/saml-configuration-options/identity-provider-initiated-single-sign-on#risks-of-using-an-identity-provider-initiated-sso-flow).

Closes #478
2021-03-30 12:56:20 -07:00
Zach Wasserman
12d292164f
Add rate-limiting to login and password reset (#543)
Prevent abuse of these endpoints with rate limiting backed by Redis. The
limits assigned should be appropriate for almost any Fleet deployment.

Closes #530
2021-03-26 11:23:29 -07:00
Zach Wasserman
d621650b78
Initial implementation of agent update management (#516)
This PR contains the initial implementation of the fleetctl updates commands, along with documentation on using this to self-host an agent update server.

Co-authored-by: Noah Talerman <noahtal@umich.edu>
Co-authored-by: Mike McNeil <mikermcneil@users.noreply.github.com>
2021-03-26 10:46:51 -07:00
Zach Wasserman
58d1d025a4
Upgrade go-bindata to maintained github.com/kevinburke/go-bindata (#512)
- This seems to be the maintained, trusted (by Homebrew, etc.) version
  of go-bindata.
- Add tools.go file to pin version with go modules.
- Use go run to run the binary, making easier configuration for new developers.
2021-03-22 11:03:03 -07:00
Zach Wasserman
db3c73d619
Test and correct semantics for MySQL withRetryTxx (#488)
- Differentiate retryable vs. non-retryable errors.
- Test functionality against mocked DB.
2021-03-17 11:23:52 -07:00
Zach Wasserman
4cfcb1b084
Upgrade fleetctl github.com/urfave/cli to v2 (#471)
This is intended to upgrade to the new API without changing fleetctl
functionality.
2021-03-12 16:42:38 -08:00
Zach Wasserman
e2e7e4707d
Update database migration tooling (#281)
- Use updated github.com/fleetdm/goose that generates templates
  requiring less editing.
- Add `make migration` to Makefile.
- Update documentation.
2021-02-05 09:48:06 -08:00
Zach Wasserman
bb124dc2b9
Improvements to fleetctl preview experience (#234)
- Make the preview directory in the default .fleet directory.
- Check for Docker daemon installed but not running.
- Add message for Chrome users on self-signed certs.
- Display login information on later invocations of command.
- Remove "Kolide" from error messages.

Closes #190

Part of #197
2021-01-28 07:57:32 -08:00