empayre/fleet
14
0
Fork 0
mirror of https://github.com/empayre/fleet.git synced 2024-11-06 17:05:18 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
7,844 Commits 3 Branches 0 Tags 561 MiB
3d6f84a737
Commit Graph

44 Commits

Author SHA1 Message Date
gillespi314
6fb3a87ae9
Enable errcheck linter for golangci-lint (#8899) 2022-12-05 16:50:49 -06:00
Frank Sievertsen
e54f066266
Add display_name to more results (#8204) 
* Make HostResponses with one dedicated function only
* more display_name for Hosts
 2022-10-14 11:14:18 -03:00
Martin Angers
afb3310937
Migrate team-related endpoints to new pattern (#3740) 2022-01-19 10:52:14 -05:00
Martin Angers
69a4985cac
Use new error handling approach in other packages (#2954) 2021-11-22 09:13:26 -05:00
Lucas Manuel Rodriguez
43d1a8c9a0
Fix races in go tests and run with -race on CI (#2712) 
* Fix races in go tests and run with -race on CI

* Fix race in pubsub

* Increase timeout to 15m for go tests

* CI takes forever, try disabling race

* Remove timeout from go tests
 2021-11-01 15:31:01 -03:00
Martin Angers
a8735d55bb
Implement async processing of hosts for label queries (#2288) 2021-11-01 14:13:16 -04:00
Martin Angers
057d4e8b2e
Add configuration and support for Redis to read from replicas (#2509) 2021-10-18 09:32:17 -04:00
Martin Angers
1fa5ce16b8
Add configurable Redis connection retries and following of cluster redirections (#2045) 
Closes #1969
 2021-09-15 08:50:32 -04:00
Tomas Touceda
b2efc9f51c
Make redis conn timeout and keep alive configurable (#1968) 
* Make redis conn timeout and keep alive configurable

* Document new configs

* Correct config name
 2021-09-08 17:55:12 -03:00
Martin Angers
9a0871a2f1
Address issues related to Redis Cluster support (#1885) 
Closes #1847 .
 2021-09-01 16:32:57 -04:00
Zach Wasserman
5993018214
Fix live query timeout handling (#1759) 
Correct cleanup of the connections was verified with the pprof goroutine
tool.

Fixes #1751
 2021-08-25 15:28:04 -07:00
Martin Angers
e61cb7e0db
Configure golangci-lint for the whole repository 
Add a relatively minimal set of linters that raise safe and 
mostly un-opinionated issues with the code. It runs 
automatically on CI via a github action.
 2021-08-24 13:35:03 -04:00
Tomas Touceda
96c0244c04
Fix redis locking receive (#1655) 
* Receive redis data with timeout to not hold the connection

* Address review comments
 2021-08-13 11:16:11 -03:00
Tomas Touceda
3d8a766ca1
Make receive calls to redis conn thread safe (#1641) 
* Make receive calls to redis conn thread safe

Also removes REDIS_TEST env var. Redis is lightweight and fast, no need
to skip these tests.

* No need to increase the wait
 2021-08-11 17:34:35 -03:00
Zach Wasserman
8723b83039
Fix data race in inmem query results test (#1369) 
Missed locking caused a race condition detected with the --race flag:

```
==================
WARNING: DATA RACE
Read at 0x00c0004b2cf0 by goroutine 67:
  runtime.mapaccess2_fast64()
      /usr/local/Cellar/go/1.16.5/libexec/src/runtime/map_fast64.go:52 +0x0
  github.com/fleetdm/fleet/v4/server/pubsub.(*inmemQueryResults).WriteResult()
      /Users/zwass/dev/fleet/server/pubsub/inmem_query_results.go:37 +0x84
  github.com/fleetdm/fleet/v4/server/service.(*Service).ingestDistributedQuery()
      /Users/zwass/dev/fleet/server/service/service_osquery.go:1020 +0x258
  github.com/fleetdm/fleet/v4/server/service.TestIngestDistributedQueryRecordCompletionError()
      /Users/zwass/dev/fleet/server/service/service_osquery_test.go:1499 +0x61c
  testing.tRunner()
      /usr/local/Cellar/go/1.16.5/libexec/src/testing/testing.go:1193 +0x202

Previous write at 0x00c0004b2cf0 by goroutine 104:
  runtime.mapassign_fast64()
      /usr/local/Cellar/go/1.16.5/libexec/src/runtime/map_fast64.go:92 +0x0
  github.com/fleetdm/fleet/v4/server/pubsub.(*inmemQueryResults).getChannel()
      /Users/zwass/dev/fleet/server/pubsub/inmem_query_results.go:31 +0x1a4
  github.com/fleetdm/fleet/v4/server/pubsub.(*inmemQueryResults).ReadChannel()
      /Users/zwass/dev/fleet/server/pubsub/inmem_query_results.go:53 +0x64
  github.com/fleetdm/fleet/v4/server/service.TestIngestDistributedQueryRecordCompletionError.func1()
      /Users/zwass/dev/fleet/server/service/service_osquery_test.go:1493 +0x117

Goroutine 67 (running) created at:
  testing.(*T).Run()
      /usr/local/Cellar/go/1.16.5/libexec/src/testing/testing.go:1238 +0x5d7
  testing.runTests.func1()
      /usr/local/Cellar/go/1.16.5/libexec/src/testing/testing.go:1511 +0xa6
  testing.tRunner()
      /usr/local/Cellar/go/1.16.5/libexec/src/testing/testing.go:1193 +0x202
  testing.runTests()
      /usr/local/Cellar/go/1.16.5/libexec/src/testing/testing.go:1509 +0x612
  testing.(*M).Run()
      /usr/local/Cellar/go/1.16.5/libexec/src/testing/testing.go:1417 +0x3b3
  main.main()
      _testmain.go:303 +0x236

Goroutine 104 (running) created at:
  github.com/fleetdm/fleet/v4/server/service.TestIngestDistributedQueryRecordCompletionError()
      /Users/zwass/dev/fleet/server/service/service_osquery_test.go:1492 +0x558
  testing.tRunner()
      /usr/local/Cellar/go/1.16.5/libexec/src/testing/testing.go:1193 +0x202
==================
==================
WARNING: DATA RACE
Read at 0x00c0000ff2d8 by goroutine 67:
  github.com/fleetdm/fleet/v4/server/pubsub.(*inmemQueryResults).WriteResult()
      /Users/zwass/dev/fleet/server/pubsub/inmem_query_results.go:37 +0xa4
  github.com/fleetdm/fleet/v4/server/service.(*Service).ingestDistributedQuery()
      /Users/zwass/dev/fleet/server/service/service_osquery.go:1020 +0x258
  github.com/fleetdm/fleet/v4/server/service.TestIngestDistributedQueryRecordCompletionError()
      /Users/zwass/dev/fleet/server/service/service_osquery_test.go:1499 +0x61c
  testing.tRunner()
      /usr/local/Cellar/go/1.16.5/libexec/src/testing/testing.go:1193 +0x202

Previous write at 0x00c0000ff2d8 by goroutine 104:
  github.com/fleetdm/fleet/v4/server/pubsub.(*inmemQueryResults).getChannel()
      /Users/zwass/dev/fleet/server/pubsub/inmem_query_results.go:31 +0x1bc
  github.com/fleetdm/fleet/v4/server/pubsub.(*inmemQueryResults).ReadChannel()
      /Users/zwass/dev/fleet/server/pubsub/inmem_query_results.go:53 +0x64
  github.com/fleetdm/fleet/v4/server/service.TestIngestDistributedQueryRecordCompletionError.func1()
      /Users/zwass/dev/fleet/server/service/service_osquery_test.go:1493 +0x117

Goroutine 67 (running) created at:
  testing.(*T).Run()
      /usr/local/Cellar/go/1.16.5/libexec/src/testing/testing.go:1238 +0x5d7
  testing.runTests.func1()
      /usr/local/Cellar/go/1.16.5/libexec/src/testing/testing.go:1511 +0xa6
  testing.tRunner()
      /usr/local/Cellar/go/1.16.5/libexec/src/testing/testing.go:1193 +0x202
  testing.runTests()
      /usr/local/Cellar/go/1.16.5/libexec/src/testing/testing.go:1509 +0x612
  testing.(*M).Run()
      /usr/local/Cellar/go/1.16.5/libexec/src/testing/testing.go:1417 +0x3b3
  main.main()
      _testmain.go:303 +0x236

Goroutine 104 (running) created at:
  github.com/fleetdm/fleet/v4/server/service.TestIngestDistributedQueryRecordCompletionError()
      /Users/zwass/dev/fleet/server/service/service_osquery_test.go:1492 +0x558
  testing.tRunner()
      /usr/local/Cellar/go/1.16.5/libexec/src/testing/testing.go:1193 +0x202
==================
--- FAIL: TestIngestDistributedQueryRecordCompletionError (0.01s)
    service_osquery_test.go:1502: PASS:	QueryCompletedByHost(string,uint)
    testing.go:1092: race detected during execution of test
==================
WARNING: DATA RACE
Read at 0x00c0000f8570 by goroutine 70:
  runtime.mapaccess2_fast64()
      /usr/local/Cellar/go/1.16.5/libexec/src/runtime/map_fast64.go:52 +0x0
  github.com/fleetdm/fleet/v4/server/pubsub.(*inmemQueryResults).WriteResult()
      /Users/zwass/dev/fleet/server/pubsub/inmem_query_results.go:37 +0x84
  github.com/fleetdm/fleet/v4/server/service.(*Service).ingestDistributedQuery()
      /Users/zwass/dev/fleet/server/service/service_osquery.go:1020 +0x258
  github.com/fleetdm/fleet/v4/server/service.TestIngestDistributedQuery()
      /Users/zwass/dev/fleet/server/service/service_osquery_test.go:1530 +0x5bc
  testing.tRunner()
      /usr/local/Cellar/go/1.16.5/libexec/src/testing/testing.go:1193 +0x202

Previous write at 0x00c0000f8570 by goroutine 71:
  runtime.mapassign_fast64()
      /usr/local/Cellar/go/1.16.5/libexec/src/runtime/map_fast64.go:92 +0x0
  github.com/fleetdm/fleet/v4/server/pubsub.(*inmemQueryResults).getChannel()
      /Users/zwass/dev/fleet/server/pubsub/inmem_query_results.go:31 +0x1a4
  github.com/fleetdm/fleet/v4/server/pubsub.(*inmemQueryResults).ReadChannel()
      /Users/zwass/dev/fleet/server/pubsub/inmem_query_results.go:53 +0x64
  github.com/fleetdm/fleet/v4/server/service.TestIngestDistributedQuery.func1()
      /Users/zwass/dev/fleet/server/service/service_osquery_test.go:1524 +0x117

Goroutine 70 (running) created at:
  testing.(*T).Run()
      /usr/local/Cellar/go/1.16.5/libexec/src/testing/testing.go:1238 +0x5d7
  testing.runTests.func1()
      /usr/local/Cellar/go/1.16.5/libexec/src/testing/testing.go:1511 +0xa6
  testing.tRunner()
      /usr/local/Cellar/go/1.16.5/libexec/src/testing/testing.go:1193 +0x202
  testing.runTests()
      /usr/local/Cellar/go/1.16.5/libexec/src/testing/testing.go:1509 +0x612
  testing.(*M).Run()
      /usr/local/Cellar/go/1.16.5/libexec/src/testing/testing.go:1417 +0x3b3
  main.main()
      _testmain.go:303 +0x236

Goroutine 71 (running) created at:
  github.com/fleetdm/fleet/v4/server/service.TestIngestDistributedQuery()
      /Users/zwass/dev/fleet/server/service/service_osquery_test.go:1523 +0x4f4
  testing.tRunner()
      /usr/local/Cellar/go/1.16.5/libexec/src/testing/testing.go:1193 +0x202
==================
==================
WARNING: DATA RACE
Read at 0x00c000c480d8 by goroutine 70:
  github.com/fleetdm/fleet/v4/server/pubsub.(*inmemQueryResults).WriteResult()
      /Users/zwass/dev/fleet/server/pubsub/inmem_query_results.go:37 +0xa4
  github.com/fleetdm/fleet/v4/server/service.(*Service).ingestDistributedQuery()
      /Users/zwass/dev/fleet/server/service/service_osquery.go:1020 +0x258
  github.com/fleetdm/fleet/v4/server/service.TestIngestDistributedQuery()
      /Users/zwass/dev/fleet/server/service/service_osquery_test.go:1530 +0x5bc
  testing.tRunner()
      /usr/local/Cellar/go/1.16.5/libexec/src/testing/testing.go:1193 +0x202

Previous write at 0x00c000c480d8 by goroutine 71:
  github.com/fleetdm/fleet/v4/server/pubsub.(*inmemQueryResults).getChannel()
      /Users/zwass/dev/fleet/server/pubsub/inmem_query_results.go:31 +0x1bc
  github.com/fleetdm/fleet/v4/server/pubsub.(*inmemQueryResults).ReadChannel()
      /Users/zwass/dev/fleet/server/pubsub/inmem_query_results.go:53 +0x64
  github.com/fleetdm/fleet/v4/server/service.TestIngestDistributedQuery.func1()
      /Users/zwass/dev/fleet/server/service/service_osquery_test.go:1524 +0x117

Goroutine 70 (running) created at:
  testing.(*T).Run()
      /usr/local/Cellar/go/1.16.5/libexec/src/testing/testing.go:1238 +0x5d7
  testing.runTests.func1()
      /usr/local/Cellar/go/1.16.5/libexec/src/testing/testing.go:1511 +0xa6
  testing.tRunner()
      /usr/local/Cellar/go/1.16.5/libexec/src/testing/testing.go:1193 +0x202
  testing.runTests()
      /usr/local/Cellar/go/1.16.5/libexec/src/testing/testing.go:1509 +0x612
  testing.(*M).Run()
      /usr/local/Cellar/go/1.16.5/libexec/src/testing/testing.go:1417 +0x3b3
  main.main()
      _testmain.go:303 +0x236

Goroutine 71 (running) created at:
  github.com/fleetdm/fleet/v4/server/service.TestIngestDistributedQuery()
      /Users/zwass/dev/fleet/server/service/service_osquery_test.go:1523 +0x4f4
  testing.tRunner()
      /usr/local/Cellar/go/1.16.5/libexec/src/testing/testing.go:1193 +0x202
==================
--- FAIL: TestIngestDistributedQuery (0.01s)
    service_osquery_test.go:1532: PASS:	QueryCompletedByHost(string,uint)
    testing.go:1092: race detected during execution of test
FAIL
FAIL	github.com/fleetdm/fleet/v4/server/service	42.743s
```
 2021-07-13 12:27:41 -07:00
Zach Wasserman
d790ff7d80
Allow Redis connection with unknown CLUSTER command (#1312) 
This improves compatibility with GCP Memorystore Redis.
 2021-07-07 15:57:32 -07:00
Zach Wasserman
c5280c0517
Add v4 suffix in go.mod (#1224) 2021-06-25 21:46:51 -07:00
Zach Wasserman
1417d01407
Make naming of host columns consistent (#1183) 
Adding consistency between API and DB helps to make it easier for users
and developers working with the API to correctly order things.

Closes #317
 2021-06-23 17:32:19 -07:00
Zach Wasserman
2ad557e3b3 Merge branch 'main' into teams 2021-06-18 09:42:20 -07:00
dsbaha
47b423ee29
Add Redis cluster support (#1045) 
This should support Redis in both cluster and non-cluster modes.

Updates were made separately to github.com/throttled/throttled to support the slight changes in types.

Co-authored-by: Joseph Macaulay <joseph.macaulay@uber.com>
Co-authored-by: Zach Wasserman <zach@fleetdm.com>
 2021-06-18 08:51:47 -07:00
Zach Wasserman
fb32f0cf40
Remove kolide types and packages from backend (#974) 
Generally renamed `kolide` -> `fleet`
 2021-06-06 15:07:29 -07:00
Zach Wasserman
6f51bd9cb5
Remove Redis read/write timeouts (#943) 
If no results were sent over the pubsub channel, the client would hang
up. This would cause the query to seem "hung".

Closes #911
 2021-06-03 10:49:25 -07:00
Zach Wasserman
1638d5e2da
Update Redigo version and timeouts (#793) 
- Use newer version of Redigo that properly handles TLS mismatch.
- Add timeouts for Redis TCP connection.

Fixes #792
 2021-05-18 15:04:14 -07:00
Josh Brower
86745ba2dc
Add ability to duplicate live query results in Redis (#762) 
This feature enables a new config option (redis.duplicate_results). When set to true, all Live Query results will be copied to an additional Redis pubsub channel named LQDuplicate

This is useful in a scenario that would involve shipping the Live Query results outside of Fleet, near-realtime.
 2021-05-13 16:01:31 -07:00
Zach Wasserman
9f71fcf440
Speed up MySQL tests (#585) 
Improves MySQL test time (on my 2020 MBP) to ~18s from ~125s.

- Use separate databases for each test to allow parallelization.
- Run migrations only once at beginning of tests and then reload
  generated schema.
- Add `--innodb-file-per-table=OFF` for ~20% additional speedup.
 2021-04-03 11:42:27 -07:00
Mike Arpaia
af96e52a00
Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
Zachary Wasserman
e8e4bc9879
Fix cleanup of orphaned queries (#2316) 
The expiration logic was incorrect leading to queries not being cleaned
up properly. Tests added for the whole subroutine.

Fixes #2302
 2020-10-06 09:30:24 -07:00
Kilian
c61ba759dd
Add redis use_tls cfg (#2311) 
Adding config parameter 'redis.use_tls' to enable tls communications with redis e.g. AWS ElastiCache

Closes #2247
 2020-10-01 16:25:48 -07:00
Stephan Miehe
cf4d8ecfee
Add redis database number support (#2269) 
Fixes #2268
 2020-07-30 08:57:25 -07:00
Zachary Wasserman
7f757d3144 Extract functionName into helper 
Cleans up some repetition in tests.
 2020-07-21 14:05:46 -07:00
Zachary Wasserman
0502412e15 Move live query operations from MySQL to Redis 
This change optimizes live queries by pushing the computation of query
targets to the creation time of the query, and efficiently caching the
targets in Redis. This results in a huge performance improvement at both
steady-state, and when running live queries.

- Live queries are stored using a bitfield in Redis, and takes
advantage of bitfield operations to be extremely efficient.

- Only run Redis live query test when REDIS_TEST is set in environment

- Ensure that live queries are only sent to hosts when there is a client
listening for results. Addresses an existing issue in Fleet along with
appropriate cleanup for the refactored live query backend.
 2020-07-21 14:05:46 -07:00
billcobbler
a83a26b279 Add ability to disable live queries (#2167) 
- Add toggle to disable live queries in advanced settings
- Add new live query status endpoint (checks for disabled via config and Redis health)
- Update QueryPage UI to use new live query status endpoint

Implements #2140
 2020-01-13 16:53:04 -08:00
Zachary Wasserman
1eccf9a874
Add warning in query UI when Redis fails (#2086) 
- Add warning message when Redis fails
- Disable query button when Redis fails
- Refactor SMTP warning banner into component for reuse

Closes #2073
 2019-08-13 09:42:58 -07:00
Zachary Wasserman
6eecaa1019
Update redigo import (#1821) 
- Use new repo path (old repo archived)
- Update version to 2.0.0

Closes #1820
 2018-06-11 12:14:21 -07:00
Mike Arpaia
018b91ab2c Rename project to Kolide Fleet (#1529) 2017-06-22 15:50:45 -04:00
Zachary Wasserman
715d908613 Update go-kit to 0.4.0 (#1411) 
Notable refactoring:
- Use stdlib "context" in place of "golang.org/x/net/context"
- Go-kit no longer wraps errors, so we remove the unwrap in transport_error.go
- Use MakeHandler when setting up endpoint tests (fixes test bug caught during
  this refactoring)

Closes #1411.
 2017-03-15 08:55:30 -07:00
Mike Arpaia
a000751bfe renaming kolide-ose to kolide (#1143) 2017-02-01 10:20:50 -07:00
Zachary Wasserman
b3213e5921 Fix failing Redis query results test (#864) 
Due to the timezone being changed on time.Time's zero value after a roundtrip
serialization, any times being compared in this test must be explicitly set.
 2017-01-10 11:56:38 -08:00
Zachary Wasserman
39ebd81dc5 Close orphaned distributed query campaign after failed publish attempt (#707) 
A distributed query campaign can be "orphaned" (left in the QueryRunning state)
if the Kolide server restarts while it is running, or other weirdness occurs.
When this happens, no subscribers are waiting to read results written by
osqueryd agents, but the agents continue to receive the query. Previously, this
would cause us to error on ingestion.

The new behavior will instead set the campaign to completed when it detects
that it is orphaned. This should prevent sending queries for which there is no
subscriber.

- New NoSubscriber error interface in pubsub
- Detect NoSubscriber errors and close campaigns
- Tests on pubsub and service methods

Fixes #695
 2016-12-27 10:35:19 -05:00
Victor Vrantchan
a47179f142 add a /healthz endpoint which checks that the app is in a healthy state (#674) 
by pinging the mysql and redis backends.

For #93
 2016-12-22 12:07:47 -05:00
John Murphy
6a825c11e3 Datastore refactor (#439) 
Removed Gorm, replaced it with Sqlx

* Added SQL bundling command to Makfile

* Using go-kit logger

* Added soft delete capability

* Changed SearchLabel to accept a variadic param for optional omit list
instead of array

* Gorm removed

* Refactor table structures to use CURRENT_TIMESTAMP mysql function

* Moved Inmem datastore into it's own package

* Updated README

* Implemented code review suggestions from @zwass

* Removed reference to Gorm from glide.yaml
 2016-11-16 21:47:49 +08:00
Zachary Wasserman
9c38d6d19e Update osquery service methods for distributed queries (#476) 2016-11-14 10:22:54 -08:00
Victor Vrantchan
2a9e27e426 add mysql and redis services (#471) 
enable redis and mysql tests in CircleCI environment
switch redis test to use UTC time - https://github.com/golang/go/issues/17885
 2016-11-10 21:04:26 -05:00
Zachary Wasserman
262a48f8eb Datastores for buffering distributed query results (#346) 
A new datastore interface is needed for buffering incoming distributed query results to be sent to the client. This PR attempts to define and implement that interface.

It is intended that the ReadChannel() method be used by the goroutine that will push query results down a websocket to the client. Passing the results through this channel will allow that goroutine to perform a select on both the channel and the websocket, in order to properly handle IO.
 2016-10-31 15:51:19 -07:00