Improves MySQL test time (on my 2020 MBP) to ~18s from ~125s.
- Use separate databases for each test to allow parallelization.
- Run migrations only once at beginning of tests and then reload
generated schema.
- Add `--innodb-file-per-table=OFF` for ~20% additional speedup.
This change optimizes live queries by pushing the computation of query
targets to the creation time of the query, and efficiently caching the
targets in Redis. This results in a huge performance improvement at both
steady-state, and when running live queries.
- Live queries are stored using a bitfield in Redis, and takes
advantage of bitfield operations to be extremely efficient.
- Only run Redis live query test when REDIS_TEST is set in environment
- Ensure that live queries are only sent to hosts when there is a client
listening for results. Addresses an existing issue in Fleet along with
appropriate cleanup for the refactored live query backend.
Notable refactoring:
- Use stdlib "context" in place of "golang.org/x/net/context"
- Go-kit no longer wraps errors, so we remove the unwrap in transport_error.go
- Use MakeHandler when setting up endpoint tests (fixes test bug caught during
this refactoring)
Closes#1411.
Due to the timezone being changed on time.Time's zero value after a roundtrip
serialization, any times being compared in this test must be explicitly set.
A distributed query campaign can be "orphaned" (left in the QueryRunning state)
if the Kolide server restarts while it is running, or other weirdness occurs.
When this happens, no subscribers are waiting to read results written by
osqueryd agents, but the agents continue to receive the query. Previously, this
would cause us to error on ingestion.
The new behavior will instead set the campaign to completed when it detects
that it is orphaned. This should prevent sending queries for which there is no
subscriber.
- New NoSubscriber error interface in pubsub
- Detect NoSubscriber errors and close campaigns
- Tests on pubsub and service methods
Fixes#695
Removed Gorm, replaced it with Sqlx
* Added SQL bundling command to Makfile
* Using go-kit logger
* Added soft delete capability
* Changed SearchLabel to accept a variadic param for optional omit list
instead of array
* Gorm removed
* Refactor table structures to use CURRENT_TIMESTAMP mysql function
* Moved Inmem datastore into it's own package
* Updated README
* Implemented code review suggestions from @zwass
* Removed reference to Gorm from glide.yaml
A new datastore interface is needed for buffering incoming distributed query results to be sent to the client. This PR attempts to define and implement that interface.
It is intended that the ReadChannel() method be used by the goroutine that will push query results down a websocket to the client. Passing the results through this channel will allow that goroutine to perform a select on both the channel and the websocket, in order to properly handle IO.
