Commit Graph

48 Commits

Author SHA1 Message Date
Tomas Touceda
825939e3dc
Improve osquery perf (#2503)
* Improve osquery perf

* Update dependencies

* Embed template so this can be run from the root of the repo

* Fix lint

* Address review comments
2021-10-14 10:09:58 -03:00
Tomas Touceda
8600d71d35
Add osquery perf (#2190)
* Add osquery perf

* Update dockerfile and gh action
2021-09-22 17:18:55 -03:00
Martin Angers
0ac4fc4c1a
Update redisc and replace EachRedisNode implementation (#1949) 2021-09-07 13:25:36 -04:00
Tomas Touceda
d6d53acda5
Issue 1600 fleetctl license expiration (#1800)
* Show banner in fleet and fleetctl if license expired

* Ignore if license is nil or tier is core

* Address review comments
2021-08-26 10:28:53 -03:00
Tomas Touceda
c6c63ab12a
Refactor app config (POC, for now) (#1685) 2021-08-20 12:27:41 -03:00
Tomas Touceda
96c0244c04
Fix redis locking receive (#1655)
* Receive redis data with timeout to not hold the connection

* Address review comments
2021-08-13 11:16:11 -03:00
Tomas Touceda
989e638cc0
Make creating dirs and files more secure by checking permissions (#1566)
* Add safe mkdirall and open

* Use secure as much as possible and merge gomodules for orbit to fleet

* Improve openfile and mkdirall to check for permissiveness instead of equality

* Don't shift

* Fix links

* Address review comments
2021-08-11 11:02:22 -03:00
Zach Wasserman
9db49a1128
Update JWT library to github.com/golang-jwt/jwt (#1568)
- The former repo has been migrated (see
  https://github.com/dgrijalva/jwt-go/issues/462).

- Dependabot alerted on a security issue in the old
  version (https://github.com/advisories/GHSA-w73w-5m7g-f7qc). This issue
  should not effect Fleet due to no use of `aud` claims.
2021-08-06 08:49:38 -07:00
Tomas Touceda
f8b7a83cc6
Process stored CPEs and store found CVEs (#1533)
* WIP

* WIP

* Make path optional and fix tests

* Add first generate

* Move to nvd package

* remove replace

* Re-add replace

* It's path, not file name

* Change how db path is set and use etag

* Fix typos

* Make db generation faster

* Remove quotes

* Doesn't like comments

* Samitize etag and save to file

* Refactor some things and improve writing of etagenv

* Compress file and truncate amount of items for faster testing

* Remove quotes

* Try to improve performance

* Ignore truncate error if not exists

* Minor cleanup and make sqlite have cpe prefix

* Simplify code and test sync

* Add VCR for sync test

* Check for nvdRelease nil

* Add test for the actual translation

* Address review comments

* Rename generate command because we'll have a cve one too

* Move to its own dir

* Add first cve db generation

* WIP but with final strategy, preparring to merge main

* Fix merge conflicts

* WIP

* wip

* Insert CVEs to the db

* Remove unused code

* Use wg instead of counting

* Call cancelFunc to avoid ctx leak

* Fix logs for better readability

* Point code to fleetdm instead of my repo
2021-08-04 18:01:39 -03:00
Tomas Touceda
46b0b7765b
Issue 1435 software to cpe (#1488)
* WIP

* WIP

* Make path optional and fix tests

* Add first generate

* Move to nvd package

* remove replace

* Re-add replace

* It's path, not file name

* Change how db path is set and use etag

* Fix typos

* Make db generation faster

* Remove quotes

* Doesn't like comments

* Samitize etag and save to file

* Refactor some things and improve writing of etagenv

* Compress file and truncate amount of items for faster testing

* Remove quotes

* Try to improve performance

* Ignore truncate error if not exists

* Minor cleanup and make sqlite have cpe prefix

* Simplify code and test sync

* Add VCR for sync test

* Check for nvdRelease nil

* Add test for the actual translation

* Address review comments

* Rename generate command because we'll have a cve one too

* Move to its own dir

* Address review comments
2021-07-29 13:10:34 -03:00
Tomas Touceda
d5e40f329e
Issue 1324 add activity feed (#1343)
* Add activities generation

* Add activities endpoint

* Fix merge error

* Fix indentation issue

* Add changes file

* Address PR review comments

* Add mock activity func

* Address codacy warings

* Set foreign key but on delete set null

* Make user_id set to null if deleted
2021-07-13 16:54:22 -03:00
Zach Wasserman
c5280c0517
Add v4 suffix in go.mod (#1224) 2021-06-25 21:46:51 -07:00
Zach Wasserman
07b89c4cbd
Implement release automation (#1215)
- Use goreleaser to automate release process.
- Add new dockerfiles for fleet (with fleetctl) and fleetctl (only).
- Add GitHub Action Workflow to run goreleaser on new tag.
- Update NPM to match new archive naming.
2021-06-25 18:40:26 -07:00
Zach Wasserman
2ad557e3b3 Merge branch 'main' into teams 2021-06-18 09:42:20 -07:00
dsbaha
47b423ee29
Add Redis cluster support (#1045)
This should support Redis in both cluster and non-cluster modes.

Updates were made separately to github.com/throttled/throttled to support the slight changes in types.

Co-authored-by: Joseph Macaulay <joseph.macaulay@uber.com>
Co-authored-by: Zach Wasserman <zach@fleetdm.com>
2021-06-18 08:51:47 -07:00
Zach Wasserman
9876dbe6b6
Implement license loading and validation (#908)
- Load license from ES256 signed JWT key.
- Parse license claims into LicenseInfo struct.
- Update contribution documentation with sample license key.

Closes #816.
2021-05-31 11:35:15 -07:00
Zach Wasserman
69f73b56f8
Upgrade dependencies and remove replace directive (#802)
Now that https://github.com/throttled/throttled/pull/87 is merged, we
can remove the stopgap `replace` directive in `go.mod`.
2021-05-18 17:00:20 -07:00
Zach Wasserman
1638d5e2da
Update Redigo version and timeouts (#793)
- Use newer version of Redigo that properly handles TLS mismatch.
- Add timeouts for Redis TCP connection.

Fixes #792
2021-05-18 15:04:14 -07:00
WangXiang
468754f2b9
Format and clean code (#774)
1. use [staticcheck](https://staticcheck.io/) to check the code, and fix some issues.
2. use `go fmt` to format the code.
3. use `go mod tidy` clean the go mod.
2021-05-17 10:29:50 -07:00
Zach Wasserman
e8669818eb
Initial backend software inventory implementation (#678)
- Maintain software inventory with detail queries.
- Associated database migrations.
- Feature flagged off by default (see documentation for details to turn on).
- Documentation.
- New test helper for slice element comparisons skipping ID.
2021-04-26 08:44:22 -07:00
Zach Wasserman
9064ef9b59
Clean up Go dependencies (#619)
Removes unused libraries and switches a use of bmizerany/assert to
stretchr/testify/assert.
2021-04-08 16:45:04 -07:00
Zach Wasserman
a79ad7d312
Improvements to built-in labels (#562)
- Fix issue with built-in labels showing multiple platforms when hosts
  are reinstalled with new platform.
- Add Red Hat Linux built-in label.
- Display more labels by default in target selector.

Fixes #546, #553
2021-03-30 15:37:33 -07:00
Zach Wasserman
2203bd81a2
Add support for IdP-initiated SSO login (#556)
This feature is off by default due to minor potential security concerns
with IdP-initiated SSO (see
https://auth0.com/docs/protocols/saml-protocol/saml-configuration-options/identity-provider-initiated-single-sign-on#risks-of-using-an-identity-provider-initiated-sso-flow).

Closes #478
2021-03-30 12:56:20 -07:00
Zach Wasserman
12d292164f
Add rate-limiting to login and password reset (#543)
Prevent abuse of these endpoints with rate limiting backed by Redis. The
limits assigned should be appropriate for almost any Fleet deployment.

Closes #530
2021-03-26 11:23:29 -07:00
Zach Wasserman
d621650b78
Initial implementation of agent update management (#516)
This PR contains the initial implementation of the fleetctl updates commands, along with documentation on using this to self-host an agent update server.

Co-authored-by: Noah Talerman <noahtal@umich.edu>
Co-authored-by: Mike McNeil <mikermcneil@users.noreply.github.com>
2021-03-26 10:46:51 -07:00
Zach Wasserman
58d1d025a4
Upgrade go-bindata to maintained github.com/kevinburke/go-bindata (#512)
- This seems to be the maintained, trusted (by Homebrew, etc.) version
  of go-bindata.
- Add tools.go file to pin version with go modules.
- Use go run to run the binary, making easier configuration for new developers.
2021-03-22 11:03:03 -07:00
Zach Wasserman
db3c73d619
Test and correct semantics for MySQL withRetryTxx (#488)
- Differentiate retryable vs. non-retryable errors.
- Test functionality against mocked DB.
2021-03-17 11:23:52 -07:00
Zach Wasserman
4cfcb1b084
Upgrade fleetctl github.com/urfave/cli to v2 (#471)
This is intended to upgrade to the new API without changing fleetctl
functionality.
2021-03-12 16:42:38 -08:00
Zach Wasserman
e2e7e4707d
Update database migration tooling (#281)
- Use updated github.com/fleetdm/goose that generates templates
  requiring less editing.
- Add `make migration` to Makefile.
- Update documentation.
2021-02-05 09:48:06 -08:00
Zach Wasserman
bb124dc2b9
Improvements to fleetctl preview experience (#234)
- Make the preview directory in the default .fleet directory.
- Check for Docker daemon installed but not running.
- Add message for Chrome users on self-signed certs.
- Display login information on later invocations of command.
- Remove "Kolide" from error messages.

Closes #190

Part of #197
2021-01-28 07:57:32 -08:00
Zach Wasserman
44be83eefd
Improved error handling in host enrollment (#203)
This will make it easier to debug issues with the enrollment process,
such as #200.
2021-01-19 14:45:58 -08:00
Zach Wasserman
57812a532e
Merge pull request from GHSA-w3wf-cfx3-6gcx
* Update github.com/russellhaering/goxmldsig

* Update signature validation to include Mattermost XML validator
2020-12-14 18:14:14 -08:00
Zach Wasserman
beb7e8b965
Add host enrollment cooldown period (#112)
This addresses an issue some users experienced in which performance
problems were encountered when hosts were "competing" for enrollment
using the same osquery host identifier. The issue is addressed by adding
a cooldown period for host enrollment, preventing the same (as judged by
osquery host identifier) host from enrolling more than once per minute.

When users end up in the problematic scenario, they will see quite a bit
of error logs due to this issue. For now that's probably a good thing as
users need to be aware of the lack of visibility. We can explore rate
limiting the logging if that becomes an issue for someone.

Fixes #102
2020-12-10 11:04:58 -08:00
Mike Arpaia
af96e52a00
Update the Go import paths to new repo name (#27) 2020-11-11 09:59:12 -08:00
Zachary Wasserman
e452cc6a8a
Add file carving support (#15)
- Add endpoints for osquery to register and continue a carve.
- Implement client functionality for retrieving carve details and contents in fleetctl.
- Add documentation on using file carving with Fleet.

Addresses kolide/fleet#1714
2020-11-04 20:45:16 -08:00
Zachary Wasserman
7c923d9d19
Implement fleetctl user create (#9)
- Allow user creation via `fleetctl user create`
- Cleanup and rename existing methods for clarity

Fixes https://github.com/kolide/fleet/issues/2306
2020-11-04 17:06:55 -08:00
Zachary Wasserman
f06c3cff93
Add pretty-printing option for query results (#4)
Use --pretty to enable this feature.

This will likely break on non-ANSI terminals.
2020-11-04 09:56:57 -08:00
Zachary Wasserman
328e8a4c8d
Update github.com/go-sql-driver/mysql to 1.5.0 (#2321) 2020-10-08 09:17:08 -07:00
Zachary Wasserman
7494513400 Clean up and comments before merge. 2020-07-21 14:05:46 -07:00
Zachary Wasserman
0502412e15 Move live query operations from MySQL to Redis
This change optimizes live queries by pushing the computation of query
targets to the creation time of the query, and efficiently caching the
targets in Redis. This results in a huge performance improvement at both
steady-state, and when running live queries.

- Live queries are stored using a bitfield in Redis, and takes
advantage of bitfield operations to be extremely efficient.

- Only run Redis live query test when REDIS_TEST is set in environment

- Ensure that live queries are only sent to hosts when there is a client
listening for results. Addresses an existing issue in Fleet along with
appropriate cleanup for the refactored live query backend.
2020-07-21 14:05:46 -07:00
Zachary Wasserman
2578466697
Upgrade SockJS JS and Go dependencies (#2256) 2020-07-06 20:04:04 -07:00
billcobbler
44c40e4255
Retry sqlx transactions (#2203)
- Add exponential backoff method to MySQL datastore
- Refactor queries using transaction to use backup.

Fixes #845
2020-03-10 18:14:02 -07:00
Zachary Wasserman
6dbc3bc77a
Implement fleetctl goquery (#2186)
- Update fleet APIs to support necessary operations in goquery
- Implement support for goquery in fleetctl
2020-01-23 21:27:20 -08:00
Zachary Wasserman
3b02640334
Update goxmldsig dependency (#2177)
Update the github.com/russellhaering/goxmldsig dependency and apply
the appropriate fixes for the API changes.

This is a preparation for integration with
github.com/AbGuthrie/goquery, which uses a newer version of the
dependency.
2020-01-13 16:15:14 -08:00
Christian Barcenas
b3a7fc83e1 Use canonical import path for gopkg.in/guregu/null (#2171)
The `gopkg.in` import path is used in the package's documentation, and
by downstream Linux distro maintainers (e.g. Debian Go Packaging Team).
2020-01-13 09:11:31 -08:00
Zachary Wasserman
bf9ae6ec0e
Update osquery-go dependency to fix thrift module download problem. (#2105)
Fixes #2102
2019-09-12 17:35:10 -07:00
Zachary Wasserman
a06ad6f4fa
Run go mod tidy (#2098) 2019-08-14 11:11:20 -07:00
Zachary Wasserman
27bbf2ae22
Migrate dependencies to go modules (#2094)
These changes should be transparent to users and developers.

Thanks to @davepgreene who made the first stab at this.
2019-08-13 10:32:23 -07:00