Commit Graph

142 Commits

Author SHA1 Message Date
gillespi314
d172128183
Add label information to host summary response (#5573) 2022-05-10 10:32:55 -05:00
Lucas Manuel Rodriguez
407d8b527e
Remove unused Datastore.SaveHost and unused Modified fields (#5245) 2022-04-26 18:33:29 -03:00
Martin Angers
7187f1adac
Queue jobs for Jira integration when enabled and new vulnerabilities are found. (#4975) 2022-04-11 16:42:16 -04:00
Martin Angers
f28dc10a51
Merge jira-integration branch to main (#4949) 2022-04-05 12:56:15 -04:00
Lucas Manuel Rodriguez
57816592ba
Add read replica testing helpers and fix non-sso login bug (#4908)
not set on the INSERT.
- OUT: Only sets the ID on the passed session and returns it. (`CreatedAt`, `AccessedAt`, are not set.)

New version:

```go
func (ds *Datastore) NewSession(ctx context.Context, userID uint, sessionKey string) (*fleet.Session, error) {
	sqlStatement := `
		INSERT INTO sessions (
			user_id,
			` + "`key`" + `
		)
		VALUES(?,?)
	`
	result, err := ds.writer.ExecContext(ctx, sqlStatement, userID, sessionKey)
	if err != nil {
		return nil, ctxerr.Wrap(ctx, err, "inserting session")
	}

	id, _ := result.LastInsertId() // cannot fail with the mysql driver
	return ds.sessionByID(ctx, ds.writer, uint(id))
}
```

- IN: Define arguments that are truly used when creating a session.
- OUT: Load and return the fleet.Session struct with all values set (using the `ds.writer` to support read replicas correctly).

PS: The new `NewSession` version mimics what we already do with other entities, like policies (`Datastore.NewGlobalPolicy`).
2022-04-04 16:52:05 -07:00
Michal Nicpon
a6902cc083
Add os versions endpoint (#4749) 2022-03-28 09:15:45 -06:00
Lucas Manuel Rodriguez
d2ba34c8fc
Add distributed discovery query support for detail queries, add orbit… (#4597)
* Add distributed discovery query support for detail queries, add orbit_info ingestion

* Amend changes file
2022-03-15 16:51:00 -03:00
Martin Angers
a1c67547b3
Add new endpoints to retrieve device information by orbit identifier (#4531) 2022-03-09 16:13:56 -05:00
Martin Angers
c8bc026d6f
Migrate special-case endpoints to new pattern (#4511) 2022-03-08 11:27:38 -05:00
Martin Angers
18e7c8e236
Implement cron cleanup job of policy membership when policy platform is updated (#4331) 2022-02-23 16:10:37 -05:00
gillespi314
089f49cc5d
Add new activity types for policy-related activities (#4043)
* Add new activities for policy create, delete, and modify, and apply policy spec
* Add PoliciesByID ds method; refactor delete svcs
2022-02-16 09:33:56 -06:00
Lucas Manuel Rodriguez
be72dc356c
Add CentOS parsing+post-processing to reduce false positives in vulnerability processing (#4037)
* Add CentOS parsing and post-processing in fleet

* Add tests and amend SyncCPEDatabase

* Add test for centosPostProcessing

* Changes from PR comments

* Amend software test

* Fix sync test

* Add index to source and vendor

* Use os.MkdirTemp

* Rearrange migrations

* Regenerate test schema

* Add support for testing migrations (#4112)

* Add support for testing migrations

* Rename migration in tests

* Changes suggested in PR

* Go mod tidy
2022-02-14 15:13:44 -03:00
Tomas Touceda
35eac78aed
Add CountsUpdatedAt for munki/mdm status (#4045)
* Add CountsUpdatedAt for munki/mdm status

* Update doc
2022-02-07 14:53:33 -03:00
Tomas Touceda
cf529e70cf
Issue 3173 debug status processlist (#4009)
* Add innodb status and process list

* Make json output a bit prettier

* Add changes file

* fix lint issues
2022-02-03 14:56:22 -03:00
Martin Angers
6e2ba62744
Trigger webhooks for recently published vulnerabilities (#3941) 2022-02-02 16:34:37 -05:00
Tomas Touceda
ffabf803a3
Aggregate munki and mdm data (#3886)
* Aggregate munki and mdm data

* Update doc

* Use reader to read

* Reader to read

* Address review comments
2022-01-26 17:55:07 -03:00
Martin Angers
9a0f749641
Add hosts_count field to "list software" endpoint (#3873) 2022-01-26 09:47:56 -05:00
Tomas Touceda
f02bef6f2c
Add platform filter to host_summary (#3845)
* Add platform filter to host_summary

* Add documentation

* Actually forward the platform param down the chain

* Update mock

* Update mock
2022-01-24 14:49:21 -03:00
Martin Angers
f19e676e62
Refactor async host processing to avoid redis SCAN keys (for policies) (#3657) 2022-01-18 09:56:43 -05:00
Lucas Manuel Rodriguez
371c533bfc
Improved Datastore usage of osquery hosts requests (#3601)
* WIP

* Amend tests

* Do not load aggregated stats for packs

* Add option to host lite

* Fix remaining TODOs

* Fix osquery_utils tests

* Fix SQL

* Fix SQL (bis)

* Restore AuthenticateHost to load once

* Code improvements and re-add deferred host save

* More fixes to the PR

* Wrap users table update on tx

* Add caching to ListPacksForHost and ListScheduledQueriesInPack

* Remove SaveHostSoftware (replaced by UpdateHostSoftware)

* Add unit tests for new functionality

* Add changes file

* Fix scheduled queries test
2022-01-17 22:52:09 -03:00
gillespi314
dc8eacc95c
Add AvailableTeams to loginResponse and getUserResponse (#3585) 2022-01-13 13:57:44 -06:00
Tomas Touceda
4bed4757fb
Remove cleanups and fk for host ids (#3607)
* Remove cleanups and fk for host ids

* Readd missing things to the schema

* Remove unused

* Add changes file and fix some error messages

* Fix test

* Use tx instead of plain writer

* Other fixes

* More not found test fixes

* Go back to getcontext
2022-01-12 14:07:51 -03:00
Lucas Manuel Rodriguez
787944482b
Global policies automation webhooks (#3378)
* Add webhook to app config

* Add redis failing policies set and webhook

* Add basic webhook test

* Store hostname in redis

* Global policy deletion to remove policy ID from set and config

* Also process new passing policies

* Fix unit test

* Sort hosts

* Add more tests

* Add ListSets to the failing policies interface

* Fix server URL and garbage collect on the triggering side

* Do not use Redis SCAN

* Fix Redis operation order

* Add API changes to doc

* Add comments

* Add more tests

* Fix tests

* Add tests for config update upon deletion of policies

* Run make dump-test-schema

* Ignore policies that failed to run

* Add proper unit tests to trigger logic

* Fix comments

* WIP

* Add tests to service_osquerty_test.go

* Use SSCAN for listing hosts instead of SMEMBERS

* Add failing policies to docs/01-Using-Fleet/configuration-files/README.md

* Remove skip

* Fix PR comments
2021-12-23 18:26:55 -03:00
Martin Angers
1e2059585a
Add support for google chrome profiles (#3423) 2021-12-21 15:36:19 -05:00
Tomas Touceda
39e71c6d77
Add mdm and munki (#3406)
* Draft for mdm and munki as direct ingest funcs

* Expose mdm/munki over API

* Add test for service and fix bug

* Update queries test

* Fix lint

* Address review comments
2021-12-21 09:37:58 -03:00
gillespi314
6ee6dba4a4
Update anonymous usage statistics to record Fleet feature adoption (#3056)
* Collect additional anonymous usage statistics

* Update fleetdm.com and docs for usage stats

* Fix lint-go errors

* Fix failing test

* update webhook inputs

* Update receive-usage-analytics.js

* fix failing test

* Update statistics_test.go

Co-authored-by: eashaw <eashaw@sailsjs.com>
2021-12-06 14:39:00 -06:00
Martin Angers
b84ef79bf4
Add the fleetctl debug {errors,db-locks} sub-commands (#3168) 2021-12-06 09:26:01 -05:00
Lucas Manuel Rodriguez
b9a408704e
Add platforms field to policies (#3181)
* Add platforms field to policies

* Fix fleetctl tests

* PR review changes

* Add missing tests

* Add changes for ListPoliciesForHost
2021-12-03 15:33:33 -03:00
Tomas Touceda
a0404b6736
Add software count API (#3105)
* Add software count API

* Fix makefile

* Fine no mock generating at this point

* Actually, one last try

* Use go install instead

* Fix go sum/mod

* Improve documentation

* Try setting node to 14
2021-12-03 10:54:17 -03:00
Lucas Manuel Rodriguez
e64a88d8b1
Add COALESCEs on all host_seen_times JOINs (#3147)
* Add COALESCEs on all host_seen_times JOINs

* Use tx instead of d.writer

* Add unit tests

* Fix compile test
2021-12-01 09:05:23 -03:00
Lucas Manuel Rodriguez
964f85b174
Amend policy creation and spec (for proprietary query), and add update APIs (#2890)
* Amend policy creation (proprietary query), add update APIs

* Fix Datastore.SavePolicy bug (and add tests)

* Add integration tests for new policy APIs

* Add author email

* Add activities

* Push breaking changes for return policy fields

* WIP

* Add integration test for host policies

* Make more improvements to policy representation

* Improve upgrade code (from PR review comments)

* PR changes

* Revert activities for policies

* Use *uint instead of uint for queryID, use fleet.PolicyPayload

* Filter out other schemas

* New policy flow (#2922)

* created new policy flow -- no API connection

* added api props

* fixed prop name

* lint fixes

* removed unused modal; fixed style

* name, desc icons; created global components

* lint fixes

* ignoring certain files and lines for prettier

* Update frontend/pages/policies/PolicyPage/PolicyPage.tsx

* Make policy names unique across deployment

* Amend upgrade script

* Fix migration for unique names

* Do not deduplicate but instead rename policies

Co-authored-by: Martavis Parker <47053705+martavis@users.noreply.github.com>
2021-11-24 14:16:42 -03:00
Lucas Manuel Rodriguez
0642bfdb1d
Add fleetctl debug migrations command to get missing migrations (#2967)
* Add fleetctl command to get missing migrations

* Fix copy paste and lint

* Detect migrations applied out of order

* Add extra bullet to changes

* Trigger creation of migration status tables

* Fix unit tests

* PR fixes

* PR comment fixes
2021-11-22 14:06:12 -03:00
Tomas Touceda
887e42650b
Further optimize host select with policies (#3013)
* Further optimize host select with policies

* Add missing row close

* Improve migration

* Skip migration if column already exists

* Add test for host with/without extras
2021-11-18 14:36:35 -03:00
Tomas Touceda
a0730ca643
Add update invite API (#2837) 2021-11-11 17:33:06 -03:00
Martin Angers
b57b64ccb2
Add total and per platform counts to host summary endpoint (#2845) 2021-11-09 09:35:36 -05:00
Tomas Touceda
7db6de7278
Serialize hosts writes per instance (#2753)
* Serialize hosts writes per instance

* Write hosts asynchronously

* Dont make the save in a goroutine

* Revert "Dont make the save in a goroutine"

This reverts commit 4a890c5271142755dec69a741582e7eca5c4c62c.

* Make all savehosts async

* Address review comments and make this approach configurable

* Address review comments

* Disable bulk seen time marking for a test

* Move host seen times to a new table

* Remove unused

* Add seen_time to list hosts

* Add some jitter to seen time flushing

* Remove unused

* Add timeout to deferred save host

* Add tests for serialSaveHost

* Update hosts in labels and policy executions in a serial way

* Address review comments and remove fk constraints in host software

* Make errCh buffered

* Add changes file

* Readd key
2021-11-08 11:42:37 -03:00
Martin Angers
a8735d55bb
Implement async processing of hosts for label queries (#2288) 2021-11-01 14:13:16 -04:00
Tomas Touceda
79f2c7c391
Issue 1798 rest api for live queries (#2675)
* wip

* wip

* Make tests not flaky

* Add changes file

* Make logging and authz thread safe

* Move cancel defer below err check
2021-10-26 11:33:31 -03:00
Tomas Touceda
c2696c1261
Issue 1892 query perf everywhere (#2580)
* wip

* wip

* wip

* Add performance stats

* Add docs and other self review fixes

* lint

* Update based on review comments

* Add quick cleanup first and then reset to 1hr

* Reduce the load in the test
2021-10-20 18:35:38 -03:00
Tomas Touceda
cbe931c0e9
Add vulnerable filter for software and also wire up the query search (#2604)
* Add vulnerable filter for software and also wire up the query search

* Add documentation

* Update to use software list options
2021-10-20 18:01:20 -03:00
Tomas Touceda
fe5660e006
Reimplement host expiration to not need mysql events (#2552)
* Reimplement host expiration to not need mysql events

* Update mocks
2021-10-19 17:47:37 -03:00
Tomas Touceda
d3a0d62902
Issue 2456 policies yaml (#2512)
* wip

* Add policy specs support

* Add documentation

* Make policy apply idempotent

* Fold in code

* Improve tests and simplify auth checks

* Lint and fix test
2021-10-15 07:34:11 -03:00
Tomas Touceda
28be73c410
Add missing software in list hosts response (#2492) 2021-10-12 15:59:01 -03:00
Tomas Touceda
876aa0e366
Add host count API (#2356)
* Add host count API

* Add rest api docs

* Add host count for labels and address review comments

* Add regex for id
2021-10-07 08:25:35 -03:00
Tomas Touceda
9b809a2d35
Issue 1890 host details policy (#2410)
* wip

* Add policies to hosts
2021-10-07 08:11:10 -03:00
Tomas Touceda
d9f34b2c19
Issue 1840 bulk delete hosts (#2268)
* wp

* Add test by ids

* Add changes file

* Update docs and remove unneeded return values

* Address review comments

* Improve integration tests

* Use TearDownTest
2021-09-29 13:13:23 -03:00
Tomas Touceda
685245c2bd
Cache app config in redis (#2205)
* Cache app config in redis

* Add changes files

* Replace string with constant

* Revert some test refactorign and duplicate a bit of test code

* Add test for AppConfig with redis failing

* Fix lint

* Use Doer so it works better in clusters

* Skip unmarshalling if we already did

* Allow to cache hosts if configured

* Omit the setting if empty

* Remove hashing, too much CPU

* Revert caching of host auth... needs a more thought through approach

* Remove config

* Remove old config

* Remove locker interface

* Fix test and address review comments
2021-09-28 10:01:53 -03:00
Tomas Touceda
1f324339f8
Add jitter to intervals (#2158)
* Add max jitter percent config

* Fix jitter calc

* Remove comment

* Reduce test jitter to make tests less flaky

* Remove jitter entirely

* Document new config

* Fix doc link
2021-09-21 14:21:44 -03:00
Tomas Touceda
4650484960
Remove fk label membership (#2157)
* Remove fk from label membership

* Add changes file

* Fix tests

* No need to IGNORE anymore
2021-09-21 11:48:20 -03:00
Tomas Touceda
b32b441c12
Issue 1512 filter observer can run queries (#2110)
* wip

* Filter queries for observers

* Update e2e test now that we filter queries
2021-09-20 13:07:51 -03:00