Martin Angers
84ac0f05a9
Grant write to policies to global maintainer ( #4321 )
2022-02-22 16:57:36 -05:00
Martin Angers
ad5146c491
Allow updating the policy platform (part 1 of the ticket) ( #4311 )
2022-02-22 13:42:03 -05:00
Tomas Touceda
4034a7ab59
Make SearchHosts behave the same as the filtering in ListHosts ( #4295 )
2022-02-22 10:19:51 -03:00
Martin Angers
2ab1b9ec85
Remove expiration of API-only user tokens ( #4314 )
2022-02-22 08:12:03 -05:00
Michal Nicpon
4ce23c7d1b
Add team targets to pack spec ( #4272 )
...
* skip flaky tests
2022-02-21 09:18:58 -07:00
Tomas Touceda
6d582cffa0
Don't depend on last inserted id to get the id for the new software ( #4298 )
...
* Don't depend on last inserted id to get the id for the new software
* Correct typo
2022-02-18 16:30:24 -03:00
Tomas Touceda
8e68ec3b96
Insert one CVE per CPE when there are multiple matches ( #4297 )
...
* Insert one CVE per CPE when there are multiple matches
* Remove comment
* No need to do sqlx.In
2022-02-18 15:25:26 -03:00
gillespi314
089f49cc5d
Add new activity types for policy-related activities ( #4043 )
...
* Add new activities for policy create, delete, and modify, and apply policy spec
* Add PoliciesByID ds method; refactor delete svcs
2022-02-16 09:33:56 -06:00
Martin Angers
80079e4fd7
Fix race in mock datastore for recent_vulns test ( #4230 )
2022-02-16 09:23:27 -05:00
Martin Angers
e29797deb0
Migrate the last batch of authenticatedUser endpoints to the new pattern ( #4210 )
2022-02-15 15:22:19 -05:00
Tomas Touceda
a49f2a06ed
Allow removal of munki versions ( #4189 )
...
* Skip any versions that are empty in munki
* Return not found if munki is uninstalled
* Fix lint
* wip
* Add deleted at for host_munki_version
2022-02-15 16:29:14 -03:00
Michal Nicpon
0be26613b3
Cache team agent options ( #4193 )
...
* use raw literal for json
* wrap cache to clone all values
2022-02-15 12:07:51 -07:00
Michal Nicpon
9009857022
Add get team by name to fleetctl ( #4202 )
2022-02-15 11:48:09 -07:00
Martin Angers
290b5d90e5
Add team target filters to rego authorization checks for running queries ( #4194 )
2022-02-15 13:41:48 -05:00
Zachary Winnerman
c5c72ed713
Add apm for testing apm ( #4053 )
...
* Add apm for testing apm
* Testing opentracing
* testing
* Testing
* go fmt
* Add config switch for tracing.
* fixup
* Update cmd/fleet/serve.go
Co-authored-by: Tomas Touceda <chiiph@gmail.com>
* Add support for both elasticapm and opentelemetry
* Fix driver stuff and config options
* Fixup
* fixup
* Add changes file
* Add config for sql driver
* fixup
* Add doc to exported field
* testing
* fixup
* fixup
* Testing again
* fixup
* testing
* Undo
Co-authored-by: Tomas Touceda <chiiph@gmail.com>
2022-02-15 12:42:22 -05:00
Lucas Manuel Rodriguez
be72dc356c
Add CentOS parsing+post-processing to reduce false positives in vulnerability processing ( #4037 )
...
* Add CentOS parsing and post-processing in fleet
* Add tests and amend SyncCPEDatabase
* Add test for centosPostProcessing
* Changes from PR comments
* Amend software test
* Fix sync test
* Add index to source and vendor
* Use os.MkdirTemp
* Rearrange migrations
* Regenerate test schema
* Add support for testing migrations (#4112 )
* Add support for testing migrations
* Rename migration in tests
* Changes suggested in PR
* Go mod tidy
2022-02-14 15:13:44 -03:00
Zachary Winnerman
3babf53cf4
Fix OSVersion to work properly when distribution does not follow symver ( #3968 )
...
* Fix OSVersion to work properly when distribution does not follow symver
Certain distros such as ClearLinux or ArchLinux do not use symver or any
dotted versioning scheme for their releases. Archlinux uses the static
string "Rolling" and ClearLinux uses a single build number such as 35550
for their versions.
In Fleet console, this shows up as a string like "Archlinux 0.0.0.0"
which makes very little sense to the user. This change makes it so that
if OSQuery cannot generate a dotted version number, we should instead
use the build id as an opaque string.
* Add /changes
* Add tests for os_version
* fixup
* fixup
2022-02-14 12:55:23 -05:00
Michal Nicpon
075702113a
Print version warning when using fleetctl ( #4139 )
...
* Remove deprecated call in fleetctl
* Remove duplicate error returned by app.Run in tests
2022-02-14 09:43:34 -07:00
Lucas Manuel Rodriguez
9c25ea1641
Prepare LoadHostByNodeKey query once ( #4128 )
...
* Prepare LoadHostByNodeKey query once
* Use a protected map for storing statements
* Add proposed test
2022-02-14 12:13:38 -03:00
dayld
c32a225104
users table to cached_users to improve performance ( #4170 )
...
* users table to cached_users to improve performance
* add changes file
Co-authored-by: dayld <>
2022-02-14 09:48:17 -03:00
Zach Wasserman
1e843f3b89
Allow short IdP name in server validation ( #4077 )
...
A customer encountered an error when setting the value to "SSO" which
seems quite reasonable.
2022-02-13 19:35:59 -08:00
Noah Talerman
67827474c2
Prepare for Fleet 4.10.0 ( #4161 )
...
Co-authored-by: Zach Wasserman <zach@fleetdm.com>
2022-02-13 18:13:06 -08:00
Zach Wasserman
e20a9b4508
Add platform filters for MDM/Munki/Chrome queries ( #4144 )
...
* Add platform filters for MDM/Munki/Chrome queries
This should help quiet warnings that users/customers have reported when
these queries try to run on platforms without the macadmins extension
tables.
For #4123
* Improve documentation
* add changes file
* revert doc formatting
* Update tests
* Yet another test fix
Co-authored-by: Tomas Touceda <chiiph@gmail.com>
2022-02-11 14:10:26 -03:00
Tomas Touceda
c45115a915
Properly handle path in CVE URL prefix ( #4174 )
2022-02-11 14:10:13 -03:00
Tomas Touceda
d167556514
Make sure we handle unenrolling properly ( #4158 )
...
* Make sure we handle unenrolling properly
* Update failing test
* Fix test
2022-02-11 08:27:15 -03:00
eashaw
9af92b23aa
Update links to documentation ( #4163 )
...
* update links to documentation
* revert .sailsrc change
* fix broken link
2022-02-10 19:26:18 -06:00
Martin Angers
cc1cf69a0f
Use a dedicated table to store hosts_count and fix pagination with vulns ( #4104 )
2022-02-09 10:16:50 -05:00
Tomas Touceda
11887f87f7
Add enable scheduled query stats to fleet config ( #4066 )
...
* Add enable scheduled query stats to fleet config as well
* Add documentation
* Revert "Allow disabling scheduled query stats via app config (#4049 )"
This reverts commit f98fd4d331
2022-02-09 08:20:29 -03:00
Michal Nicpon
578a9780f2
apply queries spec endpoint missing authorization check ( #4068 )
...
* do authorization check when updating existing query
2022-02-08 09:47:48 -07:00
Martin Angers
73d4794c55
Ignore software_id = 0 when calculating hosts count ( #4080 )
2022-02-08 10:59:17 -05:00
Martin Angers
1686bcafb8
Fix requesting subsequent pages in list software ( #4061 )
2022-02-07 15:57:55 -05:00
Tomas Touceda
35eac78aed
Add CountsUpdatedAt for munki/mdm status ( #4045 )
...
* Add CountsUpdatedAt for munki/mdm status
* Update doc
2022-02-07 14:53:33 -03:00
Martin Angers
1751c7a548
Expand linux platform to all supported linux os in generate hosts stats ( #4051 )
2022-02-07 11:50:36 -05:00
Tomas Touceda
f98fd4d331
Allow disabling scheduled query stats via app config ( #4049 )
...
* Allow disabling scheduled query stats via app config
* Update tests
* Fix test
* Moar test fixes
* Remove redundant set
* Add documentation
* Fix typo in docs
2022-02-07 13:37:54 -03:00
Martin Angers
2cdd614253
Remove todos around queries/run authorization tests ( #3992 )
2022-02-07 09:00:48 -05:00
Tharun Rajendran
2084b7d310
feat(api): add endpoint to get team by id ( #4018 )
...
* feat(api): add endpoint to get team by id
* fix review feedbacks
* add integration test in enterprise suite
2022-02-04 14:33:22 -03:00
Lucas Manuel Rodriguez
a8135aa928
Fix typo and lint checks ( #4013 )
2022-02-03 17:06:49 -03:00
Tomas Touceda
656ef07df1
Move ApplyTeamSpec to ee ( #4011 )
...
* Move ApplyTeamSpec to ee
* Update test now that apply team specs is behind premium
* Check all auth first
* Change auth call for team creation
2022-02-03 16:24:03 -03:00
Tomas Touceda
cf529e70cf
Issue 3173 debug status processlist ( #4009 )
...
* Add innodb status and process list
* Make json output a bit prettier
* Add changes file
* fix lint issues
2022-02-03 14:56:22 -03:00
Lucas Manuel Rodriguez
ab8cc6e7bc
Split hosts in bucket of minutes for the jitter ( #3767 )
...
* Split hosts in bucket of minutes
* New approach on jitter
* Use minutes to define the amount of buckets
* Add logging to jitter hash creation
* Clean up code and remove unused jitter
* Fix test
* Add docs and address review comments
* Address review comments
* Fix typo in doc
Co-authored-by: Tomas Touceda <chiiph@gmail.com>
2022-02-03 14:56:11 -03:00
Tomas Touceda
53ba8f07ea
Issue 3882 clean team packs ( #4002 )
...
* Cleanup team schedules when deleting teams
* Add changes file
* Improve code readability
* Reuse func instead of formatting twice
2022-02-03 14:55:48 -03:00
Tomas Touceda
b2d0a8c79f
Merge pull request from GHSA-ch68-7cf4-35vr
...
* Validate audience restrictions when validating SAML auth reqs
* EntityID is usually the audience
* Add coverage for failures on audience conditions
2022-02-02 15:50:09 -08:00
Martin Angers
6e2ba62744
Trigger webhooks for recently published vulnerabilities ( #3941 )
2022-02-02 16:34:37 -05:00
Tomas Touceda
b90e2e2e3d
Issue 3901 match target sw ( #3982 )
...
* Allow to search with target_sw with a period
* Remove tests for a different thign
* Add a test for programs as a source as well
* Use MATCH again and add proper tests
2022-02-02 17:17:41 -03:00
Martin Angers
6319812984
Fix column name when detecting order by ( #3978 )
2022-02-02 10:08:51 -05:00
Tomas Touceda
a63c549f07
Make software filter by counts faster ( #3975 )
...
* Make software filter by counts faster
* Sort only when it's for the agg table
2022-02-02 09:51:56 -03:00
Martin Angers
4ab7fdd6bb
Return a null timestamp when there are no software counts available ( #3955 )
2022-01-31 17:08:03 -05:00
Martin Angers
ecf6bd8907
Migrate more user-authenticated endpoints to new pattern ( #3933 )
2022-01-31 16:35:22 -05:00
Tomas Touceda
e956b0ba04
Add filter software by CVE and make osquery-perf also push vulnerable software ( #3902 )
...
* Add filter software by CVE and make osquery-perf also push vulnerable software
* Update based on review comments
2022-01-28 10:05:11 -03:00
Tomas Touceda
1667fdcf22
Add vulnerabilities webhook config ( #3897 )
...
* Add vulnerabilities webhook config
* Fix tests
* Update documentation
* Update docs
2022-01-27 10:48:46 -03:00
Tomas Touceda
ffabf803a3
Aggregate munki and mdm data ( #3886 )
...
* Aggregate munki and mdm data
* Update doc
* Use reader to read
* Reader to read
* Address review comments
2022-01-26 17:55:07 -03:00
Martin Angers
a6f3f02a85
Cleanup unused software after calculating the count of hosts ( #3887 )
2022-01-26 11:32:42 -05:00
Martin Angers
9a0f749641
Add hosts_count field to "list software" endpoint ( #3873 )
2022-01-26 09:47:56 -05:00
Martin Angers
39b34508a9
Try to fix flaky publisher-has-listener redis test ( #3876 )
2022-01-26 08:13:01 -05:00
Martin Angers
8b8cebb6fe
Migrate remaining user-authenticated endpoints ( #3796 )
2022-01-25 09:34:00 -05:00
Tomas Touceda
a18e09b613
Simplify fleetctl implementation and improve testing ( #3830 )
...
* Simplify fleetctl implementation and improve testing
* Add a few more
* Handle not founds better
* Fix tests
* Check that logout ds func is called
2022-01-24 16:40:51 -03:00
Tomas Touceda
f02bef6f2c
Add platform filter to host_summary ( #3845 )
...
* Add platform filter to host_summary
* Add documentation
* Actually forward the platform param down the chain
* Update mock
* Update mock
2022-01-24 14:49:21 -03:00
Lucas Manuel Rodriguez
81672ee50e
Add dev sql interceptor for logging of queries ( #3815 )
...
* Add dev sql interceptor for logging of queries
* Remove extra spaces and tabs from queries
* Make regex global
2022-01-21 14:28:21 -03:00
Tomas Touceda
9d572309ae
Add sentry ( #3669 )
...
* Add sentry
* Fix gosum
* More gosum fixes
* Add missing def for config
* Enrich sentry scope a bit
* Add changes file
* Add goroutine safe scope to errors
* Encapsulate sentry logic
* Add documentation for new flag
* Add sentry capturing to crons and other background tasks
* Only send to sentry when enabled
2022-01-20 16:41:02 -03:00
Zach Wasserman
4a70cd69fa
Shorten "simple" query API period to 25s ( #3775 )
...
This helps the period stay under the default request timeouts for most
load balancers.
Some default timeouts:
* AWS ALB - 60s
* Nginx - 60s
* GCP LB - 30s
2022-01-19 17:48:57 -08:00
Lucas Manuel Rodriguez
e5cb68cee9
Return 404 when listing policies for a team that does not exist ( #3793 )
...
* Return 404 when listing policies for a team that does not exist
* Set mock for auth test
2022-01-19 18:17:42 -03:00
Lucas Manuel Rodriguez
77c3a8a61e
Fix flaky TestPolicyWebhooks ( #3777 )
...
* Fix flaky TestPolicyWebhooks
* Run test redis cleanup before running tests
2022-01-19 16:17:00 -03:00
Lucas Manuel Rodriguez
47df5e83fe
Return 400 when trying to create packs, queries and policies with empty names ( #3761 )
...
* Return 400 when trying to create packs, queries and policies with empty names
* Amend sql query test
2022-01-19 16:07:58 -03:00
Martin Angers
afb3310937
Migrate team-related endpoints to new pattern ( #3740 )
2022-01-19 10:52:14 -05:00
Tomas Touceda
eee539cccc
Issue 3707 clean targets on delete ( #3739 )
...
* wip
* Delete targets when deleting teams, hosts, and labels
* Add changes file
* Fix error message
* Remove unused teamsTable
* Cleanup new pack
* Clean new packs at end of test
2022-01-19 10:28:08 -03:00
Zach Wasserman
a79d5fbfcc
Optimize users detail query ( #3754 )
...
@Smjert reported instances of Windows Domain Controllers having massive
resource utilization and being killed by the watchdog when running this
query. In his test environment, this new query performs much better.
2022-01-18 16:39:32 -08:00
Zach Wasserman
6232bfa1d6
Include browser extensions in software inventory ( #3733 )
...
Use appropriate JOINs against users table to include all results.
For #3557
2022-01-18 12:46:04 -08:00
Tomas Touceda
f85941e60c
Use time.after instead of time.tick to not leak ( #3751 )
2022-01-18 16:50:15 -03:00
Lucas Manuel Rodriguez
d4243d0a72
Team observers can browse global policies ( #3737 )
...
* Allow team observers to browse global policies
* Add integration core test for team observer
* Fix integration tests
2022-01-18 13:18:40 -03:00
Martin Angers
f19e676e62
Refactor async host processing to avoid redis SCAN keys (for policies) ( #3657 )
2022-01-18 09:56:43 -05:00
Tomas Touceda
b47cf3d2d4
Better jitter ( #3716 )
...
* Better jitter
* Fix lint
* Use milliseconds
* Make duration milliseconds
* Update based on Lucas' suggestion
* Add changes file
* Panic on error
* Fix compilation error
2022-01-18 11:29:57 -03:00
Lucas Manuel Rodriguez
371c533bfc
Improved Datastore usage of osquery hosts requests ( #3601 )
...
* WIP
* Amend tests
* Do not load aggregated stats for packs
* Add option to host lite
* Fix remaining TODOs
* Fix osquery_utils tests
* Fix SQL
* Fix SQL (bis)
* Restore AuthenticateHost to load once
* Code improvements and re-add deferred host save
* More fixes to the PR
* Wrap users table update on tx
* Add caching to ListPacksForHost and ListScheduledQueriesInPack
* Remove SaveHostSoftware (replaced by UpdateHostSoftware)
* Add unit tests for new functionality
* Add changes file
* Fix scheduled queries test
2022-01-17 22:52:09 -03:00
Martin Angers
c335272de2
Fix failing policy sets for redis cluster mode ( #3725 )
2022-01-17 15:16:54 -05:00
Martin Angers
1f185a7a8b
Refactor async host processing to avoid redis SCAN keys (for labels only) ( #3639 )
2022-01-17 14:53:59 -05:00
Zach Wasserman
72fc9dc524
Remove unused request-id in context ( #3632 )
...
This seems to be left over from the older authorization system in Fleet.
I couldn't find any other reference to the `request-id` in the code.
2022-01-13 14:12:56 -08:00
gillespi314
dc8eacc95c
Add AvailableTeams to loginResponse and getUserResponse ( #3585 )
2022-01-13 13:57:44 -06:00
gillespi314
6952653e1b
Fix usage stats to send missing numLabels field ( #3606 )
2022-01-13 11:11:13 -06:00
Tomas Touceda
4bed4757fb
Remove cleanups and fk for host ids ( #3607 )
...
* Remove cleanups and fk for host ids
* Readd missing things to the schema
* Remove unused
* Add changes file and fix some error messages
* Fix test
* Use tx instead of plain writer
* Other fixes
* More not found test fixes
* Go back to getcontext
2022-01-12 14:07:51 -03:00
Tomas Touceda
312bd840a2
Detect not found errors for queries and policies ( #3595 )
...
* Detect not found errors for queries and policies
* Fix test
2022-01-12 10:04:16 -03:00
Lucas Manuel Rodriguez
49ceee59aa
Add fixes for running tests with mysql:8 and add mysql to test-go job matrix ( #3627 )
...
* Add fixes for running tests with mysql:8
* Add getServer function
* Test github matrix
* Add changes file for the user facing fix
* Remove unused mysql8 docker-compose
2022-01-11 22:44:37 -03:00
Martin Angers
f14f97156c
Add read and write timeout options for redis ( #3624 )
2022-01-11 17:08:39 -05:00
Tomas Touceda
c662cd2b53
Add policies yaml ( #3464 )
...
* Add policies yaml
* Add documentation and address review comments
* Amend documentation
2022-01-11 11:04:29 -03:00
Martin Angers
597144bfac
Migrate most users endpoints to the new pattern ( #3366 )
2022-01-10 14:43:39 -05:00
Mike McNeil
f20762b431
update copyright year in core product transactional email templates ( #3549 )
...
* update copyright year in core product transactional email templates
* Update password_reset.html
* Update change_email_confirmation.html
* Update smtp_setup.html
2022-01-01 12:41:32 -06:00
Zach Wasserman
bda2ef0ca0
Fix regex matching email in host search ( #3539 )
...
Fixes #3528
2021-12-31 09:16:25 -08:00
gillespi314
fca1be4703
Modify /server/utils to handle all 2xx codes as POST success ( #3534 )
2021-12-30 16:00:10 -06:00
Lucas Manuel Rodriguez
aaa5b7ec3c
Allow hosts to check in even if Redis is down ( #3506 )
2021-12-29 17:06:23 -08:00
Lucas Manuel Rodriguez
30e922db07
Fix mdm direct query ingestion for non-mdm hosts ( #3483 )
...
* Fix mdm direct query ingestion for non-mdm hosts
* Amend TODO comment
* Add dummy test
2021-12-24 09:43:31 -03:00
Lucas Manuel Rodriguez
787944482b
Global policies automation webhooks ( #3378 )
...
* Add webhook to app config
* Add redis failing policies set and webhook
* Add basic webhook test
* Store hostname in redis
* Global policy deletion to remove policy ID from set and config
* Also process new passing policies
* Fix unit test
* Sort hosts
* Add more tests
* Add ListSets to the failing policies interface
* Fix server URL and garbage collect on the triggering side
* Do not use Redis SCAN
* Fix Redis operation order
* Add API changes to doc
* Add comments
* Add more tests
* Fix tests
* Add tests for config update upon deletion of policies
* Run make dump-test-schema
* Ignore policies that failed to run
* Add proper unit tests to trigger logic
* Fix comments
* WIP
* Add tests to service_osquerty_test.go
* Use SSCAN for listing hosts instead of SMEMBERS
* Add failing policies to docs/01-Using-Fleet/configuration-files/README.md
* Remove skip
* Fix PR comments
2021-12-23 18:26:55 -03:00
Zach Wasserman
fb10d50f73
Hide warnings for removed migrations ( #3449 )
...
For #3427
2021-12-23 17:07:17 -03:00
Tomas Touceda
7b46df569c
Update return values to be null if the data is not available ( #3490 )
...
* Update return values to be null if the data is not available
* Return nil in the parent object if neither is available
* Improve readability of the code
2021-12-23 16:57:43 -03:00
Tomas Touceda
06a64cdd2c
Use id instead of seen time/created_at to sort hosts ( #3482 )
...
* Use id instead of seen time/created_at to sort hosts
* Add test for ordering by id
2021-12-23 15:45:50 -03:00
Martin Angers
5cf911794f
Fix metrics test by ensuring each path has a unique name ( #3443 )
2021-12-21 16:09:20 -05:00
Martin Angers
1e2059585a
Add support for google chrome profiles ( #3423 )
2021-12-21 15:36:19 -05:00
Lucas Manuel Rodriguez
93f4577c7d
Fix policy membership migration for MariaDB ( #3418 )
...
* Fix policy membership migration for MariaDB
* Use constraintsForTable to get the proper foreign key names
2021-12-21 14:39:46 -03:00
Tomas Touceda
fe67b0486b
Finish first draft of API versions ( #3216 )
...
* Finish first draft of API versions
* wip
* Finalize tests
* Revert change in handler
* Remove made up version
* Update versioning with aliases
* Add changes file
* Address review comments
* Revert overupdated routes
* Expand life time of deprecated APIs
* Fix test
* Comment out problematic part of test
* Revert bad path changes
2021-12-21 12:23:12 -03:00
Martin Angers
a74e562893
Migrate labels endpoints to new pattern ( #3354 )
2021-12-21 09:53:15 -05:00
Tomas Touceda
39e71c6d77
Add mdm and munki ( #3406 )
...
* Draft for mdm and munki as direct ingest funcs
* Expose mdm/munki over API
* Add test for service and fix bug
* Update queries test
* Fix lint
* Address review comments
2021-12-21 09:37:58 -03:00
Martin Angers
e988d16eb3
Update the prometheus go client library ( #3140 )
2021-12-20 09:20:58 -05:00
Lucas Manuel Rodriguez
5aeb418945
Exclude old data migration from the migrations check ( #3373 )
2021-12-16 10:52:42 -03:00
