* Make preview work when run from scratch (no orbit running)
* Make vulnerability processing less RAM hungry
* Add changes file
* Only get the cpe list once
* Remove cache
* Try killing osquery as well as orbit and clear their db dir
Co-authored-by: Lucas Rodriguez <lucas@fleetdm.com>
This PR implements the status/result logger functions necessary interface with a Kafka REST Proxy service.
Specifically, this is compatible with the [Confluent KAFKA Rest Proxy Service ](https://docs.confluent.io/1.0/kafka-rest/docs/intro.html).
* Make migrations compatible with GTID replication
Fixes an issue some deployments encountered when migrations used a
statement that is unsupported in GTID replication mode (#2462).
Local dev MySQL now enforces this consistency, so it should be easier to
maintain compatibility going forward.
* Update docker-compose formatting
* if exists
* wip
* wip
* wip
* Add performance stats
* Add docs and other self review fixes
* lint
* Update based on review comments
* Add quick cleanup first and then reset to 1hr
* Reduce the load in the test
* Start a fleetctl preview test
* Add tests for fleetctl preview
* Fix setting of fleetctl auth token in test
* Add fleet instance vulnerabilities config to response of GetAppConfig
* Add checks that fleetctl preview enables vulnerability detection
* Adjust doc for get config API response
* Add the include-server-config flag to fleetctl get config
* Update test now that some of the PRs have been merged
Co-authored-by: Tomas Touceda <chiiph@gmail.com>
- Use `TRUNCATE TABLE` rather than `DELETE FROM` for improved performance.
- Move DDL statement after truncate to avoid issues with retries (due to
column already being created).
#2360
* Ignore empty host users or software inventory
* Only store additional if it's not nil
* Update label/policy updated at when we record the executions and skip saving host
* Update changes file
* Add new feature: team policies
* Continue work on team policies
* Continue work on team policies
* Continue team policies
* Revert accidental deletion
* Rename variables
* code refactored; working on runtime errors
* updated front end docs
* Update URLs from team to teams, add tests for policy auth
* Fix test
* Continue work on team policies
* Add permission checks
* mange hosts functional and cleaned up; typing
* improved label logic
* added try catch to awaits
* lint fixes
* frontend unit tests don't work for functional components
* test fix
* revert
* Address errors related to refetch on window focus
* Add loading error check
* Fix typos in loading error checks
* Guard against invariant condition in useEffect
* Update links and routes for team policies
* lint fixes
* Update frontend/pages/hosts/ManageHostsPage/helpers.ts
Co-authored-by: gillespi314 <73313222+gillespi314@users.noreply.github.com>
* Change inherited policies button, tooltip
* lint fixes
Co-authored-by: gillespi314 <73313222+gillespi314@users.noreply.github.com>
Co-authored-by: Tomas Touceda <chiiph@gmail.com>
* Cache app config in redis
* Add changes files
* Replace string with constant
* Revert some test refactorign and duplicate a bit of test code
* Add test for AppConfig with redis failing
* Fix lint
* Use Doer so it works better in clusters
* Skip unmarshalling if we already did
* Allow to cache hosts if configured
* Omit the setting if empty
* Remove hashing, too much CPU
* Revert caching of host auth... needs a more thought through approach
* Remove config
* Remove old config
* Remove locker interface
* Fix test and address review comments
* Add max jitter percent config
* Fix jitter calc
* Remove comment
* Reduce test jitter to make tests less flaky
* Remove jitter entirely
* Document new config
* Fix doc link
* Add team policies
* Add team policy documentation
* Add changes file
* Update titles
* Fix lint
* Rewrite TeamAuthorize for more clarify
* Explicitly use two slices for clarity
* Simplify switch
* Implement fleetctl get software and the underlying API
* Add documentation
* Simplify list software implementation
* Lint fixes
* Make team name unique
* Address review comments
* Fix lint
* Fix tests
* Support close `LiveQueryResultsHandler`
* Start adding test
* Make LiveQuery exit when the context is Done
* Fix lint and remove debug print
* Update server/service/client_live_query.go
Co-authored-by: Zach Wasserman <zach@fleetdm.com>
* Revert "Update server/service/client_live_query.go"
This reverts commit be67ca1512fe502503e821393c2b9e84f5e6e82e.
Co-authored-by: Tomas Touceda <chiiph@gmail.com>
Co-authored-by: Zach Wasserman <zach@fleetdm.com>
* Add extra debug logging for hosts
* Add changes file
* Ignore if appconfig is nil
* Use slice of uints instead of a string
* Debug response request for enabled hosts
* Add host-id to request/response
* Lint fixes
* Add missing AppConfigFuncs
* Rename core->free and basic->premium
* Fix lint js
* Comment out portion of test that seems to timeout
* Rename tier to premium if basic is still loaded
* fix get hosts command to properly output json/yaml based on command line flag
* add changes file
* added tests for get hosts when specifiying host
* added additional hosts to be returned in test cases
* go fmt
* wip
* Add tests and finish implementation
* Add proper default for periodicity, changes file, and documentation
* Fix tests and add defaults also to new installs
* EnableHostUsers should be true if undefined as well
* In some cases, periodicity can be zero because of the migrations
* Apply defaults when migrating appconfig
* Fix lint
* lint
* Address review comments
* Add global policies
* Update documentation and add extra parameter to config
* Fix failing tests
* Store historic policy records
* Address review comments
And also remove other inmem references I saw by chance
* Add documentation for get by id request
* Add parameter doc
* Move schema generation to a cmd instead of a test
Otherwise it messes up running all tests sometimes depending on how parallel it does
* Remove brain dump for another task
* Make migration tests a separate beast
* Make schema generation idempotent and move dbutils cmd to tools
* Allow all filters and add counts to Policy
* Add test for Policy
Add a relatively minimal set of linters that raise safe and
mostly un-opinionated issues with the code. It runs
automatically on CI via a github action.
* Make team schedule names more user friendly and hide them from host pack stats
* Delete test for a different bug and add migration
* Update name template
* Use GROUP_CONCAT instead of JSON_ARRAYAGG
* Update server/datastore/mysql/software.go
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
* Add All Linux label
* Change name to Linux instead of All Linux to see if e2e likes it better
* Revert "Change name to Linux instead of All Linux to see if e2e likes it better"
This reverts commit 26b79f214e3b744e73270c544f89bb698575f6ea.
* Fix all linux label insert
* Make receive calls to redis conn thread safe
Also removes REDIS_TEST env var. Redis is lightweight and fast, no need
to skip these tests.
* No need to increase the wait
* Fix build, add missing tests for cpe translations
Also dont fail alltogether if there's one issue translating CPEs, log it and continue
* Make it once every hour again
* Use MATCH but escape strings
* add team_id filter to fleetctl via get hosts --team flag & api via api/v1/fleet/hosts and api/v1/fleet/labels/id/hosts
* update tests & add changes file
* Add safe mkdirall and open
* Use secure as much as possible and merge gomodules for orbit to fleet
* Improve openfile and mkdirall to check for permissiveness instead of equality
* Don't shift
* Fix links
* Address review comments
* Dont delete pack stats before inserting new ones to prevent deadlocks
* Remove fk for scheduled_query_stats
* Remove fk removal
* Fix tests
* Remove unneeded comment
* modify packs api to filter non-empty pack_type from response
* change list packs store method to allow filtering for "system-level" packs, by default the api filters these packs from being returned
* add changes file
* don't allow modifications to global or team packs via apply spec
* refactor to use PackListOptions
* Expose vulnerabilities data in host software data
* Gather cves and software in one query
* Expand the test to cover all cases
* Make test less flaky
* WIP
* WIP
* Make path optional and fix tests
* Add first generate
* Move to nvd package
* remove replace
* Re-add replace
* It's path, not file name
* Change how db path is set and use etag
* Fix typos
* Make db generation faster
* Remove quotes
* Doesn't like comments
* Samitize etag and save to file
* Refactor some things and improve writing of etagenv
* Compress file and truncate amount of items for faster testing
* Remove quotes
* Try to improve performance
* Ignore truncate error if not exists
* Minor cleanup and make sqlite have cpe prefix
* Simplify code and test sync
* Add VCR for sync test
* Check for nvdRelease nil
* Add test for the actual translation
* Address review comments
* Rename generate command because we'll have a cve one too
* Move to its own dir
* Add first cve db generation
* WIP but with final strategy, preparring to merge main
* Fix merge conflicts
* WIP
* wip
* Insert CVEs to the db
* Remove unused code
* Use wg instead of counting
* Call cancelFunc to avoid ctx leak
* Fix logs for better readability
* Point code to fleetdm instead of my repo
* Don't return errors in distributed query ingestion, just log them
* Allow for multiple errors in the logging context
* Update check when loading host
* Log multiple errors and add tests for other changes
* Add missing host func
* Add another missing host func
* Add changes file
* Add basic idea
* Implement the new logging strategy everywhere
* Remove unused const
* Add tests and fix error cases
* Fix logging in osquery service
* If there are extras, log info unless force debug
* Change to info
* Fix test
* Make logging context more chainable and force info for sessions
* WIP
* WIP
* Make path optional and fix tests
* Add first generate
* Move to nvd package
* remove replace
* Re-add replace
* It's path, not file name
* Change how db path is set and use etag
* Fix typos
* Make db generation faster
* Remove quotes
* Doesn't like comments
* Samitize etag and save to file
* Refactor some things and improve writing of etagenv
* Compress file and truncate amount of items for faster testing
* Remove quotes
* Try to improve performance
* Ignore truncate error if not exists
* Minor cleanup and make sqlite have cpe prefix
* Simplify code and test sync
* Add VCR for sync test
* Check for nvdRelease nil
* Add test for the actual translation
* Address review comments
* Rename generate command because we'll have a cve one too
* Move to its own dir
* Address review comments
* update .gitattributes to be explicit about line endings with regards to the test certs
* update building-fleet guide to include python2 dependency on windows
* update configuration to default to OS specific temporary directories
* WIP
* Send usage analytics
* Improve loggin of cron tasks and fix test
* Implement appconfig method now that we are checking that as well
* Address review comments
* Migrate all mysql tests to the new form
* Only dump sql if MYSQL_TEST is on
* Removing parallel until we get rid of this code
* Move TestMain to an actual _test file
* A little experiment with tmpfs to speed up the db
* Let's make sure the dump.sql file is also in ram
* WIP
* Add get user_roles and apply for a user_roles spec to fleetctl
* Uncomment other tests
* Update test to check output
* Update test with the new struct
* Mock token so that it doesn't pick up the one in the local machine
* Address review comments
* Fix printJSON and printYaml
* Fix merge conflict error
* WIP
* wip
* wip
* Finish implementation
* Address review comments
* Fix flaky test
* WIP
* Add get user_roles and apply for a user_roles spec to fleetctl
* Uncomment other tests
* Update test to check output
* Update test with the new struct
* Mock token so that it doesn't pick up the one in the local machine
* Address review comments
* Fix printJSON and printYaml
* Fix merge conflict error
* If both roles are specified, fail
* Fix test
* Switch arguments around
* Update test with the new rule
* Fix other tests that fell through the cracks
* Add host users
* Add changes file and test removing pull_request from the on test
* Remove users and store the removal timestamp
* Improve test yml to allow for PRs from forks
* Make roles for users mandatory
* Remove nop migration
* Add missing test for wrong role
* Properly validate global and team roles
* Address codacy issues
* Address codacy review
* No need to check for nil
* First approach to diff
* Refactor things for better readability and testing
* Remove draft comment for algorithm
* Format things a bit better
* Remove unused and simplify code a bit
* Refactor for readability and testing
* Add changes file
* Implement new approach based on review comments
* Make sure to only delete from the current host
* Add single uninstall test and fix code
* Improve code based on review
* Refactor error handling for better extensibility and add more scaffolding for specific db errors
* Add integration tests to check errors from mysql are translated properly
* Address review comments
* Add changes file
* Remove username from UI code
* Remove username from tests
* Remove username from database
* Modify server endpoints for removing username
* Implement backend aspects of removing username
* Update API docs
* Add name to fleetctl
