This PR implements the status/result logger functions necessary interface with a Kafka REST Proxy service.
Specifically, this is compatible with the [Confluent KAFKA Rest Proxy Service ](https://docs.confluent.io/1.0/kafka-rest/docs/intro.html).
* Make migrations compatible with GTID replication
Fixes an issue some deployments encountered when migrations used a
statement that is unsupported in GTID replication mode (#2462).
Local dev MySQL now enforces this consistency, so it should be easier to
maintain compatibility going forward.
* Update docker-compose formatting
* if exists
* wip
* wip
* wip
* Add performance stats
* Add docs and other self review fixes
* lint
* Update based on review comments
* Add quick cleanup first and then reset to 1hr
* Reduce the load in the test
* Start a fleetctl preview test
* Add tests for fleetctl preview
* Fix setting of fleetctl auth token in test
* Add fleet instance vulnerabilities config to response of GetAppConfig
* Add checks that fleetctl preview enables vulnerability detection
* Adjust doc for get config API response
* Add the include-server-config flag to fleetctl get config
* Update test now that some of the PRs have been merged
Co-authored-by: Tomas Touceda <chiiph@gmail.com>
- Use `TRUNCATE TABLE` rather than `DELETE FROM` for improved performance.
- Move DDL statement after truncate to avoid issues with retries (due to
column already being created).
#2360
* Ignore empty host users or software inventory
* Only store additional if it's not nil
* Update label/policy updated at when we record the executions and skip saving host
* Update changes file
* Add new feature: team policies
* Continue work on team policies
* Continue work on team policies
* Continue team policies
* Revert accidental deletion
* Rename variables
* code refactored; working on runtime errors
* updated front end docs
* Update URLs from team to teams, add tests for policy auth
* Fix test
* Continue work on team policies
* Add permission checks
* mange hosts functional and cleaned up; typing
* improved label logic
* added try catch to awaits
* lint fixes
* frontend unit tests don't work for functional components
* test fix
* revert
* Address errors related to refetch on window focus
* Add loading error check
* Fix typos in loading error checks
* Guard against invariant condition in useEffect
* Update links and routes for team policies
* lint fixes
* Update frontend/pages/hosts/ManageHostsPage/helpers.ts
Co-authored-by: gillespi314 <73313222+gillespi314@users.noreply.github.com>
* Change inherited policies button, tooltip
* lint fixes
Co-authored-by: gillespi314 <73313222+gillespi314@users.noreply.github.com>
Co-authored-by: Tomas Touceda <chiiph@gmail.com>
* Cache app config in redis
* Add changes files
* Replace string with constant
* Revert some test refactorign and duplicate a bit of test code
* Add test for AppConfig with redis failing
* Fix lint
* Use Doer so it works better in clusters
* Skip unmarshalling if we already did
* Allow to cache hosts if configured
* Omit the setting if empty
* Remove hashing, too much CPU
* Revert caching of host auth... needs a more thought through approach
* Remove config
* Remove old config
* Remove locker interface
* Fix test and address review comments
* Add max jitter percent config
* Fix jitter calc
* Remove comment
* Reduce test jitter to make tests less flaky
* Remove jitter entirely
* Document new config
* Fix doc link
* Add team policies
* Add team policy documentation
* Add changes file
* Update titles
* Fix lint
* Rewrite TeamAuthorize for more clarify
* Explicitly use two slices for clarity
* Simplify switch
* Implement fleetctl get software and the underlying API
* Add documentation
* Simplify list software implementation
* Lint fixes
* Make team name unique
* Address review comments
* Fix lint
* Fix tests
* Support close `LiveQueryResultsHandler`
* Start adding test
* Make LiveQuery exit when the context is Done
* Fix lint and remove debug print
* Update server/service/client_live_query.go
Co-authored-by: Zach Wasserman <zach@fleetdm.com>
* Revert "Update server/service/client_live_query.go"
This reverts commit be67ca1512fe502503e821393c2b9e84f5e6e82e.
Co-authored-by: Tomas Touceda <chiiph@gmail.com>
Co-authored-by: Zach Wasserman <zach@fleetdm.com>
* Add extra debug logging for hosts
* Add changes file
* Ignore if appconfig is nil
* Use slice of uints instead of a string
* Debug response request for enabled hosts
* Add host-id to request/response
* Lint fixes
* Add missing AppConfigFuncs
* Rename core->free and basic->premium
* Fix lint js
* Comment out portion of test that seems to timeout
* Rename tier to premium if basic is still loaded
* fix get hosts command to properly output json/yaml based on command line flag
* add changes file
* added tests for get hosts when specifiying host
* added additional hosts to be returned in test cases
* go fmt
* wip
* Add tests and finish implementation
* Add proper default for periodicity, changes file, and documentation
* Fix tests and add defaults also to new installs
* EnableHostUsers should be true if undefined as well
* In some cases, periodicity can be zero because of the migrations
* Apply defaults when migrating appconfig
* Fix lint
* lint
* Address review comments
* Add global policies
* Update documentation and add extra parameter to config
* Fix failing tests
* Store historic policy records
* Address review comments
And also remove other inmem references I saw by chance
* Add documentation for get by id request
* Add parameter doc
* Move schema generation to a cmd instead of a test
Otherwise it messes up running all tests sometimes depending on how parallel it does
* Remove brain dump for another task
* Make migration tests a separate beast
* Make schema generation idempotent and move dbutils cmd to tools
* Allow all filters and add counts to Policy
* Add test for Policy
Add a relatively minimal set of linters that raise safe and
mostly un-opinionated issues with the code. It runs
automatically on CI via a github action.
* Make team schedule names more user friendly and hide them from host pack stats
* Delete test for a different bug and add migration
* Update name template
* Use GROUP_CONCAT instead of JSON_ARRAYAGG
* Update server/datastore/mysql/software.go
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
* Add All Linux label
* Change name to Linux instead of All Linux to see if e2e likes it better
* Revert "Change name to Linux instead of All Linux to see if e2e likes it better"
This reverts commit 26b79f214e3b744e73270c544f89bb698575f6ea.
* Fix all linux label insert
* Make receive calls to redis conn thread safe
Also removes REDIS_TEST env var. Redis is lightweight and fast, no need
to skip these tests.
* No need to increase the wait
* Fix build, add missing tests for cpe translations
Also dont fail alltogether if there's one issue translating CPEs, log it and continue
* Make it once every hour again
* Use MATCH but escape strings
* add team_id filter to fleetctl via get hosts --team flag & api via api/v1/fleet/hosts and api/v1/fleet/labels/id/hosts
* update tests & add changes file
* Add safe mkdirall and open
* Use secure as much as possible and merge gomodules for orbit to fleet
* Improve openfile and mkdirall to check for permissiveness instead of equality
* Don't shift
* Fix links
* Address review comments
* Dont delete pack stats before inserting new ones to prevent deadlocks
* Remove fk for scheduled_query_stats
* Remove fk removal
* Fix tests
* Remove unneeded comment
* modify packs api to filter non-empty pack_type from response
* change list packs store method to allow filtering for "system-level" packs, by default the api filters these packs from being returned
* add changes file
* don't allow modifications to global or team packs via apply spec
* refactor to use PackListOptions
* Expose vulnerabilities data in host software data
* Gather cves and software in one query
* Expand the test to cover all cases
* Make test less flaky
* WIP
* WIP
* Make path optional and fix tests
* Add first generate
* Move to nvd package
* remove replace
* Re-add replace
* It's path, not file name
* Change how db path is set and use etag
* Fix typos
* Make db generation faster
* Remove quotes
* Doesn't like comments
* Samitize etag and save to file
* Refactor some things and improve writing of etagenv
* Compress file and truncate amount of items for faster testing
* Remove quotes
* Try to improve performance
* Ignore truncate error if not exists
* Minor cleanup and make sqlite have cpe prefix
* Simplify code and test sync
* Add VCR for sync test
* Check for nvdRelease nil
* Add test for the actual translation
* Address review comments
* Rename generate command because we'll have a cve one too
* Move to its own dir
* Add first cve db generation
* WIP but with final strategy, preparring to merge main
* Fix merge conflicts
* WIP
* wip
* Insert CVEs to the db
* Remove unused code
* Use wg instead of counting
* Call cancelFunc to avoid ctx leak
* Fix logs for better readability
* Point code to fleetdm instead of my repo
* Don't return errors in distributed query ingestion, just log them
* Allow for multiple errors in the logging context
* Update check when loading host
* Log multiple errors and add tests for other changes
* Add missing host func
* Add another missing host func
* Add changes file
* Add basic idea
* Implement the new logging strategy everywhere
* Remove unused const
* Add tests and fix error cases
* Fix logging in osquery service
* If there are extras, log info unless force debug
* Change to info
* Fix test
* Make logging context more chainable and force info for sessions
* WIP
* WIP
* Make path optional and fix tests
* Add first generate
* Move to nvd package
* remove replace
* Re-add replace
* It's path, not file name
* Change how db path is set and use etag
* Fix typos
* Make db generation faster
* Remove quotes
* Doesn't like comments
* Samitize etag and save to file
* Refactor some things and improve writing of etagenv
* Compress file and truncate amount of items for faster testing
* Remove quotes
* Try to improve performance
* Ignore truncate error if not exists
* Minor cleanup and make sqlite have cpe prefix
* Simplify code and test sync
* Add VCR for sync test
* Check for nvdRelease nil
* Add test for the actual translation
* Address review comments
* Rename generate command because we'll have a cve one too
* Move to its own dir
* Address review comments
* update .gitattributes to be explicit about line endings with regards to the test certs
* update building-fleet guide to include python2 dependency on windows
* update configuration to default to OS specific temporary directories
* WIP
* Send usage analytics
* Improve loggin of cron tasks and fix test
* Implement appconfig method now that we are checking that as well
* Address review comments
* Migrate all mysql tests to the new form
* Only dump sql if MYSQL_TEST is on
* Removing parallel until we get rid of this code
* Move TestMain to an actual _test file
* A little experiment with tmpfs to speed up the db
* Let's make sure the dump.sql file is also in ram
* WIP
* Add get user_roles and apply for a user_roles spec to fleetctl
* Uncomment other tests
* Update test to check output
* Update test with the new struct
* Mock token so that it doesn't pick up the one in the local machine
* Address review comments
* Fix printJSON and printYaml
* Fix merge conflict error
* WIP
* wip
* wip
* Finish implementation
* Address review comments
* Fix flaky test
* WIP
* Add get user_roles and apply for a user_roles spec to fleetctl
* Uncomment other tests
* Update test to check output
* Update test with the new struct
* Mock token so that it doesn't pick up the one in the local machine
* Address review comments
* Fix printJSON and printYaml
* Fix merge conflict error
* If both roles are specified, fail
* Fix test
* Switch arguments around
* Update test with the new rule
* Fix other tests that fell through the cracks
* Add host users
* Add changes file and test removing pull_request from the on test
* Remove users and store the removal timestamp
* Improve test yml to allow for PRs from forks
* Make roles for users mandatory
* Remove nop migration
* Add missing test for wrong role
* Properly validate global and team roles
* Address codacy issues
* Address codacy review
* No need to check for nil
* First approach to diff
* Refactor things for better readability and testing
* Remove draft comment for algorithm
* Format things a bit better
* Remove unused and simplify code a bit
* Refactor for readability and testing
* Add changes file
* Implement new approach based on review comments
* Make sure to only delete from the current host
* Add single uninstall test and fix code
* Improve code based on review
* Refactor error handling for better extensibility and add more scaffolding for specific db errors
* Add integration tests to check errors from mysql are translated properly
* Address review comments
* Add changes file
* Remove username from UI code
* Remove username from tests
* Remove username from database
* Modify server endpoints for removing username
* Implement backend aspects of removing username
* Update API docs
* Add name to fleetctl
- Add enable_analytics column to database.
- Allow enable_analytics to be set via API.
- Add messaging in fleetctl setup.
Note that this defaults to off for existing installations, and defaults
on for newly set up installs.
No collection or sending of analytics yet exists, we are strictly
storing the preference at this time.
Part of #454
This may help with deadlocks on the `label_membership` table. It is not
clear from MySQL documentation whether the order of the records is
significant for locking within a single query. If it is, this should
help the problem. If it is not, this should have no negative impact.
May fix#1146
This should support Redis in both cluster and non-cluster modes.
Updates were made separately to github.com/throttled/throttled to support the slight changes in types.
Co-authored-by: Joseph Macaulay <joseph.macaulay@uber.com>
Co-authored-by: Zach Wasserman <zach@fleetdm.com>
* #511 refactored update options - new params & ts
* updated server to include agent_options for read and update
* added agent options form to org settings
* #511 finished connecting agent form to server
* #511 fixing api to save/read agent options
* #511 linted
* #511 fixed reading & updating agent options
* #511 api fixes to support agent options
* #511 removed log
* Fix json.RawMessage pointers in tests
Co-authored-by: Zach Wasserman <zach@fleetdm.com>
Reorder migrations from the long-running `teams` branch to ensure that
they can run successfully for deployments upgrading from a pre-4.0
release.
All migrations from the `teams` branch are reordered to take place
_after_ all migrations from the `main` branch, using `20210601` as the
new date, after the latest released `main` branch migration on `20210526`.
Fixes#1058
- Update links to documentation
- Update link to assets by editing `getAssetURL()`
- Prettier did some reformatting of the email template code which is resulting in a larger amount of lines changed
- Add TeamFilter to relevant host and label methods.
- Pass appropriate filter in service methods.
The dashboard should now show the appropriate hosts for a user's team membership.
- Add policy.rego file defining authorization policies.
- Add Go integrations to evaluate Rego policies (via OPA).
- Add middleware to ensure requests without authorization check are rejected (guard against programmer error).
- Add authorization checks to most service endpoints.
- Move team-related service methods to `ee/server/service`.
- Instantiate different service on startup based on license key.
- Refactor service errors into separate package.
- Add support for running E2E tests in both Core and Basic tiers.
- Add `team_id` field to secrets.
- Remove secret `name` and `active` fields (migration deletes inactive secrets).
- Assign hosts to Team based on secret provided.
- Add API for retrieving secrets by Team.
- Accept Teams as a searchable target type for the target selection API.
- Accept Teams for targets in running live queries.
- Refactoring to support these changes.
- Update API documentation.
- Move host `additional` into a separate table.
- Join when that data is needed.
- API change: `/api/v1/fleet/hosts` now returns only the requested
`additional` columns, unless `*` is provided as the sole argument.
Background:
A customer reported that MySQL binlogs grew huge and replication lag
went way up when data was stored in the `additional` column. In this
deployment MySQL was running with ROW replication. This would cause the
entire `additional` data to be copied on each update of the host checkin
time. While switching to STATEMENT or MIXED replication would likely
mitigate the issue, this was not an option in their environment.
Some datastore and service methods would return slices of structs,
rather than slices to pointers of structs (which most methods used).
Make this more consistent.
- Include only hosts that the user has access to in search targets API.
- Add parameter to specify whether `observer` hosts should be included.
- Generate counts based on which hosts user can access.
- Update API doc.
- Add config option for license key.
- Define license details data structure.
- Include license details in app config API responses.
Currently any non-empty value for `--license_key` behaves as though the
installation is licensed for `basic`. If the license key is empty,
`core` is returned.
Still to come is the appropriate parsing for the license key.
