The api/v1/fleet/queries/run_by_names endpoint was incorrectly documented as discussed in #409. This PR includes the fix.
- Add the query parameter to the queries/run_by_names endpoint and edit the example requests to include this parameter.
This PR includes two documentation fixes.
1. Remove the support property from the osquery queries spec in the fleetctl docs. The support property is not a feature of osquery or Fleet.
2. Edit the api/v1/fleet/queries/run_by_names with accurate examples. This endpoint sends query targets (hosts and labels) by name (hostname and label name). Contrast this with the api/v1/fleet/queries/run which sends query targets by id (host id and label id)
Resolves#409
This PR is part of the Complete documentation for Fleet API project #43.
The endpoint included in these changes:
- POST /api/v1/fleet/reset_password
- GET /api/v1/fleet/sessions/{id}
- DELETE /api/v1/fleet/sessions/{id}
- POST /api/v1/fleet/queries/delete
- GET /api/v1/fleet/email/change/{token}
The enrollment cooldown period was sometimes causing problems when
osquery (probably unintentionally, see
https://github.com/osquery/osquery/issues/6993) tried to enroll more
than once from the same osqueryd process.
We now set this to default to off and make it configurable. With #417
this feature may be unnecessary for most deployments.
Osquery now exposes more information during host enrollment than Fleet
previously handled. We can use this to provide more options to users in
problematic enrollment scenarios.
Users can configure --osquery_host_identifier in Fleet to set which
identifier is used to determine uniqueness of hosts. The
default (provided) replicates existing behavior in Fleet. For many
users, setting this to instance will provide better enrollment
stability.
Closes#373
Add documentation on the websocket endpoints in the Fleet API. These endpoints allow users to retrieve live query results.
Endpoints added include:
- `api/v1/fleet/results`
- `api/v1/fleet/results/websockets`
- Fix api/v1/fleet/queries/run endpoint. Prior the fix, the endpoint was incorrectly documented as api/v1/fleet/spec/queries/run
- Fix api/v1/fleet/queries/run_by_names endpoint. Prior the fix, the endpoint was incorrectly documented as api/v1/fleet/queries/run
The endpoint included in these changes:
- /api/v1/kolide/carves GET
- /api/v1/kolide/carves/{id} GET
- /api/v1/kolide/users/{id} PATCH
- /api/v1/kolide/users/{id}/enable POST
- /api/v1/kolide/users/{id}/admin POST
- /api/v1/kolide/users/{id}/require_password_reset POST
- /api/v1/kolide/users/{id}/sessions GET
- /api/v1/kolide/users/{id}/sessions DELETE
- Add query parameter to the /api/v1/fleet/hosts endpoint and edit the example request and response.
- Add query parameter to the /api/v1/fleet/labels/{id}/hosts endpoint
These changes support the ability to perform a search on the Hosts table by hostname, machine_serial, and ipv4.
This PR is part of the Complete documentation for Fleet API project #43.
The endpoint included in these changes:
- `POST api/v1/fleet/targets`
Additional changes:
- Now order_key and order_direction are included in the parameters table for endpoints that support them. Fixes#326
- Change endpoints to `api/v1/fleet` naming
This PR is part of the Complete documentation for Fleet API project #43.
The endpoints included in these changes:
- /labels POST
- /labels/{id} PATCH
- /labels/{id} GET
- /labels GET
- /labels/{id}/hosts GET
- /labels/{name} DELETE
- /labels/id/{id} DELETE
- /spec/labels POST
- /spec/labels GET
- /spec/labels/{name} GET
- Support both /api/v1/fleet and /api/v1/kolide routes in server.
- Add logging for use of deprecated routes.
- Rename routes in frontend JS.
- Rename routes and add notes in documentation.
In #212 these settings were updated and caused connectivity issues for
users in common environment configurations. The new changes are
aggressive (modern enforces TLS 1.3) and Mozilla indicates that
intermediate is an appropriate default. This will ensure better
compatibility for common deployments while still allowing the option to
use the strictest settings.
Document unintentional mismatched yaml key.
Fixes#269
- Add --dev flag that will set default flag values. This simplifies the
invocation of Fleet in a development environment.
- Change defaults in docker-compose to use `fleet` in place of `kolide`.
- Skip prompt in `prepare db` when `--dev` specified.
- Update developer documentation.
Updates to MySQL configuration in docker-compose.yml may require
existing development containers and volumes to be deleted (this will
delete data in MySQL):
```shell
docker-compose rm -sf
docker volume rm fleet_mysql-persistent-volume
```
Closes#170
Reformat the anchor links for specific endpoints by moving these links to their respective sections.
The endpoints included in these changes:
- /api/v1/kolide/packs POST
- /api/v1/kolide/packs/{id} PATCH "modify_pack"
- /api/v1/kolide/packs/{id} GET "get_pack"
- /api/v1/kolide/packs GET "list_packs"
- /api/v1/kolide/packs/{name} DELETE "delete_pack"
- /api/v1/kolide/packs/id/{id} DELETE "delete_pack_by_id"
- /api/v1/kolide/packs/{id}/scheduled GET "get_scheduled_queries_in_pack"
- /api/v1/kolide/schedule POST "schedule_query"
- /api/v1/kolide/schedule/{id} GET "get_scheduled_query"
- /api/v1/kolide/schedule/{id} PATCH "modify_scheduled_query"
- /api/v1/kolide/schedule/{id} DELETE "delete_scheduled_query"
- /api/v1/kolide/spec/packs POST "apply_pack_specs"
- /api/v1/kolide/spec/packs GET "get_pack_specs"
- /api/v1/kolide/spec/packs/{name} GET "get_pack_spec"
The endpoints included in these changes:
- /api/v1/kolide/queries/{id} GET
- /api/v1/kolide/queries/{id} PATCH
- /api/v1/kolide/queries/{name} DELETE
- /api/v1/kolide/queries GET
- /api/v1/kolide/queries POST
- /api/v1/kolide/queries/id/{id} DELETE
- /api/v1/kolide/spec/queries POST
- /api/v1/kolide/spec/queries GET
- /api/v1/kolide/spec/queries/{name} GET
- /api/v1/kolide/queries/run POST
- /api/v1/kolide/queries/run_by_names POST
The endpoints included in these changes:
- /api/v1/kolide/host_summary GET
- /api/v1/kolide/hosts/{id} GET
- /api/v1/kolide/hosts/identifier/{identifier} GET
- /api/v1/kolide/hosts/{id} DELETE
- /api/v1/kolide/spec/osquery_options POST
- /api/v1/kolide/spec/osquery_options GET
Mozilla's recommended settings have changed since this was last updated.
We now link directly to the relevant revision in the Mozilla wiki to
avoid confusion if this is updated.
This PR is part of the Complete documentation for Fleet API project #43.
The endpoints included in these changes:
- /api/v1/kolide/config/certificate GET
- /api/v1/kolide/config GET
- /api/v1/kolide/config PATCH
- /api/v1/kolide/spec/enroll_secret POST
- /api/v1/kolide/spec/enroll_secret GET
- /api/v1/kolide/invites POST
- /api/v1/kolide/invites GET
- /api/v1/kolide/invites/{id} DELETE
- /api/v1/kolide/invites/{token} GET
- Quick attempt to alleviate potential confusion for how the osquery configuration overrides option works
- Leave in the commented explanation in the sample yaml file
- Split the array of objects to separate objects for each pattern in ignorePatterns. Rename the key in each object to "pattern." The documentation for the example config file is poor.
- Add hello@fleetdm.com and /server/datastore/mysql/migrations/ to ignorePatterns
- Add 999 to aliveStatucCodes (Seems to be a LinkedIn edge case)
- Fix all broken markdown links
The current implementation of FleetDM doesn't support Docker secrets for supplying the MySQL password and JWT key. This PR provides the ability for a file path to read in secrets. The goal of this PR is to avoid storing secrets in a static config or in an environment variable.
Example config for Docker:
```yaml
mysql:
address: mysql:3306
database: fleet
username: fleet
password_path: /run/secrets/mysql-fleetdm-password
redis:
address: redis:6379
server:
address: 0.0.0.0:8080
cert: /run/secrets/fleetdm-tls-cert
key: /run/secrets/fleetdm-tls-key
auth:
jwt_key_path: /run/secrets/fleetdm-jwt-key
filesystem:
status_log_file: /var/log/osquery/status.log
result_log_file: /var/log/osquery/result.log
enable_log_rotation: true
logging:
json: true
```
This PR includes the Using Fleet section of the documentation restructure #144.
It shouldn't be merged until changes are approved for the entire restructuring (part 1, part 2, and part 3).
Update the naming convention for the files to number prefixes.
This PR includes the Deployment section of the documentation restructure #144.
- Changes include the addition of 3 overarching folders/sections named Deployment, Using fleet, and Contribution guide. Each folder includes new files for the subsections.
- Move the copy from the appropriate files into the new files in the Deployment folder. Removed old files. A detailed outline of the exact old files can be found in the linked issue above.
- Add navigation to the top of each file via links to anchors in the markdown.
The naming convention for each folder and file is up for discussion. I chose to use number prefixes (1, 2, 3) to order the folders and files. This way we have control over the order when viewing on GitHub.
- Add help text within dropdown in smaller font size underneath "Require password reset" saying "This will revoke all active Fleet API tokens for this user."
- Update API docs to use "API token" parlance instead of "Auth token"
This adds the option to set up an S3 bucket as the storage backend for file carving (partially solving #111).
It works by using the multipart upload capabilities of S3 to maintain compatibility with the "upload in blocks" protocol that osquery uses. It does this basically replacing the carve_blocks table while still maintaining the metadata in the original place (it would probably be possible to rely completely on S3 by using object tagging at the cost of listing performance). To make this pluggable, I created a new field in the service struct dedicated to the CarveStore which, if no configuration for S3 is set up will be just a reference to the standard datastore, otherwise it will point to the S3 one (effectively this separation will allow in the future to add more backends).
@ahmedmusaad added documentation for the following API endpoints:
/api/v1/kolide/users GET
/api/v1/kolide/users POST
/api/v1/kolide/users/admin POST
/api/v1/kolide/users/{id} GET
Endpoints that have been added:
- /api/v1/kolide/login POST
- /api/v1/kolide/logout POST
- /api/v1/kolide/forgot_password POST
- /api/v1/kolide/me GET
- /api/v1/kolide/change_password POST
- /api/v1/kolide/perform_required_password_reset POST
- /api/v1/kolide/sso POST
- /api/v1/kolide/sso GET
Added walkthrough for users attempting to connect to MailHog server.
Connecting to the simulated mail server allows contributors to interact with features in Fleet UI that require email configuration.
Added link to development-infrastructure docs at the end of building-the-code docs to walk contributors to the next step of serving Fleet locally.
Added --auth_jwt_key="insecure" flag to fleet serve command in development-infrastructure docs.
$ character is now removed from all shell commands in markdown documentation.
When docs are eventually compiling to we can add $ automatically to code blocks.
My best attempt at steps to solve the common database connection error users receive when installing or updating Fleet.
Inserted 'Why am I receiving a database connection error when attempting to "prepare" the database?'.
Also simplified the answer to "Is Fleet available as a SaaS product?".
- Add endpoints for osquery to register and continue a carve.
- Implement client functionality for retrieving carve details and contents in fleetctl.
- Add documentation on using file carving with Fleet.
Addresses kolide/fleet#1714
* Perform migration to delete any entries with `deleted` set, and
subsequently drop columns `deleted` and `deleted_at`.
* Remove `deleted` and `deleted_at` references.
Closes#2146
The example in the README won't follow the github redirect without `-L`, so the example as is will download an html file instead of the actual `fleet.zip`. This fixes that.
"Manual" labels can be specified by hostname, allowing users to specify
the membership of a label without having to use a dynamic query. See the
included documentation.
Additional information is collected when host details are updated using
the queries specified in the Fleet configuration. This additional
information is then available in the host API responses.
- Add the server_url_prefix flag for configuring this functionality
- Add prefix handling to the server routes
- Refactor JS to use appropriate paths from modules
- Use JS template to get URL prefix into JS environment
- Update webpack config to support prefixing
Thanks to securityonion.net for sponsoring the development of this feature.
Closes#1661
Adds Google Cloud PubSub logging for status and results.
This also changes the Write interface for logging modules to add a context.Context (only used by pubsub currently).
- Refactor configuration for logging to use separate plugins
- Move existing filesystem logging to filesystem plugin
- Create new AWS firehose plugin
- Update documentation around logging
Almost two years ago, we began referring to the project as Fleet, but there are
many occurences of the term "Kolide" throughout the UI and documentation. This
PR attempts to clear up those uses where it is easily achievable.
The term "Kolide" is used throughout the code as well, but modifying this would
be more likely to introduce bugs.
Useful for SAML login users who cannot log in with `fleetctl login`. Instead
they can pull their session token from the UI and configure the fleetctl client
to use it.
Closes#1865
Individuals unaccustomed to building Go projects from source may encounter difficulties cloning into the repo if they choose an arbitrary directory not in ~/go/src. Attempting to build the repo elsewhere will result in the `make deps` command failing. This change adds workspace prep instructions.
-Updated cp to reflect addition of fleetctl to archive
-Updated MySQL setup to reflect setting root password when installed from .rpm and taking into account password validation requirement defaults.
-Updated echo command with sudo tee due to permission denied
- Updat binary names to reflect binaries in latest archive
- Add line to 'sudo cp fleet/linux/fleetctl /usr/bin/fleetctl
- Change echo string for enroll_secret to use sudo tee due to permission denied
- Revert erroneous changes in 6442736c.
- Remove Osquery prefix from entity kinds.
- Define Query first in combination with Label and Decorator definitions.
Add a gRPC server that will interact with osquery through Launcher. This endpoint will expose the osquery configuration suitable for use via the Launcher plugin, and collect log and query results.
- Add SSH configuration to allow checkout of Kolide private repos in CI
- Add kolide/agent-api repo to glide.yaml
- Update testify version to fix broken test build
Closes#1545
Closes issue #1456 This PR adds a single sign on option to the login form, exposes single sign on to the end user, and allows an admin user to set single sign on configuration options.
This PR partially addresses #1456, providing SSO SAML support. The flow of the code is as follows.
A Kolide user attempts to access a protected resource and is directed to log in.
If SSO identity providers (IDP) have been configured by an admin, the user is presented with SSO log in.
The user selects SSO, which invokes a call the InitiateSSO passing the URL of the protected resource that the user was originally trying access. Kolide server loads the IDP metadata and caches it along with the URL. We then build an auth request URL for the IDP which is returned to the front end.
The IDP calls the server, invoking CallbackSSO with the auth response.
We extract the original request id from the response and use it to fetch the cached metadata and the URL. We check the signature of the response, and validate the timestamps. If everything passes we get the user id from the IDP response and use it to create a login session. We then build a page which executes some javascript that will write the token to web local storage, and redirect to the original URL.
I've created a test web page in tools/app/authtest.html that can be used to test and debug new IDP's which also illustrates how a front end would interact with the IDP and the server. This page can be loaded by starting Kolide with the environment variable KOLIDE_TEST_PAGE_PATH to the full path of the page and then accessed at https://localhost:8080/test
This PR contains a fix for a bug that turned up when I was testing configimporter. If the platform field is not specified, its supposed to default to all per the osquery configuration spec. The default was not properly implemented, and if the platform value was missing from the imported configuration it failed. The PR also added instructions to the api documentation describing how to import an osquery configuration.
If server is started without a JWT key, a message like the following is printed:
```
################################################################################
# ERROR:
# A value must be supplied for --auth_jwt_key. This value is used to create
# session tokens for users.
#
# Consider using the following randomly generated key:
# om3w95gMA2drT5xAdLd2Q5oE8fLw+Miz
################################################################################
```
Closes#1480.
Made log rotation for osquery results and status logs optional. This required writing the logwriter package which is a drop in replacement for lumberjack. We still use lumberjack if the log rotation flag --osquery_enable_log_rotation flag is set. Note that the performance of the default is quite a bit better than lumberjack.
BenchmarkLogger-8 2000000 747 ns/op
BenchmarkLumberjack-8 1000000 1965 ns/op
PASS
BenchmarkLogger-8 2000000 731 ns/op
BenchmarkLumberjack-8 1000000 2040 ns/op
PASS
BenchmarkLogger-8 2000000 741 ns/op
BenchmarkLumberjack-8 1000000 1970 ns/op
PASS
BenchmarkLogger-8 2000000 737 ns/op
BenchmarkLumberjack-8 1000000 1930 ns/op
PASS
When `kolide serve --debug` is used, additional handlers will be started to
provide access to profiling tools. These endpoints are authenticated with a
randomly generated token that is printed to the Kolide logs at startup. The
profiling tools are not intended for general use, but they may be useful when
providing performance-related bug reports to the Kolide developers.
Use the [SockJS Protocol](https://github.com/sockjs/sockjs-protocol) to handle
bidirectional communication instead of plain websockets. This allows
distributed queries to function in situations in which they previously failed
(Load balancers not supporting websockets, issues with Safari and self-signed
certs, etc.).
Also includes fixes to the JS message handling logic where slightly different
message delivery semantics (when using XHR) were exposing bugs.
Fixes#1241, #1327.
I think these were useful when none of us knew how to use glide except
@groob and needed copy-pasteable commands to run, but this seems like
it's much less useful now.
Adds a `make lint-license` command that will crawl through the Golang and
Javascript dependencies, trying to automatically determine the license of each
dependency. If any dependencies have incompatible licensing (or if the
automatic detection fails), the tool will print an error and exit with a
non-zero status code. After a successful run, the
docs/third-party/dependencies.md file will be updated with attribution
information as determined by the crawl.
The configuration file has been bootstrapped with the manual analysis needed for
all of the existing dependencies.
* add graceful server shutdown
Also refactors server to use `http.Server` instead of the global
http.ListenAndServe method. In the future it allows us to set
ratelimits and timeouts for http connections.
