mirror of
https://github.com/empayre/fleet.git
synced 2024-11-06 08:55:24 +00:00
Part 3 of documentation restructure. Contribution section. (#149)
This PR includes the Contribution section of the documentation restructure #144.
This commit is contained in:
noahtalerman
GitHub
parent
f9eae5e747
commit
d5f3a70152
@ -1,93 +0,0 @@
|
||||
# Monitoring Fleet
|
||||
- [Health checks](#health-checks)
|
||||
- [Metrics](#metrics)
|
||||
- [Alerting](#alerting)
|
||||
- [Graphing](#graphing)
|
||||
- [Fleet server performance](#fleet-server-performance)
|
||||
- [Horizontal scaling](#horizontal-scaling)
|
||||
- [Availability](#availability)
|
||||
- [Monitoring](#monitoring)
|
||||
- [Debugging performance issues](#debugging-performance-issues)
|
||||
- [MySQL & Redis](#mysql-&-redis)
|
||||
- [Fleet server](#fleet-server)
|
||||
|
||||
## Health checks
|
||||
|
||||
Fleet exposes a basic health check at the `/healthz` endpoint. This is the interface to use for simple monitoring and load-balancer health checks.
|
||||
|
||||
The `/healthz` endpoint will return an `HTTP 200` status if the server is running and has healthy connections to MySQL and Redis. If there are any problems, the endpoint will return an `HTTP 500` status.
|
||||
|
||||
## Metrics
|
||||
|
||||
Fleet exposes server metrics in a format compatible with [Prometheus](https://prometheus.io/). A simple example Prometheus configuration is available in [tools/app/prometheus.yml](/tools/app/prometheus.yml).
|
||||
|
||||
Prometheus can be configured to use a wide range of service discovery mechanisms within AWS, GCP, Azure, Kubernetes, and more. See the Prometheus [configuration documentation](https://prometheus.io/docs/prometheus/latest/configuration/configuration/) for more information on configuring the
|
||||
|
||||
### Alerting
|
||||
|
||||
Prometheus has built-in support for alerting through [Alertmanager](https://prometheus.io/docs/alerting/latest/overview/).
|
||||
|
||||
Consider building alerts for
|
||||
|
||||
- Changes from expected levels of host enrollment
|
||||
- Increased latency on HTTP endpoints
|
||||
- Increased error levels on HTTP endpoints
|
||||
|
||||
```
|
||||
TODO (Seeking Contributors)
|
||||
Add example alerting configurations
|
||||
```
|
||||
|
||||
### Graphing
|
||||
|
||||
Prometheus provides basic graphing capabilities, and integrates tightly with [Grafana](https://prometheus.io/docs/visualization/grafana/) for sophisticated visualizations.
|
||||
|
||||
## Fleet server performance
|
||||
|
||||
Fleet is designed to scale to hundreds of thousands of online hosts. The Fleet server scales horizontally to support higher load.
|
||||
|
||||
### Horizontal scaling
|
||||
|
||||
Scaling Fleet horizontally is as simple as running more Fleet server processes connected to the same MySQL and Redis backing stores. Typically, operators front Fleet server nodes with a load balancer that will distribute requests to the servers. All APIs in Fleet are designed to work in this arrangement by simply configuring clients to connect to the load balancer.
|
||||
|
||||
### Availability
|
||||
|
||||
The Fleet/osquery system is resilient to loss of availability. Osquery agents will continue executing the existing configuration and buffering result logs during downtime due to lack of network connectivity, server maintenance, or any other reason. Buffering in osquery can be configured with the `--buffered_log_max` flag.
|
||||
|
||||
Note that short downtimes are expected during [Fleet server upgrades](./(g)-Updating.md)-fleet.md) that require database migrations.
|
||||
|
||||
### Monitoring
|
||||
|
||||
More information on monitoring Fleet servers with Prometheus and other tools is available in the [Monitoring Fleet](./(e)-Monitoring-Fleet.md) documentation.
|
||||
|
||||
### Debugging performance issues
|
||||
|
||||
#### MySQL & Redis
|
||||
|
||||
If performance issues are encountered with the MySQL and Redis servers, use the extensive resources available online to optimize and understand these problems. Please [file an issue](https://github.com/fleetdm/fleet/issues/new/choose) with details about the problem so that Fleet developers can work to fix them.
|
||||
|
||||
#### Fleet server
|
||||
|
||||
For performance issues in the Fleet server process, please [file an issue](https://github.com/fleetdm/fleet/issues/new/choose) with details about the scenario, and attach a debug archive. Debug archives can also be submitted confidentially through other support channels.
|
||||
|
||||
##### Generate debug archive (Fleet 3.4.0+)
|
||||
|
||||
Use the `fleetctl archive` command to generate an archive of Fleet's full suite of debug profiles. See the [fleetctl setup guide](./(b)-fleetctl-CLI.md)) for details on configuring `fleetctl`.
|
||||
|
||||
The generated `.tar.gz` archive will be available in the current directory.
|
||||
|
||||
###### Targeting individual servers
|
||||
|
||||
In most configurations, the `fleetctl` client is configured to make requests to a load balancer that will proxy the requests to each server instance. This can be problematic when trying to debug a performance issue on a specific server. To target an individual server, create a new `fleetctl` context that uses the direct address of the server.
|
||||
|
||||
For example:
|
||||
|
||||
```sh
|
||||
fleetctl config set --context server-a --address https://server-a:8080
|
||||
fleetctl login --context server-a
|
||||
fleetctl debug archive --context server-a
|
||||
```
|
||||
|
||||
###### Confidential information
|
||||
|
||||
The `fleetctl archive` command retrieves information generated by Go's [`net/http/pprof`](https://golang.org/pkg/net/http/pprof/) package. In most scenarios this should not include sensitive information, however it does include command line arguments to the Fleet server. If the Fleet server receives sensitive credentials via CLI argument (not environment variables or config file), this information should be scrubbed from the archive in the `cmdline` file.
|
||||
@ -0,0 +1,210 @@
|
||||
# Building Fleet
|
||||
- [Building the code](#building-the-code)
|
||||
- [Generating the packaged JavaScript](#generating-the-packaged-javascript)
|
||||
- [Compiling the Fleet binary](#compiling-the-Fleet-binary)
|
||||
- [Development infrastructure](#development-infrastructure)
|
||||
- [Starting the local development environment](#starting-the-local-development-environment)
|
||||
- [Running Fleet using Docker development infrastructure](#running-fleet-using-docker-development-infrastructure)
|
||||
- [Setting up a Linux Development Environment](#setting-up-a-linux-development-environment)
|
||||
- [Database migrations](#database-migrations)
|
||||
- [Adding/Updating tables](#adding/updating-tables)
|
||||
- [Populating the database with default data](#populating-the-database-with-default-data)
|
||||
|
||||
## Building the code
|
||||
|
||||
Clone this repository.
|
||||
|
||||
To setup a working local development environment, you must install the following minimum toolset:
|
||||
|
||||
* [Go](https://golang.org/dl/) (1.9 or greater)
|
||||
* [Node.js](https://nodejs.org/en/download/current/) and [Yarn](https://yarnpkg.com/en/docs/install)
|
||||
* [GNU Make](https://www.gnu.org/software/make/) (probably already installed if you're on macOS/Linux)
|
||||
* [Docker](https://www.docker.com/products/overview#/install_the_platform)
|
||||
|
||||
> #### New to the Go language?
|
||||
>
|
||||
> After installing Go, your $GOPATH will probably need a little freshening up. To take care of this automatically every time a new terminal is opened, add this to your shell startup script (`~/.profile`):
|
||||
> ```bash
|
||||
> # Allow go-bindata and other Go stuff to work properly (e.g. for Fleet/osquery)
|
||||
> # More info: https://golang.org/doc/gopath_code.html#GOPATH
|
||||
> export PATH=$PATH:$(go env GOPATH)/bin
|
||||
> ```
|
||||
|
||||
Once you have those minimum requirements, you will need to install Fleet's dependencies. To do this, run the following from the root of the repository:
|
||||
|
||||
```
|
||||
make deps
|
||||
```
|
||||
|
||||
When pulling in new revisions to your working source tree, it may be necessary to re-run `make deps` if a new Go or JavaScript dependency was added.
|
||||
|
||||
### Generating the packaged JavaScript
|
||||
|
||||
To generate all necessary code (bundling JavaScript into Go, etc), run the following:
|
||||
|
||||
```
|
||||
make generate
|
||||
```
|
||||
|
||||
#### Automatic rebuilding of the JavaScript bundle
|
||||
|
||||
Normally, `make generate` takes the JavaScript code, bundles it into a single bundle via Webpack, and inlines that bundle into a generated Go source file so that all of the frontend code can be statically compiled into the binary. When you build the code after running `make generate`, all of that JavaScript is included in the binary.
|
||||
|
||||
This makes deploying Fleet a dream, since you only have to worry about a single static binary. If you are working on frontend code, it is likely that you don't want to have to manually re-run `make generate` and `make build` every time you edit JavaScript and CSS in order to see your changes in the browser. To solve this problem, before you build the Fleet binary, run the following command instead of `make generate`:
|
||||
|
||||
```
|
||||
make generate-dev
|
||||
```
|
||||
|
||||
Instead of reading the JavaScript from a inlined static bundle compiled within the binary, `make generate-dev` will generate a Go source file which reads the frontend code from disk and run Webpack in "watch mode".
|
||||
|
||||
Note that when you run `make generate-dev`, Webpack will be watching the JavaScript files that were used to generate the bundle, so the process will be long lived. Depending on your personal workflow, you might want to run this in a background terminal window.
|
||||
|
||||
After you run `make generate-dev`, run `make build` to build the binary, launch the binary and you'll be able to refresh the browser whenever you edit and save frontend code.
|
||||
|
||||
### Compiling the Fleet binary
|
||||
|
||||
Use `go build` to build the application code. For your convenience, a make command is included which builds the code:
|
||||
|
||||
```
|
||||
make build
|
||||
```
|
||||
|
||||
It's not necessary to use Make to build the code, but using Make allows us to account for cross-platform differences more effectively than the `go build` tool when writing automated tooling. Use whichever you prefer.
|
||||
|
||||
## Development infrastructure
|
||||
|
||||
### Starting the local development environment
|
||||
|
||||
To set up a canonical development environment via docker, run the following from the root of the repository:
|
||||
|
||||
```
|
||||
docker-compose up
|
||||
```
|
||||
|
||||
This requires that you have docker installed. At this point in time, automatic configuration tools are not included with this project.
|
||||
|
||||
##### Stopping the local development environment
|
||||
|
||||
If you'd like to shut down the virtual infrastructure created by docker, run the following from the root of the repository:
|
||||
|
||||
```
|
||||
docker-compose down
|
||||
```
|
||||
|
||||
##### Setting up the database tables
|
||||
|
||||
Once you `docker-compose up` and are running the databases, you can build the code and run the following command to create the database tables:
|
||||
|
||||
```
|
||||
fleet prepare db
|
||||
```
|
||||
|
||||
### Running Fleet using Docker development infrastructure
|
||||
|
||||
To start the Fleet server backed by the Docker development infrastructure, run the Fleet binary as follows:
|
||||
|
||||
```
|
||||
fleet serve --auth_jwt_key="insecure"
|
||||
```
|
||||
|
||||
By default, Fleet will try to connect to servers running on default ports on localhost. Depending on your browser's settings, you may have to click through a security warning.
|
||||
|
||||
If you're using Docker via [Docker Toolbox](https://www.docker.com/products/docker-toolbox), you may have to modify the default values use the output of `docker-machine ip` instead of `localhost`. There is an example configuration file included in this repository to make this process easier for you. Use the `--config` flag of the Fleet binary to specify the path to your config. See `fleet --help` for more options.
|
||||
|
||||
## Setting up a Linux Development Environment
|
||||
|
||||
#### Install some dependencies
|
||||
|
||||
`sudo apt-get install xzip gyp libjs-underscore libuv1-dev dep11-tools deps-tools-cli`
|
||||
|
||||
#### Create a temp directory, download and place the `node` and `golang` bins
|
||||
|
||||
```
|
||||
mkdir tmp
|
||||
cd tmp
|
||||
```
|
||||
|
||||
#### install `node` and `yarn`
|
||||
|
||||
```
|
||||
wget https://nodejs.org/dist/v9.4.0/node-v9.4.0-linux-x64.tar.xz
|
||||
xz -d node-v9.4.0-linux-x64.tar.xz
|
||||
tar -xf node-v9.4.0-linux-x64.tar
|
||||
sudo cp -rf node-v9.4.0-linux-x64/bin /usr/local/
|
||||
sudo cp -rf node-v9.4.0-linux-x64/include /usr/local
|
||||
sudo cp -rf node-v9.4.0-linux-x64/lib /usr/local
|
||||
sudo cp -rf node-v9.4.0-linux-x64/share /usr/local
|
||||
npm install -g yarn
|
||||
```
|
||||
|
||||
#### install `go`
|
||||
|
||||
```
|
||||
wget https://dl.google.com/go/go1.9.3.linux-amd64.tar.gz
|
||||
sudo tar -C /usr/local -xzf go1.9.3.linux-amd64.tar.gz
|
||||
export PATH=$PATH:/usr/local/go/bin:~/go/bin/
|
||||
```
|
||||
|
||||
#### clean-up temp directory
|
||||
|
||||
```
|
||||
cd ..
|
||||
rm -rf tmp
|
||||
```
|
||||
|
||||
#### Clone and build depenencies
|
||||
|
||||
```
|
||||
git clone https://github.com/fleetdm/fleet.git
|
||||
cd fleet
|
||||
make deps
|
||||
make generate
|
||||
make build
|
||||
sudo cp build/fleet /usr/bin/fleet
|
||||
```
|
||||
|
||||
## Database migrations
|
||||
|
||||
### Adding/Updating tables
|
||||
|
||||
Database schemas are managed by a series of migrations defined in go code. We use a customized version of the Goose migrations tool to handle these migrations.
|
||||
|
||||
Note: Once committed to the Fleet repo, table migrations should be considered immutable. Any changes to an existing table should take place in a new migration executing ALTERs.
|
||||
|
||||
From the project root run the following shell commands:
|
||||
|
||||
``` bash
|
||||
go get github.com/kolide/goose
|
||||
cd server/datastore/mysql/migrations/tables
|
||||
goose create AddColumnFooToUsers
|
||||
```
|
||||
|
||||
Find the file you created in the migrations directory and edit it:
|
||||
|
||||
* delete the import line for goose: `github.com/pressly/goose`
|
||||
* change `goose.AddMigration(...)` to `MigrationClient.AddMigration(...)`
|
||||
* add your migration code
|
||||
|
||||
You can then update the database by running the following shell commands:
|
||||
|
||||
``` bash
|
||||
make build
|
||||
build/fleet prepare db
|
||||
```
|
||||
|
||||
### Populating the database with default data
|
||||
|
||||
Populating built in data is also performed through migrations. All table migrations are performed before any data migrations.
|
||||
|
||||
Note: Data migrations can be mutable. If tables are altered in a way that would render a data migration invalid (columns changed/removed), data migrations should be updated to comply with the new schema. Data migrations will not be re-run when they have already been run against a database, but they must be updated to maintain compatibility with a fresh DB.
|
||||
|
||||
From the project root run the following shell commands:
|
||||
|
||||
``` bash
|
||||
go get github.com/kolide/goose
|
||||
cd server/datastore/mysql/migrations/data
|
||||
goose create PopulateFoo
|
||||
```
|
||||
|
||||
Proceed as for table migrations, editing and running the newly created migration file.
|
||||
@ -0,0 +1,112 @@
|
||||
# Testing
|
||||
- [Full test suite](#full-test-suite)
|
||||
- [Database tests](#database-tests)
|
||||
- [Email tests](#email-tests)
|
||||
- [Integration tests](#integration-tests)
|
||||
- [Email](#email)
|
||||
|
||||
## Full test suite
|
||||
|
||||
To execute all of the tests that CI will execute, run the following from the root of the repository:
|
||||
|
||||
```
|
||||
make test
|
||||
```
|
||||
|
||||
It is a good idea to run `make test` before submitting a Pull Request.
|
||||
|
||||
#### Go unit tests
|
||||
|
||||
To run all Go unit tests, run the following:
|
||||
|
||||
```
|
||||
make test-go
|
||||
```
|
||||
|
||||
#### JavaScript unit tests
|
||||
|
||||
To run all JavaScript unit tests, run the following:
|
||||
|
||||
```
|
||||
make test-js
|
||||
```
|
||||
|
||||
#### Go linters
|
||||
|
||||
To run all Go linters and static analyzers, run the following:
|
||||
|
||||
```
|
||||
make lint-go
|
||||
```
|
||||
|
||||
### Database tests
|
||||
|
||||
To run database tests set environment variables as follows.
|
||||
|
||||
```
|
||||
export MYSQL_PORT_3306_TCP_ADDR=192.168.99.100
|
||||
export MYSQL_TEST=1
|
||||
```
|
||||
|
||||
### Email tests
|
||||
|
||||
To run email related unit tests using MailHog set the following environment
|
||||
variable.
|
||||
|
||||
```
|
||||
export MAIL_TEST=1
|
||||
```
|
||||
|
||||
## Integration tests
|
||||
|
||||
By default, tests that require external dependecies like Mysql or Redis are skipped. The tests can be enabled by setting `MYSQL_TEST=true` and `REDIS_TEST=true` environment variables. MYSQL will try to connect with the following credentials.
|
||||
```
|
||||
user = "kolide"
|
||||
password = "kolide"
|
||||
database = "kolide"
|
||||
host = "127.0.0.1"
|
||||
```
|
||||
Redis tests expect a redis instance at `127.0.0.1:6379`.
|
||||
|
||||
|
||||
Both the Redis and MySQL tests will also be automatically enabled with Docker links. You can check out the CircleCI configuration file(`circle.yml`) for an example of how to use Docker links to run integration tests.
|
||||
#### JavaScript linters
|
||||
|
||||
To run all JavaScript linters and static analyzers, run the following:
|
||||
|
||||
```
|
||||
make lint-js
|
||||
```
|
||||
|
||||
#### Viewing test coverage
|
||||
|
||||
When you run `make test` or `make test-go` from the root of the repository, test coverage reports are generated in every subpackage. For example, the `server` subpackage will have a coverage report generated in `./server/server.cover`
|
||||
|
||||
To explore a test coverage report on a line-by-line basis in the browser, run the following:
|
||||
|
||||
```bash
|
||||
# substitute ./datastore/datastore.cover, etc
|
||||
go tool cover -html=./server/server.cover
|
||||
```
|
||||
|
||||
To view test a test coverage report in a terminal, run the following:
|
||||
|
||||
```bash
|
||||
# substitute ./datastore/datastore.cover, etc
|
||||
go tool cover -func=./server/server.cover
|
||||
```
|
||||
|
||||
### Email
|
||||
|
||||
#### Testing email using MailHog
|
||||
|
||||
To intercept sent emails while running a Fleet development environment, make sure that you've set the SMTP address to `<docker host ip>:1025` and leave the username and password blank. Then, visit `<docker host ip>:8025` in a web browser to view the [MailHog](https://github.com/mailhog/MailHog) UI.
|
||||
|
||||
For example, if docker is running natively on your `localhost`, then your mail settings should look something like:
|
||||
|
||||
```yaml
|
||||
mail:
|
||||
address: localhost:1025
|
||||
```
|
||||
|
||||
`localhost:1025` is the default configuration. You can use `fleet config_dump` to see the values which Fleet is using given your configuration.
|
||||
@ -1,12 +1,20 @@
|
||||
# A Title Which Summarizes The Decision
|
||||
# Architecture decisions
|
||||
|
||||
## Authors
|
||||
Architecturally significant changes to this project are documented as a collection of records. Documents are numbered sequentially and monotonically. If a decision is reversed, we will keep the old one around, but mark it as superseded (it's still relevant to know that it was the decision, but is no longer the decision).
|
||||
|
||||
If you'd like to make a significant change to this program, copy the template below and submit a PR which explains your proposal. Your proposal may accompany a working prototype, but significant decisions should be documented before there is significant development.
|
||||
|
||||
# Template
|
||||
|
||||
## A Title Which Summarizes The Decision
|
||||
|
||||
### Authors
|
||||
|
||||
- Your Name ([@username](https://github.com/marpaia))
|
||||
|
||||
> List all people that have contributed to the decision and can speak authoritatively about some critical aspect it.
|
||||
|
||||
## Status
|
||||
### Status
|
||||
|
||||
Accepted (March 15, 2018)
|
||||
|
||||
@ -15,18 +23,18 @@ Accepted (March 15, 2018)
|
||||
> - Accepted (the decision has been agreed upon and is the current adopted decision)
|
||||
> - Superseded (the decision has been superseded by a newer decision (include a link to the new decision))
|
||||
|
||||
## Context
|
||||
### Context
|
||||
|
||||
The context section outlines the set of conditions which have brought you here. You should outline what it is that you're talking about and why this decision is being documented. Enumerate the context you have which has gone into this decision so that others understand not only the decision itself but why it was the best decision given the context of the situation.
|
||||
|
||||
If significant conversation on this topic happened in Slack or in a meeting, do your best to summarize the context. Include links to Slack logs, GitHub issues, Pull Request discussions, etc.
|
||||
|
||||
## Decision
|
||||
### Decision
|
||||
|
||||
Summarize the decision that has been made and it's immediate practical implications. This section should be brief. Save the story-telling for Context and Consequences.
|
||||
|
||||
|
||||
## Consequences
|
||||
### Consequences
|
||||
|
||||
The closing section of the document explains what the consequences of this decision will be.
|
||||
|
||||
@ -1,5 +1,4 @@
|
||||
Releasing Fleet
|
||||
===============
|
||||
# Releasing Fleet
|
||||
|
||||
1. Update the [CHANGELOG](../../CHANGELOG.md) with the changes that have been made since the last Fleet release.
|
||||
|
||||
@ -1,4 +1,9 @@
|
||||
# Troubleshooting FAQ
|
||||
# Contribution FAQ
|
||||
- [Make errors](#make-errors)
|
||||
- [`dep: command not found`](#dep-command-not-found)
|
||||
- [`undefined: Asset`](#undefined-asset)
|
||||
- [How do I connect to the Mailhog simulated mail server?](#how-do-i-connect-to-the-mailhog-simulated-mail-server?)
|
||||
- [Adding hosts for testing](#adding-hosts-for-testing)
|
||||
|
||||
## Make errors
|
||||
|
||||
@ -19,7 +24,7 @@ server/kolide/emails.go:90:23: undefined: Asset
|
||||
make: *** [fleet] Error 2
|
||||
```
|
||||
|
||||
If you get an `undefined: Asset` error it is likely because you did not run `make generate` before `make build`. See [Building the Code](https://github.com/fleetdm/fleet/blob/master/docs/development/building-the-code.md) for additional documentation on compiling the `fleet` binary.
|
||||
If you get an `undefined: Asset` error it is likely because you did not run `make generate` before `make build`. See [Building Fleet](./1-Building-Fleet.md) for additional documentation on compiling the `fleet` binary.
|
||||
|
||||
## How do I connect to the Mailhog simulated mail server?
|
||||
|
||||
@ -1 +1,16 @@
|
||||
Contribution guide README
|
||||
# Contribution
|
||||
|
||||
### [Building Fleet](./1-Building-Fleet.md)
|
||||
Provides documentation about building the code, development infrastructure, and database migrations
|
||||
|
||||
### [Testing](./2-Testing.md)
|
||||
Includes documentation about Fleet's full test suite and integration tests
|
||||
|
||||
### [Architecture decisions](./3-Architecture-decisions.md)
|
||||
Provides a template for making architecturally significant changes to the project
|
||||
|
||||
### [Releasing Fleet](./4-Releasing-Fleet.md)
|
||||
Provides a guide for Fleet's release process
|
||||
|
||||
### [FAQ](./FAQ.md)
|
||||
Includes commonly asked questions and answers about contributing to Fleet from the Fleet community
|
||||
|
||||
@ -2,8 +2,15 @@
|
||||
|
||||
Welcome to the documentation for the Fleet osquery fleet manager.
|
||||
|
||||
- Resources for using the Fleet UI, fleetctl CLI, and Fleet REST API can all be found in [Using Fleet](./1-Using-Fleet/README.md).
|
||||
- Resources for installing Fleet's infrastructure dependencies, configuring Fleet, deploying osquery to hosts, and viewing example deployment scenarios can all be found in [Deployment](./2-Deployment/README.md).
|
||||
- Finally, if you're interested in interacting with the Fleet source code, you will find information on modifying and building the code in [Contribution](./3-Contribution/README.md).
|
||||
### [Using Fleet](./1-Using-Fleet/README.md)
|
||||
Resources for using the Fleet UI, fleetctl CLI, and Fleet REST API
|
||||
|
||||
### [Deployment](./2-Deployment/README.md)
|
||||
Resources for installing Fleet's infrastructure dependencies, configuring Fleet, deploying osquery to hosts, and viewing example deployment scenarios
|
||||
|
||||
### [Contribution](./3-Contribution/README.md)
|
||||
If you're interested in interacting with the Fleet source code, you'll find information on modifying and building the code here.
|
||||
|
||||
---
|
||||
|
||||
If you have any questions, please don't hesitate to [File a GitHub issue](https://github.com/fleetdm/fleet/issues) or [join us on Slack](https://osquery.slack.com/join/shared_invite/zt-h29zm0gk-s2DBtGUTW4CFel0f0IjTEw#/). You can find us in the `#fleet` channel.
|
||||
|
||||
@ -1,5 +0,0 @@
|
||||
# Architecture Decisions
|
||||
|
||||
Architecturally significant changes to this project are documented as a collection of records. Documents are numbered sequentially and monotonically. If a decision is reversed, we will keep the old one around, but mark it as superseded (it's still relevant to know that it was the decision, but is no longer the decision).
|
||||
|
||||
If you'd like to make a significant change to this program, copy [the template](./1970-01-01_template.md) and submit a PR which explains your proposal. Your proposal may accompany a working prototype, but significant decisions should be documented before there is significant development.
|
||||
@ -1,48 +0,0 @@
|
||||
Development Documentation
|
||||
=========================
|
||||
|
||||
The Fleet application is a Go API server which serves a React/Redux single-page application for the frontend. The development documentation contains documents on the following topics:
|
||||
|
||||
## Frequently Asked Questions
|
||||
|
||||
For FAQs on common Fleet problems, see the [FAQ](./faq.md).
|
||||
|
||||
## Building and contributing code
|
||||
|
||||
- For documentation on building the Fleet source code, see the [Building The Code](./building-the-code.md) guide.
|
||||
- To learn about database migrations and populating the application with default seed data, see the [Database Migrations](./database-migrations.md) document.
|
||||
|
||||
## Running tests
|
||||
|
||||
For information on running the various tests that Fleet application contains (JavaScript unit tests, Go unit tests, linters, integration tests, etc), see the [Testing](./testing.md) guide.
|
||||
|
||||
## Using development infrastructure and tooling
|
||||
|
||||
The Fleet application uses a lot of docker tooling to make setting up a development environment quick and easy. For information on this, see the [Development Infrastructure](./development-infrastructure.md) document.
|
||||
|
||||
#### Setting up a Launcher environment
|
||||
|
||||
It's helpful to have a local build of the Launcher and it's included package building tools when reasoning about connecting the Launcher to Fleet. Both Launcher and Fleet have a similar repository interface that should be familiar.
|
||||
|
||||
If you have installed Go, but have never used it before (ie: you have not configured a `$GOPATH` environment variable), then there's good news: you don't need to do this anymore. By default, the Go compiler now looks in `~/go` for your Go source code. So, let's clone the launcher directory where it's supposed to go:
|
||||
|
||||
```
|
||||
mkdir -p $GOPATH/src/github.com/kolide
|
||||
cd $GOPATH/src/github.com/kolide
|
||||
git clone git@github.com:kolide/launcher.git
|
||||
cd launcher
|
||||
```
|
||||
|
||||
Once you're in the root of the repository (and you have a recent Go toolchain installed), you can follow the directions included with the Launcher repository:
|
||||
|
||||
```
|
||||
make deps
|
||||
make generate
|
||||
make test
|
||||
make
|
||||
./build/launcher --help
|
||||
```
|
||||
|
||||
## Releasing Fleet
|
||||
|
||||
The release process for Fleet is documented in [release.md](./release.md).
|
||||
@ -1,66 +0,0 @@
|
||||
Building The Code
|
||||
=================
|
||||
|
||||
## Building the Code
|
||||
|
||||
Clone this repository.
|
||||
|
||||
To setup a working local development environment, you must install the following minimum toolset:
|
||||
|
||||
* [Go](https://golang.org/dl/) (1.9 or greater)
|
||||
* [Node.js](https://nodejs.org/en/download/current/) and [Yarn](https://yarnpkg.com/en/docs/install)
|
||||
* [GNU Make](https://www.gnu.org/software/make/) (probably already installed if you're on macOS/Linux)
|
||||
* [Docker](https://www.docker.com/products/overview#/install_the_platform)
|
||||
|
||||
> #### New to the Go language?
|
||||
>
|
||||
> After installing Go, your $GOPATH will probably need a little freshening up. To take care of this automatically every time a new terminal is opened, add this to your shell startup script (`~/.profile`):
|
||||
> ```bash
|
||||
> # Allow go-bindata and other Go stuff to work properly (e.g. for Fleet/osquery)
|
||||
> # More info: https://golang.org/doc/gopath_code.html#GOPATH
|
||||
> export PATH=$PATH:$(go env GOPATH)/bin
|
||||
> ```
|
||||
|
||||
Once you have those minimum requirements, you will need to install Fleet's dependencies. To do this, run the following from the root of the repository:
|
||||
|
||||
```
|
||||
make deps
|
||||
```
|
||||
|
||||
When pulling in new revisions to your working source tree, it may be necessary to re-run `make deps` if a new Go or JavaScript dependency was added.
|
||||
|
||||
## Generating the packaged JavaScript
|
||||
|
||||
To generate all necessary code (bundling JavaScript into Go, etc), run the following:
|
||||
|
||||
```
|
||||
make generate
|
||||
```
|
||||
|
||||
### Automatic rebuilding of the JavaScript bundle
|
||||
|
||||
Normally, `make generate` takes the JavaScript code, bundles it into a single bundle via Webpack, and inlines that bundle into a generated Go source file so that all of the frontend code can be statically compiled into the binary. When you build the code after running `make generate`, all of that JavaScript is included in the binary.
|
||||
|
||||
This makes deploying Fleet a dream, since you only have to worry about a single static binary. If you are working on frontend code, it is likely that you don't want to have to manually re-run `make generate` and `make build` every time you edit JavaScript and CSS in order to see your changes in the browser. To solve this problem, before you build the Fleet binary, run the following command instead of `make generate`:
|
||||
|
||||
```
|
||||
make generate-dev
|
||||
```
|
||||
|
||||
Instead of reading the JavaScript from a inlined static bundle compiled within the binary, `make generate-dev` will generate a Go source file which reads the frontend code from disk and run Webpack in "watch mode".
|
||||
|
||||
Note that when you run `make generate-dev`, Webpack will be watching the JavaScript files that were used to generate the bundle, so the process will be long lived. Depending on your personal workflow, you might want to run this in a background terminal window.
|
||||
|
||||
After you run `make generate-dev`, run `make build` to build the binary, launch the binary and you'll be able to refresh the browser whenever you edit and save frontend code.
|
||||
|
||||
## Compiling the Fleet binary
|
||||
|
||||
Use `go build` to build the application code. For your convenience, a make command is included which builds the code:
|
||||
|
||||
```
|
||||
make build
|
||||
```
|
||||
|
||||
It's not necessary to use Make to build the code, but using Make allows us to account for cross-platform differences more effectively than the `go build` tool when writing automated tooling. Use whichever you prefer.
|
||||
|
||||
Once you're successful in building the code, head to the [development infrastrucutre](./development-infrastructure.md) documentation to use the local development Docker Compose infrastructure to run Fleet locally.
|
||||
@ -1,45 +0,0 @@
|
||||
Database Migrations
|
||||
===================
|
||||
|
||||
## Adding/Updating tables
|
||||
|
||||
Database schemas are managed by a series of migrations defined in go code. We use a customized version of the Goose migrations tool to handle these migrations.
|
||||
|
||||
Note: Once committed to the Fleet repo, table migrations should be considered immutable. Any changes to an existing table should take place in a new migration executing ALTERs.
|
||||
|
||||
From the project root run the following shell commands:
|
||||
|
||||
``` bash
|
||||
go get github.com/kolide/goose
|
||||
cd server/datastore/mysql/migrations/tables
|
||||
goose create AddColumnFooToUsers
|
||||
```
|
||||
|
||||
Find the file you created in the migrations directory and edit it:
|
||||
|
||||
* delete the import line for goose: `github.com/pressly/goose`
|
||||
* change `goose.AddMigration(...)` to `MigrationClient.AddMigration(...)`
|
||||
* add your migration code
|
||||
|
||||
You can then update the database by running the following shell commands:
|
||||
|
||||
``` bash
|
||||
make build
|
||||
build/fleet prepare db
|
||||
```
|
||||
|
||||
## Populating the database with default data
|
||||
|
||||
Populating built in data is also performed through migrations. All table migrations are performed before any data migrations.
|
||||
|
||||
Note: Data migrations can be mutable. If tables are altered in a way that would render a data migration invalid (columns changed/removed), data migrations should be updated to comply with the new schema. Data migrations will not be re-run when they have already been run against a database, but they must be updated to maintain compatibility with a fresh DB.
|
||||
|
||||
From the project root run the following shell commands:
|
||||
|
||||
``` bash
|
||||
go get github.com/kolide/goose
|
||||
cd server/datastore/mysql/migrations/data
|
||||
goose create PopulateFoo
|
||||
```
|
||||
|
||||
Proceed as for table migrations, editing and running the newly created migration file.
|
||||
@ -1,40 +0,0 @@
|
||||
Development Infrastructure
|
||||
==========================
|
||||
|
||||
## Starting the local development environment
|
||||
|
||||
To set up a canonical development environment via docker, run the following from the root of the repository:
|
||||
|
||||
```
|
||||
docker-compose up
|
||||
```
|
||||
|
||||
This requires that you have docker installed. At this point in time, automatic configuration tools are not included with this project.
|
||||
|
||||
#### Stopping the local development environment
|
||||
|
||||
If you'd like to shut down the virtual infrastructure created by docker, run the following from the root of the repository:
|
||||
|
||||
```
|
||||
docker-compose down
|
||||
```
|
||||
|
||||
#### Setting up the database tables
|
||||
|
||||
Once you `docker-compose up` and are running the databases, you can build the code and run the following command to create the database tables:
|
||||
|
||||
```
|
||||
fleet prepare db
|
||||
```
|
||||
|
||||
## Running Fleet using Docker development infrastructure
|
||||
|
||||
To start the Fleet server backed by the Docker development infrastructure, run the Fleet binary as follows:
|
||||
|
||||
```
|
||||
fleet serve --auth_jwt_key="insecure"
|
||||
```
|
||||
|
||||
By default, Fleet will try to connect to servers running on default ports on localhost. Depending on your browser's settings, you may have to click through a security warning.
|
||||
|
||||
If you're using Docker via [Docker Toolbox](https://www.docker.com/products/docker-toolbox), you may have to modify the default values use the output of `docker-machine ip` instead of `localhost`. There is an example configuration file included in this repository to make this process easier for you. Use the `--config` flag of the Fleet binary to specify the path to your config. See `fleet --help` for more options.
|
||||
@ -1,51 +0,0 @@
|
||||
## Setting up a Linux Development Environment
|
||||
|
||||
### Install some dependencies
|
||||
|
||||
`sudo apt-get install xzip gyp libjs-underscore libuv1-dev dep11-tools deps-tools-cli`
|
||||
|
||||
### Create a temp directory, download and place the `node` and `golang` bins
|
||||
|
||||
```
|
||||
mkdir tmp
|
||||
cd tmp
|
||||
```
|
||||
|
||||
#### install `node` and `yarn`
|
||||
|
||||
```
|
||||
wget https://nodejs.org/dist/v9.4.0/node-v9.4.0-linux-x64.tar.xz
|
||||
xz -d node-v9.4.0-linux-x64.tar.xz
|
||||
tar -xf node-v9.4.0-linux-x64.tar
|
||||
sudo cp -rf node-v9.4.0-linux-x64/bin /usr/local/
|
||||
sudo cp -rf node-v9.4.0-linux-x64/include /usr/local
|
||||
sudo cp -rf node-v9.4.0-linux-x64/lib /usr/local
|
||||
sudo cp -rf node-v9.4.0-linux-x64/share /usr/local
|
||||
npm install -g yarn
|
||||
```
|
||||
|
||||
#### install `go`
|
||||
|
||||
```
|
||||
wget https://dl.google.com/go/go1.9.3.linux-amd64.tar.gz
|
||||
sudo tar -C /usr/local -xzf go1.9.3.linux-amd64.tar.gz
|
||||
export PATH=$PATH:/usr/local/go/bin:~/go/bin/
|
||||
```
|
||||
|
||||
#### clean-up temp directory
|
||||
|
||||
```
|
||||
cd ..
|
||||
rm -rf tmp
|
||||
```
|
||||
|
||||
### Clone and build depenencies
|
||||
|
||||
```
|
||||
git clone https://github.com/fleetdm/fleet.git
|
||||
cd fleet
|
||||
make deps
|
||||
make generate
|
||||
make build
|
||||
sudo cp build/fleet /usr/bin/fleet
|
||||
```
|
||||
@ -1,108 +0,0 @@
|
||||
Testing
|
||||
=======
|
||||
|
||||
## Full test suite
|
||||
|
||||
To execute all of the tests that CI will execute, run the following from the root of the repository:
|
||||
|
||||
```
|
||||
make test
|
||||
```
|
||||
|
||||
It is a good idea to run `make test` before submitting a Pull Request.
|
||||
|
||||
#### Go unit tests
|
||||
|
||||
To run all Go unit tests, run the following:
|
||||
|
||||
```
|
||||
make test-go
|
||||
```
|
||||
|
||||
### Database Tests
|
||||
|
||||
To run database tests set environment variables as follows.
|
||||
|
||||
```
|
||||
export MYSQL_PORT_3306_TCP_ADDR=192.168.99.100
|
||||
export MYSQL_TEST=1
|
||||
```
|
||||
|
||||
### Email Tests
|
||||
|
||||
To run email related unit tests using MailHog set the following environment
|
||||
variable.
|
||||
|
||||
```
|
||||
export MAIL_TEST=1
|
||||
```
|
||||
|
||||
#### JavaScript unit tests
|
||||
|
||||
To run all JavaScript unit tests, run the following:
|
||||
|
||||
```
|
||||
make test-js
|
||||
```
|
||||
|
||||
#### Go linters
|
||||
|
||||
To run all Go linters and static analyzers, run the following:
|
||||
|
||||
```
|
||||
make lint-go
|
||||
```
|
||||
|
||||
# Integration Tests
|
||||
|
||||
By default, tests that require external dependecies like Mysql or Redis are skipped. The tests can be enabled by setting `MYSQL_TEST=true` and `REDIS_TEST=true` environment variables. MYSQL will try to connect with the following credentials.
|
||||
```
|
||||
user = "kolide"
|
||||
password = "kolide"
|
||||
database = "kolide"
|
||||
host = "127.0.0.1"
|
||||
```
|
||||
Redis tests expect a redis instance at `127.0.0.1:6379`.
|
||||
|
||||
|
||||
Both the Redis and MySQL tests will also be automatically enabled with Docker links. You can check out the CircleCI configuration file(`circle.yml`) for an example of how to use Docker links to run integration tests.
|
||||
#### JavaScript linters
|
||||
|
||||
To run all JavaScript linters and static analyzers, run the following:
|
||||
|
||||
```
|
||||
make lint-js
|
||||
```
|
||||
|
||||
#### Viewing test coverage
|
||||
|
||||
When you run `make test` or `make test-go` from the root of the repository, test coverage reports are generated in every subpackage. For example, the `server` subpackage will have a coverage report generated in `./server/server.cover`
|
||||
|
||||
To explore a test coverage report on a line-by-line basis in the browser, run the following:
|
||||
|
||||
```bash
|
||||
# substitute ./datastore/datastore.cover, etc
|
||||
go tool cover -html=./server/server.cover
|
||||
```
|
||||
|
||||
To view test a test coverage report in a terminal, run the following:
|
||||
|
||||
```bash
|
||||
# substitute ./datastore/datastore.cover, etc
|
||||
go tool cover -func=./server/server.cover
|
||||
```
|
||||
|
||||
### Email
|
||||
|
||||
#### Testing email using MailHog
|
||||
|
||||
To intercept sent emails while running a Fleet development environment, make sure that you've set the SMTP address to `<docker host ip>:1025` and leave the username and password blank. Then, visit `<docker host ip>:8025` in a web browser to view the [MailHog](https://github.com/mailhog/MailHog) UI.
|
||||
|
||||
For example, if docker is running natively on your `localhost`, then your mail settings should look something like:
|
||||
|
||||
```yaml
|
||||
mail:
|
||||
address: localhost:1025
|
||||
```
|
||||
|
||||
`localhost:1025` is the default configuration. You can use `fleet config_dump` to see the values which Fleet is using given your configuration.
|
||||
Loading…
Reference in New Issue
Block a user