7afef3f035
* Re-IP Loadtesting for TGW+VPN * Further restrict VPN IPs for LoadTesting * Update VPN IPs for Loadtesting |
||
|---|---|---|
| .. | ||
| docker | ||
| shared | ||
| state | ||
| .gitignore | ||
| .terraform-version | ||
| alb.tf | ||
| ecr.tf | ||
| ecs-iam.tf | ||
| ecs-sgs.tf | ||
| ecs.tf | ||
| firehose.tf | ||
| init.tf | ||
| loadtesting.tf | ||
| locals.tf | ||
| outputs.tf | ||
| rds.tf | ||
| readme.md | ||
| redis.tf | ||
| variables.tf | ||
Terraform for Loadtesting Environment
The interface into this code is designed to be minimal. If you require changes beyond whats described here, contact @zwinnerman-fleetdm.
Deploying your code to the loadtesting environment
- Push your branch to https://github.com/fleetdm/fleet and wait for the build to complete (https://github.com/fleetdm/fleet/actions).
- Initialize your terraform environment with
terraform init. - Select a workspace for your test:
terraform workspace new WORKSPACE-NAME; terraform workspace select WORKSPACE-NAME. Ensure yourWORKSPACE-NAMEcontains only alphanumeric characters and hyphens, as it is used to generate names for AWS resources. - Apply terraform with your branch name with
terraform apply -var tag=BRANCH_NAMEand typeyesto approve execution of the plan. This takes a while to complete (many minutes). - Perform your tests (see next sections). Your deployment will be available at
https://WORKSPACE-NAME.loadtest.fleetdm.com. - When you're done, clean up the environment with
terraform destroy.
Running migrations
After applying terraform with the commands above and before performing your tests, run the following command:
aws ecs run-task --region us-east-2 --cluster fleet-"$(terraform workspace show)"-backend --task-definition fleet-"$(terraform workspace show)"-migrate:"$(terraform output -raw fleet_migration_revision)" --launch-type FARGATE --network-configuration "awsvpcConfiguration={subnets="$(terraform output -raw fleet_migration_subnets)",securityGroups="$(terraform output -raw fleet_migration_security_groups)"}"
Running a loadtest
We run simulated hosts in containers of 5,000 at a time. Once the infrastructure is running, you can run the following command:
terraform apply -var tag=BRANCH_NAME -var loadtest_containers=8
With the variable loadtest_containers you can specify how many containers of 5,000 hosts you want to start. In the example above, it will run 40,000. If the fleet instances need special configuration, you can pass them as environment variables to the fleet_config terraform variable, which is a map, using the following syntax (note the use of single quotes around the whole fleet_config variable assignment, and the use of double quotes inside its map value):
terraform apply -var tag=BRANCH_NAME -var loadtest_containers=8 -var='fleet_config={"FLEET_OSQUERY_ENABLE_ASYNC_HOST_PROCESSING":"host_last_seen=true","FLEET_OSQUERY_ASYNC_HOST_COLLECT_INTERVAL":"host_last_seen=10s"}'
Monitoring the infrastructure
There are a few main places of interest to monitor the load and resource usage:
- The Application Performance Monitoring (APM) dashboard: access it on your Fleet load-testing URL on port
:5601and path/app/apm, e.g.https://loadtest.fleetdm.com:5601/app/apm. - To monitor mysql database load, go to AWS RDS, select "Performance Insights" and the database instance to monitor (you may want to turn off auto-refresh).
- To monitor Redis load, go to Amazon ElastiCache, select the redis cluster to monitor, and go to "Metrics".
Troubleshooting
If terraform fails for some reason, you can make it output extra information to stderr by setting the TF_LOG environment variable to "DEBUG" or "TRACE", e.g.:
TF_LOG=DEBUG terraform apply ...
See https://www.terraform.io/internals/debugging for more details.