Terraform module example update (#17466)

terraform/example/.header.md

@@ -1,7 +1,40 @@
 # Fleet Terraform Module Example
-This code provides some example usage of the Fleet Terraform module, including how some addons can be used to extend functionality.
+This code provides some example usage of the Fleet Terraform module, including how some addons can be used to extend functionality.  Prior to applying, edit the locals in `main.tf` to match the settings you want for your Fleet instance including:
+
+ - domain name
+ - route53 zone name (may match the domain name)
+ - license key (if premium)
+ - uncommenting the mdm module if mdm is desired
+ - any extra settings to be passed to Fleet via ENV var.
 
 Due to Terraform issues, this code requires 3 applies "from scratch":
 1. `terraform apply -target module.fleet.module.vpc`
-2. `terraform apply -target module.fleet`
+2. `terraform apply -target module.osquery-carve -target module.firehose-logging`
-3. `terraform apply`
+3. If enabling mdm: `terraform apply -target module.mdm`.  It will need to be uncommented as well as the KMS section below it.
+4. `terraform apply -target module.fleet`
+5. `terraform apply`
+6. If enabling mdm do the following:
+ - Record the KMS key from step 5 output.
+ - Use `fleetctl` to obtain all of the mdm certs.  Use https://fleetdm.com/docs/using-fleet/mdm-macos-setup#apple-push-notification-service-apns and https://fleetdm.com/docs/using-fleet/mdm-macos-setup#apple-business-manager-abm for reference.
+ - Place the certificates in the `resources` folder with the following names based upon their function:
+```
+scep.crt
+scep.key
+apns.crt
+apns.key
+abm.crt
+abm.key
+abm_token.p7m
+```
+ - Using the `encrypt.sh` script, KMS encrypt all of these secrets as follows:
+```
+cd resources
+for i in *; do ../scripts/encrypt.sh <kms-key-id-from-terraform-output> $i $i.encrypted; done
+for i in *.encrypted; do rm ${i/.encrypted/}; done
+```
+This will encrypt all of the mdm secrets and add the .encrypted extension to them. It will also remove the non-encrypted version of the secrets so that they are encrypted at rest even locally.
+
+ - Uncomment all of the resources and data sources in `mdm-secrets.tf`.
+ - Re-run `terraform apply` to populate the Secrets Manager secrets.
+ - Uncomment the sections in the `fleet_config` portion of `main.tf` for mdm and run a final `terraform apply`.  Services will restart with mdm enabled.
+

terraform/example/README.md

@@ -1,10 +1,42 @@
 # Fleet Terraform Module Example
-This code provides some example usage of the Fleet Terraform module, including how some addons can be used to extend functionality.
+This code provides some example usage of the Fleet Terraform module, including how some addons can be used to extend functionality.  Prior to applying, edit the locals in `main.tf` to match the settings you want for your Fleet instance including:
+
+ - domain name
+ - route53 zone name (may match the domain name)
+ - license key (if premium)
+ - uncommenting the mdm module if mdm is desired
+ - any extra settings to be passed to Fleet via ENV var.
 
 Due to Terraform issues, this code requires 3 applies "from scratch":
 1. `terraform apply -target module.fleet.module.vpc`
-2. `terraform apply -target module.fleet`
+2. `terraform apply -target module.osquery-carve -target module.firehose-logging`
-3. `terraform apply`
+3. If enabling mdm: `terraform apply -target module.mdm`.  It will need to be uncommented as well as the KMS section below it.
+4. `terraform apply -target module.fleet`
+5. `terraform apply`
+6. If enabling mdm do the following:
+ - Record the KMS key from step 5 output.
+ - Use `fleetctl` to obtain all of the mdm certs.  Use https://fleetdm.com/docs/using-fleet/mdm-macos-setup#apple-push-notification-service-apns and https://fleetdm.com/docs/using-fleet/mdm-macos-setup#apple-business-manager-abm for reference.
+ - Place the certificates in the `resources` folder with the following names based upon their function:
+```
+scep.crt
+scep.key
+apns.crt
+apns.key
+abm.crt
+abm.key
+abm_token.p7m
+```
+ - Using the `encrypt.sh` script, KMS encrypt all of these secrets as follows:
+```
+cd resources
+for i in *; do ../scripts/encrypt.sh <kms-key-id-from-terraform-output> $i $i.encrypted; done
+for i in *.encrypted; do rm ${i/.encrypted/}; done
+```
+This will encrypt all of the mdm secrets and add the .encrypted extension to them. It will also remove the non-encrypted version of the secrets so that they are encrypted at rest even locally.
+
+ - Uncomment all of the resources and data sources in `mdm-secrets.tf`.
+ - Re-run `terraform apply` to populate the Secrets Manager secrets.
+ - Uncomment the sections in the `fleet_config` portion of `main.tf` for mdm and run a final `terraform apply`.  Services will restart with mdm enabled.
 
 ## Requirements
 
@@ -23,8 +55,10 @@ Due to Terraform issues, this code requires 3 applies "from scratch":
 | Name | Source | Version |
 |------|--------|---------|
 | <a name="module_acm"></a> [acm](#module\_acm) | terraform-aws-modules/acm/aws | 4.3.1 |
+| <a name="module_firehose-logging"></a> [firehose-logging](#module\_firehose-logging) | github.com/fleetdm/fleet//terraform/addons/logging-destination-firehose | tf-mod-addon-logging-destination-firehose-v1.1.0 |
 | <a name="module_fleet"></a> [fleet](#module\_fleet) | github.com/fleetdm/fleet//terraform | tf-mod-root-v1.7.1 |
 | <a name="module_migrations"></a> [migrations](#module\_migrations) | github.com/fleetdm/fleet//terraform/addons/migrations | tf-mod-addon-migrations-v2.0.0 |
+| <a name="module_osquery-carve"></a> [osquery-carve](#module\_osquery-carve) | github.com/fleetdm/fleet//terraform/addons/osquery-carve | tf-mod-addon-osquery-carve-v1.0.1 |
 
 ## Resources
 
@@ -35,12 +69,10 @@ Due to Terraform issues, this code requires 3 applies "from scratch":
 
 ## Inputs
 
-| Name | Description | Type | Default | Required |
+No inputs.
-|------|-------------|------|---------|:--------:|
-| <a name="input_domain_name"></a> [domain\_name](#input\_domain\_name) | domain name to host fleet under | `string` | n/a | yes |
-| <a name="input_vpc_name"></a> [vpc\_name](#input\_vpc\_name) | name of the vpc to provision | `string` | `"fleet"` | no |
-| <a name="input_zone_name"></a> [zone\_name](#input\_zone\_name) | the name to give to your hosted zone | `string` | `"fleet"` | no |
 
 ## Outputs
 
-No outputs.
+| Name | Description |
+|------|-------------|
+| <a name="output_route53_name_servers"></a> [route53\_name\_servers](#output\_route53\_name\_servers) | Ensure that these records are added to the parent DNS zone Delete this output if you switched the route53 zone above to a data source. |

terraform/example/main.tf

@@ -1,3 +1,11 @@
+# This example doesn't cover using a remote backend for storing the current
+# terraform state in S3 with a lock in DynamoDB (ideal for AWS) or other 
+# methods. If using automation to apply the configuration or if multiple people
+# will be managing these resources, this is recommended.
+#
+# See https://developer.hashicorp.com/terraform/language/settings/backends/s3
+# for reference.
+
 terraform {
   required_providers {
     aws = {
@@ -7,39 +15,97 @@ terraform {
   }
 }
 
-variable "domain_name" {
+locals {
-  type        = string
+  # Change these to match your environment.
-  description = "domain name to host fleet under"
+  domain_name = "fleet.example.com"
-}
+  vpc_name = "fleet-vpc"
+  # This creates a subdomain in AWS to manage DNS Records.
+  # This allows for easy validation of TLS Certificates via ACM and
+  # the use of alias records to the load balancer.  Please note if
+  # this is a subdomain that NS records will be needed to be created
+  # in the primary zone.  These NS records will be included in the outputs
+  # of this terraform run.
+  zone_name = "fleet.example.com"
 
-variable "vpc_name" {
+  # Bucket names need to be unique across AWS.  Change this to a friendly
-  type        = string
+  # name to make finding carves in s3 easier later.
-  description = "name of the vpc to provision"
+  osquery_carve_bucket_name   = "fleet-osquery-carve"
-  default     = "fleet"
+  osquery_results_bucket_name = "fleet-osquery-results"
-}
+  osquery_status_bucket_name  = "fleet-osquery-status"
 
-variable "zone_name" {
+  # Extra ENV Vars for Fleet customization can be set here.
-  type        = string
+  fleet_environment_variables = {
-  description = "the name to give to your hosted zone"
+    # Uncomment and provide license key to unlock premium features.
-  default     = "fleet"
+    #      FLEET_LICENSE_KEY = "<enter_license_key>"
+    # JSON logging improves the experience with Cloudwatch Log Insights
+    FLEET_LOGGING_JSON                      = "true"
+    FLEET_MYSQL_MAX_OPEN_CONNS              = "10"
+    FLEET_MYSQL_READ_REPLICA_MAX_OPEN_CONNS = "10"
+    # Vulnerabilities is a premium feature.
+    # Uncomment as this is a writable location in the container.
+    # FLEET_VULNERABILITIES_DATABASES_PATH    = "/home/fleet"
+    FLEET_REDIS_MAX_OPEN_CONNS = "500"
+    FLEET_REDIS_MAX_IDLE_CONNS = "500"
+  }
 }
 
 module "fleet" {
   source          = "github.com/fleetdm/fleet//terraform?ref=tf-mod-root-v1.7.1"
   certificate_arn = module.acm.acm_certificate_arn
 
-  vpc_config = {
+  vpc = {
-    name = var.vpc_name
+    name = local.vpc_name
   }
 
   fleet_config = {
-    image = "fleetdm/fleet:v4.46.1" # override default to deploy the image you desire
+    # To avoid pull-rate limiting from dockerhub, consider using our quay.io mirror
-    extra_environment_variables = {
+    # for the Fleet image. e.g. "quay.io/fleetdm/fleet:v4.47.0"
-      #      FLEET_LICENSE_KEY = "<enter_license_key>"
+    image = "fleetdm/fleet:v4.47.0" # override default to deploy the image you desire
+    # See https://fleetdm.com/docs/deploy/reference-architectures#aws for appropriate scaling
+    # memory and cpu.
+    autoscaling = {
+      min_capacity = 2
+      max_capacity = 5
     }
+    # 4GB Required for vulnerability scanning.  512MB works without.
+    mem = 4096
+    cpu = 512
+    extra_environment_variables = local.fleet_environment_variables
+    # Uncomment if enabling mdm module below.
+    # extra_secrets = module.mdm.extra_secrets
+    # extra_execution_iam_policies = module.mdm.extra_execution_iam_policies
+    extra_iam_policies = concat(
+      module.osquery-carve.fleet_extra_iam_policies,
+      module.firehose-logging.fleet_extra_iam_policies,
+    )
+  }
+  rds_config = {
+    # See https://fleetdm.com/docs/deploy/reference-architectures#aws for instance classes.
+    instance_class = "db.t4g.medium"
+  }
+  redis_config = {
+    # See https://fleetdm.com/docs/deploy/reference-architectures#aws for instance types.
+    instance_type = "cache.t4g.small"
+    # Note these parameters help performance with large/complex live queries.
+    # See https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Troubleshooting-live-queries.md#1-redis for details.
+    parameter = [
+      { name = "client-output-buffer-limit-pubsub-hard-limit", value = 0 },
+      { name = "client-output-buffer-limit-pubsub-soft-limit", value = 0 },
+      { name = "client-output-buffer-limit-pubsub-soft-seconds", value = 0 },
+    ]
+  }
+  alb_config = {
+    # Script execution can run for up to 300s plus overhead.
+    # Ensure the load balancer does not 5XX before we have results.
+    idle_timeout = 305
   }
 }
 
+# Migrations will handle scaling Fleet to 0 running containers before running the DB migration task.
+# This module will also handle scaling back up once migrations complete.
+# NOTE: This requires the aws cli to be installed on the device running terraform as terraform
+# doesn't directly support all the features required.  the aws cli is invoked via a null-resource.
+
 module "migrations" {
   source                   = "github.com/fleetdm/fleet//terraform/addons/migrations?ref=tf-mod-addon-migrations-v2.0.0"
   ecs_cluster              = module.fleet.byo-vpc.byo-db.byo-ecs.service.cluster
@@ -52,23 +118,87 @@ module "migrations" {
   min_capacity             = module.fleet.byo-vpc.byo-db.byo-ecs.appautoscaling_target.min_capacity
 }
 
+module "osquery-carve" {
+  source = "github.com/fleetdm/fleet//terraform/addons/osquery-carve?ref=tf-mod-addon-osquery-carve-v1.0.1"
+  osquery_carve_s3_bucket = {
+    name = local.osquery_carve_bucket_name
+  }   
+} 
+
+module "firehose-logging" {
+  source = "github.com/fleetdm/fleet//terraform/addons/logging-destination-firehose?ref=tf-mod-addon-logging-destination-firehose-v1.1.0"
+  osquery_results_s3_bucket = {
+    name = local.osquery_results_bucket_name
+  }
+  osquery_status_s3_bucket = {
+    name = local.osquery_status_bucket_name
+  }
+}
+
+## MDM
+
+# MDM Secrets must be populated with JSON data including the payload from the certs, keys, challenge, etc.
+# These can be populated via terraform with a secret-version, or manually after terraform is applied.
+# Note: Services will not start if the mdm module is enabled and the secrets are applied but not populated.
+
+
+## MDM Secret payload
+
+# See https://github.com/fleetdm/fleet/blob/tf-mod-addon-mdm-v2.0.0/terraform/addons/mdm/README.md#abm
+# Per that document, both Windows and Mac will use the same SCEP secret under the hood.
+
+
+# module "mdm" {
+#   source             = "github.com/fleetdm/fleet//terraform/addons/mdm?ref=tf-mod-addon-mdm-v2.0.0"
+#   # Set apn_secret_name = null if not using mac mdm
+#   apn_secret_name    = "fleet-apn"
+#   scep_secret_name   = "fleet-scep"
+#   # Set abm_secret_name = null if customer is not using dep
+#   abm_secret_name    = "fleet-dep"
+#   enable_apple_mdm   = true
+#   enable_windows_mdm = true
+# }
+
+# If you want to supply the MDM secrets via terraform, I recommend that you do not store the secrets in the clear
+# on the device that applies the terraform.  For the example here, terraform will create a KMS key, which will then
+# be used to encrypt the secrets. The included mdm-secrets.tf file will then use the KMS key to dercrypt the secrets
+# on the filesystem to generate the 
+
+# resource "aws_kms_key" "fleet_data_key" {
+#   description = "key used to encrypt sensitive data stored in terraform"
+# }
+#
+# resource "aws_kms_alias" "alias" {
+#   name          = "alias/fleet-terraform-encrypted"
+#   target_key_id = aws_kms_key.fleet_data_key.id
+# }
+#
+# output "kms_key_id" {
+#   value = aws_kms_key.fleet_data_key.id
+# }
+
 module "acm" {
   source  = "terraform-aws-modules/acm/aws"
   version = "4.3.1"
 
-  domain_name = var.domain_name
+  domain_name = local.domain_name
+  # If you change the route53 zone to a data source this needs to become "data.aws_route53_zone.main.id"
   zone_id     = aws_route53_zone.main.id
 
   wait_for_validation = true
 }
 
+# If you already are managing your zone in AWS in the same account,
+# this resource could be swapped with a data source instead to
+# read the properties of that resource.
 resource "aws_route53_zone" "main" {
-  name = var.zone_name
+  name = local.zone_name
 }
 
 resource "aws_route53_record" "main" {
+  # If you change the route53_zone to a data source this also needs to become "data.aws_route53_zone.main.id"
   zone_id = aws_route53_zone.main.id
-  name    = var.domain_name
+  name    = local.domain_name
   type    = "A"
 
   alias {
@@ -77,3 +207,9 @@ resource "aws_route53_record" "main" {
     evaluate_target_health = true
   }
 }
+
+# Ensure that these records are added to the parent DNS zone
+# Delete this output if you switched the route53 zone above to a data source.
+output "route53_name_servers" {
+  value = aws_route53_zone.main.name_servers
+}

terraform/example/mdm-secrets.tf

@@ -0,0 +1,124 @@
+# Note: Everything is commented out here as mdm is not enabled by default.
+# Uncomment to use.
+
+# This section expects all kms-encrypted secrets to live in the resources/
+# subdirectory.  The list of expected filenames is as follows:
+
+locals {
+  mdm_resource_path = "${path.module}/resources"
+  scep_cert = "${local.mdm_resource_path}/scep.crt.encrypted"
+  scep_key  = "${local.mdm_resource_path}/scep.key.encrypted"
+  apns_cert = "${local.mdm_resource_path}/apns.crt.encrypted"
+  apns_key  = "${local.mdm_resource_path}/apns.key.encrypted"
+  abm_cert  = "${local.mdm_resource_path}/abm.crt.encrypted"
+  abm_key   = "${local.mdm_resource_path}/abm.key.encrypted"
+  abm_token = "${local.mdm_resource_path}/abm_token.p7m.encrypted"
+}
+
+# To ease the process of encrypting and decrypting secrets, see
+# scripts/encrypt.sh and scripts/decrypt.sh
+
+# Place your non-encrypted files in the resources folder and
+# run the following:
+#
+# cd resources 
+# for i in *; do ../scripts/encrypt.sh <kms-key-id-from-terraform-output> $i $i.encrypted; done
+# for i in *.encrypted; do rm ${i/.encrypted/}; done
+
+# The SCEP challenge will be randomly generated by terraform. We do not
+# need to know what it is.  For troubleshooting, it can always be found
+# in the SCEP secret in AWS.
+
+# resource "random_password" "challenge" {
+#   length  = 12
+#   special = false
+# }
+# 
+# resource "aws_secretsmanager_secret_version" "scep" {
+#   secret_id = module.mdm.scep.id
+#   secret_string = jsonencode(
+#     {
+#       FLEET_MDM_APPLE_SCEP_CERT_BYTES = data.aws_kms_secrets.scep_cert.plaintext["FLEET_MDM_APPLE_SCEP_CERT_BYTES"]
+#       FLEET_MDM_APPLE_SCEP_KEY_BYTES  = data.aws_kms_secrets.scep_key.plaintext["FLEET_MDM_APPLE_SCEP_KEY_BYTES"]
+#       FLEET_MDM_APPLE_SCEP_CHALLENGE  = random_password.challenge.result
+#     }
+#   )
+# }
+# 
+#
+# data "aws_kms_secrets" "scep_cert" {
+#   secret {
+#     name    = "FLEET_MDM_APPLE_SCEP_CERT_BYTES"
+#     key_id  = aws_kms_key.fleet_data_key.id
+#     payload = file(local.scep_cert)
+#   }
+# }
+# 
+# data "aws_kms_secrets" "scep_key" {
+#   secret {
+#     name    = "FLEET_MDM_APPLE_SCEP_KEY_BYTES"
+#     key_id  = aws_kms_key.fleet_data_key.id
+#     payload = file(local.scep_key)
+#   }
+# }
+# 
+# resource "aws_secretsmanager_secret_version" "apn" {
+#   secret_id = module.mdm.apn.id
+#   secret_string = jsonencode(
+#     {
+#       FLEET_MDM_APPLE_APNS_CERT_BYTES = data.aws_kms_secrets.apns_cert.plaintext["FLEET_MDM_APPLE_APNS_CERT_BYTES"]
+#       FLEET_MDM_APPLE_APNS_KEY_BYTES  = data.aws_kms_secrets.apns_key.plaintext["FLEET_MDM_APPLE_APNS_KEY_BYTES"]
+#     }
+#   )
+# }
+# 
+# data "aws_kms_secrets" "apns_cert" {
+#   secret {
+#     name    = "FLEET_MDM_APPLE_APNS_CERT_BYTES"
+#     key_id  = aws_kms_key.fleet_data_key.id
+#     payload = file(local.apns_cert)
+#   }
+# }
+# 
+# data "aws_kms_secrets" "apns_key" {
+#   secret {
+#     name    = "FLEET_MDM_APPLE_APNS_KEY_BYTES"
+#     key_id  = aws_kms_key.fleet_data_key.id
+#     payload = file(local.apns_key)
+#   }
+# }
+# 
+# resource "aws_secretsmanager_secret_version" "abm" {
+#   secret_id = module.mdm.abm.id
+#   secret_string = jsonencode(
+#     {
+#       FLEET_MDM_APPLE_BM_CERT_BYTES = data.aws_kms_secrets.abm_cert.plaintext["FLEET_MDM_APPLE_BM_CERT_BYTES"]
+#       FLEET_MDM_APPLE_BM_KEY_BYTES = data.aws_kms_secrets.abm_key.plaintext["FLEET_MDM_APPLE_BM_KEY_BYTES"]
+#       FLEET_MDM_APPLE_BM_SERVER_TOKEN_BYTES = data.aws_kms_secrets.token.plaintext["FLEET_MDM_APPLE_BM_SERVER_TOKEN_BYTES"]
+#     }
+#   )
+# }
+# 
+# data "aws_kms_secrets" "abm_cert" {
+#   secret {
+#     name    = "FLEET_MDM_APPLE_BM_CERT_BYTES"
+#     key_id  = aws_kms_key.fleet_data_key.id
+#     payload = file(local.abm_cert)
+#   }
+# }
+# 
+# data "aws_kms_secrets" "abm_key" {
+#   secret {
+#     name    = "FLEET_MDM_APPLE_BM_KEY_BYTES"
+#     key_id  = aws_kms_key.fleet_data_key.id
+#     payload = file(local.abm_key)
+#   }
+# }
+# 
+# data "aws_kms_secrets" "token" {
+#   secret {
+#     name    = "FLEET_MDM_APPLE_BM_SERVER_TOKEN_BYTES"
+#     key_id  = aws_kms_key.fleet_data_key.id
+#     payload = file(local.abm_token)
+#   }
+# }

terraform/example/scripts/decrypt.sh

@@ -0,0 +1,27 @@
+#!/bin/bash
+
+set -e
+
+function usage() {
+	cat <<-EOUSAGE
+	
+	Usage: $(basename ${0}) <KMS_KEY_ID> <SOURCE> <DESTINATION> [AWS_PROFILE]
+	
+		This script decrypts an AWS KMS encrypted file from the desired
+		SOURCE and places it it as the DESTINATION file.  Optionally you
+		may provide the AWS_PROFILE you wish to use to run the aws kms
+		commands.
+
+		Hint: You can use /dev/stdout for the destination to just view the
+		output.
+	EOUSAGE
+	exit 1
+}
+
+[ $# -lt 3 ] && usage
+
+if [ -n "${4}" ]; then
+	export AWS_PROFILE=${4}
+fi
+
+aws kms decrypt --key-id "${1:?}" --ciphertext-blob fileb://<(cat "${2:?}" | base64 -d) --output text --query Plaintext | base64 --decode > "${3:?}"

terraform/example/scripts/encrypt.sh

@@ -0,0 +1,25 @@
+#!/bin/bash
+
+set -e
+
+function usage() {
+	cat <<-EOUSAGE
+	
+	Usage: $(basename ${0}) <KMS_KEY_ID> <SOURCE> <DESTINATION> [AWS_PROFILE]
+	
+		This script encrypts an plaintext file from SOURCE into an
+		AWS KMS encrypted DESTINATION file.  Optionally you
+		may provide the AWS_PROFILE you wish to use to run the aws kms
+		commands.
+
+	EOUSAGE
+	exit 1
+}
+
+[ $# -lt 3 ] && usage
+
+if [ -n "${4}" ]; then
+	export AWS_PROFILE=${4}
+fi
+
+aws kms encrypt --key-id "${1:?}" --plaintext fileb://<(cat "${2:?}") --output text --query CiphertextBlob > "${3:?}"