fleet/docs/Contributing/Testing-and-local-development.md

Testing and local development

• Testing and local development
  • License key
  • Simulated hosts
  • Test suite
    • Go unit tests
    • Go linters
    • Javascript unit and integration tests
    • Javascript linters
    • MySQL tests
    • Email tests
    • Network tests
    • Viewing test coverage
  • End-to-end tests
    • Preparation
    • Run tests
    • Interactive
    • Command line
  • Test hosts
  • Email
    • Manually testing email with MailHog
  • Development database management
  • MySQL shell
  • Redis REPL
  • Testing SSO
    • Configuration
  • Testing Kinesis Logging
  • Testing pre-built installers
  • Telemetry
  • MDM setup and testing
    • ABM setup
      • Private key + certificate
      • Encrypted token
    • APNs and SCEP setup
    • Running the server
    • Testing MDM
      • Testing manual enrollment
      • Testing DEP enrollment

License key

Do you need to test Fleet Premium features locally?

Use the --dev_license flag to use the default development license key.

For example:

./build/fleet serve --dev --dev_license

Simulated hosts

It can be helpful to quickly populate the UI with simulated hosts when developing or testing features that require host information.

Check out /tools/osquery directory instructions for starting up simulated hosts in your development environment.

Test suite

You must install the golangci-lint command to run make test[-go] or make lint[-go], using:

go install github.com/golangci/golangci-lint/cmd/golangci-lint@v1.51.1

Make sure it is available in your PATH. To execute the basic unit and integration tests, run the following from the root of the repository:

REDIS_TEST=1 MYSQL_TEST=1 make test

Go unit tests

To run all Go unit tests, run the following:

REDIS_TEST=1 MYSQL_TEST=1 MINIO_STORAGE_TEST=1 SAML_IDP_TEST=1 make test-go

Go linters

To run all Go linters and static analyzers, run the following:

make lint-go

Javascript unit and integration tests

To run all JS unit tests, run the following:

make test-js

or

yarn test

Javascript linters

To run all JS linters and static analyzers, run the following:

make lint-js

or

yarn lint

MySQL tests

To run MySQL integration tests, set environment variables as follows:

MYSQL_TEST=1 make test-go

Email tests

To run email related integration tests using MailHog set environment as follows:

MAIL_TEST=1 make test-go

Network tests

A few tests require network access as they make requests to external hosts. Given that the network is unreliable and may not be available. Those hosts may also be unavailable so these tests are skipped by default. They are opt-in via the NETWORK_TEST environment variable. To run them:

NETWORK_TEST=1 make test-go

Viewing test coverage

When you run make test or make test-go from the root of the repository, a test coverage report is generated at the root of the repo in a filed named coverage.txt

To explore a test coverage report on a line-by-line basis in the browser, run the following:

go tool cover -html=coverage.txt

To view test a test coverage report in a terminal, run the following:

go tool cover -func=coverage.txt

End-to-end tests

E2E tests are run using Docker and Cypress.

E2E tests are constantly evolving, and running them or examining CI results is the best way to understand what they cover, but at a high level, they cover:

1. Setup
2. Log in/out flows
3. Host page Add hosts Label flows
4. Query flows
5. Policy flows
6. Schedule flows scheduling packs
7. Permissions Admin Observer (global and team) Maintainer
8. Organizational Settings Settings adjustments Users

Preparation

Make sure dependencies are up to date and to build the Fleet binaries locally.

For Fleet Free tests:

make e2e-reset-db
make e2e-serve-free

For Fleet Premium tests:

make e2e-reset-db
make e2e-serve-premium

This will start a local Fleet server connected to the E2E database. Leave this server running for the duration of end-to-end testing.

make e2e-setup

This will initialize the E2E instance with a user.

Run tests

Tests can be run in interactive mode or from the command line.

Interactive

For Fleet Free tests:

yarn e2e-browser:free

For Fleet Premium tests:

yarn e2e-browser:premium

Use the graphical UI controls to run and view tests.

Command line

For Fleet Free tests:

yarn e2e-cli:free

For Fleet Premium tests:

yarn e2e-cli:premium

Tests will run automatically, and results are reported to the shell.

Test hosts

The Fleet repo includes tools to start testing osquery hosts. Please see the documentation in /tools/osquery for more information.

Email

Manually testing email with MailHog

To intercept sent emails while running a Fleet development environment, first, as an Admin in the Fleet UI, navigate to the Organization settings.

Then, in the "SMTP options" section, enter any email address in the "Sender address" field, set the "SMTP server" to localhost on port 1025, and set "Authentication type" to None. Note that you may use any active or inactive sender address.

Visit localhost:8025 to view MailHog's admin interface displaying all emails sent using the simulated mail server.

Development database management

In the course of development (particularly when crafting database migrations), it may be useful to backup, restore, and reset the MySQL database. This can be achieved with the following commands:

Backup:

make db-backup

The database dump is stored in backup.sql.gz.

Restore:

make db-restore

Note that a "restore" will replace the state of the development database with the state from the backup.

Reset:

make db-reset

MySQL shell

Connect to the MySQL shell to view and interact directly with the contents of the development database.

To connect via Docker:

docker-compose exec mysql mysql -uroot -ptoor -Dfleet

Redis REPL

Connect to the redis-cli in REPL mode to view and interact directly with the contents stored in Redis.

docker-compose exec redis redis-cli

Testing SSO

Fleet's docker-compose file includes a SAML identity provider (IdP) for testing SAML-based SSO locally.

Configuration

Configure SSO on the Organization Settings page with the following:

Identity Provider Name: SimpleSAML
Entity ID: https://localhost:8080
Issuer URI: http://localhost:8080/simplesaml/saml2/idp/SSOService.php
Metadata URL: http://localhost:9080/simplesaml/saml2/idp/metadata.php

The identity provider is configured with two users:

Username: sso_user
Email: sso_user@example.com
Password: user123#

Username: sso_user2
Email: sso_user2@example.com
Password: user123#

Use the Fleet UI to invite one of these users with the associated email. Be sure the "Enable single sign-on" box is checked for that user. Now, after accepting the invitation, you should be able to log in as that user by clicking "Sign on with SimpleSAML" on the login page.

To add additional users, modify tools/saml/users.php and restart the simplesaml container.

Testing Kinesis Logging

Tip: Install AwsLocal to ease interaction with LocalStack. Alternatively, you can use the aws client and use --endpoint-url=http://localhost:4566 on all invocations.

The following guide assumes you have server dependencies running:

docker-compose up
#
# (Starts LocalStack with kinesis enabled.)
#

First, create one stream for "status" logs and one for "result" logs (see https://osquery.readthedocs.io/en/stable/deployment/logging/ for more information around the two types of logs):

$ awslocal kinesis create-stream --stream-name "sample_status" --shard-count 1
$ awslocal kinesis create-stream --stream-name "sample_result" --shard-count 1
$ awslocal kinesis list-streams
{
    "StreamNames": [
        "sample_result",
        "sample_status"
    ]
}

Use the following configuration to run Fleet:

FLEET_OSQUERY_RESULT_LOG_PLUGIN=kinesis
FLEET_OSQUERY_STATUS_LOG_PLUGIN=kinesis
FLEET_KINESIS_REGION=us-east-1
FLEET_KINESIS_ENDPOINT_URL=http://localhost:4566
FLEET_KINESIS_ACCESS_KEY_ID=default
FLEET_KINESIS_SECRET_ACCESS_KEY=default
FLEET_KINESIS_STATUS_STREAM=sample_status
FLEET_KINESIS_RESULT_STREAM=sample_result

Here's a sample command for running fleet serve:

make fleet && FLEET_OSQUERY_RESULT_LOG_PLUGIN=kinesis FLEET_OSQUERY_STATUS_LOG_PLUGIN=kinesis FLEET_KINESIS_REGION=us-east-1 FLEET_KINESIS_ENDPOINT_URL=http://localhost:4566 FLEET_KINESIS_ACCESS_KEY_ID=default FLEET_KINESIS_SECRET_ACCESS_KEY=default FLEET_KINESIS_STATUS_STREAM=sample_status FLEET_KINESIS_RESULT_STREAM=sample_result ./build/fleet serve --dev --dev_license --logging_debug

Fleet will now be relaying "status" and "result" logs from osquery agents to the LocalStack's kinesis.

Let's work on inspecting "status" logs received by Kinesis ("status" logs are easier to verify, to generate "result" logs so you need to configure "schedule queries").

Get "status" logging stream shard ID:

$ awslocal kinesis list-shards --stream-name sample_status

{
    "Shards": [
        {
            "ShardId": "shardId-000000000000",
            "HashKeyRange": {
                "StartingHashKey": "0",
                "EndingHashKey": "340282366920938463463374607431768211455"
            },
            "SequenceNumberRange": {
                "StartingSequenceNumber": "49627262640659126499334026974892685537161954570981605378"
            }
        }
    ]
}

Get the shard-iterator for the status logging stream:

awslocal kinesis get-shard-iterator --shard-id shardId-000000000000 --shard-iterator-type TRIM_HORIZON --stream-name sample_status

{
    "ShardIterator": "AAAAAAAAAAERtiUrWGI0sq99TtpKnmDu6haj/80llVpP80D4A5XSUBFqWqcUvlwWPsTAiGin/pDYt0qJ683PeuSFP0gkNISIkGZVcW3cLvTYtERGh2QYVv+TrAlCs6cMpNvPuW0LwILTJDFlwWXdkcRaFMjtFUwikuOmWX7N4hIJA+1VsTx4A0kHfcDxHkjYi1WDe+8VMfYau+fB1XTEJx9AerfxdTBm"
}

Finally, you can use the above ShardIterator to get "status" log records:

awslocal kinesis get-records --shard-iterator AAAAAAAAAAERtiUrWGI0sq99TtpKnmDu6haj/80llVpP80D4A5XSUBFqWqcUvlwWPsTAiGin/pDYt0qJ683PeuSFP0gkNISIkGZVcW3cLvTYtERGh2QYVv+TrAlCs6cMpNvPuW0LwILTJDFlwWXdkcRaFMjtFUwikuOmWX7N4hIJA+1VsTx4A0kHfcDxHkjYi1WDe+8VMfYau+fB1XTEJx9AerfxdTBm
[...]
        {
            "SequenceNumber": "49627262640659126499334026986980734807488684740304699394",
            "ApproximateArrivalTimestamp": "2022-03-02T19:45:54-03:00",
            "Data": "eyJob3N0SWRlbnRpZmllciI6Ijg3OGE2ZWRmLTcxMzEtNGUyOC05NWEyLWQzNDQ5MDVjYWNhYiIsImNhbGVuZGFyVGltZSI6IldlZCBNYXIgIDIgMjI6MDI6NTQgMjAyMiBVVEMiLCJ1bml4VGltZSI6IjE2NDYyNTg1NzQiLCJzZXZlcml0eSI6IjAiLCJmaWxlbmFtZSI6Imdsb2dfbG9nZ2VyLmNwcCIsImxpbmUiOiI0OSIsIm1lc3NhZ2UiOiJDb3VsZCBub3QgZ2V0IFJQTSBoZWFkZXIgZmxhZy4iLCJ2ZXJzaW9uIjoiNC45LjAiLCJkZWNvcmF0aW9ucyI6eyJob3N0X3V1aWQiOiJlYjM5NDZiMi0wMDAwLTAwMDAtYjg4OC0yNTkxYTFiNjY2ZTkiLCJob3N0bmFtZSI6ImUwMDg4ZDI4YTYzZiJ9fQo=",
            "PartitionKey": "149",
            "EncryptionType": "NONE"
        }
    ],
[...]

The Data field is base64 encoded. You can use the following command to decode:

echo eyJob3N0SWRlbnRpZmllciI6Ijg3OGE2ZWRmLTcxMzEtNGUyOC05NWEyLWQzNDQ5MDVjYWNhYiIsImNhbGVuZGFyVGltZSI6IldlZCBNYXIgIDIgMjI6MDI6NTQgMjAyMiBVVEMiLCJ1bml4VGltZSI6IjE2NDYyNTg1NzQiLCJzZXZlcml0eSI6IjAiLCJmaWxlbmFtZSI6Imdsb2dfbG9nZ2VyLmNwcCIsImxpbmUiOiI0OSIsIm1lc3NhZ2UiOiJDb3VsZCBub3QgZ2V0IFJQTSBoZWFkZXIgZmxhZy4iLCJ2ZXJzaW9uIjoiNC45LjAiLCJkZWNvcmF0aW9ucyI6eyJob3N0X3V1aWQiOiJlYjM5NDZiMi0wMDAwLTAwMDAtYjg4OC0yNTkxYTFiNjY2ZTkiLCJob3N0bmFtZSI6ImUwMDg4ZDI4YTYzZiJ9fQo= | base64 -d
{"hostIdentifier":"878a6edf-7131-4e28-95a2-d344905cacab","calendarTime":"Wed Mar  2 22:02:54 2022 UTC","unixTime":"1646258574","severity":"0","filename":"glog_logger.cpp","line":"49","message":"Could not get RPM header flag.","version":"4.9.0","decorations":{"host_uuid":"eb3946b2-0000-0000-b888-2591a1b666e9","hostname":"e0088d28a63f"}}

Testing pre-built installers

Pre-built installers are kept in a blob storage like AWS S3. As part of your your local development there's a MinIO instance running on http://localhost:9000. To test the pre-built installers functionality locally:

1. Build the installers you want using fleetctl package. Be sure to include the --insecure flag for local testing.
2. Use the installerstore tool to upload them to your MinIO instance.
3. Configure your fleet server setting FLEET_PACKAGING_GLOBAL_ENROLL_SECRET to match your global enroll secret.
4. Set FLEET_SERVER_SANDBOX_ENABLED=1, as the endpoint to retrieve the installer is only available in the sandbox.

FLEET_SERVER_SANDBOX_ENABLED=1 FLEET_PACKAGING_GLOBAL_ENROLL_SECRET=xyz  ./build/fleet serve --dev

Be sure to replace the FLEET_PACKAGING_GLOBAL_ENROLL_SECRET value above with the global enroll secret from the fleetctl package command used to build the installers.

MinIO also offers a web interface at http://localhost:9001. Credentials are minio / minio123!.

Telemetry

You can configure the server to record and report trace data using OpenTelemetry or Elastic APM and use a tracing system like Jaeger to consume this data and inspect the traces locally.

Please refer to tools/telemetry for instructions.

MDM setup and testing

To run your local server with the MDM features enabled, you need to get certificates and keys.

• ABM setup
• APNs and SCEP setup
• Running the server
• Testing MDM

ABM setup

To enable the DEP enrollment flow, the Fleet server needs three things:

1. A private key.
2. A certificate.
3. An encrypted token generated by Apple.

Private key + certificate

You can generate the private key and the certificate using fleetctl:

fleetctl generate mdm-apple-bm

This will output two files fleet-apple-mdm-bm-public-key.crt and fleet-apple-mdm-bm-private.key, save them in a safe place.

Encrypted token

Ask @zwass to create an account for you in ABM

Once you have access:

1. Go to https://business.apple.com/#/main/preferences/myprofile
2. Click on "+ Add" to create a new MDM server.
3. Use a name that allows you to identify the server.
4. Under "Upload Public Key," upload the fleet-apple-mdm-bm-public-key.crt you generated before.
5. Click "Save."
6. Click on the "Download Token" button at the top and confirm the download in the modal.
7. Save the token in a safe place.

APNs and SCEP setup

The server also needs a private key + certificate to identify with Apple's APNs servers, and another for SCEP.

Both can be generated using the command below. The email must be a company email address. It cannot be an email address from commercial email providers like gmail.com.

$ fleetctl generate mdm-apple --email <email> --org <organization>

Using the above syntax, a command would look like this:

$ fleetctl generate mdm-apple --email it@example.com --org "Acme Co."

Successful output would look like this:

Sending certificate signing request (CSR) for Apple Push Notification service (APNs) to <email>...
Generating APNs key, Simple Certificate Enrollment Protocol (SCEP) certificate, and SCEP key...

Success!

Generated your APNs key at fleet-mdm-apple-apns.key

Generated your SCEP certificate at fleet-mdm-apple-scep.crt

Generated your SCEP key at fleet-mdm-apple-scep.key

Go to your email to download a CSR from Fleet. Then, visit https://identity.apple.com/pushcert to upload the CSR. You should receive an APNs certificate in return from Apple.

Next, use the generated certificates to deploy Fleet with `mdm` configuration: https://fleetdm.com/docs/deploying/configuration#mobile-device-management-mdm

Note that:

1. Fleet must be running for the command to succeed.
2. You must be logged in to fleetctl using a global admin account. See Building Fleet for details on getting Fleet setup locally.
3. To login into https://identity.apple.com/pushcert you can use your ABM account generated in the previous step.
4. Save all the certificates and keys in a safe place.

Another option, if for some reason the fleetctl generate command fails or you don't have a supported email address handy is to use openssl to generate your SCEP key pair:

$ openssl genrsa -out fleet-mdm-apple-scep.key 4096

$ openssl req -x509 -new -nodes -key fleet-mdm-apple-scep.key -sha256 -days 1826 -out fleet-mdm-apple-scep.crt -subj '/CN=Fleet Root CA/C=US/O=Fleet DM.'

Running the server

Try to store all the certificates and tokens you generated in the earlier steps together in a safe place outside of the repo, then start the server with:

FLEET_MDM_APPLE_ENABLE=1 \
FLEET_MDM_APPLE_SCEP_CHALLENGE=scepchallenge \
FLEET_MDM_APPLE_SCEP_CERT=/path/to/fleet-mdm-apple-scep.crt \
FLEET_MDM_APPLE_SCEP_KEY=/path/to/fleet-mdm-apple-scep.key \
FLEET_MDM_APPLE_BM_SERVER_TOKEN=/path/to/dep_encrypted_token.p7m \
FLEET_MDM_APPLE_BM_CERT=/path/to/fleet-apple-mdm-bm-public-key.crt \
FLEET_MDM_APPLE_BM_KEY=/path/to/fleet-apple-mdm-bm-private.key \
FLEET_MDM_APPLE_APNS_CERT=/path/to/mdmcert.download.push.pem \
FLEET_MDM_APPLE_APNS_KEY=/path/to/mdmcert.download.push.key \
 ./build/fleet serve --dev --dev_license --logging_debug

Note: if you need to enroll VMs using MDM, the server needs to run behind TLS with a valid certificate. In a separate terminal window/tab, create a local tunnel to your server using ngrok (brew install ngrok/ngrok/ngrok if you don't have it.)

ngrok http https://localhost:8080

NOTE: If this is your first time using ngrok this command will fail and you will see a message about signing up. Open the sign up link and complete the sign up flow. You can rerun the same command and ngrok should work this time. After this open the forwarding link, you will be asked to confirm that you'd like to be forwarded to your local server and should accept.

Don't forget to edit your Fleet server settings (through the UI or fleetctl) to use the URL ngrok provides to you. You need to do this whenever you restart ngrok.

Testing MDM

To test MDM, you'll need one or more virtual machines (VMs) that you can use to enroll to your server.

Choose and download a VM software, some options:

• VMware Fusion: https://www.vmware.com/products/fusion.html
• UTM: https://mac.getutm.app/

If you need a license please use your Brex card (and submit the receipt on Brex.)

With the software in place, you need to create a VM and install macOS, the steps to do this vary depending on your software of choice.

If you are using VMWare, we've used this guide in the past, please reach out in #g-mdm before starting so you can get the right serial numbers.

If you are using UTM, you can simply click "Create a New Virtual Machine" button with the default settings. This creates a VM running the latest macOS.

Testing manual enrollment

1. Create a manual profile with:

fleetctl apple-mdm enrollment-profiles create-manual

2. Open the URL that the command outputs in your VM, download and install the configuration profile.

Testing DEP enrollment

NOTE: Currently this is not possible for M1 Mac machines.

1. Create a DEP profile with:

fleetctl apple-mdm enrollment-profiles create-automatic --dep-profile ./tools/mdm/apple/dep_sample_profile.json

2. In ABM, look for the computer with the serial number that matches the one your VM has, click on it and click on "Edit MDM Server" to assign that computer to your MDM server.

3. Boot the machine, it should automatically enroll into MDM.