Test DB migrations with Percona XtraDB MySQL server 5.7.25 (#16320)

#15881

This PR adds a script to test DB migrations with Percona XtraDB 5.7.25.

PS: To run this test before we merge this PR to `main` you will need to
change step 2 (`Make sure to be on latest main`), instead of `main` use
this branch `15881-test-migrations-with-percona`.

.github/ISSUE_TEMPLATE/release-qa.md

@@ -131,6 +131,11 @@ Using the migration scripts located in fleet/test/upgrade/
 2. Upgrade test returns an 'OK' response.
 </td><td>pass/fail</td></tr>
 
+<tr><td>Migration Test with Percona XtraDB MySQL Server</td><td>Verify Fleet can migrate to the next version without issues when using a specific version of Percona XtraDB Server.</td><td>
+
+Run the instructions in [tools/percona/test/README.md](../../tools/percona/test/README.md)
+</td><td>pass/fail</td></tr>
+  
 <tr><td>Release blockers</td><td>Verify there are no outstanding release blocking tickets.</td><td>
   
 1. Check [this](https://github.com/fleetdm/fleet/labels/~release%20blocker) filter to view all open `~release blocker` tickets.

.github/workflows/test-go.yaml

@@ -12,6 +12,7 @@ on:
       - 'go.sum'
       - '.github/workflows/test-go.yaml'
       - 'server/authz/policy.rego'
+      - 'docker-compose.yml'
   pull_request:
     paths:
       - '**.go'
@@ -19,6 +20,7 @@ on:
       - 'go.sum'
       - '.github/workflows/test-go.yaml'
       - 'server/authz/policy.rego'
+      - 'docker-compose.yml'
   workflow_dispatch: # Manual
   schedule:
     - cron: '0 4 * * *'

docker-compose.yml

@@ -25,6 +25,8 @@ services:
       MYSQL_DATABASE: fleet
       MYSQL_USER: fleet
       MYSQL_PASSWORD: insecure
+      # This is required by Percona XtraDB server.
+      CLUSTER_NAME: fleet
     ports:
       - "3306:3306"
 

tools/percona/test/README.md

@@ -0,0 +1,30 @@
+# Test migrations with Percona Server XtraDB 5.7.25
+
+> IMPORTANT: 
+> - The test performed here will clear your local database.
+> - This test was developed and tested on a macOS Intel device.
+
+Following are the instructions to test Fleet DB migrations with a specific version of Percona Server XtraDB (5.7.25). We need to run this specific test for users running this specific version of Percona Server.
+The test will run migrations with `pxc_strict_mode=PERMISSIVE` up until `fleet-v4.42.0` and then run the remaining migrations with `pxc_strict_mode=ENFORCING` (default) because starting in `fleet-v4.44.0` we will attempt to make every migration compatible with running this specific version of Percona Server with the default setting.
+
+Dependencies:
+- Docker for Mac.
+- `mysql` client (`brew install mysql-client`).
+
+Everything should be executed at the root of the repository.
+
+1. Backup first by running `make db-backup`.
+1. Make sure to be on latest `main`:
+```sh
+git checkout main
+git pull origin main
+```
+1. Run the upgrade test script: `./tools/percona/test/upgrade.sh`.
+1. Once the script finishes (you should see `Migrations completed.` at the very end), run `fleet serve` and perform smoke tests as usual.
+1. Restore your previous setup by running the following:
+```sh
+docker compose down
+docker volume rm fleet_mysql-persistent-volume
+docker compose up
+make db-restore
+```

tools/percona/test/upgrade.sh

@@ -0,0 +1,42 @@
+#!/bin/bash
+
+set -x
+set -e
+
+# Up to `fleet-v4.40.0` there are no migration issues with Percona Server XtraDB's `pxc_strict_mode=ENFORCING` default value.
+# We introduced issues with `pxc_strict_mode=ENFORCING` in DB migrations in `fleet-v4.41.0` and in `fleet-v4.42.0`.
+
+# Bring everything down.
+docker compose down
+docker volume rm fleet_mysql-persistent-volume
+
+# Start dependencies using Percona XtraDB as MySQL server.
+# NOTE: To troubleshoot, remove `>/dev/null`.
+FLEET_MYSQL_IMAGE=percona/percona-xtradb-cluster:5.7.25 docker compose up >/dev/null 2>&1 &
+
+export MYSQL_PWD=toor
+
+until mysql --host 127.0.0.1 --port 3306 -uroot -e 'SELECT 1=1;' ; do
+    echo "Waiting for Percona XtraDB MySQL Server..."
+    sleep 10
+done
+echo "Percona XtraDB MySQL Server is up and running, continuing..."
+
+# Checkout and build `fleet-4.42.0`.
+git checkout fleet-v4.42.0
+make generate && make fleet
+
+# Set pxc_strict_mode=PERMISSIVE to run migrations up to fleet-v4.42.0,
+# which was the last migration released with `pxc_strict_mode=ENFORCING` issues.
+mysql --host 127.0.0.1 --port 3306 -uroot -e 'SET GLOBAL pxc_strict_mode=PERMISSIVE;'
+
+# Run migrations up to fleet-v4.42.0.
+make db-reset
+
+# Set `pxc_strict_mode` back to the `ENFORCING` default.
+mysql --host 127.0.0.1 --port 3306 -uroot -e 'SET GLOBAL pxc_strict_mode=ENFORCING;'
+
+# Run migrations from fleet-v4.42.0 up to latest to catch any future bugs when running with `pxc_strict_mode=ENFORCING`.
+git checkout main
+make generate && make fleet
+./build/fleet prepare db --dev --logging_debug