Roll a deploy back when it never gets healthy, flag migrations that break the running version, and keep server action IDs across builds

The prod and staging deploys counted a container as started once Docker said
"running", so a release that crashed after booting, or never reached the
database, went out green. They now wait for /api/v1/health through a Docker
health check, and put the image that ran before back when the new one never
gets healthy after its migrations. They leave it alone when a migration
failed, the database cannot be reached or a migration is still running: the
old image would stop at the same place, or a migration would be cut halfway.
The database is never touched, and the job fails either way. The manual
rollbacks wait for healthy too. The scripts live in .github/scripts.

A rollback runs the previous version against the new schema, so a new
Migration check fails a pull request whose migrations drop, rename, retype or
require something the old code uses. The allow-breaking-migration label lets
one through. Of the 88 migrations so far it flags the 5 that did.

Server action IDs are hashed from the build's key, and every image build made
a random one, so each release gave every action a new ID and a tab opened
before the deploy failed its next save. The image builds now take a fixed key
from the NEXT_SERVER_ACTIONS_BUILD_KEY secret; without it they build as
before.

.github/scripts/check-migrations.sh

@@ -0,0 +1,72 @@
+#!/usr/bin/env bash
+# Flags migration statements that break the version of the app still running.
+#
+#   check-migrations.sh <migration.sql>...
+#
+# A deploy migrates the database while the previous version may still be
+# serving, and a rollback runs the previous version against the new schema.
+# Both work only while a migration leaves the old code working. Adding tables
+# and nullable columns, or columns with a default, is fine. Dropping, renaming,
+# retyping or tightening something the old code uses is not: stop using it in
+# one release, and change it in a later one.
+#
+# Prints GitHub annotations, and exits 1 when it finds any of the errors.
+set -uo pipefail
+
+S='[[:space:]]'
+errors=0
+warnings=0
+
+flag() { # level file line title message
+  echo "::$1 file=$2,line=$3,title=$4::$5"
+  if [ "$1" = "error" ]; then errors=$((errors + 1)); else warnings=$((warnings + 1)); fi
+}
+
+for file in "$@"; do
+  [ -f "$file" ] || continue
+  n=0
+  while IFS= read -r raw || [ -n "$raw" ]; do
+    n=$((n + 1))
+    line="${raw%%--*}" # drop comments
+    up="${line^^}"
+
+    if [[ $up =~ DROP$S+TABLE ]]; then
+      flag error "$file" "$n" "Drops a table" "The version still running may read or write this table."
+    fi
+    if [[ $up =~ DROP$S+COLUMN ]]; then
+      flag error "$file" "$n" "Drops a column" "The version still running selects this column; Prisma reads every column it knows, so every query on the table fails."
+    fi
+    if [[ $up =~ RENAME$S+COLUMN ]]; then
+      flag error "$file" "$n" "Renames a column" "The version still running uses the old name. Add the new column, write both, and drop the old one in a later release."
+    fi
+    if [[ $up =~ ALTER$S+(TABLE|TYPE)$S.*RENAME$S+TO ]]; then
+      flag error "$file" "$n" "Renames a table or type" "The version still running uses the old name. For an enum, this is usually a value being removed or renamed."
+    fi
+    if [[ $up =~ ALTER$S+COLUMN$S+[^[:space:]]+$S+(SET$S+DATA$S+)?TYPE ]]; then
+      flag error "$file" "$n" "Changes a column type" "The version still running reads and writes the old type."
+    fi
+    if [[ $up =~ SET$S+NOT$S+NULL ]]; then
+      flag error "$file" "$n" "Makes a column required" "The version still running may insert rows without it."
+    fi
+    if [[ $up =~ ADD$S+COLUMN ]] && [[ $up =~ NOT$S+NULL ]] && ! [[ $up =~ DEFAULT ]]; then
+      flag error "$file" "$n" "Adds a required column with no default" "The version still running inserts rows without it. Give it a default, or add it nullable and require it in a later release."
+    fi
+    if [[ $up =~ DROP$S+DEFAULT ]]; then
+      flag error "$file" "$n" "Removes a column default" "The version still running leaves the column out of inserts and relies on the default."
+    fi
+    if [[ $up =~ ALTER$S+TYPE$S.*ADD$S+VALUE ]]; then
+      flag warning "$file" "$n" "Adds an enum value" "The version still running cannot read rows that hold the new value. Fine as long as nothing writes it before the deploy is done, and nothing needs a rollback after."
+    fi
+  done < "$file"
+done
+
+if [ "$errors" -gt 0 ]; then
+  echo
+  echo "$errors statement(s) would break the version still running during a deploy, and the one a rollback puts back."
+  echo "Split the change across releases. If it is safe anyway (nothing uses what is dropped, say), add the"
+  echo "allow-breaking-migration label to the pull request and say why in its description."
+  exit 1
+fi
+note=""
+[ "$warnings" -gt 0 ] && note=" ($warnings warning(s) above)"
+echo "Checked $# migration file(s): nothing that breaks the running version$note."

.github/scripts/roll-back-if-unhealthy.sh

@@ -0,0 +1,77 @@
+#!/usr/bin/env bash
+# Runs right after a deploy's `docker compose up`, from the deploy folder that
+# holds docker-compose.yml and .env. Waits for the new container, and when it
+# never gets healthy puts back the image that ran before.
+#
+#   roll-back-if-unhealthy.sh <container> <new tag> <previous image id> <previous image ref> <name>
+#
+# Only the image goes back. The database is never touched: when the old code
+# cannot run against the new schema, the manual rollback workflow restores the
+# dump the deploy took. Either way the job fails, so a rollback is never quiet.
+set -uo pipefail
+
+container="$1"
+new_tag="$2"
+prev_id="$3"
+prev_ref="$4"
+name="$5"
+# The image compose names as image: <repo>:${APP_TAG}. Set only by the tests.
+IMAGE_REPO="${IMAGE_REPO:-ghcr.io/torqvoice/torqvoice}"
+here="$(cd "$(dirname "$0")" && pwd)"
+
+bash "$here/wait-healthy.sh" "$container"
+code=$?
+if [ "$code" = 0 ]; then
+  echo "Deployed $new_tag."
+  exit 0
+fi
+
+echo "--- Last 80 lines from $container"
+docker logs "$container" --tail 80 2>&1
+
+case "$code" in
+  2)
+    echo "::error title=Not rolled back::$new_tag stopped or got stuck before its migrations finished. The previous version waits for the database and migrates the same way at start, so it would stop there too. Read the log above. If a migration failed, resolve it (prisma migrate resolve) before either version can start."
+    exit 1
+    ;;
+  3)
+    echo "::error title=Not rolled back::$new_tag is still migrating. It is left running so the migration can finish. Check on it, and roll back by hand afterwards if it still does not get healthy."
+    exit 1
+    ;;
+esac
+
+if [ -z "$prev_id" ]; then
+  echo "::error title=Not rolled back::Nothing was running before $new_tag, so there is nothing to put back."
+  exit 1
+fi
+if [ "$(docker inspect --format '{{.Image}}' "$container" 2>/dev/null)" = "$prev_id" ]; then
+  echo "::error title=Not rolled back::The version that ran before is this same image."
+  exit 1
+fi
+
+# Compose names its image by tag. Reuse the old tag while it still points at
+# the image that ran; a floating tag such as latest has moved on since, so the
+# old image gets a local tag of its own.
+prev_tag="${prev_ref##*:}"
+prev_name="$prev_ref"
+if [ "$(docker image inspect --format '{{.Id}}' "$prev_ref" 2>/dev/null)" != "$prev_id" ]; then
+  short_id="${prev_id#sha256:}"
+  prev_name="$prev_ref as it was before this deploy (image ${short_id:0:12})"
+  prev_tag="rollback-$name"
+  docker tag "$prev_id" "$IMAGE_REPO:$prev_tag" || {
+    echo "::error title=Not rolled back::The image that ran before ($prev_name) is no longer on this host."
+    exit 1
+  }
+fi
+
+echo "::warning title=Rolling back::$new_tag did not get healthy. Putting back $prev_name."
+APP_TAG="$prev_tag" docker compose up -d
+
+if bash "$here/wait-healthy.sh" "$container"; then
+  echo "::error title=Rolled back::$new_tag did not get healthy, so $prev_name is running again. The database was not touched."
+else
+  echo "--- Last 80 lines from $container after the rollback"
+  docker logs "$container" --tail 80 2>&1
+  echo "::error title=Rollback is not healthy either::$new_tag did not get healthy, and neither did $prev_name after it was put back. If the old code cannot run against the new schema, use the Rollback workflow with restore_database."
+fi
+exit 1

.github/scripts/wait-healthy.sh

@@ -0,0 +1,98 @@
+#!/usr/bin/env bash
+# Waits for a freshly started app container to answer its health check.
+#
+#   wait-healthy.sh <container>
+#
+# A container counts as started only once it is "healthy": the server answers
+# /api/v1/health and reaches the database. "running" is not enough; a release
+# that boots and then crashes, or never reaches the database, is running too.
+#
+# The exit code tells a deploy whether putting the previous image back helps:
+#
+#   0  healthy
+#   1  the migrations finished but the app never got healthy. The previous
+#      image can run against this schema, so rolling back is safe.
+#   2  it stopped before the migrations finished, or they failed. The previous
+#      image runs the same migrate step at start and would fail the same way,
+#      so rolling back does not help.
+#   3  still migrating when the wait ran out. Rolling back now would stop a
+#      migration halfway, so it is left running.
+#
+# It follows the lines init-db.sh prints as it goes. Keep the two in step.
+set -uo pipefail
+
+container="${1:?usage: wait-healthy.sh <container>}"
+DB_TIMEOUT="${DB_TIMEOUT:-120}"           # to reach the database
+MIGRATE_TIMEOUT="${MIGRATE_TIMEOUT:-1800}" # for the migrations to finish
+APP_TIMEOUT="${APP_TIMEOUT:-180}"         # from migrated to healthy
+
+DB_READY='PostgreSQL is ready!'
+MIGRATED='Migrations applied successfully!'
+
+db_ready_at=""
+migrated_at=""
+
+fail() {
+  echo "::error title=$container is not healthy::$2"
+  exit "$1"
+}
+
+echo "Waiting for $container to be healthy..."
+while :; do
+  if ! state=$(docker inspect --format '{{.State.Status}} {{.RestartCount}} {{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}' "$container" 2>/dev/null); then
+    fail 2 "No container named $container."
+  fi
+  read -r status restarts health <<<"$state"
+  logs=$(docker logs "$container" 2>&1)
+
+  if [ -z "$db_ready_at" ] && grep -qF "$DB_READY" <<<"$logs"; then
+    db_ready_at=$SECONDS
+    echo "Database reachable after ${SECONDS}s, migrating..."
+  fi
+  if [ -z "$migrated_at" ] && grep -qF "$MIGRATED" <<<"$logs"; then
+    migrated_at=$SECONDS
+    echo "Migrations done after ${SECONDS}s, starting..."
+  fi
+
+  if [ "$health" = "healthy" ]; then
+    echo "$container is healthy after ${SECONDS}s."
+    exit 0
+  fi
+
+  # A compose file written before health checks were added has none. Fall
+  # back to what the deploys used to check, with a short wait for a crash.
+  if [ "$health" = "none" ] && [ "$status" = "running" ] && [ "$restarts" = "0" ] \
+    && [ -n "$migrated_at" ] && [ $((SECONDS - migrated_at)) -ge 15 ]; then
+    echo "::warning::$container has no health check; it has been running for 15s since its migrations, which is all that can be checked."
+    exit 0
+  fi
+
+  problem=""
+  if [ "$restarts" != "0" ]; then
+    problem="it restarted $restarts time(s)"
+  elif [ "$status" = "exited" ] || [ "$status" = "dead" ]; then
+    problem="it $status"
+  elif [ "$health" = "unhealthy" ]; then
+    problem="its health check failed"
+  fi
+  if [ -n "$problem" ]; then
+    if [ -n "$migrated_at" ]; then
+      fail 1 "After its migrations finished, $problem."
+    fi
+    fail 2 "Before its migrations finished, $problem."
+  fi
+
+  if [ -n "$migrated_at" ]; then
+    if [ $((SECONDS - migrated_at)) -ge "$APP_TIMEOUT" ]; then
+      fail 1 "Not healthy ${APP_TIMEOUT}s after its migrations finished."
+    fi
+  elif [ -n "$db_ready_at" ]; then
+    if [ $((SECONDS - db_ready_at)) -ge "$MIGRATE_TIMEOUT" ]; then
+      fail 3 "Still migrating after ${MIGRATE_TIMEOUT}s."
+    fi
+  elif [ "$SECONDS" -ge "$DB_TIMEOUT" ]; then
+    fail 2 "Could not reach the database within ${DB_TIMEOUT}s."
+  fi
+
+  sleep 3
+done

.github/workflows/deploy-demo.yml

@@ -55,6 +55,9 @@ jobs:
             ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:demo-${{ steps.sha.outputs.short }}
           build-args: |
             APP_VERSION=demo-${{ steps.sha.outputs.short }}
+          # Keeps server action IDs the same from build to build (see the Dockerfile).
+          secrets: |
+            next_actions_key=${{ secrets.NEXT_SERVER_ACTIONS_BUILD_KEY }}
           cache-from: type=gha,scope=demo
           cache-to: type=gha,scope=demo,mode=min
 

.github/workflows/deploy-prod.yml

@@ -13,6 +13,11 @@ jobs:
     runs-on: [self-hosted, Linux, X64, hetzner]
 
     steps:
+      # Only for .github/scripts; the app itself arrives as an image.
+      - uses: actions/checkout@v4
+        with:
+          sparse-checkout: .github/scripts
+
       - name: Resolve version tag
         id: tag
         run: |
@@ -78,11 +83,11 @@ jobs:
           # Optional, a repository variable rather than a secret: the largest backup data
           # file a restore reads into memory, in MiB (128 unless set, see .env.example).
           BACKUP_MAX_DATA_FILE_MB: ${{ vars.CLOUD_BACKUP_MAX_DATA_FILE_MB }}
-          # Optional. Next encrypts what a server action closes over with a key
-          # made fresh by every build, so a tab left open across a deploy can
-          # no longer be read by the new container. A fixed key keeps those tabs
-          # working. 32 random bytes, base64: `openssl rand -base64 32`. Left
-          # unset, the build's own key is used, as before.
+          # Optional. Next encrypts what a server action closes over with this
+          # key; left unset, the key the image was built with is used. It does
+          # not keep open tabs working across a deploy: action IDs are fixed
+          # when the image is built, from the build's key (see the Dockerfile).
+          # 32 random bytes, base64: `openssl rand -base64 32`.
           NEXT_SERVER_ACTIONS_ENCRYPTION_KEY: ${{ secrets.CLOUD_NEXT_SERVER_ACTIONS_ENCRYPTION_KEY }}
           GOOGLE_INTEGRATION_CLIENT_ID: ${{ secrets.CLOUD_GOOGLE_INTEGRATION_CLIENT_ID }}
           GOOGLE_INTEGRATION_CLIENT_SECRET: ${{ secrets.CLOUD_GOOGLE_INTEGRATION_CLIENT_SECRET }}
@@ -160,6 +165,17 @@ jobs:
                 QUICKBOOKS_INTEGRATION_CLIENT_SECRET: ${QUICKBOOKS_INTEGRATION_CLIENT_SECRET}
                 FIKEN_INTEGRATION_CLIENT_ID: ${FIKEN_INTEGRATION_CLIENT_ID}
                 FIKEN_INTEGRATION_CLIENT_SECRET: ${FIKEN_INTEGRATION_CLIENT_SECRET}
+              # Healthy once the server answers and reaches the database. The
+              # deploy waits for it and puts the previous version back when it
+              # never comes. Failures in start_period do not count, so a long
+              # migration at start is not taken for a crash. nginx-proxy routes
+              # by container, not by health, so this changes no traffic.
+              healthcheck:
+                test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:3000/api/v1/health"]
+                interval: 10s
+                timeout: 10s
+                retries: 3
+                start_period: 30m
               networks:
                 - proxy
           COMPOSE
@@ -242,26 +258,33 @@ jobs:
           # folder belong to the backup cron and age out there.
           ls -t "$BACKUP_DIR"/*-pre-*.dump 2>/dev/null | tail -n +11 | xargs -r rm --
 
+      # What to put back if the new version never gets healthy. By image ID,
+      # since a tag can have moved by the time it is needed.
+      - name: Note the running version
+        id: previous
+        run: |
+          if docker inspect torqvoice-app > /dev/null 2>&1; then
+            echo "image_id=$(docker inspect --format '{{.Image}}' torqvoice-app)" >> "$GITHUB_OUTPUT"
+            echo "image=$(docker inspect --format '{{.Config.Image}}' torqvoice-app)" >> "$GITHUB_OUTPUT"
+            echo "Running now: $(docker inspect --format '{{.Config.Image}}' torqvoice-app)"
+          else
+            echo "Nothing running yet, so there is nothing to roll back to."
+          fi
+
       - name: Deploy
         run: |
           cd $HOME/torqvoice-deploy/prod
           APP_TAG=${{ steps.tag.outputs.value }} docker compose up -d --pull always
 
-      - name: Health check
+      - name: Health check, rolling back if it fails
         run: |
-          echo "Waiting for prod container to be ready..."
-          for i in $(seq 1 12); do
-            STATUS=$(docker inspect --format='{{.State.Status}}' torqvoice-app 2>/dev/null || echo "not found")
-            if [ "$STATUS" = "running" ]; then
-              echo "Deployed ${{ steps.tag.outputs.value }}."
-              exit 0
-            fi
-            echo "Status: $STATUS, retrying in 5s..."
-            sleep 5
-          done
-          echo "::error::Container failed to start"
-          docker logs torqvoice-app --tail 50
-          exit 1
+          cd "$HOME/torqvoice-deploy/prod"
+          bash "$GITHUB_WORKSPACE/.github/scripts/roll-back-if-unhealthy.sh" \
+            torqvoice-app \
+            "${{ steps.tag.outputs.value }}" \
+            "${{ steps.previous.outputs.image_id }}" \
+            "${{ steps.previous.outputs.image }}" \
+            prod
 
       - name: Prune old images and build cache
         run: docker image prune -f && docker builder prune -f

.github/workflows/deploy-staging.yml

@@ -15,6 +15,11 @@ jobs:
     runs-on: [self-hosted, Linux, X64, hetzner]
 
     steps:
+      # Only for .github/scripts; the app itself arrives as an image.
+      - uses: actions/checkout@v4
+        with:
+          sparse-checkout: .github/scripts
+
       - name: Log in to GitHub Container Registry
         run: echo "${{ secrets.GHCR_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
 
@@ -139,6 +144,17 @@ jobs:
                 QUICKBOOKS_INTEGRATION_CLIENT_SECRET: ${QUICKBOOKS_INTEGRATION_CLIENT_SECRET}
                 FIKEN_INTEGRATION_CLIENT_ID: ${FIKEN_INTEGRATION_CLIENT_ID}
                 FIKEN_INTEGRATION_CLIENT_SECRET: ${FIKEN_INTEGRATION_CLIENT_SECRET}
+              # Healthy once the server answers and reaches the database. The
+              # deploy waits for it and puts the previous version back when it
+              # never comes. Failures in start_period do not count, so a long
+              # migration at start is not taken for a crash. nginx-proxy routes
+              # by container, not by health, so this changes no traffic.
+              healthcheck:
+                test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:3000/api/v1/health"]
+                interval: 10s
+                timeout: 10s
+                retries: 3
+                start_period: 30m
               networks:
                 - proxy
           COMPOSE
@@ -226,27 +242,34 @@ jobs:
           # folder belong to the backup cron and age out there.
           ls -t "$BACKUP_DIR"/*-pre-*.dump 2>/dev/null | tail -n +11 | xargs -r rm --
 
+      # What to put back if the new version never gets healthy. By image ID,
+      # since staging usually runs latest, and latest moves with the pull.
+      - name: Note the running version
+        id: previous
+        run: |
+          if docker inspect torqvoice-app-staging > /dev/null 2>&1; then
+            echo "image_id=$(docker inspect --format '{{.Image}}' torqvoice-app-staging)" >> "$GITHUB_OUTPUT"
+            echo "image=$(docker inspect --format '{{.Config.Image}}' torqvoice-app-staging)" >> "$GITHUB_OUTPUT"
+            echo "Running now: $(docker inspect --format '{{.Config.Image}}' torqvoice-app-staging)"
+          else
+            echo "Nothing running yet, so there is nothing to roll back to."
+          fi
+
       - name: Deploy staging
         run: |
           cd $HOME/torqvoice-deploy/staging
           TAG=${{ inputs.tag || 'latest' }}
           APP_TAG=$TAG docker compose up -d --pull always
 
-      - name: Health check
+      - name: Health check, rolling back if it fails
         run: |
-          echo "Waiting for staging container to be ready..."
-          for i in $(seq 1 12); do
-            STATUS=$(docker inspect --format='{{.State.Status}}' torqvoice-app-staging 2>/dev/null || echo "not found")
-            if [ "$STATUS" = "running" ]; then
-              echo "Container is running!"
-              exit 0
-            fi
-            echo "Status: $STATUS, retrying in 5s..."
-            sleep 5
-          done
-          echo "::error::Container failed to start"
-          docker logs torqvoice-app-staging --tail 50
-          exit 1
+          cd "$HOME/torqvoice-deploy/staging"
+          bash "$GITHUB_WORKSPACE/.github/scripts/roll-back-if-unhealthy.sh" \
+            torqvoice-app-staging \
+            "${{ inputs.tag || 'latest' }}" \
+            "${{ steps.previous.outputs.image_id }}" \
+            "${{ steps.previous.outputs.image }}" \
+            staging
 
       - name: Prune old images and build cache
         run: docker image prune -f && docker builder prune -f

.github/workflows/docker-dev.yml

@@ -49,5 +49,8 @@ jobs:
           labels: ${{ steps.meta.outputs.labels }}
           build-args: |
             APP_VERSION=dev-${{ steps.sha.outputs.short }}
+          # Keeps server action IDs the same from build to build (see the Dockerfile).
+          secrets: |
+            next_actions_key=${{ secrets.NEXT_SERVER_ACTIONS_BUILD_KEY }}
           cache-from: type=gha
           cache-to: type=gha,mode=max

.github/workflows/docker-publish.yml

@@ -66,5 +66,9 @@ jobs:
           labels: ${{ steps.meta.outputs.labels }}
           build-args: |
             APP_VERSION=${{ steps.version.outputs.value }}
+          # Keeps server action IDs the same from release to release, so tabs
+          # opened before a deploy keep working (see the Dockerfile).
+          secrets: |
+            next_actions_key=${{ secrets.NEXT_SERVER_ACTIONS_BUILD_KEY }}
           cache-from: type=gha
           cache-to: type=gha,mode=max

.github/workflows/migration-check.yml

@@ -0,0 +1,40 @@
+name: Migration check
+
+# A deploy migrates the database while the previous version may still be
+# serving, and a rollback runs the previous version against the new schema.
+# Both work only while a migration leaves the old code working, so this fails
+# a pull request whose new migrations drop, rename, retype or tighten
+# something (see .github/scripts/check-migrations.sh). When such a change is
+# safe anyway, add the allow-breaking-migration label and say why in the
+# description; the label reruns this check.
+
+on:
+  pull_request:
+    branches: [main]
+    types: [opened, synchronize, reopened, labeled, unlabeled]
+    paths: ["prisma/migrations/**"]
+
+concurrency:
+  group: migration-check-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  check:
+    name: Migrations keep the running version working
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    steps:
+      # The merge commit and its first parent, the base branch as it is now.
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 2
+
+      - name: Check new and changed migrations
+        if: ${{ !contains(github.event.pull_request.labels.*.name, 'allow-breaking-migration') }}
+        run: |
+          mapfile -t files < <(git diff --name-only --diff-filter=AM HEAD^1 HEAD -- 'prisma/migrations/*/migration.sql')
+          bash .github/scripts/check-migrations.sh "${files[@]}"
+
+      - name: Allowed by label
+        if: ${{ contains(github.event.pull_request.labels.*.name, 'allow-breaking-migration') }}
+        run: echo "::notice title=Migration check skipped::The allow-breaking-migration label is on this pull request."

.github/workflows/rollback-cloud.yml

@@ -36,6 +36,11 @@ jobs:
     runs-on: [self-hosted, Linux, X64, hetzner]
 
     steps:
+      # Only for .github/scripts; the app itself arrives as an image.
+      - uses: actions/checkout@v4
+        with:
+          sparse-checkout: .github/scripts
+
       - name: Refuse floating tags
         run: |
           if [ "${{ inputs.tag }}" = "latest" ] || [ -z "${{ inputs.tag }}" ]; then
@@ -117,16 +122,10 @@ jobs:
 
       - name: Health check
         run: |
-          echo "Waiting for prod container to be ready..."
-          for i in $(seq 1 12); do
-            STATUS=$(docker inspect --format='{{.State.Status}}' torqvoice-app 2>/dev/null || echo "not found")
-            if [ "$STATUS" = "running" ]; then
-              echo "Rolled back to ${{ inputs.tag }}."
-              exit 0
-            fi
-            echo "Status: $STATUS, retrying in 5s..."
-            sleep 5
-          done
-          echo "::error::Container failed to start after rollback"
-          docker logs torqvoice-app --tail 50
+          if bash "$GITHUB_WORKSPACE/.github/scripts/wait-healthy.sh" torqvoice-app; then
+            echo "Rolled back to ${{ inputs.tag }}."
+            exit 0
+          fi
+          echo "::error::Not healthy after rolling back to ${{ inputs.tag }}"
+          docker logs torqvoice-app --tail 80
           exit 1

.github/workflows/rollback-staging.yml

@@ -30,6 +30,11 @@ jobs:
     runs-on: [self-hosted, Linux, X64, hetzner]
 
     steps:
+      # Only for .github/scripts; the app itself arrives as an image.
+      - uses: actions/checkout@v4
+        with:
+          sparse-checkout: .github/scripts
+
       - name: Show available dumps
         run: ls -lht "${{ secrets.DATA_PATH }}/db-backups/torqvoice-staging" 2>/dev/null | head -20 || echo "No dumps yet."
 
@@ -110,16 +115,10 @@ jobs:
 
       - name: Health check
         run: |
-          echo "Waiting for staging container to be ready..."
-          for i in $(seq 1 12); do
-            STATUS=$(docker inspect --format='{{.State.Status}}' torqvoice-app-staging 2>/dev/null || echo "not found")
-            if [ "$STATUS" = "running" ]; then
-              echo "Rolled back to ${{ inputs.tag }}."
-              exit 0
-            fi
-            echo "Status: $STATUS, retrying in 5s..."
-            sleep 5
-          done
-          echo "::error::Container failed to start after rollback"
-          docker logs torqvoice-app-staging --tail 50
+          if bash "$GITHUB_WORKSPACE/.github/scripts/wait-healthy.sh" torqvoice-app-staging; then
+            echo "Rolled back to ${{ inputs.tag }}."
+            exit 0
+          fi
+          echo "::error::Not healthy after rolling back to ${{ inputs.tag }}"
+          docker logs torqvoice-app-staging --tail 80
           exit 1

CONTRIBUTING.md

@@ -46,6 +46,7 @@ npm run dev               # http://localhost:3000
 - Store dates in UTC, render in the workshop's timezone via `src/lib/timezone.ts` and `src/lib/workshop-datetime.ts`. No `setHours`/`getDay` on raw `Date` server-side.
 - Money carries a currency code; countries ISO 3166-1, languages ISO 639-1.
 - Schema changes: `npx prisma migrate dev --name <change>`, migration folder in the PR. Wrap data migrations in `BEGIN`/`COMMIT`.
+- A migration must leave the previous release working: add tables and columns (nullable or with a default), and drop, rename, retype or require them in a later release, once no code uses the old shape. CI flags migrations that do otherwise.
 - Tests assert visible behaviour, under `src/__tests__/**`, `*.test.ts(x)`.
 - No secrets. New env vars go in `.env.example` with a comment.
 - Country-specific features (registries, tax rules, parts marketplaces) ship as connectors or settings, never as defaults everyone sees.

Dockerfile

@@ -22,7 +22,18 @@ RUN npx prisma generate
 
 ENV NEXT_TELEMETRY_DISABLED=1
 
-RUN npm run build
+# Server action IDs are hashed from this key, the action's file path and its
+# name. Without one, Next makes a random key for every build, so each release
+# gives every action a new ID and a tab opened before the deploy fails its next
+# save. The same key on every build keeps the IDs, and those tabs keep working.
+# Optional: without the secret the build makes its own key, as before. Next
+# writes the key into the build output, so it is not secret once built; never
+# reuse the key a deployment sets at run time (NEXT_SERVER_ACTIONS_ENCRYPTION_KEY).
+RUN --mount=type=secret,id=next_actions_key \
+    if [ -s /run/secrets/next_actions_key ]; then \
+      export NEXT_SERVER_ACTIONS_ENCRYPTION_KEY="$(cat /run/secrets/next_actions_key)"; \
+    fi && \
+    npm run build
 
 # What the container runs beside the server: the Prisma CLI for the migrations
 # at start, and tsx with a full Prisma client for the demo seed. Installed in

init-db.sh

@@ -10,6 +10,9 @@ if [ -z "$DATABASE_URL" ]; then
   export DATABASE_URL="postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@${POSTGRES_HOST}:${POSTGRES_PORT}/${POSTGRES_DB}"
 fi
 
+# The cloud deploys follow "PostgreSQL is ready!" and "Migrations applied
+# successfully!" in this log to tell a failed migration from a failed start
+# (.github/scripts/wait-healthy.sh).
 echo "Waiting for PostgreSQL to be ready..."
 until pg_isready -d "$DATABASE_URL" -q; do
   echo "PostgreSQL is unavailable - sleeping 2s..."