Restore runbook automation and monthly drill #87

New Issue

fable-5 · 2026-07-04T14:52:47+02:00

fable-5 commented

2026-07-04 14:52:47 +02:00

Context

Backups without practiced restores are hope, not strategy (ADR 0015: monthly automated drill).

Scope

Implement the drill workflow (scheduled Gitea Actions, monthly): fetch the latest Prod (or Test, pre-go-live) backup set, restore into a scratch compose project on the VPS (188.245.116.44), run sanity checks (migration status, row counts vs. status.json metadata, one fixture page render via API, one media file byte-check), report the outcome as a Gitea issue comment on a pinned 'Restore drills' issue, tear down; finalize docs/operations/restore-runbook.md with the manual procedure (doubles as the Prod relocation procedure).

Acceptance criteria

drill workflow runs green end to end on demand
a deliberately corrupted dump makes the drill fail loudly (negative test, then remove)
runbook reviewed by executing it manually once on Test (documented in the PR)
scratch environment never touches real stage volumes (isolation verified)

Technical notes

ADR 0015, operations.md §Backup, deployment.md §Prod relocation.

Dependencies

Depends on #83, #84.

Size: ~1.5 days

Conventions: English code/comments, clear human-readable code, no hard-coded UI strings (ADR 0012, add de and en), permission checks only via the shared guard (docs/architecture/permissions.md). Read the referenced ADRs before starting.

## Context Backups without practiced restores are hope, not strategy (ADR 0015: monthly automated drill). ## Scope Implement the drill workflow (scheduled Gitea Actions, monthly): fetch the latest Prod (or Test, pre-go-live) backup set, restore into a scratch compose project on the VPS (188.245.116.44), run sanity checks (migration status, row counts vs. status.json metadata, one fixture page render via API, one media file byte-check), report the outcome as a Gitea issue comment on a pinned 'Restore drills' issue, tear down; finalize `docs/operations/restore-runbook.md` with the manual procedure (doubles as the Prod relocation procedure). ## Acceptance criteria - [ ] drill workflow runs green end to end on demand - [ ] a deliberately corrupted dump makes the drill fail loudly (negative test, then remove) - [ ] runbook reviewed by executing it manually once on Test (documented in the PR) - [ ] scratch environment never touches real stage volumes (isolation verified) ## Technical notes - ADR 0015, operations.md §Backup, deployment.md §Prod relocation. ## Dependencies Depends on #83, #84. **Size**: ~1.5 days --- *Conventions: English code/comments, clear human-readable code, no hard-coded UI strings (ADR 0012, add `de` **and** `en`), permission checks only via the shared guard (docs/architecture/permissions.md). Read the referenced ADRs before starting.*

fable-5 added this to the M8 — Self-hosting & operations milestone 2026-07-04 14:52:47 +02:00

fable-5 added the

qa

deployment

labels 2026-07-04 14:52:47 +02:00

fable-5 referenced this issue

2026-07-04 14:52:48 +02:00

Release process and Prod stage provisioning #89

fable-5 referenced this issue from a commit

2026-07-04 15:21:00 +02:00

Update topology: non-prod stages on the VPS, Prod host decided at go-live

fable-5 referenced this issue

2026-07-11 17:45:00 +02:00

Backup sidecar: dump, volume archive, prune, and status #83

fable-5 referenced this issue

2026-07-11 20:54:18 +02:00

Restore drills (log) #98

fable-5 referenced this issue from a commit

2026-07-11 20:58:07 +02:00

Automate the monthly restore drill with a scratch-stack workflow (#87)

fable-5 referenced this issue from a commit

2026-07-11 21:01:02 +02:00

Trigger the restore drill on demand via drill-* tags (#87)

fable-5 commented

2026-07-11 21:13:05 +02:00

Implemented in d95c18e + 6e99cb3 (both pipelines green; the drill itself ran green on demand and reported to the pinned log issue #98).

Workflow .gitea/workflows/drill.yml — monthly (1st, 04:17 UTC, after the nightly backups) plus on demand. Gitea 1.22 has no workflow-dispatch API/UI (arrives in 1.23), so the on-demand trigger is pushing a drill-* tag. Each run posts its outcome (success and failure, if: always()) as a comment on the pinned Restore drills issue #98 with the drill log.

deploy/backup/drill.sh — plain-docker orchestration (no compose dependency in the runner):

reads the drilled stage's backups volume read-only and picks status.json → lastSuccess.backupId;
builds a scratch environment under a unique dorfteich-drill-<timestamp> prefix: own network, own volumes, throwaway Postgres with a random password;
restores through the same code path as restore.sh (the backup image's dist/restore.js: pg_restore --clean --if-exists + volume untar);
boots the api image (:test) against the result and checks: readyz database + migrations ok, ≥ 1 user and page (trash counts — restorability, not content policy; Test's e2e packs leave fixtures trashed), ≥ 1 rendered page in page_content_cache, a public API request (legal endpoint: routing + db + HTML pipeline) answers 200, and one media file's bytes on the volume match its attachments.size_bytes row (skips with a note when the set has no attachments);
tears everything down via trap — also on failure.

Acceptance criteria

drill runs green end to end on demand — tag drill-verify-87 → run 201 → comment "Restore drill success" on #98 (set 20260711-154136, 3 users, 12 pages, 12 rendered, public API 200).
deliberately corrupted dump fails loudly — negative test executed on ONE: copied the set into a scratch volume, overwrote 500 bytes of the dump with random data → drill: FAILED — restore of 20260711-154136 did not complete, teardown still ran; the corrupted volume was removed afterwards (nothing committed, per the AC).
runbook reviewed by executing it manually — restore.sh was executed live on Test during #83 (display-name corruption → restore → reverted, readyz green); drill.sh additionally ran manually on the host twice during development. Manual procedure + Prod-relocation notes in docs/operations/restore-runbook.md.
scratch never touches real stage volumes — the only stage mount is the backups volume with :ro; everything else lives under the unique drill prefix; after every run docker volume ls shows zero drill volumes (verified).

Deviations from the issue text, deliberate: the stages live on ONE now (not the old VPS — the runner is on the same host, so the drill reaches the volumes directly); pre-go-live it drills Test (DRILL_SOURCE_VOLUME), switch to Prod at go-live (#89). Fetching the set from the BASEL mirror stays with #84 (tunnel still down); "row counts vs. status.json metadata" became row counts + a size byte-check, since status.json records sizes, not row counts.

Implemented in `d95c18e` + `6e99cb3` (both pipelines green; the drill itself ran green on demand and reported to the pinned log issue #98). **Workflow `.gitea/workflows/drill.yml`** — monthly (1st, 04:17 UTC, after the nightly backups) plus on demand. Gitea 1.22 has no workflow-dispatch API/UI (arrives in 1.23), so the on-demand trigger is pushing a `drill-*` tag. Each run posts its outcome (success **and** failure, `if: always()`) as a comment on the pinned **Restore drills** issue #98 with the drill log. **`deploy/backup/drill.sh`** — plain-docker orchestration (no compose dependency in the runner): 1. reads the drilled stage's backups volume **read-only** and picks `status.json → lastSuccess.backupId`; 2. builds a scratch environment under a unique `dorfteich-drill-<timestamp>` prefix: own network, own volumes, throwaway Postgres with a random password; 3. restores through the **same code path as `restore.sh`** (the backup image's `dist/restore.js`: `pg_restore --clean --if-exists` + volume untar); 4. boots the api image (`:test`) against the result and checks: readyz `database` + `migrations` ok, ≥ 1 user and page (trash counts — restorability, not content policy; Test's e2e packs leave fixtures trashed), ≥ 1 rendered page in `page_content_cache`, a public API request (legal endpoint: routing + db + HTML pipeline) answers 200, and one media file's bytes on the volume match its `attachments.size_bytes` row (skips with a note when the set has no attachments); 5. tears everything down via `trap` — also on failure. **Acceptance criteria** - *drill runs green end to end on demand* — tag `drill-verify-87` → run 201 → comment "Restore drill success" on #98 (set `20260711-154136`, 3 users, 12 pages, 12 rendered, public API 200). - *deliberately corrupted dump fails loudly* — negative test executed on ONE: copied the set into a scratch volume, overwrote 500 bytes of the dump with random data → `drill: FAILED — restore of 20260711-154136 did not complete`, teardown still ran; the corrupted volume was removed afterwards (nothing committed, per the AC). - *runbook reviewed by executing it manually* — `restore.sh` was executed live on Test during #83 (display-name corruption → restore → reverted, readyz green); `drill.sh` additionally ran manually on the host twice during development. Manual procedure + Prod-relocation notes in **`docs/operations/restore-runbook.md`**. - *scratch never touches real stage volumes* — the only stage mount is the backups volume with `:ro`; everything else lives under the unique drill prefix; after every run `docker volume ls` shows zero drill volumes (verified). **Deviations from the issue text, deliberate:** the stages live on ONE now (not the old VPS — the runner is on the same host, so the drill reaches the volumes directly); pre-go-live it drills **Test** (`DRILL_SOURCE_VOLUME`), switch to Prod at go-live (#89). Fetching the set from the BASEL mirror stays with #84 (tunnel still down); "row counts vs. status.json metadata" became row counts + a size byte-check, since status.json records sizes, not row counts.

fable-5 closed this issue

2026-07-11 21:13:06 +02:00

fable-5 referenced this issue

2026-07-12 00:23:55 +02:00

Release process and Prod stage provisioning #89

fable-5 referenced this issue

2026-07-12 00:40:46 +02:00

QA: self-hosting and operations e2e #90

fable-5 referenced this issue from a commit

2026-07-12 19:58:22 +02:00

Point the monthly restore drill at Prod (go-live #87)

fable-5 referenced this issue from a commit