From 18239e2fa9735efd553f1f16c12877ac7300b1d6 Mon Sep 17 00:00:00 2001 From: Claude Fable 5 Date: Fri, 31 Jul 2026 17:26:50 +0200 Subject: [PATCH] #221: offline update path incl. migrations, rehearsed with rollback MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds docs/operations/update-runbook.md (obtain, verify by digest, back up, apply, verify, roll back) with the migration behaviour stated explicitly: a failed migration rolls back its own transaction but is recorded in _prisma_migrations and blocks every further migrate deploy (P3009) — including a re-deployed old image — until migrate resolve --rolled-back; semantically irreversible migrations have exactly one way back, the pre-update backup set. No rolling updates on a compose stage. Rehearsed in the isolated environment of #220: regular update to a v2 image set, then a deliberate failed-update (P3018 division by zero, schema change proven rolled back) with image-rollback-alone shown insufficient and the documented recovery executed. Protocol: docs/vs-nfd/98-update-rollback-protokoll.md. ADR 0024 decisions 5+6 recorded as executed; operations handbook and restore runbook updated; plan checkbox P1-3 ticked. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01AUtYMxwTCMHG9mVHnwbFg8 --- .../0024-reproducible-offline-deployment.md | 20 +++++ docs/operations/restore-runbook.md | 4 + docs/operations/update-runbook.md | 69 +++++++++++++++++ docs/vs-nfd/20-massnahmenplan.md | 3 +- docs/vs-nfd/70-betriebshandbuch.md | 16 ++-- docs/vs-nfd/98-update-rollback-protokoll.md | 77 +++++++++++++++++++ 6 files changed, 183 insertions(+), 6 deletions(-) create mode 100644 docs/operations/update-runbook.md create mode 100644 docs/vs-nfd/98-update-rollback-protokoll.md diff --git a/docs/architecture/adr/0024-reproducible-offline-deployment.md b/docs/architecture/adr/0024-reproducible-offline-deployment.md index aaf9d54..98bb903 100644 --- a/docs/architecture/adr/0024-reproducible-offline-deployment.md +++ b/docs/architecture/adr/0024-reproducible-offline-deployment.md @@ -62,6 +62,26 @@ documented mirror or update path. offline (`CI=1` skips the fetch, as in CI); for an offline ZIP build the tarball is pre-seeded into `packages/plugins/drawio/vendor/`. +## Decisions taken in #220 and #221 + +- **The airgap claim is proven** (decision 5 executed): a full deployment + ran in an environment with no egress path at all; the complete capture + shows the application makes exactly one outbound attempt — SMTP — which + fails contained in the outbox while the instance stays functional. + Evidence: `docs/vs-nfd/97-isolationslauf-protokoll.md`. The run + surfaced and fixed one real defect (#288, restore on partitioned + tables). +- **The update path is defined and rehearsed** (decision 6 executed): + procedure in `docs/operations/update-runbook.md`; rehearsed in the same + isolated environment including one deliberate failed-update rollback. + Migration irreversibility is stated explicitly: failed migrations roll + back their own transaction but block every further `migrate deploy` + (P3009) until `migrate resolve --rolled-back`; successfully applied, + semantically irreversible migrations have exactly one way back — the + pre-update backup set. No rolling updates on a compose stage; updates + happen in a short maintenance window. Evidence: + `docs/vs-nfd/98-update-rollback-protokoll.md`. + ## Consequences - Digest pinning creates recurring maintenance: security updates now diff --git a/docs/operations/restore-runbook.md b/docs/operations/restore-runbook.md index b761902..59cca69 100644 --- a/docs/operations/restore-runbook.md +++ b/docs/operations/restore-runbook.md @@ -28,6 +28,10 @@ Consistency model (ADR 0015): the volume archive is taken minutes after the dump — a page referencing a file uploaded in between shows a missing image, never corruption. +Restore is also the designated way back from an update whose migrations +succeeded but must be undone — the pre-update backup set is step 3 of +`docs/operations/update-runbook.md` (#221). + **Relocation to a new host:** provision the stage directory (compose + `.env`, deploy/stages.md), start only `db` and `backup` (`docker compose up -d db backup`), copy the set into the backups volume diff --git a/docs/operations/update-runbook.md b/docs/operations/update-runbook.md new file mode 100644 index 0000000..08eb0e8 --- /dev/null +++ b/docs/operations/update-runbook.md @@ -0,0 +1,69 @@ +# Update runbook (ADR 0024, issue #221) + +How to update a Dorfteich stage — connected or airgapped — and how to get +back when an update fails. Rehearsed end-to-end (including one deliberate +failed-update rollback) in the isolated environment of #220; evidence: +`docs/vs-nfd/98-update-rollback-protokoll.md`. + +## Procedure + +On the stage host, from the stage directory (`/srv/DOCKER/dorfteich-/`): + +1. **Obtain the update.** Connected: `docker compose pull` after step 3. + Airgapped: mirror the release images into the internal registry — + generated image list + digest-preserving copy per `deploy/stages.md` + §5b (procedure evidence: `docs/vs-nfd/95-mirror-protokoll.md`). +2. **Verify it.** Third-party images are digest-pinned in the compose + file — the pin **is** the verification. Own images: compare the + mirrored digest against the release digest before switching `TAG`. + Release notes state manual steps and carry the `migration` label when + a release contains migrations. +3. **Back up first.** Trigger a backup run (admin UI or + `POST /api/v1/admin/system/backup/run`) and confirm the new set id in + the sets list. The pre-update set is the guaranteed way back. +4. **Apply.** Set the new `TAG` in the stage `.env`, then + `docker compose up -d`. All services are replaced together — there is + **no rolling update** on a compose stage; plan a short maintenance + window (seconds to low minutes). The api applies migrations on start + (`MIGRATE_ON_START`, `prisma migrate deploy`); web/api/collab version + skew therefore lasts only for the container replacement itself and is + not a supported operating state. +5. **Verify health.** `/readyz` fully green; spot-check a page, a file + download, and one export. +6. **Roll back if needed** — see below, the path depends on what failed. + +## Migration behaviour (explicit) + +`prisma migrate deploy` applies pending migrations in order, each in its +own transaction, and stops at the first failure: + +- **The failing migration itself is rolled back** — its partial DDL/DML + does not persist. +- **It is recorded as `failed` in `_prisma_migrations`** and from then on + every `migrate deploy` aborts with error P3009 — including the one in a + re-deployed **old** image. A plain image rollback alone therefore does + not recover a failed migration; clear the record first: + + ```sh + docker compose run --rm --no-deps api \ + node node_modules/prisma/build/index.js migrate resolve \ + --rolled-back + ``` + +- **Migrations that succeeded are not undone by an image rollback.** The + supported downgrade window is one minor release + (`docs/architecture/operations.md` §Update strategy): schema additions + tolerate the previous minor. Anything older, or a migration that is + semantically irreversible (dropped/rewritten data), has exactly one way + back: **restore the pre-update backup set** + (`docs/operations/restore-runbook.md`) — which is why step 3 is not + optional. `prisma migrate reset` is not part of any procedure here. + +## Rollback paths + +| Failure | Way back | +| ------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | +| App misbehaves, no migration in the release | previous `TAG` in `.env`, `docker compose up -d` | +| Migration failed (api restarts on P3009) | `migrate resolve --rolled-back ` (see above), then previous `TAG` + `up -d`; verify `/readyz` and content | +| Migration succeeded but must be undone | restore the pre-update set (`restore-runbook.md`), then previous `TAG` + `up -d` | +| Update bundle broken / images fail to start | previous `TAG` + `up -d` — the db was never touched before the api booted | diff --git a/docs/vs-nfd/20-massnahmenplan.md b/docs/vs-nfd/20-massnahmenplan.md index 48f5e39..3047279 100644 --- a/docs/vs-nfd/20-massnahmenplan.md +++ b/docs/vs-nfd/20-massnahmenplan.md @@ -75,7 +75,8 @@ gehen" ist dort eine schlechtere Antwort als „getestet, hier ist die Anleitung Prebuilt-Images) · 2–3 AT · #219 - [x] Testlauf in netzisolierter Umgebung, Protokoll als Beleg · 2 AT · #220 → `97-isolationslauf-protokoll.md` -- [ ] Offline-Update-Pfad inkl. Migrationen · 2–3 AT · #221 +- [x] Offline-Update-Pfad inkl. Migrationen · 2–3 AT · #221 + → `docs/operations/update-runbook.md` + `98-update-rollback-protokoll.md` --- diff --git a/docs/vs-nfd/70-betriebshandbuch.md b/docs/vs-nfd/70-betriebshandbuch.md index be58289..90fe1a1 100644 --- a/docs/vs-nfd/70-betriebshandbuch.md +++ b/docs/vs-nfd/70-betriebshandbuch.md @@ -71,14 +71,18 @@ dokumentiert die realen Instanzen). Egress bleibt die Instanz voll funktionsfähig, nur die Mail-Zustellung scheitert kontrolliert (Outbox, 5 Versuche, dann FAILED). Nachweis: `97-isolationslauf-protokoll.md`. -- ⏳ offen: Offline-Update-Pfad (#221); Meilenstein M28. Bereits - vorhanden als Grundlage: alle Dritt-Images digest-gepinnt (#203), ein +- **Offline-Update-Pfad (#221): ✅ live verifiziert.** Prozedur + `docs/operations/update-runbook.md`; in der isolierten Umgebung + geprobt inkl. eines absichtlichen Migrations-Fehlschlags mit + Rollback. Nachweis: `98-update-rollback-protokoll.md`. Grundlage + weiterhin: alle Dritt-Images digest-gepinnt (#203), ein authoritativer Node-Pin (#236), SBOMs je Release (#202). ## 2 Update und Rollback -Referenz: `docs/architecture/operations.md` §Update strategy, -`deploy/stages.md`. +Referenz: `docs/operations/update-runbook.md` (maßgeblich, inkl. +explizitem Migrationsverhalten und Rollback-Pfaden), +`docs/architecture/operations.md` §Update strategy, `deploy/stages.md`. - **Stages:** Merge auf `main` → CD baut Images, deployt Test, führt Smoke-Tests aus, promotet Int. CD synct **keine** Compose-Dateien — @@ -101,7 +105,9 @@ Referenz: `docs/architecture/operations.md` §Update strategy, (imagetools inspect → Compose-Referenz ändern → CI bestätigt → Stage- Composes von Hand nachziehen → `docker inspect` verifiziert). Belegstufe: ✅ erprobt (Rollout #203 am 31.07.2026). -- **Offline-Update:** ⏳ offen (#221, M28). +- **Offline-Update:** ✅ geprobt (#221) — Bundle-Beschaffung über das + Mirror-Verfahren (§1), dann identische Prozedur; Fehlschlag-Rollback + einmal durchgespielt (`98-update-rollback-protokoll.md`). ## 3 Backup und Restore diff --git a/docs/vs-nfd/98-update-rollback-protokoll.md b/docs/vs-nfd/98-update-rollback-protokoll.md new file mode 100644 index 0000000..8768440 --- /dev/null +++ b/docs/vs-nfd/98-update-rollback-protokoll.md @@ -0,0 +1,77 @@ +# Update-/Rollback-Protokoll — Probe in der isolierten Umgebung (Issue #221) + +Nachweis für Maßnahmenplan P1-3 (ADR 0024): der Update-Pfad inklusive +Migrationen und geprobtem Fehlschlag-Rollback, ausgeführt in derselben +netzisolierten Umgebung wie der Isolationslauf +(`97-isolationslauf-protokoll.md` — Compose-Projekt `dorfteich-isolated`, +alle Netze `internal: true`, kein Egress-Pfad). Belegstufe: **live +verifiziert**. Die daraus destillierte Prozedur steht in +`docs/operations/update-runbook.md`. + +- **Datum:** 2026-07-31 +- **Ausgangsstand („v1"):** Images `dorfteich-iso-*:isolated` + (`main@a758c9d`, backup mit #288-Fix aus `4f6596e`), Instanz mit + Inhalt aus dem Isolationslauf. +- **Update-Ziel („v2"):** Images `dorfteich-iso-*:isolated-v2`, gebaut + aus dem Kettenstand `ccffcaa` (#288 + #220) — als Stellvertreter eines + regulären Release-Bundles; die Beschaffung/Digest-Prüfung eines echten + Bundles ist das Mirror-Verfahren (#218, `95-mirror-protokoll.md`) und + wurde dort bereits nachgewiesen. +- **Fehlschlag-Kandidat („bad"):** api-Image mit einer absichtlich + fehlschlagenden Migration + (`20260731230000_deliberately_failing_update`: `ALTER TABLE` + + `SELECT 1/0`), nur für die Probe gebaut, nie committet. + +## Phase A — reguläres Update + +| Schritt | Ergebnis | +| ------------------- | --------------------------------------------------------------------------------------------- | +| Pre-Update-Backup | ✅ Set `20260731-150833` über die Admin-API erzeugt und in der Set-Liste bestätigt | +| `TAG` → v2, `up -d` | ✅ alle vier eigenen Services zusammen ersetzt (Ausfallfenster Sekunden, kein Rolling Update) | +| Migrationen | ✅ migrate-on-start läuft durch (dieses „Release" enthält keine neuen Migrationen) | +| `/readyz` | ✅ vollständig grün | +| Inhalt | ✅ Seite „Isolationstest Seite" unverändert vorhanden | + +## Phase B — geprobter Fehlschlag + Rollback + +1. **Bad-Update einspielen** (`TAG` → bad, `up -d`): api scheitert beim + Boot in `migrate deploy` — **P3018**, `ERROR: division by zero` bei + `20260731230000_deliberately_failing_update`; Container geht in den + Restart-Loop, jeder Folge-Boot meldet **P3009** („migrate found + failed migrations"). +2. **Datenbank-Zustand geprüft:** die Migration steht als `failed` in + `_prisma_migrations` (Logs erfasst), aber `broken_column` existiert + **nicht** — die per-Migration-Transaktion hat den Schema-Eingriff + zurückgerollt. Genau das dokumentierte Verhalten. +3. **Image-Rollback allein reicht nicht** (bewusst vorgeführt): `TAG` + zurück auf v2, `up -d` — der alte Stand scheitert weiter mit + **P3009** („The 20260731230000_deliberately_failing_update migration + started at 2026-07-31 15:10:50 UTC failed"). +4. **Recovery wie im Runbook:** + `migrate resolve --rolled-back 20260731230000_deliberately_failing_update` + (im api-Image ausgeführt) → „marked as rolled back" → api-Neustart: + `/readyz` vollständig grün, Inhalt unverändert, Migrations-Row trägt + `rolled_back_at`. +5. **Alternativer Weg zurück** (für semantisch irreversible, aber + erfolgreich gelaufene Migrationen): Restore des Pre-Update-Sets — der + Restore-Pfad selbst ist im Isolationslauf verifiziert + (`97-isolationslauf-protokoll.md`, Fix #288). + +## Versions-Skew (Akzeptanzkriterium) + +Ein Compose-Stage kennt **kein Rolling Update**: `docker compose up -d` +ersetzt web/api/collab/backup zusammen; Mischbetrieb alter und neuer +Versionen ist kein unterstützter Betriebszustand und dauert nur die +Container-Ersetzung selbst (Sekunden). Für VS-Zonen heißt das: Update im +angekündigten Wartungsfenster; die Migrationen wendet ausschließlich der +neue api-Container beim Start an (`MIGRATE_ON_START`). + +## Rückfluss + +- **`docs/operations/update-runbook.md`** — die Prozedur (Beschaffen, + Prüfen, Sichern, Einspielen, Verifizieren, Rollback) inkl. explizitem + Migrationsverhalten; aus dieser Probe destilliert. +- **`docs/operations/restore-runbook.md`** — Querverweis: Restore als + einziger Weg zurück nach irreversiblen Migrationen. +- **#229** — Betriebshandbuch §1 (Airgap) und §2 (Update/Rollback) + verweisen auf Runbook und dieses Protokoll.