Claude Fable 5
18239e2fa9
All checks were successful
CD / Deploy to Test (push) Successful in 12s
CD / Smoke tests against Test (push) Successful in 1m15s
CD / Promote to Int (push) Successful in 12s
CI / Lint, typecheck, test (push) Successful in 6m19s
CI / Build container images (push) Has been skipped
CI / Auth e2e pack (push) Successful in 8m13s
CI / Import/export fidelity gate (push) Successful in 54s
CI / Lint, typecheck, test (pull_request) Successful in 6m20s
CI / Build container images (pull_request) Successful in 1m12s
CI / Auth e2e pack (pull_request) Successful in 8m24s
CI / Import/export fidelity gate (pull_request) Successful in 58s
CD / Build and push images (push) Successful in 22s
Adds docs/operations/update-runbook.md (obtain, verify by digest, back up, apply, verify, roll back) with the migration behaviour stated explicitly: a failed migration rolls back its own transaction but is recorded in _prisma_migrations and blocks every further migrate deploy (P3009) — including a re-deployed old image — until migrate resolve --rolled-back; semantically irreversible migrations have exactly one way back, the pre-update backup set. No rolling updates on a compose stage. Rehearsed in the isolated environment of #220: regular update to a v2 image set, then a deliberate failed-update (P3018 division by zero, schema change proven rolled back) with image-rollback-alone shown insufficient and the documented recovery executed. Protocol: docs/vs-nfd/98-update-rollback-protokoll.md. ADR 0024 decisions 5+6 recorded as executed; operations handbook and restore runbook updated; plan checkbox P1-3 ticked. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AUtYMxwTCMHG9mVHnwbFg8
70 lines
4.0 KiB
Markdown
70 lines
4.0 KiB
Markdown
# Update runbook (ADR 0024, issue #221)
|
|
|
|
How to update a Dorfteich stage — connected or airgapped — and how to get
|
|
back when an update fails. Rehearsed end-to-end (including one deliberate
|
|
failed-update rollback) in the isolated environment of #220; evidence:
|
|
`docs/vs-nfd/98-update-rollback-protokoll.md`.
|
|
|
|
## Procedure
|
|
|
|
On the stage host, from the stage directory (`/srv/DOCKER/dorfteich-<stage>/`):
|
|
|
|
1. **Obtain the update.** Connected: `docker compose pull` after step 3.
|
|
Airgapped: mirror the release images into the internal registry —
|
|
generated image list + digest-preserving copy per `deploy/stages.md`
|
|
§5b (procedure evidence: `docs/vs-nfd/95-mirror-protokoll.md`).
|
|
2. **Verify it.** Third-party images are digest-pinned in the compose
|
|
file — the pin **is** the verification. Own images: compare the
|
|
mirrored digest against the release digest before switching `TAG`.
|
|
Release notes state manual steps and carry the `migration` label when
|
|
a release contains migrations.
|
|
3. **Back up first.** Trigger a backup run (admin UI or
|
|
`POST /api/v1/admin/system/backup/run`) and confirm the new set id in
|
|
the sets list. The pre-update set is the guaranteed way back.
|
|
4. **Apply.** Set the new `TAG` in the stage `.env`, then
|
|
`docker compose up -d`. All services are replaced together — there is
|
|
**no rolling update** on a compose stage; plan a short maintenance
|
|
window (seconds to low minutes). The api applies migrations on start
|
|
(`MIGRATE_ON_START`, `prisma migrate deploy`); web/api/collab version
|
|
skew therefore lasts only for the container replacement itself and is
|
|
not a supported operating state.
|
|
5. **Verify health.** `/readyz` fully green; spot-check a page, a file
|
|
download, and one export.
|
|
6. **Roll back if needed** — see below, the path depends on what failed.
|
|
|
|
## Migration behaviour (explicit)
|
|
|
|
`prisma migrate deploy` applies pending migrations in order, each in its
|
|
own transaction, and stops at the first failure:
|
|
|
|
- **The failing migration itself is rolled back** — its partial DDL/DML
|
|
does not persist.
|
|
- **It is recorded as `failed` in `_prisma_migrations`** and from then on
|
|
every `migrate deploy` aborts with error P3009 — including the one in a
|
|
re-deployed **old** image. A plain image rollback alone therefore does
|
|
not recover a failed migration; clear the record first:
|
|
|
|
```sh
|
|
docker compose run --rm --no-deps api \
|
|
node node_modules/prisma/build/index.js migrate resolve \
|
|
--rolled-back <migration-name>
|
|
```
|
|
|
|
- **Migrations that succeeded are not undone by an image rollback.** The
|
|
supported downgrade window is one minor release
|
|
(`docs/architecture/operations.md` §Update strategy): schema additions
|
|
tolerate the previous minor. Anything older, or a migration that is
|
|
semantically irreversible (dropped/rewritten data), has exactly one way
|
|
back: **restore the pre-update backup set**
|
|
(`docs/operations/restore-runbook.md`) — which is why step 3 is not
|
|
optional. `prisma migrate reset` is not part of any procedure here.
|
|
|
|
## Rollback paths
|
|
|
|
| Failure | Way back |
|
|
| ------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
|
|
| App misbehaves, no migration in the release | previous `TAG` in `.env`, `docker compose up -d` |
|
|
| Migration failed (api restarts on P3009) | `migrate resolve --rolled-back <name>` (see above), then previous `TAG` + `up -d`; verify `/readyz` and content |
|
|
| Migration succeeded but must be undone | restore the pre-update set (`restore-runbook.md`), then previous `TAG` + `up -d` |
|
|
| Update bundle broken / images fail to start | previous `TAG` + `up -d` — the db was never touched before the api booted |
|