dorfteich/docs/operations/update-runbook.md

Update runbook (ADR 0024, issue #221)

How to update a Dorfteich stage — connected or airgapped — and how to get back when an update fails. Rehearsed end-to-end (including one deliberate failed-update rollback) in the isolated environment of #220; evidence: docs/vs-nfd/98-update-rollback-protokoll.md.

Procedure

On the stage host, from the stage directory (/srv/DOCKER/dorfteich-<stage>/):

1. Obtain the update. Connected: docker compose pull after step 3. Airgapped: mirror the release images into the internal registry — generated image list + digest-preserving copy per deploy/stages.md §5b (procedure evidence: docs/vs-nfd/95-mirror-protokoll.md).
2. Verify it. Third-party images are digest-pinned in the compose file — the pin is the verification. Own images: compare the mirrored digest against the release digest before switching TAG. Release notes state manual steps and carry the migration label when a release contains migrations.
3. Back up first. Trigger a backup run (admin UI or POST /api/v1/admin/system/backup/run) and confirm the new set id in the sets list. The pre-update set is the guaranteed way back.
4. Apply. Set the new TAG in the stage .env, then docker compose up -d. All services are replaced together — there is no rolling update on a compose stage; plan a short maintenance window (seconds to low minutes). The api applies migrations on start (MIGRATE_ON_START, prisma migrate deploy); web/api/collab version skew therefore lasts only for the container replacement itself and is not a supported operating state.
5. Verify health. /readyz fully green; spot-check a page, a file download, and one export.
6. Roll back if needed — see below, the path depends on what failed.

Migration behaviour (explicit)

prisma migrate deploy applies pending migrations in order, each in its own transaction, and stops at the first failure:

• The failing migration itself is rolled back — its partial DDL/DML does not persist.

• It is recorded as failed in _prisma_migrations and from then on every migrate deploy aborts with error P3009 — including the one in a re-deployed old image. A plain image rollback alone therefore does not recover a failed migration; clear the record first:

  docker compose run --rm --no-deps api \
    node node_modules/prisma/build/index.js migrate resolve \
    --rolled-back <migration-name>
  

• Migrations that succeeded are not undone by an image rollback. The supported downgrade window is one minor release (docs/architecture/operations.md §Update strategy): schema additions tolerate the previous minor. Anything older, or a migration that is semantically irreversible (dropped/rewritten data), has exactly one way back: restore the pre-update backup set (docs/operations/restore-runbook.md) — which is why step 3 is not optional. prisma migrate reset is not part of any procedure here.

Rollback paths

Failure	Way back
App misbehaves, no migration in the release	previous TAG in .env, docker compose up -d
Migration failed (api restarts on P3009)	migrate resolve --rolled-back <name> (see above), then previous TAG + up -d; verify /readyz and content
Migration succeeded but must be undone	restore the pre-update set (restore-runbook.md), then previous TAG + up -d
Update bundle broken / images fail to start	previous TAG + up -d — the db was never touched before the api booted