stwaidele/dorfteich
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases 22 Wiki Activity

Backup sidecar: dump, volume archive, prune, and status #83

New Issue
Closed
opened 2026-07-04 14:52:44 +02:00 by fable-5 · 1 comment
fable-5 commented 2026-07-04 14:52:44 +02:00
Collaborator
Copy Link

Context

ADR 0015 defines the nightly backup set; the sidecar makes it real for every stage and self-hoster.

Scope

Build the dorfteich-backup sidecar image: nightly (cron env-configurable) pg_dump -Fc + uploads/plugins volume tar with shared backup id, prune by retention (env: 30 d default, 7 d Test/Int), status.json (last run, outcome, sizes, backup id) on the backups volume, failure mail via the instance SMTP (direct nodemailer call — the API may be the thing that is broken), restore helper script (restore.sh <backup-id>) implementing the runbook steps; compose integration for all stages.

Acceptance criteria

  • nightly run produces dump + matching archive + updated status.json (time-accelerated test)
  • prune removes only expired sets, never the newest complete one
  • restore.sh on a scratch stack reproduces fixture content (automated test on Test)
  • failed dump → failure mail + status reflects the error

Technical notes

  • ADR 0015, operations.md §Backup.

Dependencies

Depends on #6.

Size: ~2 days

Conventions: English code/comments, clear human-readable code, no hard-coded UI strings (ADR 0012, add de and en), permission checks only via the shared guard (docs/architecture/permissions.md). Read the referenced ADRs before starting.

## Context ADR 0015 defines the nightly backup set; the sidecar makes it real for every stage and self-hoster. ## Scope Build the `dorfteich-backup` sidecar image: nightly (cron env-configurable) `pg_dump -Fc` + uploads/plugins volume tar with shared backup id, prune by retention (env: 30 d default, 7 d Test/Int), `status.json` (last run, outcome, sizes, backup id) on the backups volume, failure mail via the instance SMTP (direct nodemailer call — the API may be the thing that is broken), restore helper script (`restore.sh <backup-id>`) implementing the runbook steps; compose integration for all stages. ## Acceptance criteria - [ ] nightly run produces dump + matching archive + updated status.json (time-accelerated test) - [ ] prune removes only expired sets, never the newest complete one - [ ] restore.sh on a scratch stack reproduces fixture content (automated test on Test) - [ ] failed dump → failure mail + status reflects the error ## Technical notes - ADR 0015, operations.md §Backup. ## Dependencies Depends on #6. **Size**: ~2 days --- *Conventions: English code/comments, clear human-readable code, no hard-coded UI strings (ADR 0012, add `de` **and** `en`), permission checks only via the shared guard (docs/architecture/permissions.md). Read the referenced ADRs before starting.*
fable-5 added this to the M8 — Self-hosting & operations milestone 2026-07-04 14:52:44 +02:00
fable-5 added the
deployment
 label 2026-07-04 14:52:44 +02:00
fable-5 commented 2026-07-11 17:45:00 +02:00
Author
Collaborator
Copy Link

Implemented in 8dbff86 (pipeline green, 8/8 contexts; provisioned and drilled on Test + Int).

New workspace app apps/backup (@dorfteich/backup) — the dorfteich-backup sidecar image (ADR 0015):

  • Nightly run (default 03:00, BACKUP_TIME/TZ env; no cron dependency — one re-armed setTimeout to the next HH:MM): pg_dump -Fc first, then one tar of the uploads + plugins volumes, both under a shared backup id YYYYMMDD-HHMMSS (UTC). Artifacts are staged as .partial and renamed on completion, so a crash never leaves something that looks restorable.
  • Prune by BACKUP_RETENTION_DAYS (default 30; Test/Int set 7 per ADR 0015) — and never the newest complete set, even when expired: after long downtime the last restorable set survives until a new success replaces it.
  • status.json on the new backups volume (atomic write): schemaVersion, lastRun (id, timestamps, duration, outcome, error, sizes) and lastSuccess carried across failed runs — the consumer contract for the #85 freshness check and the #86 admin card.
  • Failure mail directly via nodemailer (the api may be the broken part), texts in the shared mails i18n catalog (de + en, BACKUP_MAIL_LOCALE), recipient BACKUP_MAIL_TO (unset = log only). SMTP resolves exactly like the api: explicit env wins, the wizard-written secret store fills gaps, empty strings count as unset — the pure secret-store helpers moved to @dorfteich/shared for that (api re-exports them, import paths unchanged).
  • Restore: deploy/backup/restore.sh <backup-id> automates the runbook (stop web/api/collab — db stays up —, pg_restore --clean --if-exists + volume untar via the sidecar image, docker compose up -d, wait for /readyz). On-demand backup: docker compose run --rm -e BACKUP_RUN_ONCE=1 backup (exit code = outcome, the hook for #86's manual trigger).

Acceptance criteria

  • nightly run produces dump + matching archive + updated status.json (time-accelerated test) — runner.test.ts drives 30 simulated nights through an injected clock against a real temp filesystem; the real pg_dump/tar path verified locally against a fixture DB and live on both stages (Test dump 61 KB, Int 247 KB).
  • prune removes only expired sets, never the newest complete one — unit-tested (backup-set.test.ts, incl. "newer incomplete set does not shield it") plus the 60-days-downtime scenario in runner.test.ts.
  • restore.sh on a scratch stack reproduces fixture content — locally: fixture DB + files → corrupt → restore → byte-identical; live drill on Test: changed fixture-user's display name, ./restore.sh 20260711-154136, value reverted, /readyz fully green. (The recurring automated monthly drill is #87.)
  • failed dump → failure mail + status reflects the error — verified end-to-end against a local SMTP sink (dead DB → exit 1, status.json outcome failed with lastSuccess preserved, mail delivered); covered in tests including "alert transport down never breaks the run".

CI/CD: backup image added to the PR image-build job, the CD build-push (:sha + :test) and the Int retag. Stages: composes replaced (backups *.bak-pre83), .env + TZ=Europe/Berlin, BACKUP_RETENTION_DAYS=7, BACKUP_MAIL_TO/BACKUP_MAIL_LOCALE; restore.sh deployed to both stage directories. .env.example, operations.md and deployment.md updated.

Out of scope, as planned: BASEL mirror (#84), readyz freshness degradation (#85), admin panel card/trigger + dump download (#86), automated monthly drill (#87).

Implemented in `8dbff86` (pipeline green, 8/8 contexts; provisioned and drilled on Test + Int). **New workspace app `apps/backup` (`@dorfteich/backup`)** — the `dorfteich-backup` sidecar image (ADR 0015): - **Nightly run** (default 03:00, `BACKUP_TIME`/`TZ` env; no cron dependency — one re-armed `setTimeout` to the next HH:MM): `pg_dump -Fc` first, then **one tar of the uploads + plugins volumes**, both under a shared backup id `YYYYMMDD-HHMMSS` (UTC). Artifacts are staged as `.partial` and renamed on completion, so a crash never leaves something that looks restorable. - **Prune** by `BACKUP_RETENTION_DAYS` (default 30; Test/Int set 7 per ADR 0015) — and **never the newest complete set**, even when expired: after long downtime the last restorable set survives until a new success replaces it. - **`status.json`** on the new `backups` volume (atomic write): `schemaVersion`, `lastRun` (id, timestamps, duration, outcome, error, sizes) and `lastSuccess` carried across failed runs — the consumer contract for the #85 freshness check and the #86 admin card. - **Failure mail directly via nodemailer** (the api may be the broken part), texts in the shared `mails` i18n catalog (de + en, `BACKUP_MAIL_LOCALE`), recipient `BACKUP_MAIL_TO` (unset = log only). SMTP resolves exactly like the api: explicit env wins, the wizard-written secret store fills gaps, empty strings count as unset — the pure secret-store helpers moved to `@dorfteich/shared` for that (api re-exports them, import paths unchanged). - **Restore**: `deploy/backup/restore.sh <backup-id>` automates the runbook (stop web/api/collab — db stays up —, `pg_restore --clean --if-exists` + volume untar via the sidecar image, `docker compose up -d`, wait for `/readyz`). On-demand backup: `docker compose run --rm -e BACKUP_RUN_ONCE=1 backup` (exit code = outcome, the hook for #86's manual trigger). **Acceptance criteria** - *nightly run produces dump + matching archive + updated status.json (time-accelerated test)* — `runner.test.ts` drives 30 simulated nights through an injected clock against a real temp filesystem; the real `pg_dump`/`tar` path verified locally against a fixture DB and live on both stages (Test dump 61 KB, Int 247 KB). - *prune removes only expired sets, never the newest complete one* — unit-tested (`backup-set.test.ts`, incl. "newer incomplete set does not shield it") plus the 60-days-downtime scenario in `runner.test.ts`. - *restore.sh on a scratch stack reproduces fixture content* — locally: fixture DB + files → corrupt → restore → byte-identical; **live drill on Test**: changed `fixture-user`'s display name, `./restore.sh 20260711-154136`, value reverted, `/readyz` fully green. (The recurring automated monthly drill is #87.) - *failed dump → failure mail + status reflects the error* — verified end-to-end against a local SMTP sink (dead DB → exit 1, `status.json` outcome `failed` with `lastSuccess` preserved, mail delivered); covered in tests including "alert transport down never breaks the run". **CI/CD**: backup image added to the PR image-build job, the CD build-push (`:sha` + `:test`) and the Int retag. **Stages**: composes replaced (backups `*.bak-pre83`), `.env` + `TZ=Europe/Berlin`, `BACKUP_RETENTION_DAYS=7`, `BACKUP_MAIL_TO`/`BACKUP_MAIL_LOCALE`; `restore.sh` deployed to both stage directories. `.env.example`, operations.md and deployment.md updated. Out of scope, as planned: BASEL mirror (#84), readyz freshness degradation (#85), admin panel card/trigger + dump download (#86), automated monthly drill (#87).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels   
area:auth

   
area:docs

   
area:export

   
area:ops

   
area:storage

   
area:supply-chain

   
auth

Authentication, permissions, quotas

  
backend

API server (apps/api)

  
blocked

Blocked on external dependency/infrastructure

  
collab

Collaboration server (apps/collab)

  
deployment

Docker, CI/CD, hosting

  
docs

Documentation

  
effort:L

   
effort:M

   
effort:S

   
frontend

SPA (apps/web)

  
plugins

Plugin system and SDK

  
qa

Tests, fixtures, e2e

  
vs-nfd

   
vs-nfd:blocker
No Label
area:auth
area:docs
area:export
area:ops
area:storage
area:supply-chain
auth
backend
blocked
collab
deployment
docs
effort:L
effort:M
effort:S
frontend
plugins
qa
vs-nfd
vs-nfd:blocker
Milestone
Clear milestone
No items
No Milestone
M8 — Self-hosting & operations
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: stwaidele/dorfteich#83
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?