64f2deb40f
6 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
| 18239e2fa9 |
#221: offline update path incl. migrations, rehearsed with rollback
All checks were successful
CD / Deploy to Test (push) Successful in 12s
CD / Smoke tests against Test (push) Successful in 1m15s
CD / Promote to Int (push) Successful in 12s
CI / Lint, typecheck, test (push) Successful in 6m19s
CI / Build container images (push) Has been skipped
CI / Auth e2e pack (push) Successful in 8m13s
CI / Import/export fidelity gate (push) Successful in 54s
CI / Lint, typecheck, test (pull_request) Successful in 6m20s
CI / Build container images (pull_request) Successful in 1m12s
CI / Auth e2e pack (pull_request) Successful in 8m24s
CI / Import/export fidelity gate (pull_request) Successful in 58s
CD / Build and push images (push) Successful in 22s
Adds docs/operations/update-runbook.md (obtain, verify by digest, back up, apply, verify, roll back) with the migration behaviour stated explicitly: a failed migration rolls back its own transaction but is recorded in _prisma_migrations and blocks every further migrate deploy (P3009) — including a re-deployed old image — until migrate resolve --rolled-back; semantically irreversible migrations have exactly one way back, the pre-update backup set. No rolling updates on a compose stage. Rehearsed in the isolated environment of #220: regular update to a v2 image set, then a deliberate failed-update (P3018 division by zero, schema change proven rolled back) with image-rollback-alone shown insufficient and the documented recovery executed. Protocol: docs/vs-nfd/98-update-rollback-protokoll.md. ADR 0024 decisions 5+6 recorded as executed; operations handbook and restore runbook updated; plan checkbox P1-3 ticked. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AUtYMxwTCMHG9mVHnwbFg8 |
|||
| 4f6596e8a2 |
#288: reset schema before pg_restore — partitioned tables broke --clean
Some checks failed
CI / Lint, typecheck, test (pull_request) Successful in 6m48s
CI / Build container images (pull_request) Successful in 1m46s
CI / Auth e2e pack (pull_request) Successful in 8m22s
CI / Import/export fidelity gate (pull_request) Successful in 1m8s
CD / Deploy to Test (push) Blocked by required conditions
CD / Smoke tests against Test (push) Blocked by required conditions
CD / Promote to Int (push) Blocked by required conditions
CI / Auth e2e pack (push) Blocked by required conditions
CI / Import/export fidelity gate (push) Blocked by required conditions
CI / Build container images (push) Blocked by required conditions
CD / Build and push images (push) Has been cancelled
CI / Lint, typecheck, test (push) Has been cancelled
Since #224 read_events is partitioned; the dump carries per-partition primary keys as own entries, and pg_restore --clean emitted DROP CONSTRAINT against inherited constraints, which PostgreSQL refuses. The restore then reported FAILED although the content was restored. Dropping and recreating the public schema first makes every --clean drop a no-op and the restore faithful: objects created after the backup no longer survive. Verified in the isolated environment of #220 (set 20260731-132200, exit 0, readyz green). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AUtYMxwTCMHG9mVHnwbFg8 |
|||
| 69e7b1406b |
Move the stage directories to /srv/DOCKER (host consolidation)
Some checks failed
CD / Build and push images (push) Successful in 1m9s
CI / Lint, typecheck, test (push) Failing after 1m11s
CI / Auth e2e pack (push) Has been skipped
CI / Import/export fidelity gate (push) Has been skipped
CI / Build container images (push) Has been skipped
CD / Deploy to Test (push) Successful in 9s
CD / Smoke tests against Test (push) Successful in 1m13s
CD / Promote to Int (push) Successful in 11s
ONE consolidated every Docker stack under /srv/DOCKER (BASEL convention, tracked in stwaidele/infrastructure-one); the three Dorfteich stages follow. Deploy targets in cd.yml (test/int) and prod-deploy.yml plus the docs now point at /srv/DOCKER/dorfteich-<stage>. Data lives in named volumes keyed by the unchanged compose project name, so the directory move carries no data migration. The go-live checklist's uptime-kuma path already lives under /srv/DOCKER — the doc just catches up. Deliberately committed together with the host-side move: this commit must not deploy before the directories exist at the new path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
|||
| 5cef359b8f |
Nextcloud backup target: admin-configured, manual + scheduled uploads, in-app restore (#103)
All checks were successful
CI / Lint, typecheck, test (push) Successful in 3m45s
CD / Build and push images (push) Successful in 3m49s
CI / Build container images (push) Has been skipped
CD / Deploy to Test (push) Successful in 9s
CD / Smoke tests against Test (push) Successful in 1m18s
CD / Promote to Int (push) Successful in 11s
CI / Auth e2e pack (push) Successful in 5m35s
CI / Import/export fidelity gate (push) Successful in 47s
Off-host backups for every self-hoster, configured entirely in the admin UI — supersedes the host-specific mirror plan behind #84. shared: - webdav.ts (new package entry like token-crypto): minimal WebDAV client with basic auth — PROPFIND (tolerant multistatus parser), MKCOL, PUT (streamed), GET, DELETE; Nextcloud DAV path derived from the plain server URL, explicit DAV bases pass through - backup-status.ts: additive remote-upload status in status.json, the restore-status.json contract (running/succeeded/failed + staleness bound), the backup_command/backup_maintenance NOTIFY channels, and the one-bundle-per-set naming (dorfteich-backup-<id>.tar.gz) - backup-set.ts moved here from apps/backup (api lists local sets) backup sidecar: - reads the backup.* instance settings directly from the database (admin changes apply next run; local retention row overrides the env) and the app password from the secret store - after each successful set: bundle dump + files archive + manifest into ONE self-contained tar.gz, upload via WebDAV per schedule (off/daily/weekly; manual runs always upload), prune remote bundles — never the newest — and record the outcome in status.json; upload failures alert via a new backupUploadFailed mail (de+en) - command listener on backup_command (run / restore) with a serial queue against the nightly timer - restore orchestrator: restore-status.json → maintenance NOTIFY → grace → (remote: download + manifest-verify bundle) → terminate other DB connections → shared perform-restore path (same code as restore.sh) → final status + maintenance exit api: - MaintenanceGuard (global, registered before the setup gate): 503 maintenance_mode while restore-status says running; health endpoints and the new public GET /backup/restore-status stay exempt; a stale running state (crashed sidecar) unblocks after 30 min - MaintenanceStateService watches the file and restarts the api after a successful restore (fresh caches, migrate-on-start for older dumps); main.ts refuses to touch the database while a restore runs — a container restarting mid-restore must not race pg_restore with migrate deploy - worker sweeps (conversion, mail outbox, scheduler) catch transient database failures instead of dying on an unhandled rejection — the restore's connection termination crashed the api in verification - backup admin endpoints under /admin/system/backup: settings (live connection test before save, password write-only into the secret store), nextcloud/test, sets (local via the ro backups mount + remote via WebDAV), run + restore (type-to-confirm backstop, source validation) — commands travel as NOTIFY payloads; audit actions backup.settings_changed/run_triggered/restore_requested - readyz: new warning-level backup_remote check while a target is configured (26 h daily / 170 h weekly bound) collab: - maintenance listener: on enter, persist + close every live session and refuse new connections until exit (failsafe timeout 30 min) — no in-memory document may write pre-restore content back afterwards web: - Admin → System backup section: status card with remote facts and a "Back up now" button, the Nextcloud settings form with test button, and the restore picker (local + remote sets, type-to-confirm) - global maintenance screen: any 503 maintenance_mode flips the SPA to a status page polling the exempt endpoint, reloading when the instance returns Verified end-to-end against a live stack (fresh DB, native api + sidecar, fake WebDAV server): configure → test → manual backup → bundle upload → readyz/sets/status surfaces → remote restore with maintenance gate, marker rollback and api restart; suites: shared 21, backup 9, collab 11, api 58 files green, lint + i18n:check + typecheck clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EwZ4jR4KFAPvpjWevfUGX1 |
|||
| 6e99cb35fc |
Trigger the restore drill on demand via drill-* tags (#87)
All checks were successful
Restore drill / Restore the latest backup into a scratch stack (push) Successful in 28s
CD / Build and push images (push) Successful in 1m4s
CD / Deploy to Test (push) Successful in 11s
CD / Smoke tests against Test (push) Successful in 1m9s
CD / Promote to Int (push) Successful in 12s
CI / Lint, typecheck, test (push) Successful in 3m19s
CI / Build container images (push) Has been skipped
CI / Import/export fidelity gate (push) Successful in 45s
CI / Auth e2e pack (push) Successful in 5m9s
Gitea 1.22 cannot dispatch workflows through the API or UI (that lands in 1.23), so pushing a drill-* tag is the on-demand path next to the monthly schedule. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EwZ4jR4KFAPvpjWevfUGX1 |
|||
| d95c18e9e8 |
Automate the monthly restore drill with a scratch-stack workflow (#87)
All checks were successful
CD / Build and push images (push) Successful in 1m5s
CD / Deploy to Test (push) Successful in 10s
CD / Smoke tests against Test (push) Successful in 1m7s
CD / Promote to Int (push) Successful in 11s
CI / Lint, typecheck, test (push) Successful in 3m18s
CI / Build container images (push) Has been skipped
CI / Auth e2e pack (push) Successful in 5m14s
CI / Import/export fidelity gate (push) Successful in 45s
New scheduled workflow (monthly + on demand) runs deploy/backup/drill.sh: it reads the drilled stage's backups volume strictly read-only, restores the latest successful set into a throwaway Postgres and volumes under a unique drill prefix via the backup image's restore path, boots the api against the result, and verifies readyz (database + migrations), row counts, rendered content in the page cache, a public API request, and a media byte-check against the attachments table — then tears everything down, also on failure. Each run reports its outcome as a comment on the pinned "Restore drills" issue (#98). docs/operations/restore-runbook.md carries the manual procedure, which doubles as the Prod relocation path; pre-go-live the drill restores the Test set (switch the source volume at go-live, #89 — off-host fetch from the BASEL mirror stays with #84). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EwZ4jR4KFAPvpjWevfUGX1 |
