Claude Fable 5
52192eb05f
All checks were successful
CD / Build and push images (push) Successful in 3m51s
CI / Lint, typecheck, test (push) Successful in 4m5s
CD / Deploy to Test (push) Successful in 11s
CI / Build container images (push) Has been skipped
CD / Smoke tests against Test (push) Successful in 1m11s
CD / Promote to Int (push) Successful in 12s
CI / Auth e2e pack (push) Successful in 5m52s
CI / Import/export fidelity gate (push) Successful in 47s
The operator-level extra beside the admin-configured Nextcloud target (#103), unblocked now that the ONE→BASEL tunnel is stable again. - sidecar: optional mirror step (mirror.ts) driven purely by env — BACKUP_MIRROR_TARGET (rsync-over-ssh), BACKUP_MIRROR_SSH_KEY (private key on the secrets volume, never in image or repo), BACKUP_MIRROR_SSH_PORT. Runs after the prune of every successful run, so --delete aligns the remote retention with the local one (the newest-complete-set guarantee carries over). Only set files travel (db-*.dump, files-*.tar.gz); status files and bundles stay local. Host key pinned via accept-new into .mirror_known_hosts on the backups volume; fixed remote modes (dirs 750, files 640, symbolic --chmod — octal needs rsync ≥ 3, macOS dev machines ship 2.6.9). rsync + openssh-client added to the sidecar image. - status: additive `mirror` block in status.json (outcome, transferred count, lastSuccessAt carried across failures) — shown on the admin backup card; failures alert via a new backupMirrorFailed mail (de+en) while the local run still counts as succeeded. - deploy/backup-basel.md: complete BASEL-side walkthrough — dedicated user dorfteich-backup with a /home/ home and a bash login shell, explicitly avoiding the Debian backup-user (UID 34) pitfalls (nologin shell rejects rsync sessions, /var/backups home), key placement through the api container onto the secrets volume, .env values, on-demand verification. - tests: rsync-arg/stats-parsing units plus an integration suite against the real rsync binary (local target; skips where rsync is absent) — transfer, idempotent re-run (0 files), retention alignment, failure path carrying lastSuccessAt. Verified live against the real BASEL host from a native sidecar run: initial transfer, host-key pinning, retention alignment after a local prune, idempotency, and the failure path (surfaced in status.json while the local run stayed green). BASEL side provisioned per the doc. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EwZ4jR4KFAPvpjWevfUGX1
96 lines
3.9 KiB
Markdown
96 lines
3.9 KiB
Markdown
# Backup mirror to BASEL (issue #84)
|
|
|
|
Nightly rsync of the Prod backup sets to the private BASEL host over the
|
|
WireGuard tunnel (ADR 0015) — the operator-level extra beside the
|
|
admin-configured Nextcloud target (#103). The sidecar mirrors the set
|
|
files (`db-*.dump`, `files-*.tar.gz`) after every successful local run;
|
|
`--delete` keeps the remote retention aligned with the local prune, so
|
|
the newest-complete-set guarantee carries over. Mirror outcome lands in
|
|
`status.json` (`mirror` block, shown on the admin backup card); failures
|
|
alert through the backup failure mail while local backups continue.
|
|
|
|
## BASEL-side setup (once, as root on BASEL)
|
|
|
|
A **dedicated user with a home under `/home/`** — deliberately NOT the
|
|
Debian `backup` system user (UID 34), whose `/var/backups` home and
|
|
`nologin` shell are documented foot-guns in the operator conventions:
|
|
|
|
```sh
|
|
useradd --create-home --shell /bin/bash dorfteich-backup
|
|
mkdir -p /home/dorfteich-backup/.ssh
|
|
# authorized_keys: the public half of the key generated below
|
|
install -m 600 -o dorfteich-backup -g dorfteich-backup authorized_keys \
|
|
/home/dorfteich-backup/.ssh/authorized_keys
|
|
chmod 700 /home/dorfteich-backup/.ssh
|
|
chown dorfteich-backup:dorfteich-backup /home/dorfteich-backup/.ssh
|
|
|
|
mkdir -p /home/RAID/BACKUPS/dorfteich-prod
|
|
chown dorfteich-backup:dorfteich-backup /home/RAID/BACKUPS/dorfteich-prod
|
|
chmod 750 /home/RAID/BACKUPS/dorfteich-prod
|
|
```
|
|
|
|
**Pitfalls (learned on the wochenplan setup):**
|
|
|
|
- The login shell MUST be `/bin/bash` (or `/bin/sh`) — with
|
|
`/usr/sbin/nologin` sshd rejects every session, including rsync's.
|
|
Security comes from the key-only login, not from nologin; if you want
|
|
to lock it down further, prefix the `authorized_keys` line with
|
|
`command="rsync --server ..."` restrictions.
|
|
- `.ssh` must be mode 700 and owned by the user; `authorized_keys` 600.
|
|
- The home directory itself may be root-owned but must not be
|
|
group/world-writable (sshd's StrictModes).
|
|
|
|
## Stack-side setup (per stage that mirrors)
|
|
|
|
1. **Generate a keypair** (on the docker host, never in the repo):
|
|
|
|
```sh
|
|
ssh-keygen -t ed25519 -N '' -C dorfteich-backup-mirror -f backup_mirror_ed25519
|
|
# the .pub half goes into BASEL's authorized_keys (above)
|
|
```
|
|
|
|
2. **Put the private key on the `secrets` volume** (the sidecar mounts it
|
|
read-only at `/data/secrets`; copy through the api container, which
|
|
mounts it read-write):
|
|
|
|
```sh
|
|
docker compose cp backup_mirror_ed25519 api:/data/secrets/backup_mirror_ed25519
|
|
docker compose exec api chmod 600 /data/secrets/backup_mirror_ed25519
|
|
shred -u backup_mirror_ed25519
|
|
```
|
|
|
|
3. **Configure the stage `.env`** and recreate the sidecar:
|
|
|
|
```sh
|
|
BACKUP_MIRROR_TARGET=dorfteich-backup@172.30.1.10:/home/RAID/BACKUPS/dorfteich-prod/
|
|
BACKUP_MIRROR_SSH_KEY=/data/secrets/backup_mirror_ed25519
|
|
#BACKUP_MIRROR_SSH_PORT=22 # default
|
|
```
|
|
|
|
```sh
|
|
docker compose up -d backup
|
|
```
|
|
|
|
4. **Verify** with an on-demand run — the first mirror pins BASEL's host
|
|
key (`accept-new`) into `.mirror_known_hosts` on the backups volume:
|
|
|
|
```sh
|
|
docker compose run --rm -e BACKUP_RUN_ONCE=1 backup
|
|
docker compose exec backup sh -c \
|
|
'grep -A4 \"mirror\" /backups/status.json'
|
|
# re-run: "transferredFiles": 0 proves idempotency
|
|
```
|
|
|
|
## Behaviour
|
|
|
|
- Mirror runs after the local prune of every **successful** run (nightly,
|
|
manual button, `BACKUP_RUN_ONCE`); a failed local run never mirrors.
|
|
- A mirror failure sets `status.json → mirror.lastRun.outcome = "failed"`
|
|
(visible on the admin backup card) and sends the `backupMirrorFailed`
|
|
alert mail — the local run still counts as succeeded.
|
|
- Restoring from the mirror: copy the set pair back into the stack's
|
|
backups volume and run `./restore.sh <backup-id>` — same procedure as
|
|
the Nextcloud path in `docs/operations/restore-runbook.md`.
|
|
- The mirror carries only the raw set files; `status.json` and the
|
|
Nextcloud bundles stay local.
|