dorfteich/deploy/backup-basel.md

Backup mirror to BASEL (issue #84)

Nightly rsync of the Prod backup sets to the private BASEL host over the WireGuard tunnel (ADR 0015) — the operator-level extra beside the admin-configured Nextcloud target (#103). The sidecar mirrors the set files (db-*.dump, files-*.tar.gz) after every successful local run; --delete keeps the remote retention aligned with the local prune, so the newest-complete-set guarantee carries over. Mirror outcome lands in status.json (mirror block, shown on the admin backup card); failures alert through the backup failure mail while local backups continue.

BASEL-side setup (once, as root on BASEL)

A dedicated user with a home under /home/ — deliberately NOT the Debian backup system user (UID 34), whose /var/backups home and nologin shell are documented foot-guns in the operator conventions:

useradd --create-home --shell /bin/bash dorfteich-backup
mkdir -p /home/dorfteich-backup/.ssh
# authorized_keys: the public half of the key generated below
install -m 600 -o dorfteich-backup -g dorfteich-backup authorized_keys \
  /home/dorfteich-backup/.ssh/authorized_keys
chmod 700 /home/dorfteich-backup/.ssh
chown dorfteich-backup:dorfteich-backup /home/dorfteich-backup/.ssh

mkdir -p /home/RAID/BACKUPS/dorfteich-prod
chown dorfteich-backup:dorfteich-backup /home/RAID/BACKUPS/dorfteich-prod
chmod 750 /home/RAID/BACKUPS/dorfteich-prod

Pitfalls (learned on the wochenplan setup):

• The login shell MUST be /bin/bash (or /bin/sh) — with /usr/sbin/nologin sshd rejects every session, including rsync's. Security comes from the key-only login, not from nologin; if you want to lock it down further, prefix the authorized_keys line with command="rsync --server ..." restrictions.
• .ssh must be mode 700 and owned by the user; authorized_keys 600.
• The home directory itself may be root-owned but must not be group/world-writable (sshd's StrictModes).

Stack-side setup (per stage that mirrors)

1. Generate a keypair (on the docker host, never in the repo):

   ssh-keygen -t ed25519 -N '' -C dorfteich-backup-mirror -f backup_mirror_ed25519
   # the .pub half goes into BASEL's authorized_keys (above)
   

2. Put the private key on the secrets volume (the sidecar mounts it read-only at /data/secrets; copy through the api container, which mounts it read-write):

   docker compose cp backup_mirror_ed25519 api:/data/secrets/backup_mirror_ed25519
   docker compose exec api chmod 600 /data/secrets/backup_mirror_ed25519
   shred -u backup_mirror_ed25519
   

3. Configure the stage .env and recreate the sidecar:

   BACKUP_MIRROR_TARGET=dorfteich-backup@172.30.1.10:/home/RAID/BACKUPS/dorfteich-prod/
   BACKUP_MIRROR_SSH_KEY=/data/secrets/backup_mirror_ed25519
   #BACKUP_MIRROR_SSH_PORT=22   # default
   

   docker compose up -d backup
   

4. Verify with an on-demand run — the first mirror pins BASEL's host key (accept-new) into .mirror_known_hosts on the backups volume:

   docker compose run --rm -e BACKUP_RUN_ONCE=1 backup
   docker compose exec backup sh -c \
     'grep -A4 \"mirror\" /backups/status.json'
   # re-run: "transferredFiles": 0 proves idempotency
   

Behaviour

• Mirror runs after the local prune of every successful run (nightly, manual button, BACKUP_RUN_ONCE); a failed local run never mirrors.
• A mirror failure sets status.json → mirror.lastRun.outcome = "failed" (visible on the admin backup card) and sends the backupMirrorFailed alert mail — the local run still counts as succeeded.
• Restoring from the mirror: copy the set pair back into the stack's backups volume and run ./restore.sh <backup-id> — same procedure as the Nextcloud path in docs/operations/restore-runbook.md.
• The mirror carries only the raw set files; status.json and the Nextcloud bundles stay local.