Backup mirror to BASEL: rsync of the sets after every successful run (#84)
All checks were successful
CD / Build and push images (push) Successful in 3m51s
CI / Lint, typecheck, test (push) Successful in 4m5s
CD / Deploy to Test (push) Successful in 11s
CI / Build container images (push) Has been skipped
CD / Smoke tests against Test (push) Successful in 1m11s
CD / Promote to Int (push) Successful in 12s
CI / Auth e2e pack (push) Successful in 5m52s
CI / Import/export fidelity gate (push) Successful in 47s
All checks were successful
CD / Build and push images (push) Successful in 3m51s
CI / Lint, typecheck, test (push) Successful in 4m5s
CD / Deploy to Test (push) Successful in 11s
CI / Build container images (push) Has been skipped
CD / Smoke tests against Test (push) Successful in 1m11s
CD / Promote to Int (push) Successful in 12s
CI / Auth e2e pack (push) Successful in 5m52s
CI / Import/export fidelity gate (push) Successful in 47s
The operator-level extra beside the admin-configured Nextcloud target (#103), unblocked now that the ONE→BASEL tunnel is stable again. - sidecar: optional mirror step (mirror.ts) driven purely by env — BACKUP_MIRROR_TARGET (rsync-over-ssh), BACKUP_MIRROR_SSH_KEY (private key on the secrets volume, never in image or repo), BACKUP_MIRROR_SSH_PORT. Runs after the prune of every successful run, so --delete aligns the remote retention with the local one (the newest-complete-set guarantee carries over). Only set files travel (db-*.dump, files-*.tar.gz); status files and bundles stay local. Host key pinned via accept-new into .mirror_known_hosts on the backups volume; fixed remote modes (dirs 750, files 640, symbolic --chmod — octal needs rsync ≥ 3, macOS dev machines ship 2.6.9). rsync + openssh-client added to the sidecar image. - status: additive `mirror` block in status.json (outcome, transferred count, lastSuccessAt carried across failures) — shown on the admin backup card; failures alert via a new backupMirrorFailed mail (de+en) while the local run still counts as succeeded. - deploy/backup-basel.md: complete BASEL-side walkthrough — dedicated user dorfteich-backup with a /home/ home and a bash login shell, explicitly avoiding the Debian backup-user (UID 34) pitfalls (nologin shell rejects rsync sessions, /var/backups home), key placement through the api container onto the secrets volume, .env values, on-demand verification. - tests: rsync-arg/stats-parsing units plus an integration suite against the real rsync binary (local target; skips where rsync is absent) — transfer, idempotent re-run (0 files), retention alignment, failure path carrying lastSuccessAt. Verified live against the real BASEL host from a native sidecar run: initial transfer, host-key pinning, retention alignment after a local prune, idempotency, and the failure path (surfaced in status.json while the local run stayed green). BASEL side provisioned per the doc. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EwZ4jR4KFAPvpjWevfUGX1
This commit is contained in:
parent
04e21a0aac
commit
52192eb05f
@ -22,8 +22,9 @@ ENV NODE_ENV=production APP_VERSION=${APP_VERSION} \
|
||||
BACKUPS_DIR=/backups UPLOADS_DIR=/data/uploads PLUGINS_DIR=/data/plugins \
|
||||
SECRETS_FILE=/data/secrets/secrets.env
|
||||
# pg_dump/pg_restore matching the stack's postgres:17 server, GNU tar for the
|
||||
# volume archives, tzdata so BACKUP_TIME honors a configured TZ.
|
||||
RUN apk add --no-cache postgresql17-client tar tzdata \
|
||||
# volume archives, tzdata so BACKUP_TIME honors a configured TZ, and
|
||||
# rsync + ssh for the optional off-host mirror (issue #84).
|
||||
RUN apk add --no-cache postgresql17-client tar tzdata rsync openssh-client \
|
||||
&& mkdir -p /backups && chown node:node /backups
|
||||
WORKDIR /app
|
||||
COPY --from=build --chown=node:node /out /app
|
||||
|
||||
@ -8,6 +8,7 @@ import { createArchive } from './archive.js';
|
||||
import { createCommandListener } from './commands.js';
|
||||
import { loadBackupEnv } from './config.js';
|
||||
import { sendFailureMail } from './mail.js';
|
||||
import { mirrorSets, resolveMirrorConfig } from './mirror.js';
|
||||
import { performRestore } from './perform-restore.js';
|
||||
import { pgDump } from './pg.js';
|
||||
import { fetchRemoteSet, resolveRemoteTarget, uploadDue, uploadSet } from './remote.js';
|
||||
@ -43,6 +44,7 @@ function readSecrets(): Record<string, string> {
|
||||
async function buildRunnerDeps(trigger: 'scheduled' | 'manual'): Promise<RunnerDeps> {
|
||||
const settings = await readBackupDbSettings(env.DATABASE_URL);
|
||||
const target = resolveRemoteTarget(settings, readSecrets());
|
||||
const mirrorConfig = resolveMirrorConfig(env);
|
||||
return {
|
||||
backupsDir: env.BACKUPS_DIR,
|
||||
retentionDays: settings.localRetentionDays ?? env.BACKUP_RETENTION_DAYS,
|
||||
@ -76,6 +78,17 @@ async function buildRunnerDeps(trigger: 'scheduled' | 'manual'): Promise<RunnerD
|
||||
});
|
||||
}
|
||||
: undefined,
|
||||
mirror: mirrorConfig
|
||||
? ({ previous }) =>
|
||||
mirrorSets({
|
||||
env,
|
||||
config: mirrorConfig,
|
||||
backupsDir: env.BACKUPS_DIR,
|
||||
previous,
|
||||
now: () => new Date(),
|
||||
log,
|
||||
})
|
||||
: undefined,
|
||||
log,
|
||||
};
|
||||
}
|
||||
|
||||
@ -28,7 +28,7 @@ export interface RenderedFailureMail {
|
||||
const CATALOGS = { de: deMails, en: enMails } as const;
|
||||
|
||||
/** Both alert mails share one key shape; the namespace picks the texts. */
|
||||
type AlertNamespace = 'backupFailed' | 'backupUploadFailed';
|
||||
type AlertNamespace = 'backupFailed' | 'backupUploadFailed' | 'backupMirrorFailed';
|
||||
|
||||
function t(
|
||||
locale: 'de' | 'en',
|
||||
@ -100,6 +100,17 @@ export async function sendUploadFailureMail(
|
||||
return sendAlertMail(env, renderUploadFailureMail(input, env.BACKUP_MAIL_LOCALE, label(env)));
|
||||
}
|
||||
|
||||
/** Same delivery path for the rsync-mirror alert (issue #84). */
|
||||
export async function sendMirrorFailureMail(
|
||||
env: BackupEnv,
|
||||
input: FailureMailInput,
|
||||
): Promise<boolean> {
|
||||
return sendAlertMail(
|
||||
env,
|
||||
renderAlertMail('backupMirrorFailed', input, env.BACKUP_MAIL_LOCALE, label(env)),
|
||||
);
|
||||
}
|
||||
|
||||
function label(env: BackupEnv): string {
|
||||
return env.BACKUP_INSTANCE_LABEL || 'Dorfteich';
|
||||
}
|
||||
|
||||
148
apps/backup/src/mirror.test.ts
Normal file
148
apps/backup/src/mirror.test.ts
Normal file
@ -0,0 +1,148 @@
|
||||
import { execFileSync } from 'node:child_process';
|
||||
import { existsSync } from 'node:fs';
|
||||
import { mkdtemp, readdir, rm, writeFile } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
|
||||
import type { BackupEnv } from '@dorfteich/shared';
|
||||
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
|
||||
|
||||
import {
|
||||
buildRsyncArgs,
|
||||
mirrorSets,
|
||||
parseTransferredFiles,
|
||||
resolveMirrorConfig,
|
||||
} from './mirror.js';
|
||||
|
||||
const silentLog = { info: () => {}, warn: () => {}, error: () => {} };
|
||||
const noMailEnv = { BACKUP_MAIL_TO: undefined } as unknown as BackupEnv;
|
||||
|
||||
function hasRsync(): boolean {
|
||||
try {
|
||||
execFileSync('rsync', ['--version'], { stdio: 'ignore' });
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
describe('resolveMirrorConfig', () => {
|
||||
it('requires both target and key; port defaults to 22', () => {
|
||||
const base = { BACKUP_MIRROR_SSH_PORT: 22 } as unknown as BackupEnv;
|
||||
expect(resolveMirrorConfig(base)).toBeNull();
|
||||
expect(
|
||||
resolveMirrorConfig({ ...base, BACKUP_MIRROR_TARGET: 'u@h:/x/' } as BackupEnv),
|
||||
).toBeNull();
|
||||
expect(
|
||||
resolveMirrorConfig({
|
||||
...base,
|
||||
BACKUP_MIRROR_TARGET: 'u@h:/x/',
|
||||
BACKUP_MIRROR_SSH_KEY: '/data/secrets/key',
|
||||
} as BackupEnv),
|
||||
).toEqual({ target: 'u@h:/x/', sshKeyFile: '/data/secrets/key', sshPort: 22 });
|
||||
});
|
||||
});
|
||||
|
||||
describe('buildRsyncArgs', () => {
|
||||
it('transfers only set files, deletes within the filter, uses the pinned ssh', () => {
|
||||
const args = buildRsyncArgs(
|
||||
{ target: 'u@h:/backups/', sshKeyFile: '/k', sshPort: 2222 },
|
||||
'/backups',
|
||||
);
|
||||
expect(args).toContain('--delete');
|
||||
expect(args).toContain('--include=db-*.dump');
|
||||
expect(args).toContain('--include=files-*.tar.gz');
|
||||
expect(args).toContain('--exclude=*');
|
||||
const ssh = args[args.indexOf('-e') + 1]!;
|
||||
expect(ssh).toContain('-i /k');
|
||||
expect(ssh).toContain('-p 2222');
|
||||
expect(ssh).toContain('BatchMode=yes');
|
||||
expect(args.at(-2)).toBe('/backups/');
|
||||
expect(args.at(-1)).toBe('u@h:/backups/');
|
||||
});
|
||||
});
|
||||
|
||||
describe('parseTransferredFiles', () => {
|
||||
it('reads the rsync stats line, tolerating thousands separators', () => {
|
||||
expect(parseTransferredFiles('Number of regular files transferred: 4\n')).toBe(4);
|
||||
expect(parseTransferredFiles('Number of regular files transferred: 1,234\n')).toBe(1234);
|
||||
expect(parseTransferredFiles('no stats here')).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe.skipIf(!hasRsync())('mirrorSets (real rsync, local target)', () => {
|
||||
let source: string;
|
||||
let target: string;
|
||||
let keyFile: string;
|
||||
|
||||
beforeEach(async () => {
|
||||
source = await mkdtemp(join(tmpdir(), 'dorfteich-mirror-src-'));
|
||||
target = await mkdtemp(join(tmpdir(), 'dorfteich-mirror-dst-'));
|
||||
// rsync to a local path ignores -e ssh; a dummy key satisfies the check.
|
||||
keyFile = join(source, '.dummy-key');
|
||||
await writeFile(keyFile, 'dummy');
|
||||
await writeFile(join(source, 'db-20260712-030000.dump'), 'dump-1');
|
||||
await writeFile(join(source, 'files-20260712-030000.tar.gz'), 'files-1');
|
||||
await writeFile(join(source, 'db-20260711-030000.dump'), 'dump-0');
|
||||
await writeFile(join(source, 'files-20260711-030000.tar.gz'), 'files-0');
|
||||
await writeFile(join(source, 'status.json'), '{}');
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
await rm(source, { recursive: true, force: true });
|
||||
await rm(target, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
const run = () =>
|
||||
mirrorSets({
|
||||
env: noMailEnv,
|
||||
config: { target: `${target}/`, sshKeyFile: keyFile, sshPort: 22 },
|
||||
backupsDir: source,
|
||||
previous: undefined,
|
||||
now: () => new Date('2026-07-12T03:05:00Z'),
|
||||
log: silentLog,
|
||||
});
|
||||
|
||||
it('transfers set files only, is idempotent, and aligns retention', async () => {
|
||||
const first = await run();
|
||||
expect(first.lastRun.outcome).toBe('succeeded');
|
||||
expect(first.lastRun.transferredFiles).toBe(4);
|
||||
expect(first.lastSuccessAt).not.toBeNull();
|
||||
expect((await readdir(target)).sort()).toEqual([
|
||||
'db-20260711-030000.dump',
|
||||
'db-20260712-030000.dump',
|
||||
'files-20260711-030000.tar.gz',
|
||||
'files-20260712-030000.tar.gz',
|
||||
]);
|
||||
|
||||
// Idempotent re-run: nothing travels.
|
||||
const second = await run();
|
||||
expect(second.lastRun.outcome).toBe('succeeded');
|
||||
expect(second.lastRun.transferredFiles).toBe(0);
|
||||
|
||||
// A locally pruned set disappears remotely too (retention alignment).
|
||||
await rm(join(source, 'db-20260711-030000.dump'));
|
||||
await rm(join(source, 'files-20260711-030000.tar.gz'));
|
||||
await run();
|
||||
expect((await readdir(target)).sort()).toEqual([
|
||||
'db-20260712-030000.dump',
|
||||
'files-20260712-030000.tar.gz',
|
||||
]);
|
||||
});
|
||||
|
||||
it('reports a failure without throwing and carries the last success', async () => {
|
||||
const good = await run();
|
||||
const failed = await mirrorSets({
|
||||
env: noMailEnv,
|
||||
config: { target: `${target}/`, sshKeyFile: '/nonexistent-key', sshPort: 22 },
|
||||
backupsDir: source,
|
||||
previous: good,
|
||||
now: () => new Date('2026-07-12T03:10:00Z'),
|
||||
log: silentLog,
|
||||
});
|
||||
expect(failed.lastRun.outcome).toBe('failed');
|
||||
expect(failed.lastRun.error).toContain('not found');
|
||||
expect(failed.lastSuccessAt).toBe(good.lastSuccessAt);
|
||||
expect(existsSync(join(target, 'db-20260712-030000.dump'))).toBe(true);
|
||||
});
|
||||
});
|
||||
130
apps/backup/src/mirror.ts
Normal file
130
apps/backup/src/mirror.ts
Normal file
@ -0,0 +1,130 @@
|
||||
import { execFile } from 'node:child_process';
|
||||
import { existsSync } from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
import { promisify } from 'node:util';
|
||||
|
||||
import type { BackupEnv, BackupMirrorStatus } from '@dorfteich/shared';
|
||||
|
||||
import { sendMirrorFailureMail } from './mail.js';
|
||||
import type { RemoteLogger } from './remote.js';
|
||||
|
||||
const execFileAsync = promisify(execFile);
|
||||
|
||||
/**
|
||||
* The rsync mirror to a private host (issue #84, ADR 0015) — the operator
|
||||
* extra beside the admin-configured Nextcloud target (#103). After every
|
||||
* successful local run the set artifacts are rsynced to
|
||||
* `BACKUP_MIRROR_TARGET`; `--delete` keeps the remote retention aligned
|
||||
* with the local prune (the newest-set guarantee therefore carries over).
|
||||
* Only set files travel — status files and staging artifacts stay local.
|
||||
* rsync's delta transfer makes re-runs idempotent (0 files transferred).
|
||||
*/
|
||||
|
||||
export interface MirrorConfig {
|
||||
target: string;
|
||||
sshKeyFile: string;
|
||||
sshPort: number;
|
||||
}
|
||||
|
||||
/** The mirror configuration, or null when the env does not enable it. */
|
||||
export function resolveMirrorConfig(env: BackupEnv): MirrorConfig | null {
|
||||
if (!env.BACKUP_MIRROR_TARGET || !env.BACKUP_MIRROR_SSH_KEY) return null;
|
||||
return {
|
||||
target: env.BACKUP_MIRROR_TARGET,
|
||||
sshKeyFile: env.BACKUP_MIRROR_SSH_KEY,
|
||||
sshPort: env.BACKUP_MIRROR_SSH_PORT,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* The rsync invocation: only complete-set artifacts (and the remote bundle
|
||||
* naming is local-only, so just dumps + archives), `--delete` inside that
|
||||
* filter for retention alignment. The known-hosts file lives on the
|
||||
* backups volume so the host key pins across container recreations;
|
||||
* `accept-new` covers the very first contact (inside the WireGuard tunnel).
|
||||
*/
|
||||
export function buildRsyncArgs(config: MirrorConfig, backupsDir: string): string[] {
|
||||
const ssh = [
|
||||
'ssh',
|
||||
`-i ${config.sshKeyFile}`,
|
||||
`-p ${config.sshPort}`,
|
||||
'-o StrictHostKeyChecking=accept-new',
|
||||
`-o UserKnownHostsFile=${join(backupsDir, '.mirror_known_hosts')}`,
|
||||
'-o BatchMode=yes',
|
||||
].join(' ');
|
||||
return [
|
||||
'--archive',
|
||||
// Fixed modes on the mirror host (dirs 750, files 640) — `--archive`
|
||||
// would otherwise copy the container-side modes onto the target.
|
||||
// Symbolic form: octal `--chmod` needs rsync ≥ 3, which not every
|
||||
// dev machine has (macOS ships 2.6.9); the symbolic one works on both.
|
||||
'--chmod=Du=rwx,Dg=rx,Do-rwx,Fu=rw,Fg=r,Fo-rwx',
|
||||
'--delete',
|
||||
'--stats',
|
||||
'--include=db-*.dump',
|
||||
'--include=files-*.tar.gz',
|
||||
'--exclude=*',
|
||||
'-e',
|
||||
ssh,
|
||||
`${backupsDir}/`,
|
||||
config.target,
|
||||
];
|
||||
}
|
||||
|
||||
/** The transferred-file count from `rsync --stats` output. rsync 3 prints
|
||||
* "Number of regular files transferred", the ancient 2.6 (macOS) drops
|
||||
* "regular" — accept both. */
|
||||
export function parseTransferredFiles(stats: string): number | undefined {
|
||||
const match = /Number of (?:regular )?files transferred:\s*([\d,.]+)/.exec(stats);
|
||||
if (!match) return undefined;
|
||||
return Number(match[1]!.replace(/[,.]/g, ''));
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs one mirror pass and returns the new mirror status. Failures are
|
||||
* reported in the status and alert by mail — never thrown: the local
|
||||
* backup succeeded and must count (issue #84 acceptance criteria).
|
||||
*/
|
||||
export async function mirrorSets(deps: {
|
||||
env: BackupEnv;
|
||||
config: MirrorConfig;
|
||||
backupsDir: string;
|
||||
previous: BackupMirrorStatus | undefined;
|
||||
now(): Date;
|
||||
log: RemoteLogger;
|
||||
}): Promise<BackupMirrorStatus> {
|
||||
const previousSuccessAt = deps.previous?.lastSuccessAt ?? null;
|
||||
try {
|
||||
if (!existsSync(deps.config.sshKeyFile)) {
|
||||
throw new Error(`mirror ssh key not found: ${deps.config.sshKeyFile}`);
|
||||
}
|
||||
const { stdout } = await execFileAsync('rsync', buildRsyncArgs(deps.config, deps.backupsDir), {
|
||||
maxBuffer: 16 * 1024 * 1024,
|
||||
});
|
||||
const finishedAt = deps.now().toISOString();
|
||||
const transferredFiles = parseTransferredFiles(stdout);
|
||||
deps.log.info({ target: deps.config.target, transferredFiles }, 'mirror run succeeded');
|
||||
return {
|
||||
lastRun: { finishedAt, outcome: 'succeeded', transferredFiles },
|
||||
lastSuccessAt: finishedAt,
|
||||
};
|
||||
} catch (error) {
|
||||
const stderr = (error as { stderr?: string }).stderr?.trim();
|
||||
const message = stderr || (error instanceof Error ? error.message : String(error));
|
||||
deps.log.error({ target: deps.config.target, error: message }, 'mirror run failed');
|
||||
try {
|
||||
const sent = await sendMirrorFailureMail(deps.env, {
|
||||
backupId: 'mirror',
|
||||
error: message,
|
||||
lastSuccessAt: previousSuccessAt,
|
||||
});
|
||||
if (!sent) deps.log.warn({}, 'no BACKUP_MAIL_TO configured, mirror alert not sent');
|
||||
} catch (mailError) {
|
||||
deps.log.error({ error: String(mailError) }, 'mirror alert could not be sent');
|
||||
}
|
||||
return {
|
||||
lastRun: { finishedAt: deps.now().toISOString(), outcome: 'failed', error: message },
|
||||
lastSuccessAt: previousSuccessAt,
|
||||
};
|
||||
}
|
||||
}
|
||||
@ -2,7 +2,7 @@ import { mkdir, readdir, rename, rm, stat } from 'node:fs/promises';
|
||||
import { join } from 'node:path';
|
||||
|
||||
import { archiveFileName, dumpFileName, expiredSets, listSets, newBackupId } from './backup-set.js';
|
||||
import type { BackupRemoteStatus } from '@dorfteich/shared';
|
||||
import type { BackupMirrorStatus, BackupRemoteStatus } from '@dorfteich/shared';
|
||||
|
||||
import {
|
||||
readStatus,
|
||||
@ -43,6 +43,14 @@ export interface RunnerDeps {
|
||||
backupId: string;
|
||||
previous: BackupRemoteStatus | undefined;
|
||||
}): Promise<BackupRemoteStatus | undefined>;
|
||||
/**
|
||||
* rsync mirror to a private host (issue #84, mirror.ts). Runs after the
|
||||
* prune so the remote retention aligns with the local one. Failures are
|
||||
* reported inside the returned status, never thrown.
|
||||
*/
|
||||
mirror?(input: {
|
||||
previous: BackupMirrorStatus | undefined;
|
||||
}): Promise<BackupMirrorStatus | undefined>;
|
||||
log: {
|
||||
info(details: object, message: string): void;
|
||||
error(details: object, message: string): void;
|
||||
@ -55,6 +63,7 @@ export async function runBackup(deps: RunnerDeps): Promise<BackupStatus> {
|
||||
const previous = readStatus(deps.backupsDir);
|
||||
const lastSuccess = previous?.lastSuccess ?? null;
|
||||
let remote = previous?.remote;
|
||||
let mirror = previous?.mirror;
|
||||
await mkdir(deps.backupsDir, { recursive: true });
|
||||
|
||||
const dumpFile = join(deps.backupsDir, dumpFileName(backupId));
|
||||
@ -111,6 +120,18 @@ export async function runBackup(deps: RunnerDeps): Promise<BackupStatus> {
|
||||
|
||||
await prune(deps);
|
||||
|
||||
// After the prune, so `--delete` aligns the remote retention with the
|
||||
// local one — including the newest-complete-set guarantee.
|
||||
if (run.outcome === 'succeeded' && deps.mirror) {
|
||||
try {
|
||||
mirror = (await deps.mirror({ previous: mirror })) ?? mirror;
|
||||
} catch (mirrorError) {
|
||||
// Defensive like the upload hook: mirror.ts reports failures in its
|
||||
// return value; a throw here must never fail the local run.
|
||||
deps.log.error({ error: String(mirrorError) }, 'mirror hook threw unexpectedly');
|
||||
}
|
||||
}
|
||||
|
||||
const status: BackupStatus = {
|
||||
schemaVersion: 1,
|
||||
updatedAt: deps.now().toISOString(),
|
||||
@ -118,6 +139,7 @@ export async function runBackup(deps: RunnerDeps): Promise<BackupStatus> {
|
||||
lastRun: run,
|
||||
lastSuccess: success,
|
||||
...(remote ? { remote } : {}),
|
||||
...(mirror ? { mirror } : {}),
|
||||
};
|
||||
await writeStatus(deps.backupsDir, status);
|
||||
return status;
|
||||
|
||||
@ -121,6 +121,23 @@ function StatusCard({ view }: { view: SystemBackupView }): React.JSX.Element {
|
||||
</>
|
||||
)}
|
||||
</dd>
|
||||
{view.status.mirror && (
|
||||
<>
|
||||
<dt>{t('backup.mirror.title')}</dt>
|
||||
<dd>
|
||||
{t('backup.mirror.lastSuccess')}:{' '}
|
||||
{view.status.mirror.lastSuccessAt
|
||||
? new Date(view.status.mirror.lastSuccessAt).toLocaleString()
|
||||
: t('backup.never')}
|
||||
{view.status.mirror.lastRun.outcome === 'failed' && (
|
||||
<>
|
||||
{' — '}
|
||||
{t('backup.mirror.failed')}: <code>{view.status.mirror.lastRun.error}</code>
|
||||
</>
|
||||
)}
|
||||
</dd>
|
||||
</>
|
||||
)}
|
||||
</dl>
|
||||
<p className="system-backup__retention">
|
||||
{t('backup.retention', { days: view.status.retentionDays })}
|
||||
|
||||
95
deploy/backup-basel.md
Normal file
95
deploy/backup-basel.md
Normal file
@ -0,0 +1,95 @@
|
||||
# Backup mirror to BASEL (issue #84)
|
||||
|
||||
Nightly rsync of the Prod backup sets to the private BASEL host over the
|
||||
WireGuard tunnel (ADR 0015) — the operator-level extra beside the
|
||||
admin-configured Nextcloud target (#103). The sidecar mirrors the set
|
||||
files (`db-*.dump`, `files-*.tar.gz`) after every successful local run;
|
||||
`--delete` keeps the remote retention aligned with the local prune, so
|
||||
the newest-complete-set guarantee carries over. Mirror outcome lands in
|
||||
`status.json` (`mirror` block, shown on the admin backup card); failures
|
||||
alert through the backup failure mail while local backups continue.
|
||||
|
||||
## BASEL-side setup (once, as root on BASEL)
|
||||
|
||||
A **dedicated user with a home under `/home/`** — deliberately NOT the
|
||||
Debian `backup` system user (UID 34), whose `/var/backups` home and
|
||||
`nologin` shell are documented foot-guns in the operator conventions:
|
||||
|
||||
```sh
|
||||
useradd --create-home --shell /bin/bash dorfteich-backup
|
||||
mkdir -p /home/dorfteich-backup/.ssh
|
||||
# authorized_keys: the public half of the key generated below
|
||||
install -m 600 -o dorfteich-backup -g dorfteich-backup authorized_keys \
|
||||
/home/dorfteich-backup/.ssh/authorized_keys
|
||||
chmod 700 /home/dorfteich-backup/.ssh
|
||||
chown dorfteich-backup:dorfteich-backup /home/dorfteich-backup/.ssh
|
||||
|
||||
mkdir -p /home/RAID/BACKUPS/dorfteich-prod
|
||||
chown dorfteich-backup:dorfteich-backup /home/RAID/BACKUPS/dorfteich-prod
|
||||
chmod 750 /home/RAID/BACKUPS/dorfteich-prod
|
||||
```
|
||||
|
||||
**Pitfalls (learned on the wochenplan setup):**
|
||||
|
||||
- The login shell MUST be `/bin/bash` (or `/bin/sh`) — with
|
||||
`/usr/sbin/nologin` sshd rejects every session, including rsync's.
|
||||
Security comes from the key-only login, not from nologin; if you want
|
||||
to lock it down further, prefix the `authorized_keys` line with
|
||||
`command="rsync --server ..."` restrictions.
|
||||
- `.ssh` must be mode 700 and owned by the user; `authorized_keys` 600.
|
||||
- The home directory itself may be root-owned but must not be
|
||||
group/world-writable (sshd's StrictModes).
|
||||
|
||||
## Stack-side setup (per stage that mirrors)
|
||||
|
||||
1. **Generate a keypair** (on the docker host, never in the repo):
|
||||
|
||||
```sh
|
||||
ssh-keygen -t ed25519 -N '' -C dorfteich-backup-mirror -f backup_mirror_ed25519
|
||||
# the .pub half goes into BASEL's authorized_keys (above)
|
||||
```
|
||||
|
||||
2. **Put the private key on the `secrets` volume** (the sidecar mounts it
|
||||
read-only at `/data/secrets`; copy through the api container, which
|
||||
mounts it read-write):
|
||||
|
||||
```sh
|
||||
docker compose cp backup_mirror_ed25519 api:/data/secrets/backup_mirror_ed25519
|
||||
docker compose exec api chmod 600 /data/secrets/backup_mirror_ed25519
|
||||
shred -u backup_mirror_ed25519
|
||||
```
|
||||
|
||||
3. **Configure the stage `.env`** and recreate the sidecar:
|
||||
|
||||
```sh
|
||||
BACKUP_MIRROR_TARGET=dorfteich-backup@172.30.1.10:/home/RAID/BACKUPS/dorfteich-prod/
|
||||
BACKUP_MIRROR_SSH_KEY=/data/secrets/backup_mirror_ed25519
|
||||
#BACKUP_MIRROR_SSH_PORT=22 # default
|
||||
```
|
||||
|
||||
```sh
|
||||
docker compose up -d backup
|
||||
```
|
||||
|
||||
4. **Verify** with an on-demand run — the first mirror pins BASEL's host
|
||||
key (`accept-new`) into `.mirror_known_hosts` on the backups volume:
|
||||
|
||||
```sh
|
||||
docker compose run --rm -e BACKUP_RUN_ONCE=1 backup
|
||||
docker compose exec backup sh -c \
|
||||
'grep -A4 \"mirror\" /backups/status.json'
|
||||
# re-run: "transferredFiles": 0 proves idempotency
|
||||
```
|
||||
|
||||
## Behaviour
|
||||
|
||||
- Mirror runs after the local prune of every **successful** run (nightly,
|
||||
manual button, `BACKUP_RUN_ONCE`); a failed local run never mirrors.
|
||||
- A mirror failure sets `status.json → mirror.lastRun.outcome = "failed"`
|
||||
(visible on the admin backup card) and sends the `backupMirrorFailed`
|
||||
alert mail — the local run still counts as succeeded.
|
||||
- Restoring from the mirror: copy the set pair back into the stack's
|
||||
backups volume and run `./restore.sh <backup-id>` — same procedure as
|
||||
the Nextcloud path in `docs/operations/restore-runbook.md`.
|
||||
- The mirror carries only the raw set files; `status.json` and the
|
||||
Nextcloud bundles stay local.
|
||||
@ -79,6 +79,13 @@ SMTP_FROM=Dorfteich <wiki@example.com>
|
||||
#BACKUP_MAIL_TO=ops@example.com
|
||||
#BACKUP_MAIL_LOCALE=en
|
||||
#BACKUP_INSTANCE_LABEL=dorfteich-test
|
||||
# Optional rsync mirror of the backup sets to a private host (issue #84):
|
||||
# rsync-over-ssh target plus the private key file INSIDE the container —
|
||||
# put the key on the secrets volume (docker compose cp), never in the repo.
|
||||
# Full setup walkthrough: deploy/backup-basel.md. Unset = no mirror.
|
||||
#BACKUP_MIRROR_TARGET=dorfteich-backup@172.30.1.10:/home/RAID/BACKUPS/dorfteich-prod/
|
||||
#BACKUP_MIRROR_SSH_KEY=/data/secrets/backup_mirror_ed25519
|
||||
#BACKUP_MIRROR_SSH_PORT=22
|
||||
|
||||
# --- first-run setup (optional pre-seeding, issue #80) ------------------------
|
||||
# A fresh (empty) database makes the instance require the browser setup
|
||||
|
||||
@ -156,6 +156,11 @@ services:
|
||||
BACKUP_MAIL_TO: ${BACKUP_MAIL_TO:-}
|
||||
BACKUP_MAIL_LOCALE: ${BACKUP_MAIL_LOCALE:-}
|
||||
BACKUP_INSTANCE_LABEL: ${BACKUP_INSTANCE_LABEL:-${COMPOSE_PROJECT_NAME:-dorfteich}}
|
||||
# Optional rsync mirror to a private host (issue #84); the SSH key
|
||||
# lives on the secrets volume (see deploy/backup-basel.md).
|
||||
BACKUP_MIRROR_TARGET: ${BACKUP_MIRROR_TARGET:-}
|
||||
BACKUP_MIRROR_SSH_KEY: ${BACKUP_MIRROR_SSH_KEY:-}
|
||||
BACKUP_MIRROR_SSH_PORT: ${BACKUP_MIRROR_SSH_PORT:-}
|
||||
# Same SMTP resolution as the api: explicit env wins, the wizard-written
|
||||
# secret store fills the gaps (issue #80).
|
||||
SMTP_HOST: ${SMTP_HOST:-}
|
||||
|
||||
@ -50,7 +50,11 @@ monitoring, structured logs, backup alerting — no dedicated metrics stack.
|
||||
7 Test+Int; a Site-Admin setting overrides the env; the newest complete
|
||||
set always survives) → `status.json` on the `backups` volume → on
|
||||
failure a mail directly via the instance SMTP to `BACKUP_MAIL_TO`.
|
||||
Mirror to BASEL is issue #84.
|
||||
- **Private mirror** (issue #84): with `BACKUP_MIRROR_TARGET` +
|
||||
`BACKUP_MIRROR_SSH_KEY` set (Prod: BASEL over WireGuard), every
|
||||
successful run rsyncs the set files after the prune (`--delete` aligns
|
||||
the remote retention); outcome in `status.json → mirror`, failures mail
|
||||
like run failures. Setup: `deploy/backup-basel.md`.
|
||||
- **Off-host copies** (issue #103): with a Nextcloud target configured in
|
||||
the admin UI (WebDAV base URL + username + folder in instance settings,
|
||||
app password in the secret store), each successful set is bundled into
|
||||
|
||||
@ -40,6 +40,15 @@
|
||||
"lastSuccessNever": "Letzter erfolgreicher Upload: noch keiner",
|
||||
"hint": "Prüfe die Nextcloud-Verbindungseinstellungen im Admin-Bereich (Verbindungstest) und die Sidecar-Logs (docker compose logs backup)."
|
||||
},
|
||||
"backupMirrorFailed": {
|
||||
"subject": "[{{instance}}] Backup-Spiegelung zum privaten Host fehlgeschlagen",
|
||||
"intro": "Das lokale Backup auf {{instance}} war erfolgreich, aber die Spiegelung der Sets per rsync ist fehlgeschlagen.",
|
||||
"backupId": "Komponente: rsync-Mirror (Issue #84)",
|
||||
"error": "Fehler: {{error}}",
|
||||
"lastSuccess": "Letzte erfolgreiche Spiegelung: {{finishedAt}}",
|
||||
"lastSuccessNever": "Letzte erfolgreiche Spiegelung: noch keine",
|
||||
"hint": "Prüfe den Tunnel zum Mirror-Host, den SSH-Key im Secrets-Volume und die Sidecar-Logs (docker compose logs backup). Einrichtung: deploy/backup-basel.md."
|
||||
},
|
||||
"digest": {
|
||||
"subject": "Dorfteich: {{count}} Neuigkeiten für dich",
|
||||
"intro": "Das ist auf von dir beobachteten Seiten passiert ({{count}} Neuigkeiten):",
|
||||
|
||||
@ -111,6 +111,11 @@
|
||||
"succeeded": "Wiederherstellung von {{id}} war erfolgreich ({{finishedAt}}).",
|
||||
"failed": "Wiederherstellung von {{id}} ist fehlgeschlagen: {{error}}"
|
||||
}
|
||||
},
|
||||
"mirror": {
|
||||
"title": "Privater Spiegel",
|
||||
"lastSuccess": "Letzte erfolgreiche Spiegelung",
|
||||
"failed": "letzter Lauf fehlgeschlagen"
|
||||
}
|
||||
},
|
||||
"audit": {
|
||||
|
||||
@ -40,6 +40,15 @@
|
||||
"lastSuccessNever": "Last successful upload: none yet",
|
||||
"hint": "Check the Nextcloud connection settings in the admin panel (test connection) and the sidecar logs (docker compose logs backup)."
|
||||
},
|
||||
"backupMirrorFailed": {
|
||||
"subject": "[{{instance}}] Backup mirror to the private host failed",
|
||||
"intro": "The local backup on {{instance}} succeeded, but mirroring the sets via rsync failed.",
|
||||
"backupId": "Component: rsync mirror (issue #84)",
|
||||
"error": "Error: {{error}}",
|
||||
"lastSuccess": "Last successful mirror: {{finishedAt}}",
|
||||
"lastSuccessNever": "Last successful mirror: none yet",
|
||||
"hint": "Check the tunnel to the mirror host, the SSH key in the secrets volume, and the sidecar logs (docker compose logs backup). Setup: deploy/backup-basel.md."
|
||||
},
|
||||
"digest": {
|
||||
"subject": "Dorfteich: {{count}} updates for you",
|
||||
"intro": "Here is what happened on pages you watch ({{count}} updates):",
|
||||
|
||||
@ -111,6 +111,11 @@
|
||||
"succeeded": "Restore of {{id}} succeeded ({{finishedAt}}).",
|
||||
"failed": "Restore of {{id}} failed: {{error}}"
|
||||
}
|
||||
},
|
||||
"mirror": {
|
||||
"title": "Private mirror",
|
||||
"lastSuccess": "Last successful mirror",
|
||||
"failed": "last run failed"
|
||||
}
|
||||
},
|
||||
"audit": {
|
||||
|
||||
@ -53,6 +53,8 @@ export interface BackupStatus {
|
||||
* visible.
|
||||
*/
|
||||
remote?: BackupRemoteStatus;
|
||||
/** rsync mirror state (issue #84); absent until a target is configured. */
|
||||
mirror?: BackupMirrorStatus;
|
||||
}
|
||||
|
||||
/** One WebDAV upload attempt of a complete restore set (issue #103). */
|
||||
@ -72,6 +74,25 @@ export interface BackupRemoteStatus {
|
||||
lastSuccessfulUpload: { backupId: string; finishedAt: string; sizeBytes: number } | null;
|
||||
}
|
||||
|
||||
/**
|
||||
* The rsync mirror to a private host (issue #84, ADR 0015) — an
|
||||
* operator-level extra beside the admin-configured Nextcloud target
|
||||
* (#103). Present once a mirror target is configured; carried across
|
||||
* failed runs like the other blocks.
|
||||
*/
|
||||
export interface BackupMirrorStatus {
|
||||
lastRun: {
|
||||
finishedAt: string;
|
||||
outcome: 'succeeded' | 'failed';
|
||||
/** Present on failure: the first error the mirror hit. */
|
||||
error?: string;
|
||||
/** Present on success: files rsync actually transferred (0 = idempotent re-run). */
|
||||
transferredFiles?: number;
|
||||
};
|
||||
/** Carried across failed runs — when the mirror last matched the local sets. */
|
||||
lastSuccessAt: string | null;
|
||||
}
|
||||
|
||||
/**
|
||||
* The remote bundle naming scheme (issue #103): one self-contained archive
|
||||
* per restore set, everything needed to rebuild an instance after total
|
||||
|
||||
@ -172,6 +172,17 @@ export const backupEnvSchema = z.object({
|
||||
BACKUP_MAIL_LOCALE: z.enum(['de', 'en']).default('en'),
|
||||
/** Instance label in the mail subject, e.g. "dorfteich-test". */
|
||||
BACKUP_INSTANCE_LABEL: z.string().optional(),
|
||||
/**
|
||||
* Optional rsync mirror to a private host (issue #84, ADR 0015), e.g.
|
||||
* `dorfteich-backup@172.30.1.10:/home/RAID/BACKUPS/dorfteich-prod/`.
|
||||
* Unset disables the mirror entirely. Deliberately env-only (operator
|
||||
* territory), unlike the admin-configured Nextcloud target (#103).
|
||||
*/
|
||||
BACKUP_MIRROR_TARGET: z.string().optional(),
|
||||
/** Private SSH key file for the mirror; mount it via the secrets volume
|
||||
* (e.g. `/data/secrets/basel_ed25519`), mode 600, never in the image. */
|
||||
BACKUP_MIRROR_SSH_KEY: z.string().optional(),
|
||||
BACKUP_MIRROR_SSH_PORT: z.coerce.number().int().min(1).max(65535).default(22),
|
||||
...smtpFields,
|
||||
/** Read-only view of the wizard-written secret store (issue #80). */
|
||||
SECRETS_FILE: z.string().min(1).default('./data/secrets.env'),
|
||||
|
||||
Loading…
Reference in New Issue
Block a user