20 Commits

Author	SHA1	Message	Date
Claude Fable 5	4f6596e8a2	#288: reset schema before pg_restore — partitioned tables broke --clean ... Since #224 read_events is partitioned; the dump carries per-partition primary keys as own entries, and pg_restore --clean emitted DROP CONSTRAINT against inherited constraints, which PostgreSQL refuses. The restore then reported FAILED although the content was restored. Dropping and recreating the public schema first makes every --clean drop a no-op and the restore faithful: objects created after the backup no longer survive. Verified in the isolated environment of #220 (set 20260731-132200, exit 0, readyz green). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AUtYMxwTCMHG9mVHnwbFg8	2026-07-31 17:00:05 +02:00
Claude Fable 5	2bdb0ec2cf	#224: read-trail storage — partitioning, retention, admin query path ... Convert read_events to monthly RANGE partitions on occurred_at, with a DEFAULT partition as safety net: a lagging maintenance job must never turn the trail's hard-failure semantics into an outage for classified reads. The dedup unique pair (#223) moves to per-partition indexes (PostgreSQL cannot carry it on the parent); a bucket spanning a month boundary may record one duplicate — over-recording is acceptable, gaps are not. New daily job read-trail-maintenance (job-count fence 9 -> 10) creates months ahead — each with its dedup index — and applies the trail's own retention readTrail.retentionDays (default 365, deliberately independent of audit.retentionDays): whole expired months are DROPped without scanning, remainders deleted by range, every run audited as read_trail.pruned (catalogue v1.2; the fence regex now admits an underscore namespace). Site-Admin query path GET /admin/system/read-events answers "who read page X" and "what did user Y read" within a period — API-only by design, documented. Growth measured and documented in data-model.md: ~1 MB per 1000 events including indexes. Tests: retention pruning + audited deletion + admin queries on the shared database; the partitioned shape, per-partition P2002 dedup, months-ahead creation and DROP-based pruning against a fresh database built by the real migration chain. Refs #224. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AUtYMxwTCMHG9mVHnwbFg8	2026-07-31 12:21:45 +02:00
Claude Fable 5	e505fc74dc	#212: mark attachment downloads by filename prefix and companion file ... Downloads whose effective classification is vs_nfd carry the documented VS-NfD_ filename prefix (single source classificationFilenamePrefix() in shared; ADR 0022 records the short form for file names). Effective classification: the linked page's level; an attachment with unset pageId (paste-then-insert, pond-level) FAILS CLOSED to the highest level of any live page in its pond. The pond export ZIP adds a sibling <file>.classification.txt companion with the full marking for classified media, next to the manifest entry (#210). Documented in operations.md, incl. the deliberate residual risk: the file's own content carries no marking (recorded on #231, not hidden). Tests: prefixed classified download, unchanged open download, fail-closed orphan both ways, ZIP companion + manifest level. Co-Authored-By: Claude Fable 5 (1M context) <noreply@anthropic.com>	2026-07-31 07:29:16 +02:00
Claude Fable 5	74970f6073	#199: SHA-256 integrity hashes for attachments ... Every upload stores the SHA-256 of its bytes, computed from the in-memory buffer that is written — never by re-reading disk. Every download re-hashes the stored object BEFORE the first byte leaves (memory bounded by the max_file_bytes quota that gated the upload) and fails closed on mismatch with attachment_integrity_failure; the mismatch lands in the audit trail as file.integrity_failed with both hashes. Detection of payload manipulation is the one integrity duty par. 52 VSA leaves with the application — only it knows what the file should be. Pre-#199 rows are hashed by a bounded, idempotent backfill that rides the existing nightly orphan-file-sweep job (no new scheduler job, job fence untouched); unreadable files are logged and retried, never silently skipped, and null-hash rows are served unverified only until the backfill reaches them. Operator runbook note in security.md (restore from backup, re-download, audit entry carries both hashes). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0168Ph5uBmHm8X28CSVpbpnJ	2026-07-31 04:32:29 +02:00
Claude Fable 5	6a520e27b1	#236: pin the Node version ... .node-version (22.15.1) becomes the single authoritative Node version: CI/CD select Node only via node-version-file, every Dockerfile pins node:22.15.1-alpine, and the engines floor in package.json states the same version (open-ended upwards so a newer local Node keeps working — reproducibility rests on images and CI). An early CI step fails on any drift between those places; update procedure in operations.md (Update strategy). Precondition for the reproducibility claim in #219. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0168Ph5uBmHm8X28CSVpbpnJ	2026-07-31 04:14:55 +02:00
Claude Fable 5	9a43a2f6bb	#235: keep page_links rows pointing at purged pages — recorded decision ... The row is only the index of a wikilink whose text (slug = title) remains visible in the linking page's own content either way; deleting the index would remove nothing the system still shows while breaking phantom-link re-resolution. Kept as an accepted residue, reasoning in operations.md (deletion/purge section) and recorded on #231. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0168Ph5uBmHm8X28CSVpbpnJ	2026-07-30 22:18:39 +02:00
Claude Fable 5	69d9072d2c	#234: retention for mail_outbox ... Sent mails were kept forever, and digest bodies name page titles and actors — an unbounded copy of content-adjacent data. A new daily mail-outbox-retention job deletes SENT rows (by sentAt) and permanently FAILED rows (by nextAttemptAt, the last attempt's stamp) once they pass mail.outboxRetentionDays (instance setting, default 30). PENDING rows — including failed-but-retryable ones — stay the retry loop's alone. Decision recorded (security.md §Privacy, residual-risk note for #231): digest mails keep carrying page titles for now — there is no per-page classification marking yet to key a suppression on (ADR 0022 / M32 revisits), and a VS-NfD reference configuration can leave SMTP unconfigured entirely. Job-count fence in system.spec: 8 -> 9. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0168Ph5uBmHm8X28CSVpbpnJ	2026-07-30 22:17:28 +02:00
Claude Fable 5	ff505bc752	#233: prune conversion job payloads for every job kind ... The raw input/result bytes of import/export conversion jobs were kept forever; a deleted classified page could live on inside its last export. A new daily conversion-payload-prune job nulls both once a finished (succeeded or failed) job passes conversion.payloadRetentionDays (instance setting, default 30) — the row survives for status/audit. PENDING and RUNNING rows keep their payload, so the worker's stale-lock recovery path is untouched; a hand-requeued pruned job fails finally via conversionInputOf instead of crashing the worker. The input column becomes nullable; the migration backfills by clearing payloads of jobs already finished longer ago than the default period (recent results stay downloadable until they age out). Job-count fence in system.spec: 7 -> 8 (new scheduler registration). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0168Ph5uBmHm8X28CSVpbpnJ	2026-07-30 22:12:32 +02:00
Claude Fable 5	ed2225bb77	#196: audit-trail retention job ... audit.retentionDays (instance setting, default 365) bounds the audit_log: the daily audit-retention job deletes entries past the period and records the deletion itself (audit.pruned with count, cutoff and period) so a gap in the trail is always explainable. Lives in its own AuditRetentionService because the settings service audits its writes - folding retention into AuditService would close a constructor cycle. The read-access trail (#222-#225) is deliberately not covered; it gets its own period. security.md gains the Logging section the schema has cited for a while; the maintenance-job fence moves 6 -> 7 (the deliberate new row). Refs #196 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0168Ph5uBmHm8X28CSVpbpnJ	2026-07-30 14:59:28 +02:00
Claude Fable 5	0bc36aa58c	#194: orphan-file sweep, drop the unused Attachment.deletedAt ... Nightly sweep with two directions: attachments still unclaimed (pageId null) after a 24 h grace period - claimed by no collab persist, page upload, or import - are reclaimed (row, file, quota released); files on the uploads volume without a database row (drift after a crashed upload) are removed once older than the grace period. The grace period protects the paste-then-insert window. Deliberate deviation from the issue's content-reference idea, documented in schema comment and operations.md: claimed attachments whose page content no longer embeds them are NOT auto-deleted. The page attachments panel lists claimed files as user-managed objects (inserting into the document is optional there), so 'not embedded' is not 'unused' - an auto-delete would destroy panel assets. Humans clean those up in the panel or the pond file manager, which flags orphans already. Attachment.deletedAt is removed by migration - deletion is hard everywhere (sweep, purge, manual), there is no soft-delete state; the never-true deletedAt:null filters in files/export queries went with it. Refs #194 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0168Ph5uBmHm8X28CSVpbpnJ	2026-07-30 13:19:58 +02:00
Claude Fable 5	402b22e05f	#193: pond purge — retention job and manual Site-Admin endpoint ... Deletion now actually deletes: a trashed pond past the trash retention (same clock as pages, extended trash-purge job) or purged manually via DELETE /ponds/:id/purge (Site-Admin-only, like pond restore) is removed with everything it holds. Files go first (idempotent rm, resumable on a crash), then one transaction ordered around the FK actions: attachments and labels (Restrict) precede the pond; the page delete cascades versions, comments, content cache incl. the search vector, update log, mentions, label assignments, favorites, outgoing links and open collab sessions; the pond delete cascades grants, usage counters (that is the quota correction), pond-plugin opt-ins and conversion jobs; polymorphic watches and pond quota overrides are deleted explicitly. A purge racing a restore or another purge is a no-op; both paths record a pond.purged audit event. Known residues by design, documented in operations.md: target_slug in other ponds' page links (#235) and backups within their retention. Refs #193 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0168Ph5uBmHm8X28CSVpbpnJ	2026-07-30 12:44:52 +02:00
Claude Fable 5	394d1c811d	#192: deploy-level backup target allowlist ... BACKUP_ALLOWED_TARGETS (comma-separated destination hosts) constrains where backups may go, enforced twice: the api rejects settings writes and connection tests towards non-allowlisted hosts with admin-visible error codes and resolves a non-allowlisted configured target to null, and the sidecar enforces the same policy at the point of egress for the WebDAV upload and the rsync mirror alike (shared policy helpers in packages/shared/src/backup-target-policy.ts). BREAKING: the empty default disables every remote target - backups stay local only, the VS-NfD reference configuration (ADR 0026). Existing deployments with a remote target must list its host or uploads and mirror stop. The admin UI distinguishes unavailable-by-policy from unconfigured (i18n de+en) and shows the permitted hosts. Refs #192 (ADR 0026) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0168Ph5uBmHm8X28CSVpbpnJ	2026-07-30 12:17:14 +02:00
Claude Fable 5	52192eb05f	Backup mirror to BASEL: rsync of the sets after every successful run (#84) ... The operator-level extra beside the admin-configured Nextcloud target (#103), unblocked now that the ONE→BASEL tunnel is stable again. - sidecar: optional mirror step (mirror.ts) driven purely by env — BACKUP_MIRROR_TARGET (rsync-over-ssh), BACKUP_MIRROR_SSH_KEY (private key on the secrets volume, never in image or repo), BACKUP_MIRROR_SSH_PORT. Runs after the prune of every successful run, so --delete aligns the remote retention with the local one (the newest-complete-set guarantee carries over). Only set files travel (db-*.dump, files-*.tar.gz); status files and bundles stay local. Host key pinned via accept-new into .mirror_known_hosts on the backups volume; fixed remote modes (dirs 750, files 640, symbolic --chmod — octal needs rsync ≥ 3, macOS dev machines ship 2.6.9). rsync + openssh-client added to the sidecar image. - status: additive `mirror` block in status.json (outcome, transferred count, lastSuccessAt carried across failures) — shown on the admin backup card; failures alert via a new backupMirrorFailed mail (de+en) while the local run still counts as succeeded. - deploy/backup-basel.md: complete BASEL-side walkthrough — dedicated user dorfteich-backup with a /home/ home and a bash login shell, explicitly avoiding the Debian backup-user (UID 34) pitfalls (nologin shell rejects rsync sessions, /var/backups home), key placement through the api container onto the secrets volume, .env values, on-demand verification. - tests: rsync-arg/stats-parsing units plus an integration suite against the real rsync binary (local target; skips where rsync is absent) — transfer, idempotent re-run (0 files), retention alignment, failure path carrying lastSuccessAt. Verified live against the real BASEL host from a native sidecar run: initial transfer, host-key pinning, retention alignment after a local prune, idempotency, and the failure path (surfaced in status.json while the local run stayed green). BASEL side provisioned per the doc. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EwZ4jR4KFAPvpjWevfUGX1	2026-07-12 12:20:32 +02:00
Claude Fable 5	5cef359b8f	Nextcloud backup target: admin-configured, manual + scheduled uploads, in-app restore (#103) ... Off-host backups for every self-hoster, configured entirely in the admin UI — supersedes the host-specific mirror plan behind #84. shared: - webdav.ts (new package entry like token-crypto): minimal WebDAV client with basic auth — PROPFIND (tolerant multistatus parser), MKCOL, PUT (streamed), GET, DELETE; Nextcloud DAV path derived from the plain server URL, explicit DAV bases pass through - backup-status.ts: additive remote-upload status in status.json, the restore-status.json contract (running/succeeded/failed + staleness bound), the backup_command/backup_maintenance NOTIFY channels, and the one-bundle-per-set naming (dorfteich-backup-<id>.tar.gz) - backup-set.ts moved here from apps/backup (api lists local sets) backup sidecar: - reads the backup.* instance settings directly from the database (admin changes apply next run; local retention row overrides the env) and the app password from the secret store - after each successful set: bundle dump + files archive + manifest into ONE self-contained tar.gz, upload via WebDAV per schedule (off/daily/weekly; manual runs always upload), prune remote bundles — never the newest — and record the outcome in status.json; upload failures alert via a new backupUploadFailed mail (de+en) - command listener on backup_command (run / restore) with a serial queue against the nightly timer - restore orchestrator: restore-status.json → maintenance NOTIFY → grace → (remote: download + manifest-verify bundle) → terminate other DB connections → shared perform-restore path (same code as restore.sh) → final status + maintenance exit api: - MaintenanceGuard (global, registered before the setup gate): 503 maintenance_mode while restore-status says running; health endpoints and the new public GET /backup/restore-status stay exempt; a stale running state (crashed sidecar) unblocks after 30 min - MaintenanceStateService watches the file and restarts the api after a successful restore (fresh caches, migrate-on-start for older dumps); main.ts refuses to touch the database while a restore runs — a container restarting mid-restore must not race pg_restore with migrate deploy - worker sweeps (conversion, mail outbox, scheduler) catch transient database failures instead of dying on an unhandled rejection — the restore's connection termination crashed the api in verification - backup admin endpoints under /admin/system/backup: settings (live connection test before save, password write-only into the secret store), nextcloud/test, sets (local via the ro backups mount + remote via WebDAV), run + restore (type-to-confirm backstop, source validation) — commands travel as NOTIFY payloads; audit actions backup.settings_changed/run_triggered/restore_requested - readyz: new warning-level backup_remote check while a target is configured (26 h daily / 170 h weekly bound) collab: - maintenance listener: on enter, persist + close every live session and refuse new connections until exit (failsafe timeout 30 min) — no in-memory document may write pre-restore content back afterwards web: - Admin → System backup section: status card with remote facts and a "Back up now" button, the Nextcloud settings form with test button, and the restore picker (local + remote sets, type-to-confirm) - global maintenance screen: any 503 maintenance_mode flips the SPA to a status page polling the exempt endpoint, reloading when the instance returns Verified end-to-end against a live stack (fresh DB, native api + sidecar, fake WebDAV server): configure → test → manual backup → bundle upload → readyz/sets/status surfaces → remote restore with maintenance gate, marker rollback and api restart; suites: shared 21, backup 9, collab 11, api 58 files green, lint + i18n:check + typecheck clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EwZ4jR4KFAPvpjWevfUGX1	2026-07-12 10:39:18 +02:00
Claude Fable 5	d95c18e9e8	Automate the monthly restore drill with a scratch-stack workflow (#87) ... New scheduled workflow (monthly + on demand) runs deploy/backup/drill.sh: it reads the drilled stage's backups volume strictly read-only, restores the latest successful set into a throwaway Postgres and volumes under a unique drill prefix via the backup image's restore path, boots the api against the result, and verifies readyz (database + migrations), row counts, rendered content in the page cache, a public API request, and a media byte-check against the attachments table — then tears everything down, also on failure. Each run reports its outcome as a comment on the pinned "Restore drills" issue (#98). docs/operations/restore-runbook.md carries the manual procedure, which doubles as the Prod relocation path; pre-go-live the drill restores the Test set (switch the source volume at go-live, #89 — off-host fetch from the BASEL mirror stays with #84). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EwZ4jR4KFAPvpjWevfUGX1	2026-07-11 20:58:06 +02:00
Claude Fable 5	c8aac13dfb	Add Site-Admin system panel with persistent audit trail (#86) ... New /admin/system panel (operations.md §Maintenance jobs): the maintenance job list shows every registered job with truthful last-run data (new Job.lastDurationMs recorded by the scheduler) and a manual trigger that respects the run-mutex and is itself audit-logged; a backup card mirrors the sidecar's status.json including the freshness verdict; an audit-log viewer filters by actor, action, and time range with pagination; and a storage overview lists the largest ponds. Auth events and admin actions (grants, members, user/quota admin, plugins, settings, setup) now land in a new audit_log table through a central AuditService — which keeps emitting the established stdout log line — while content activity stays log-only by design. All endpoints are Site-Admin-only; covered by API DB tests and a Playwright pack in CI. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EwZ4jR4KFAPvpjWevfUGX1	2026-07-11 20:03:05 +02:00
Claude Fable 5	0ef96147e0	Extend readyz with backup freshness and a degraded status level (#85) ... readyz now enumerates database/migrations (hard failures, HTTP 503), converter/renderer, and a new backup check that reads the sidecar's status.json from the read-only backups mount and warns when the last successful backup is older than 26 h. Warning-level checks surface as overall status "degraded" while staying HTTP 200 — monitors alert on the body keyword, Docker healthchecks keep using the liveness endpoints so a degraded instance is never restart-looped. The status.json shape moved to @dorfteich/shared as the contract between the sidecar and its readers (#85/#86); deploy/monitoring.md defines the Uptime-Kuma monitor set per stage. The api image also pre-creates /data/backups node-owned so the shared backups volume stays writable for the sidecar regardless of which container initializes it, and the sidecar's scheduler survives runs that cannot even record their status. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EwZ4jR4KFAPvpjWevfUGX1	2026-07-11 19:02:59 +02:00
Claude Fable 5	8dbff86537	Add backup sidecar: nightly dump, volume archive, prune, status, restore (#83) ... New apps/backup service (ADR 0015): nightly pg_dump -Fc plus one tar of the uploads/plugins volumes as a consistent restore set on a new backups volume, retention prune that never removes the newest complete set, atomic status.json for the readiness/admin consumers (#85/#86), and a failure mail sent directly via nodemailer (the api may be the broken part) with de/en texts in the shared mails catalog. BACKUP_RUN_ONCE=1 gives the on-demand path; deploy/backup/restore.sh automates the documented restore runbook. The pure secret-store helpers moved to @dorfteich/shared so the sidecar resolves the wizard-written SMTP relay exactly like the api. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EwZ4jR4KFAPvpjWevfUGX1	2026-07-11 17:29:15 +02:00
Claude Fable 5	b16d23297e	Scaffold pnpm monorepo with lint, format, and test tooling ... pnpm workspace with apps/web, apps/api, apps/collab, and packages/shared; strict TypeScript base config, repo-wide ESLint (flat) + Prettier, Vitest per package, and root scripts lint/typecheck/test/ build. @dorfteich/shared ships a first health-response helper consumed by apps/api to prove workspace linking. Existing markdown docs are reformatted once by the new Prettier setup. Closes #1 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-04 19:06:27 +02:00
Claude Fable 5	0629411966	Add architecture documentation, ADRs, and operations concept ... Initial deliverable of the architecture phase: 16 ADRs (stack, CRDT collaboration, plugin sandbox, import/export, backups, CI/CD), data model, permission model, real-time collaboration and plugin concepts, deployment/operations/security documentation, and the milestone roadmap that the implementation issues are derived from. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-04 14:36:16 +02:00