7 Commits

Author	SHA1	Message	Date
Claude Fable 5	8ae010218e	Obsidian vault import: endpoint, job orchestration, rollback (#117) ... POST /ponds/:pondId/import/vault (pond-admin-gated; a vault import creates a subtree, uploads files, and creates labels — administration, not everyday editing) takes the ZIP plus a JSON options field {parentPageId?, labelIds?, frontmatterMode}. The archive is parsed at enqueue for fast 400s; the job (new kind import_vault, riding the existing isImportKind worker routing) re-parses and runs the #116 transform, then: containers top-down → notes (asset placeholders → uploaded pond files; non-images become page attachments) → tags to labels (nested tags build a label hierarchy via LabelsService, so locking and cache invalidation apply) plus the dialog labels. All-or-nothing: any failure hard-deletes the created pages (children first) and removes the stored files (quota restored), then surfaces as import_vault_invalid_zip / import_vault_too_large / quota_exceeded / conversion_failed — and makes the worker's retry policy safe. Supporting changes: - conversion_jobs gains a nullable options jsonb column; enqueue takes kind-specific options and a maxInputBytes override (the 25 MiB default protects the pandoc sidecar, which a vault never touches — vaults use the 64 MiB upload limit). - insertPage accepts a pre-reserved slug (the batch reserves all slugs up front against pond ∪ batch). - NEW: pages born with content seed their outgoing page_links rows (deriveContent now returns wikilinkSlugs) — imported pages would otherwise stay invisible to backlinks and the graph until their first collab save. Collab still rewrites the rows on every save, and the existing phantom resolution heals batch creation order. import-vault.e2e.db.test.ts (4 tests, real worker drained): gating + input rejection, the full fixture import (tree under a mount page, collision suffixes, link rows incl. phantom, nested tag labels, extra label everywhere, frontmatter stripped, image embedded + PDF attached), complete quota rollback, and a clean re-import with fresh suffixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-14 17:13:11 +02:00
Claude Opus 4.8	462eca9699	Add self-service GDPR data export (#68) ... A signed-in account can export all of its own data — profile, a list of its memberships/grants, and the Markdown of its personal pond plus the shared ponds it owns — as one ZIP. Foreign content never appears: only owned ponds are bundled and the per-page read filter (reused from #65) runs for each. - Reuse the conversion-job queue as the async carrier: a `data_export` job whose worker branch resolves DataExportService via a token (no DI cycle), builds the ZIP, and stores it with an `expiresAt`. The download link 404s past expiry and an hourly scheduled purge drops the bytes (data minimization, security.md §Privacy). - Extract ExportService.appendPondMarkdown so the pond ZIP (#65) and the data export share one read-filtered pond archiver. - Rate-limit requests per account (RateLimitService); POST /users/me/data-export enqueues, GET /jobs/:id(/result) poll/download. - Settings UI "Export my data" (de+en); web share pollJob/downloadJobResult between the document and data export hooks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EwZ4jR4KFAPvpjWevfUGX1	2026-07-10 13:01:51 +02:00
Claude Opus 4.8	8a68ef68e7	Add PDF export via Gotenberg (#67) ... Server-side PDF export for reading/sharing (ADR 0009), rendered by a new internal Gotenberg (headless Chromium) sidecar. - Sidecar: `gotenberg/gotenberg:8` in the compose stack (internal, pinned, healthcheck); api `GOTENBERG_URL` env; a `renderer` readyz check at warning-level (mirrors the converter) so PDF export degrades gracefully when Gotenberg is down without failing readyz. - Export HTML: `buildPdfHtml` renders a self-contained document (no app chrome) — the page's content with images inlined as data URIs, the pond's fonts inlined as base64 `@font-face` + applied via CSS variables (ADR 0016), print CSS (A4, page-break rules, a title header), and page numbers from Gotenberg's footer. Plugin-block fallbacks are a marked TODO(#79) for M7. - Fonts in the api image: the api Dockerfile now bakes the font catalog in (`build-fonts.mjs` with FONTS_OUT) so the exporter can read a pond's chosen WOFF2 and inline them; a missing file falls back to the system stack. - Job flow: `POST /pages/:id/export {format: pdf}` builds the HTML (read permission checked by the guard) and enqueues an `export_pdf` job on the #62 queue with the HTML as input; the worker branches `to === 'pdf'` to the `GotenbergRenderer` (html → pdf) instead of pandoc, retrying an unreachable sidecar and failing a refused render (`renderer_unavailable`/`render_failed`, de+en). The client polls and downloads `GET /jobs/:id/result`. - Frontend: the page-menu PDF button is now a real export (PDF added to EXPORT_FORMATS; the disabled placeholder removed). - Tests: export.service.db PDF cases (HTML has title/font-variable/inlined image; renderer-down fails with `render_failed`); e2e PDF export self-skips without a Gotenberg sidecar (like the .docx case). Verified locally against real Gotenberg — a valid PDF with the pond font embedded. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EwZ4jR4KFAPvpjWevfUGX1	2026-07-10 12:11:12 +02:00
Claude Opus 4.8	699c003d04	Add pond ZIP + per-page docx/odt export (#65) ... Two export paths, both permission-aware (permissions.md): - `GET /ponds/:id/export/markdown` streams a ZIP of the pond's readable pages as Markdown (one `<slug>.md` per page, a `media/` directory, wikilinks rewritten to relative `[text](slug.md)` links, image sources to `media/<id>.<ext>`). The `reader` guard is "may see the pond"; the service filters to the pages the requester may actually read, so a label-restricted reader gets only their slice. Media is appended as read streams and pages as small strings, so memory stays bounded for a large pond (500-page test). - `POST /pages/:id/export {format: docx|odt}` enqueues a `markdown → pandoc → file` conversion job (the #62 queue): embedded images are inlined as data URIs so the sidecar embeds them, wikilinks flatten to text. The client polls `GET /jobs/:id` and downloads `GET /jobs/:id/result`. Frontend: office-export buttons in the page menu (`.docx`/`.odt` run the job and download the result; PDF is a disabled placeholder for Gotenberg, #67) and a "Download pond as ZIP" link in pond settings. New `export` i18n namespace (de+en). Markdown copy/download stay as-is (#30). Robustness: the pond ZIP skips an attachment whose bytes are missing on disk (data drift) rather than letting an unhandled read-stream error crash the api; `FileStorageService.exists` gates inclusion, with a defensive stream error handler. The per-page export drops an unreadable image the same way. - shared: EXPORT_FORMATS + pageExportInputSchema; export-markdown transform helpers (image/wikilink rewrites, MIME→extension). - deps: archiver (streaming ZIP; v7 for CommonJS compat), fflate (dev, reads ZIPs in tests). - tests: export-markdown unit + export.service.db (ZIP contents & relative links, label-restricted omission, docx job with inlined images, 500-page streaming, missing-media skip); e2e export pack (ZIP download; `.docx` self-skips without a pandoc sidecar, as in the import pack, #64). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EwZ4jR4KFAPvpjWevfUGX1	2026-07-10 10:31:19 +02:00
Claude Opus 4.8	e2f942c0ff	Add document import UI in the sidebar (#64) ... An "Import document" action in the pond sidebar: pick a .docx/.odt/.md file (or several), upload with per-file progress, and open the new page. A .docx/.odt polls the conversion job (queued → converting → done); a .md imports directly and comes back already succeeded. Failures stay listed with the localized error and a retry; concurrent imports all complete and appear. - web apps/web/src/import/: useImport hook (upload via apiUploadFile → poll GET /jobs/:id → resolve the page slug → navigate; first success of a batch navigates, every success refreshes the sidebar) and ImportControl (hidden file input, accept from shared IMPORT_EXTENSIONS, per-file status list). Wired into Sidebar next to "new page"; `import` i18n namespace (de+en). - api: ImportService accepts .md/.markdown and imports in-process (no job), returning a succeeded ConversionJobView with the created resultPageId ("Markdown imports directly"); the media+parse+create tail is now shared between the job path and the sync path (createPageFromMarkdown), and a conversion error on the sync path maps to an HTTP status. shared IMPORT_EXTENSIONS gains md/markdown. - e2e apps/web/e2e/import.spec.ts + CI step: .docx corpus fixture opens the converted page (self-skips without a reachable pandoc sidecar — CI's e2e stack has none, same as #63; verified locally + on stage), .md opens directly, an unsupported .txt shows the localized error with no page created, and two concurrent .md imports both complete. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EwZ4jR4KFAPvpjWevfUGX1	2026-07-10 09:38:31 +02:00
Claude Opus 4.8	546e8279ac	Import .docx and .odt documents as new pages (#63) ... Uploading a Word/OpenOffice document to POST /ponds/:id/import enqueues a conversion job (the #62 queue) that produces a new page in the pond; the client polls GET /jobs/:id for the created resultPageId. Pipeline (ImportService, ADR 0009): pandoc-server is stateless and hands back a document's media no other way, so we convert in two passes — docx/odt → html with embed-resources inlines every image as a data: URI, then html → gfm produces clean structural Markdown with those data URIs still inline. Embedded images are stored as pond files (with quota accounting) and their references rewritten to file ids on the Markdown text before parsing (the editor parser only admits png/jpeg/gif/webp data URIs); an image whose bytes the upload pipeline rejects is dropped, not fatal. The title comes from a leading top-level heading (removed from the body) else the file name. The page is created from the resulting Yjs state. The shared conversion worker routes import-kind jobs to the pipeline via a token (breaking a module cycle), so import inherits the queue's locking, retry, and restart-survival. Media stored during a failed attempt is rolled back; a pond that runs out of storage fails the job with quota_exceeded. - schema: ConversionJob gains pond_id / source_name / result_page_id (migration 20260710041215_import_pages_conversion); ConversionJobView gains resultPageId. - PagesService.createWithState / yjs-content docToState build a page from a prepared document; FilesService.linkAttachmentsToPage links import media. - fixtures/import/: representative .docx/.odt corpus (headings, lists, nested lists, tables, images, links, bold/italic) with expected-Markdown snapshots; scripts/gen-import-fixtures.mjs regenerates them. - tests: import.service.db.test.ts drives the full pipeline with a fake converter (CI); import.fixtures.test.ts runs the real two-pass conversion over the corpus and a 50-page timing check against a reachable sidecar. - i18n: import_unsupported_format (de+en). Limits documented (25 MiB input, 60 s per pass). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EwZ4jR4KFAPvpjWevfUGX1	2026-07-10 07:34:43 +02:00
Claude Opus 4.8	4755c18ef5	Add conversion job queue and pandoc sidecar integration (#62) ... Import/export conversions run asynchronously against an internal pandoc-server sidecar with limits and graceful failure (ADR 0009). This is the plumbing; the import (#63) and export (#65) features enqueue jobs onto it. Sidecar & config: - pandoc/core:3.6 in HTTP server mode added to the Compose stack, internal network only, with a wget healthcheck on /version; the api depends on it healthy and reaches it via the new PANDOC_URL env (default http://pandoc:3030). - readyz gains a warning-level `converter` check: an unreachable sidecar degrades import/export but never flips the instance to unready (new `warn` status on ReadinessCheck). Conversion flow (apps/api/src/import-export/): - ConversionJob table (per-request work queue, distinct from the name-keyed maintenance Job table): owner, formats, input/result bytes, status, attempts, lockedAt. Migration + owner cascade. - PandocConverter (abstract) + PandocServerConverter: POST / with {text,from,to,standalone}; binary input formats (docx/odt/…) are base64-encoded in `text`; 60 s AbortController timeout; input/output size caps. Failures map to distinct localized codes — converter_unavailable / converter_timeout (retryable) and conversion_failed (final). - ConversionWorker: claims one job at a time with `FOR UPDATE SKIP LOCKED` (safe against overlapping sweeps and a second process), recovers a stale RUNNING lock, retries transient failures up to 3 attempts then fails. A 2 s sweep plus wake-on-enqueue means a queued job survives an API restart. - ConversionJobService.enqueue (size-limited) + owner-scoped GET /jobs/:id (poll) and GET /jobs/:id/result (stream the output); a foreign/unknown id is 404. ConversionJobView in @dorfteich/shared. Tests: - conversion-job.e2e.db.test.ts (fake converter injected via a new createTestApp override hook): enqueue→convert→poll→result; foreign/unknown job 404; a persisted PENDING job picked up by a fresh app's worker (restart survival); sidecar-down fails after 3 retries while the API stays healthy. - pandoc.converter.test.ts: success, non-200→conversion_failed, refused→ converter_unavailable, and a delay-injecting server→converter_timeout. - Verified locally against a real pandoc/core:3.6 container: markdown→html, markdown→docx (valid PK/OOXML bytes), and a docx→markdown round-trip. Local: typecheck, lint, i18n:check, build all green; api 193 tests (9 new), shared 121, web 50. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EwZ4jR4KFAPvpjWevfUGX1	2026-07-10 04:06:27 +02:00