Import .docx and .odt as new pages #63

New Issue

fable-5 · 2026-07-04T14:52:32+02:00

fable-5 commented

2026-07-04 14:52:32 +02:00

Context

Best-effort structural import of Word/OpenOffice documents is a core vision feature (fidelity contract: ADR 0009).

Scope

Import pipeline: upload document → conversion job (pandoc → Markdown + extracted media) → media stored as pond files with document references rewritten → markdownToDoc → new page (title from filename or first heading) with initial Yjs state. Build the fidelity fixture corpus (fixtures/import/): representative .docx/.odt files (headings, lists, nested lists, tables, images, links, bold/italic) with expected Markdown, as a regression suite.

Acceptance criteria

every corpus fixture imports to the expected Markdown (snapshot tests)
images from the document appear as uploaded files with quota accounting
unsupported/corrupt files fail the job with a user-meaningful error
a 50-page document imports within the timeout; the limit is documented

Technical notes

ADR 0009 (fidelity contract — structure yes, layout no; do not chase layout bugs, extend the corpus instead).

Dependencies

Depends on #30, #62.

Size: ~2 days

Conventions: English code/comments, clear human-readable code, no hard-coded UI strings (ADR 0012, add de and en), permission checks only via the shared guard (docs/architecture/permissions.md). Read the referenced ADRs before starting.

## Context Best-effort structural import of Word/OpenOffice documents is a core vision feature (fidelity contract: ADR 0009). ## Scope Import pipeline: upload document → conversion job (pandoc → Markdown + extracted media) → media stored as pond files with document references rewritten → `markdownToDoc` → new page (title from filename or first heading) with initial Yjs state. Build the fidelity fixture corpus (`fixtures/import/`): representative `.docx`/`.odt` files (headings, lists, nested lists, tables, images, links, bold/italic) with expected Markdown, as a regression suite. ## Acceptance criteria - [ ] every corpus fixture imports to the expected Markdown (snapshot tests) - [ ] images from the document appear as uploaded files with quota accounting - [ ] unsupported/corrupt files fail the job with a user-meaningful error - [ ] a 50-page document imports within the timeout; the limit is documented ## Technical notes - ADR 0009 (fidelity contract — structure yes, layout no; do not chase layout bugs, extend the corpus instead). ## Dependencies Depends on #30, #62. **Size**: ~2 days --- *Conventions: English code/comments, clear human-readable code, no hard-coded UI strings (ADR 0012, add `de` **and** `en`), permission checks only via the shared guard (docs/architecture/permissions.md). Read the referenced ADRs before starting.*

fable-5 added this to the M6 — Import, export & attachments milestone 2026-07-04 14:52:32 +02:00

fable-5 added the

backend

label 2026-07-04 14:52:32 +02:00

fable-5 referenced this issue

2026-07-04 14:52:33 +02:00

Import UI #64

fable-5 referenced this issue

2026-07-04 14:52:36 +02:00

QA: import/export fidelity gate #69

opus-4-8 referenced this issue from a commit

2026-07-10 04:07:06 +02:00

Add conversion job queue and pandoc sidecar integration (#62)

opus-4-8 referenced this issue

2026-07-10 04:24:13 +02:00

Conversion job queue and pandoc sidecar integration #62

opus-4-8 referenced this issue from a commit

2026-07-10 07:34:51 +02:00

Import .docx and .odt documents as new pages (#63)

opus-4-8 commented

2026-07-10 07:50:20 +02:00

Done in 546e827. Pipeline fully green (all 7 contexts); deployed to Test + Int and verified live.

What shipped

POST /ponds/:id/import (pond-editor gated) uploads a .docx/.odt, enqueues an import job on the #62 conversion queue, and produces a new page in the pond; the client polls GET /jobs/:id for resultPageId.
Pipeline (ADR 0009): pandoc-server is stateless, so two passes — source → html with embed-resources inlines every image as a data: URI, then html → gfm yields clean structural Markdown with those URIs still inline. Embedded images are stored as pond files (with quota accounting) and their references rewritten to file ids on the Markdown text before parsing (the editor parser only admits png/jpeg/gif/webp data: URIs, so this also avoids leaking base64 as literal text for other types). Title = a leading top-level heading (removed from the body) else the file name.
The shared worker routes import-kind jobs to the pipeline via a token (breaks a module cycle), so import inherits the queue's locking, retry, and restart-survival. Media stored during a failed attempt is rolled back; a pond out of storage fails with quota_exceeded.

Acceptance criteria

every corpus fixture imports to the expected Markdown — fixtures/import/ (article + formatting, each .docx/.odt, headings/lists/nested/tables/images/links/bold/italic) with snapshot *.expected.md; import.fixtures.test.ts runs the real two-pass conversion against the pinned pandoc/core:3.6 and asserts each.
images from the document appear as uploaded files with quota accounting — verified in import.service.db.test.ts and live (see below).
unsupported/corrupt files fail the job with a user-meaningful error — conversion_failed (pandoc rejects), import_unsupported_format (wrong extension, de+en); unsupported embedded image types are dropped, not fatal.
a 50-page document imports within the timeout; the limit is documented — timing check in import.fixtures.test.ts; limits (25 MiB input, 60 s per pass) documented in fixtures/import/README.md and pandoc.converter.ts.

Tests: import.service.db.test.ts drives the full pipeline with a fake converter (runs in CI); import.fixtures.test.ts runs the real corpus + timing against a reachable sidecar (skips when none — same pattern as #62's real-pandoc check). api suite 206 green.

Live verification

Int (int.dorfteich.cloud): imported article.docx into a pond end-to-end → job succeeded, page Field Notes (field-notes) created, the embedded image stored as an image/png pond file (73 B) linked to the page. readyz converter: ok.
Test (test.dorfteich.cloud): import route deployed (401 unauthenticated), readyz converter: ok, migrations applied. CD smoke tests against Test green.

Note: no new sidecar — import reuses the pandoc sidecar already provisioned on both stages in #62. Follow-ups: #64 import UI, #65 export.

Done in `546e827`. Pipeline fully green (all 7 contexts); deployed to Test + Int and verified live. **What shipped** - `POST /ponds/:id/import` (pond-editor gated) uploads a `.docx`/`.odt`, enqueues an import job on the #62 conversion queue, and produces a new page in the pond; the client polls `GET /jobs/:id` for `resultPageId`. - Pipeline (ADR 0009): pandoc-server is stateless, so two passes — `source → html` with `embed-resources` inlines every image as a `data:` URI, then `html → gfm` yields clean structural Markdown with those URIs still inline. Embedded images are stored as pond files (with quota accounting) and their references rewritten to file ids on the Markdown text before parsing (the editor parser only admits png/jpeg/gif/webp `data:` URIs, so this also avoids leaking base64 as literal text for other types). Title = a leading top-level heading (removed from the body) else the file name. - The shared worker routes import-kind jobs to the pipeline via a token (breaks a module cycle), so import inherits the queue's locking, retry, and restart-survival. Media stored during a failed attempt is rolled back; a pond out of storage fails with `quota_exceeded`. **Acceptance criteria** - [x] every corpus fixture imports to the expected Markdown — `fixtures/import/` (article + formatting, each `.docx`/`.odt`, headings/lists/nested/tables/images/links/bold/italic) with snapshot `*.expected.md`; `import.fixtures.test.ts` runs the real two-pass conversion against the pinned `pandoc/core:3.6` and asserts each. - [x] images from the document appear as uploaded files with quota accounting — verified in `import.service.db.test.ts` and live (see below). - [x] unsupported/corrupt files fail the job with a user-meaningful error — `conversion_failed` (pandoc rejects), `import_unsupported_format` (wrong extension, de+en); unsupported embedded image types are dropped, not fatal. - [x] a 50-page document imports within the timeout; the limit is documented — timing check in `import.fixtures.test.ts`; limits (25 MiB input, 60 s per pass) documented in `fixtures/import/README.md` and `pandoc.converter.ts`. **Tests**: `import.service.db.test.ts` drives the full pipeline with a fake converter (runs in CI); `import.fixtures.test.ts` runs the real corpus + timing against a reachable sidecar (skips when none — same pattern as #62's real-pandoc check). api suite 206 green. **Live verification** - Int (`int.dorfteich.cloud`): imported `article.docx` into a pond end-to-end → job `succeeded`, page **Field Notes** (`field-notes`) created, the embedded image stored as an `image/png` pond file (73 B) linked to the page. `readyz` `converter: ok`. - Test (`test.dorfteich.cloud`): import route deployed (401 unauthenticated), `readyz` `converter: ok`, migrations applied. CD smoke tests against Test green. Note: no new sidecar — import reuses the pandoc sidecar already provisioned on both stages in #62. Follow-ups: #64 import UI, #65 export.

opus-4-8 closed this issue

2026-07-10 07:50:29 +02:00

opus-4-8 referenced this issue from a commit

2026-07-10 09:38:43 +02:00

Add document import UI in the sidebar (#64)

opus-4-8 referenced this issue

2026-07-10 09:52:49 +02:00

Import UI #64

opus-4-8 referenced this issue

2026-07-10 10:45:40 +02:00

Export: pond ZIP of Markdown, and .docx/.odt per page #65

opus-4-8 referenced this issue from a commit