Claude Opus 4.8
546e8279ac
Uploading a Word/OpenOffice document to POST /ponds/:id/import enqueues a conversion job (the #62 queue) that produces a new page in the pond; the client polls GET /jobs/:id for the created resultPageId. Pipeline (ImportService, ADR 0009): pandoc-server is stateless and hands back a document's media no other way, so we convert in two passes — docx/odt → html with embed-resources inlines every image as a data: URI, then html → gfm produces clean structural Markdown with those data URIs still inline. Embedded images are stored as pond files (with quota accounting) and their references rewritten to file ids on the Markdown text before parsing (the editor parser only admits png/jpeg/gif/webp data URIs); an image whose bytes the upload pipeline rejects is dropped, not fatal. The title comes from a leading top-level heading (removed from the body) else the file name. The page is created from the resulting Yjs state. The shared conversion worker routes import-kind jobs to the pipeline via a token (breaking a module cycle), so import inherits the queue's locking, retry, and restart-survival. Media stored during a failed attempt is rolled back; a pond that runs out of storage fails the job with quota_exceeded. - schema: ConversionJob gains pond_id / source_name / result_page_id (migration 20260710041215_import_pages_conversion); ConversionJobView gains resultPageId. - PagesService.createWithState / yjs-content docToState build a page from a prepared document; FilesService.linkAttachmentsToPage links import media. - fixtures/import/: representative .docx/.odt corpus (headings, lists, nested lists, tables, images, links, bold/italic) with expected-Markdown snapshots; scripts/gen-import-fixtures.mjs regenerates them. - tests: import.service.db.test.ts drives the full pipeline with a fake converter (CI); import.fixtures.test.ts runs the real two-pass conversion over the corpus and a 50-page timing check against a reachable sidecar. - i18n: import_unsupported_format (de+en). Limits documented (25 MiB input, 60 s per pass). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EwZ4jR4KFAPvpjWevfUGX1
2.0 KiB
Import fixture corpus (issue #63, ADR 0009)
Representative .docx/.odt documents and the Markdown our import pipeline is
expected to produce from them. import.fixtures.test.ts runs the real two-pass
pandoc conversion over each and asserts the result, so a change in behaviour
(ours or pandoc's) surfaces as a snapshot diff to review — not a silent
regression.
Files
For each document <name> and format <ext> (docx, odt):
<name>.<ext>— the source document.<name>.<ext>.expected.md— the Markdown the pipeline produces, with every embedded image'sdata:URI normalised to the literaldata:embedded-image(the base64 payload is volatile and not what the snapshot is pinning).
article covers headings, paragraphs, bold/italic, a link, a bullet list
with nesting, an ordered list, an embedded image, and a table. formatting
covers strikethrough, inline code, a blockquote, and three levels of nesting.
Fidelity notes (structure, not layout — ADR 0009)
- ODT images lose their alt text and ODT tables lose their header row: pandoc's HTML→ODT writer does not encode either, so the source documents genuinely lack them. The DOCX variants keep both. This is a pinned pandoc limitation, not an import bug — extend the corpus rather than chasing it.
Limits (ADR 0009)
An uploaded document may be at most 25 MiB (MAX_CONVERSION_INPUT_BYTES),
and each of the two conversion passes may run for at most 60 s
(CONVERSION_TIMEOUT_MS). A ~50-page document converts well inside that ceiling
(import.fixtures.test.ts asserts it); a document that exceeds the timeout fails
the job with converter_timeout.
Regenerating
The documents and snapshots are generated with the pinned pandoc/core:3.6
(the production sidecar) so the snapshots match CI; a different pandoc version
wraps lists and pads tables differently. scripts/gen-import-fixtures.mjs
regenerates everything against a reachable pandoc sidecar (PANDOC_URL).