dorfteich/apps/api/src/import-export/import.fixtures.test.ts
Claude Opus 4.8 546e8279ac
All checks were successful
CD / Build and push images (push) Successful in 3m19s
CI / Lint, typecheck, test (push) Successful in 2m55s
CI / Auth e2e pack (push) Successful in 3m45s
CI / Build container images (push) Has been skipped
CD / Deploy to Test (push) Successful in 9s
CD / Smoke tests against Test (push) Successful in 1m13s
CD / Promote to Int (push) Successful in 11s
Import .docx and .odt documents as new pages (#63)
Uploading a Word/OpenOffice document to POST /ponds/:id/import enqueues a
conversion job (the #62 queue) that produces a new page in the pond; the
client polls GET /jobs/:id for the created resultPageId.

Pipeline (ImportService, ADR 0009): pandoc-server is stateless and hands
back a document's media no other way, so we convert in two passes —
docx/odt → html with embed-resources inlines every image as a data: URI,
then html → gfm produces clean structural Markdown with those data URIs
still inline. Embedded images are stored as pond files (with quota
accounting) and their references rewritten to file ids on the Markdown
text before parsing (the editor parser only admits png/jpeg/gif/webp data
URIs); an image whose bytes the upload pipeline rejects is dropped, not
fatal. The title comes from a leading top-level heading (removed from the
body) else the file name. The page is created from the resulting Yjs state.

The shared conversion worker routes import-kind jobs to the pipeline via a
token (breaking a module cycle), so import inherits the queue's locking,
retry, and restart-survival. Media stored during a failed attempt is rolled
back; a pond that runs out of storage fails the job with quota_exceeded.

- schema: ConversionJob gains pond_id / source_name / result_page_id
  (migration 20260710041215_import_pages_conversion); ConversionJobView
  gains resultPageId.
- PagesService.createWithState / yjs-content docToState build a page from a
  prepared document; FilesService.linkAttachmentsToPage links import media.
- fixtures/import/: representative .docx/.odt corpus (headings, lists,
  nested lists, tables, images, links, bold/italic) with expected-Markdown
  snapshots; scripts/gen-import-fixtures.mjs regenerates them.
- tests: import.service.db.test.ts drives the full pipeline with a fake
  converter (CI); import.fixtures.test.ts runs the real two-pass conversion
  over the corpus and a 50-page timing check against a reachable sidecar.
- i18n: import_unsupported_format (de+en). Limits documented (25 MiB input,
  60 s per pass).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwZ4jR4KFAPvpjWevfUGX1
2026-07-10 07:34:43 +02:00

79 lines
3.3 KiB
TypeScript

import { readFileSync } from 'node:fs';
import { join } from 'node:path';
import { markdownToDoc } from '@dorfteich/shared';
import { beforeAll, describe, expect, it, TestContext } from 'vitest';
import { AppConfig } from '../config/app-config.service';
import { convertImportedDocument } from './import.service';
import { CONVERSION_TIMEOUT_MS, PandocServerConverter } from './pandoc.converter';
/**
* Import fidelity regression (issue #63, ADR 0009): runs the real two-pass
* pandoc conversion over the committed `.docx`/`.odt` corpus and asserts each
* produces its expected Markdown. Needs a reachable pandoc sidecar (the pinned
* `pandoc/core:3.6`, so output matches the snapshots) — each test skips itself
* when none is configured, and CI starts one and points `PANDOC_URL` at it.
*/
const PANDOC_URL = process.env.PANDOC_URL ?? 'http://localhost:3030';
// The test runner's cwd is `apps/api`; the corpus lives at the repo root.
const FIXTURES = join(process.cwd(), '../../fixtures/import');
const converter = new PandocServerConverter({
env: { PANDOC_URL },
} as unknown as AppConfig);
let reachable = false;
/** Normalise embedded image `data:` URIs to the stable token the snapshots use
* (the base64 payload is volatile and not what we are pinning). */
function normalize(markdown: string): string {
return markdown.replace(
/data:image\/[a-zA-Z0-9.+-]+;base64,[A-Za-z0-9+/=]+/g,
'data:embedded-image',
);
}
const CORPUS = ['article.docx', 'article.odt', 'formatting.docx', 'formatting.odt'];
describe('import fixture corpus (real pandoc, issue #63)', () => {
beforeAll(async () => {
reachable = await converter.reachable().catch(() => false);
});
for (const fixture of CORPUS) {
it(`converts ${fixture} to its expected Markdown`, async (ctx: TestContext) => {
if (!reachable) ctx.skip();
const format = fixture.endsWith('.odt') ? 'odt' : 'docx';
const document = readFileSync(join(FIXTURES, fixture));
const expected = readFileSync(join(FIXTURES, `${fixture}.expected.md`), 'utf8');
const markdown = await convertImportedDocument(converter, format, document);
expect(normalize(markdown)).toBe(expected);
// The Markdown must also parse into a valid editor document (no schema
// surprises from real-world structure).
expect(() => markdownToDoc(markdown)).not.toThrow();
});
}
it('imports a 50-page document within the conversion timeout', async (ctx: TestContext) => {
if (!reachable) ctx.skip();
// Build a ~50-page document by repeating a page of structured content, then
// convert it to docx once and time the import conversion of that document.
const onePage =
'# Section\n\n' + 'A paragraph of survey notes about the pond. '.repeat(20) + '\n\n';
const large = Array.from({ length: 50 }, () => onePage).join('\n---\n\n');
const docx = await converter.convert({ from: 'gfm', to: 'docx', input: Buffer.from(large) });
const started = Date.now();
const markdown = await convertImportedDocument(converter, 'docx', docx.output);
const elapsed = Date.now() - started;
expect(markdown.length).toBeGreaterThan(1000);
// Comfortably inside the documented 60 s per-conversion ceiling (ADR 0009).
expect(elapsed).toBeLessThan(CONVERSION_TIMEOUT_MS);
});
});