# Import/export fidelity corpus (issue #69, ADR 0009)

The fidelity contract for document conversion is **structure-true best effort**:
headings, paragraphs, lists, tables, images, links, and inline emphasis are
preserved; layout (columns, text boxes, exact spacing) is explicitly out of
scope (ADR 0009). "Best effort" only stays meaningful if it is executable, so
this corpus **is** the contract:

> **The corpus is green ⇒ current fidelity is acceptable.**

The gate runs in CI (`.gitea/workflows/ci.yml`, job _Import/export fidelity
gate_) against the **pinned** sidecar images the stages run — `pandoc/core:3.6`
and `gotenberg/gotenberg:8`. It is deliberately a small, separate job that stays
well under five minutes; the same suites self-skip in the main `checks` job,
which has no sidecars.

## What the gate covers

| Suite            | File                                                 | What it pins                                                                         |
| ---------------- | ---------------------------------------------------- | ------------------------------------------------------------------------------------ |
| Import snapshots | `apps/api/src/import-export/import.fixtures.test.ts` | `.docx`/`.odt` → Markdown for the [`import/`](./import) corpus                       |
| Export snapshots | `apps/api/src/import-export/export.fidelity.test.ts` | Markdown → `.docx`/`.odt` → Markdown round trip for the [`export/`](./export) corpus |
| PDF smoke        | `apps/api/src/import-export/pdf.fidelity.test.ts`    | a rendered PDF contains the expected text, with a sane page count                    |

Each corpus directory has its own README describing its files.

## The fixture-first bug process (the contract in practice)

A fidelity problem is not fixed by hand-patching output until one document looks
right — that lets "best effort" quietly erode. Instead, **every fidelity bug
enters the corpus first**:

1. **Reproduce it as a fixture.** Add the smallest document that exhibits the
   problem to the relevant corpus (`import/` a source document, `export/` a
   source `.md`). If the shape is new, extend an existing source rather than
   adding a near-duplicate.
2. **Regenerate the snapshot** with the pinned sidecar so it records _today's_
   real output:
   - import: `PANDOC_URL=… node scripts/gen-import-fixtures.mjs`
   - export: `PANDOC_URL=… node scripts/gen-export-fixtures.mjs`
     (start the sidecar with `docker run --rm -p 3030:3030 pandoc/core:3.6 server`).
3. **Commit the fixture and its snapshot together**, and review the snapshot
   diff — it is the human-readable statement of what the conversion does. If the
   snapshot encodes the _bug_, fix the pipeline and regenerate until the diff
   shows the intended structure; if it encodes a genuine pinned-tool limitation
   (see below), the snapshot _is_ the accepted behaviour and the fix is
   documentation, not code.
4. **The gate now defends it.** Any later regression — ours or a tool's —
   changes that document's output and fails the suite.

## Why pinning matters (and how a version bump fails the gate)

The snapshots are byte-exact and generated with the pinned sidecar versions.
pandoc changes its writers between releases (3.10 wraps lists and pads tables
differently from 3.6), so **bumping `pandoc/core` — or `gotenberg/gotenberg` —
without regenerating shifts the output and fails the suite.** That is the point:
a version change becomes a reviewed snapshot diff, never a silent drift. To bump
a sidecar: change the pin in `deploy/compose/docker-compose.yml`, the CI job, and
these tests' defaults; regenerate the corpus; review the diff; commit together.

## Known pinned-tool limitations (accepted, not bugs)

- **ODT loses image alt text and table header rows**: pandoc's HTML→ODT _writer_
  does not encode either, so the ODT sources genuinely lack them (the DOCX
  variants keep both).
- **ODT loosens tight lists and flattens fenced code blocks** to plain
  paragraphs on the export round trip; DOCX preserves both.

These are recorded in the corpus snapshots on purpose — extend the corpus rather
than chasing layout parity.