Claude Opus 4.8
546e8279ac
All checks were successful
CD / Build and push images (push) Successful in 3m19s
CI / Lint, typecheck, test (push) Successful in 2m55s
CI / Auth e2e pack (push) Successful in 3m45s
CI / Build container images (push) Has been skipped
CD / Deploy to Test (push) Successful in 9s
CD / Smoke tests against Test (push) Successful in 1m13s
CD / Promote to Int (push) Successful in 11s
Uploading a Word/OpenOffice document to POST /ponds/:id/import enqueues a conversion job (the #62 queue) that produces a new page in the pond; the client polls GET /jobs/:id for the created resultPageId. Pipeline (ImportService, ADR 0009): pandoc-server is stateless and hands back a document's media no other way, so we convert in two passes — docx/odt → html with embed-resources inlines every image as a data: URI, then html → gfm produces clean structural Markdown with those data URIs still inline. Embedded images are stored as pond files (with quota accounting) and their references rewritten to file ids on the Markdown text before parsing (the editor parser only admits png/jpeg/gif/webp data URIs); an image whose bytes the upload pipeline rejects is dropped, not fatal. The title comes from a leading top-level heading (removed from the body) else the file name. The page is created from the resulting Yjs state. The shared conversion worker routes import-kind jobs to the pipeline via a token (breaking a module cycle), so import inherits the queue's locking, retry, and restart-survival. Media stored during a failed attempt is rolled back; a pond that runs out of storage fails the job with quota_exceeded. - schema: ConversionJob gains pond_id / source_name / result_page_id (migration 20260710041215_import_pages_conversion); ConversionJobView gains resultPageId. - PagesService.createWithState / yjs-content docToState build a page from a prepared document; FilesService.linkAttachmentsToPage links import media. - fixtures/import/: representative .docx/.odt corpus (headings, lists, nested lists, tables, images, links, bold/italic) with expected-Markdown snapshots; scripts/gen-import-fixtures.mjs regenerates them. - tests: import.service.db.test.ts drives the full pipeline with a fake converter (CI); import.fixtures.test.ts runs the real two-pass conversion over the corpus and a 50-page timing check against a reachable sidecar. - i18n: import_unsupported_format (de+en). Limits documented (25 MiB input, 60 s per pass). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EwZ4jR4KFAPvpjWevfUGX1
70 lines
2.7 KiB
JavaScript
70 lines
2.7 KiB
JavaScript
#!/usr/bin/env node
|
|
// Regenerate the import fixture corpus (issue #63, ADR 0009) from the committed
|
|
// `*.src.html` sources: for each source, write `<name>.docx`, `<name>.odt`, and
|
|
// the expected Markdown snapshot for each format. Run against a reachable pandoc
|
|
// sidecar (the pinned `pandoc/core:3.6`, so snapshots match CI):
|
|
//
|
|
// docker run --rm -p 3030:3030 pandoc/core:3.6 server
|
|
// PANDOC_URL=http://localhost:3030 node scripts/gen-import-fixtures.mjs
|
|
//
|
|
// The pipeline mirrors ImportService: `html → <ext>` builds the document, then
|
|
// `<ext> → html` (embed-resources) → `html → gfm` reproduces what an import
|
|
// would parse, with image data URIs normalised to a stable token.
|
|
|
|
import { readFileSync, writeFileSync, readdirSync } from 'node:fs';
|
|
import { fileURLToPath } from 'node:url';
|
|
import { dirname, join } from 'node:path';
|
|
|
|
const FIXTURES = join(dirname(fileURLToPath(import.meta.url)), '..', 'fixtures', 'import');
|
|
const PANDOC_URL = process.env.PANDOC_URL ?? 'http://localhost:3030';
|
|
const MARKDOWN_FORMAT = 'gfm-implicit_figures-raw_html';
|
|
|
|
/** POST a conversion to pandoc-server. Without an `Accept` header the server
|
|
* returns text for text writers and raw bytes for binary ones (docx/odt). */
|
|
async function pandoc(params, binaryOut = false) {
|
|
const res = await fetch(`${PANDOC_URL}/`, {
|
|
method: 'POST',
|
|
headers: { 'Content-Type': 'application/json' },
|
|
body: JSON.stringify(params),
|
|
});
|
|
if (!res.ok) throw new Error(`pandoc ${res.status}: ${await res.text()}`);
|
|
return binaryOut ? Buffer.from(await res.arrayBuffer()) : res.text();
|
|
}
|
|
|
|
async function buildDocument(html, ext) {
|
|
return pandoc({ text: html, from: 'html', to: ext, standalone: true }, true);
|
|
}
|
|
|
|
async function toExpectedMarkdown(documentBytes, ext) {
|
|
const embedded = await pandoc({
|
|
text: documentBytes.toString('base64'),
|
|
from: ext,
|
|
to: 'html',
|
|
standalone: false,
|
|
'embed-resources': true,
|
|
});
|
|
const md = await pandoc({
|
|
text: embedded,
|
|
from: 'html',
|
|
to: MARKDOWN_FORMAT,
|
|
standalone: false,
|
|
wrap: 'none',
|
|
});
|
|
return md.replace(/data:image\/[a-zA-Z0-9.+-]+;base64,[A-Za-z0-9+/=]+/g, 'data:embedded-image');
|
|
}
|
|
|
|
const sources = readdirSync(FIXTURES).filter((f) => f.endsWith('.src.html'));
|
|
for (const source of sources) {
|
|
const name = source.replace(/\.src\.html$/, '');
|
|
const html = readFileSync(join(FIXTURES, source), 'utf8');
|
|
for (const ext of ['docx', 'odt']) {
|
|
const bytes = await buildDocument(html, ext);
|
|
writeFileSync(join(FIXTURES, `${name}.${ext}`), bytes);
|
|
writeFileSync(
|
|
join(FIXTURES, `${name}.${ext}.expected.md`),
|
|
await toExpectedMarkdown(bytes, ext),
|
|
);
|
|
console.log(`wrote ${name}.${ext} (+ expected.md)`);
|
|
}
|
|
}
|