f8c241b11a
5 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
| 74a9e495e4 |
#209: pandoc reference documents carry the VS-NfD marking for DOCX/ODT
reference-vs-nfd.docx/.odt ship as derived binaries: the pinned pandoc's
default reference documents plus a header and footer with the marking —
part of the document's page setup, so it repeats on every page in Word
and LibreOffice and is not deletable body text. Source of truth is
scripts/gen-classified-reference-docs.mjs (wording from shared
classificationMarking(); maintenance documented in assets/README.md).
The converter passes reference docs to pandoc-server via in-request
files + reference-doc; the worker attaches them for marked docx/odt jobs
(job option {marking}, as in #208). Unclassified exports pass nothing
and are unchanged (pinned by fake-converter test). Fidelity suite
asserts against real pandoc 3.6 that marked outputs carry the
header/footer parts and unmarked ones do not; per-page repetition
verified via LibreOffice 25.8 headless PDF (5/5 pages, 2 markings each,
both formats). Word: quick manual look pending (sample files in the
workspace), procedure documented in assets/README.md.
Co-Authored-By: Claude Fable 5 (1M context) <noreply@anthropic.com>
|
|||
| ff505bc752 |
#233: prune conversion job payloads for every job kind
Some checks failed
CI / Lint, typecheck, test (pull_request) Successful in 5m12s
CI / Build container images (pull_request) Successful in 3m28s
CI / Auth e2e pack (pull_request) Successful in 8m33s
CI / Import/export fidelity gate (pull_request) Successful in 1m2s
CD / Build and push images (push) Successful in 29s
CD / Deploy to Test (push) Successful in 12s
CD / Smoke tests against Test (push) Successful in 1m22s
CD / Promote to Int (push) Successful in 11s
CI / Lint, typecheck, test (push) Failing after 5m9s
CI / Auth e2e pack (push) Has been skipped
CI / Import/export fidelity gate (push) Has been skipped
CI / Build container images (push) Has been skipped
The raw input/result bytes of import/export conversion jobs were kept forever; a deleted classified page could live on inside its last export. A new daily conversion-payload-prune job nulls both once a finished (succeeded or failed) job passes conversion.payloadRetentionDays (instance setting, default 30) — the row survives for status/audit. PENDING and RUNNING rows keep their payload, so the worker's stale-lock recovery path is untouched; a hand-requeued pruned job fails finally via conversionInputOf instead of crashing the worker. The input column becomes nullable; the migration backfills by clearing payloads of jobs already finished longer ago than the default period (recent results stay downloadable until they age out). Job-count fence in system.spec: 7 -> 8 (new scheduler registration). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0168Ph5uBmHm8X28CSVpbpnJ |
|||
| 8ae010218e |
Obsidian vault import: endpoint, job orchestration, rollback (#117)
Some checks failed
CD / Build and push images (push) Successful in 4m5s
CD / Deploy to Test (push) Successful in 9s
CI / Lint, typecheck, test (push) Failing after 4m21s
CI / Auth e2e pack (push) Has been skipped
CI / Import/export fidelity gate (push) Has been skipped
CI / Build container images (push) Has been skipped
CD / Smoke tests against Test (push) Successful in 1m22s
CD / Promote to Int (push) Successful in 11s
POST /ponds/:pondId/import/vault (pond-admin-gated; a vault import
creates a subtree, uploads files, and creates labels — administration,
not everyday editing) takes the ZIP plus a JSON options field
{parentPageId?, labelIds?, frontmatterMode}. The archive is parsed at
enqueue for fast 400s; the job (new kind import_vault, riding the
existing isImportKind worker routing) re-parses and runs the #116
transform, then: containers top-down → notes (asset placeholders →
uploaded pond files; non-images become page attachments) → tags to
labels (nested tags build a label hierarchy via LabelsService, so
locking and cache invalidation apply) plus the dialog labels.
All-or-nothing: any failure hard-deletes the created pages (children
first) and removes the stored files (quota restored), then surfaces as
import_vault_invalid_zip / import_vault_too_large / quota_exceeded /
conversion_failed — and makes the worker's retry policy safe.
Supporting changes:
- conversion_jobs gains a nullable options jsonb column; enqueue takes
kind-specific options and a maxInputBytes override (the 25 MiB
default protects the pandoc sidecar, which a vault never touches —
vaults use the 64 MiB upload limit).
- insertPage accepts a pre-reserved slug (the batch reserves all slugs
up front against pond ∪ batch).
- NEW: pages born with content seed their outgoing page_links rows
(deriveContent now returns wikilinkSlugs) — imported pages would
otherwise stay invisible to backlinks and the graph until their
first collab save. Collab still rewrites the rows on every save, and
the existing phantom resolution heals batch creation order.
import-vault.e2e.db.test.ts (4 tests, real worker drained): gating +
input rejection, the full fixture import (tree under a mount page,
collision suffixes, link rows incl. phantom, nested tag labels, extra
label everywhere, frontmatter stripped, image embedded + PDF attached),
complete quota rollback, and a clean re-import with fresh suffixes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
|||
| 546e8279ac |
Import .docx and .odt documents as new pages (#63)
All checks were successful
CD / Build and push images (push) Successful in 3m19s
CI / Lint, typecheck, test (push) Successful in 2m55s
CI / Auth e2e pack (push) Successful in 3m45s
CI / Build container images (push) Has been skipped
CD / Deploy to Test (push) Successful in 9s
CD / Smoke tests against Test (push) Successful in 1m13s
CD / Promote to Int (push) Successful in 11s
Uploading a Word/OpenOffice document to POST /ponds/:id/import enqueues a conversion job (the #62 queue) that produces a new page in the pond; the client polls GET /jobs/:id for the created resultPageId. Pipeline (ImportService, ADR 0009): pandoc-server is stateless and hands back a document's media no other way, so we convert in two passes — docx/odt → html with embed-resources inlines every image as a data: URI, then html → gfm produces clean structural Markdown with those data URIs still inline. Embedded images are stored as pond files (with quota accounting) and their references rewritten to file ids on the Markdown text before parsing (the editor parser only admits png/jpeg/gif/webp data URIs); an image whose bytes the upload pipeline rejects is dropped, not fatal. The title comes from a leading top-level heading (removed from the body) else the file name. The page is created from the resulting Yjs state. The shared conversion worker routes import-kind jobs to the pipeline via a token (breaking a module cycle), so import inherits the queue's locking, retry, and restart-survival. Media stored during a failed attempt is rolled back; a pond that runs out of storage fails the job with quota_exceeded. - schema: ConversionJob gains pond_id / source_name / result_page_id (migration 20260710041215_import_pages_conversion); ConversionJobView gains resultPageId. - PagesService.createWithState / yjs-content docToState build a page from a prepared document; FilesService.linkAttachmentsToPage links import media. - fixtures/import/: representative .docx/.odt corpus (headings, lists, nested lists, tables, images, links, bold/italic) with expected-Markdown snapshots; scripts/gen-import-fixtures.mjs regenerates them. - tests: import.service.db.test.ts drives the full pipeline with a fake converter (CI); import.fixtures.test.ts runs the real two-pass conversion over the corpus and a 50-page timing check against a reachable sidecar. - i18n: import_unsupported_format (de+en). Limits documented (25 MiB input, 60 s per pass). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EwZ4jR4KFAPvpjWevfUGX1 |
|||
| 4755c18ef5 |
Add conversion job queue and pandoc sidecar integration (#62)
All checks were successful
CD / Build and push images (push) Successful in 4m9s
CI / Lint, typecheck, test (push) Successful in 2m50s
CI / Auth e2e pack (push) Successful in 3m44s
CI / Build container images (push) Has been skipped
CD / Deploy to Test (push) Successful in 8s
CD / Smoke tests against Test (push) Successful in 1m15s
CD / Promote to Int (push) Successful in 11s
Import/export conversions run asynchronously against an internal pandoc-server sidecar with limits and graceful failure (ADR 0009). This is the plumbing; the import (#63) and export (#65) features enqueue jobs onto it. Sidecar & config: - pandoc/core:3.6 in HTTP server mode added to the Compose stack, internal network only, with a wget healthcheck on /version; the api depends on it healthy and reaches it via the new PANDOC_URL env (default http://pandoc:3030). - readyz gains a warning-level `converter` check: an unreachable sidecar degrades import/export but never flips the instance to unready (new `warn` status on ReadinessCheck). Conversion flow (apps/api/src/import-export/): - ConversionJob table (per-request work queue, distinct from the name-keyed maintenance Job table): owner, formats, input/result bytes, status, attempts, lockedAt. Migration + owner cascade. - PandocConverter (abstract) + PandocServerConverter: POST / with {text,from,to,standalone}; binary input formats (docx/odt/…) are base64-encoded in `text`; 60 s AbortController timeout; input/output size caps. Failures map to distinct localized codes — converter_unavailable / converter_timeout (retryable) and conversion_failed (final). - ConversionWorker: claims one job at a time with `FOR UPDATE SKIP LOCKED` (safe against overlapping sweeps and a second process), recovers a stale RUNNING lock, retries transient failures up to 3 attempts then fails. A 2 s sweep plus wake-on-enqueue means a queued job survives an API restart. - ConversionJobService.enqueue (size-limited) + owner-scoped GET /jobs/:id (poll) and GET /jobs/:id/result (stream the output); a foreign/unknown id is 404. ConversionJobView in @dorfteich/shared. Tests: - conversion-job.e2e.db.test.ts (fake converter injected via a new createTestApp override hook): enqueue→convert→poll→result; foreign/unknown job 404; a persisted PENDING job picked up by a fresh app's worker (restart survival); sidecar-down fails after 3 retries while the API stays healthy. - pandoc.converter.test.ts: success, non-200→conversion_failed, refused→ converter_unavailable, and a delay-injecting server→converter_timeout. - Verified locally against a real pandoc/core:3.6 container: markdown→html, markdown→docx (valid PK/OOXML bytes), and a docx→markdown round-trip. Local: typecheck, lint, i18n:check, build all green; api 193 tests (9 new), shared 121, web 50. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EwZ4jR4KFAPvpjWevfUGX1 |
