stwaidele/dorfteich
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases 22 Wiki Activity

Conversion job queue and pandoc sidecar integration #62

New Issue
Closed
opened 2026-07-04 14:52:32 +02:00 by fable-5 · 1 comment
fable-5 commented 2026-07-04 14:52:32 +02:00
Collaborator
Copy Link

Context

Import/export conversions run asynchronously against internal sidecars with limits and graceful failure (ADR 0009).

Scope

Add pandoc (server mode) to the Compose stack (internal network, pinned image, healthcheck); implement the conversion job flow on the jobs table: enqueue → worker calls sidecar with timeout (60 s) and size limit → store result/error → GET /jobs/:id polling endpoint with job ownership check; typed client for pandoc-server in a dedicated service; failure modes (sidecar down, timeout, unsupported content) mapped to distinct localized errors.

Acceptance criteria

  • a queued conversion survives an API restart and completes
  • sidecar down → job fails with a clear error after retries; API stays healthy
  • timeout kills the request and fails the job (test with a delay-injecting fake)
  • compose healthcheck covers pandoc; readyz includes converter reachability (warning-level)

Technical notes

  • ADR 0009, deployment.md (service table), data-model.md (jobs).

Dependencies

Depends on #6, #31.

Size: ~1.5 days

Conventions: English code/comments, clear human-readable code, no hard-coded UI strings (ADR 0012, add de and en), permission checks only via the shared guard (docs/architecture/permissions.md). Read the referenced ADRs before starting.

## Context Import/export conversions run asynchronously against internal sidecars with limits and graceful failure (ADR 0009). ## Scope Add `pandoc` (server mode) to the Compose stack (internal network, pinned image, healthcheck); implement the conversion job flow on the `jobs` table: enqueue → worker calls sidecar with timeout (60 s) and size limit → store result/error → `GET /jobs/:id` polling endpoint with job ownership check; typed client for pandoc-server in a dedicated service; failure modes (sidecar down, timeout, unsupported content) mapped to distinct localized errors. ## Acceptance criteria - [ ] a queued conversion survives an API restart and completes - [ ] sidecar down → job fails with a clear error after retries; API stays healthy - [ ] timeout kills the request and fails the job (test with a delay-injecting fake) - [ ] compose healthcheck covers pandoc; readyz includes converter reachability (warning-level) ## Technical notes - ADR 0009, deployment.md (service table), data-model.md (`jobs`). ## Dependencies Depends on #6, #31. **Size**: ~1.5 days --- *Conventions: English code/comments, clear human-readable code, no hard-coded UI strings (ADR 0012, add `de` **and** `en`), permission checks only via the shared guard (docs/architecture/permissions.md). Read the referenced ADRs before starting.*
fable-5 added this to the M6 — Import, export & attachments milestone 2026-07-04 14:52:32 +02:00
fable-5 added the
deployment
backend
 labels 2026-07-04 14:52:32 +02:00
opus-4-8 commented 2026-07-10 04:24:13 +02:00
Collaborator
Copy Link

#62 verifiziert & abgeschlossen — Conversion-Job-Queue + pandoc-Sidecar (ADR 0009).

Pipeline (4755c18): alle 7 Kontexte grün — CI (Lint/Typecheck/Test, i18n:check, api-db 193), Auth-e2e, Build → Deploy-Test → Smoke → Promote-Int. Der Deploy lief sauber durch, obwohl die Stages den Sidecar noch nicht hatten: die api degradiert bei fehlendem pandoc auf warn-level (readyz converter: warn, Instanz bleibt ready) — genau das zeigt der Smoke-Test-Durchlauf.

Stage-Provisioning (Test + Int, VPS 188.245.116.44): neue docker-compose.yml (mit pandoc-Service) in /home/DOCKER/dorfteich-{test,int}/ eingespielt (Backup *.bak-pre62), docker compose pull pandoc && up -d.

  • pandoc-Container healthy auf beiden Stages (Compose-Healthcheck wget /version).
  • readyz converter: ok auf Test und Int (vorher warn (fetch failed) → Beweis für die warn-level-Degradation).
  • Interner Pfad live geprüft: dorfteich-test-api-1 → http://pandoc:3030/ konvertiert markdown→html (korrektes HTML) und markdown→docx (gültige PK/OOXML-Bytes, 9925 B) — derselbe Pfad, den der Worker nutzt.

Umfang:

  • Sidecar pandoc/core:3.6 (server mode) im internen Netz, gepinnt, Healthcheck; api depends_on pandoc + neue Env PANDOC_URL (default http://pandoc:3030). readyz converter-Check (warn-level, nie unready).
  • ConversionJob-Tabelle (Per-Request-Queue, getrennt von der name-keyed Maintenance-Job-Tabelle): owner, Formate, input/result-Bytes, status, attempts, lockedAt.
  • PandocServerConverter: POST / mit {text,from,to,standalone}, Binär-Input-Formate (docx/odt) base64 im text, 60s-Timeout (AbortController), Input/Output-Size-Caps; Fehler → converter_unavailable/converter_timeout (retryable) vs conversion_failed (final).
  • ConversionWorker: claimt je einen Job via FOR UPDATE SKIP LOCKED (safe gegen überlappende Sweeps + 2. Prozess), Stale-Lock-Recovery, retry-then-fail (3 Versuche); 2s-Sweep + wake-on-enqueue → überlebt API-Neustart.
  • ConversionJobService.enqueue (size-limited) + owner-scoped GET /jobs/:id (Poll) + GET /jobs/:id/result (Stream); fremd/unbekannt → 404.

Akzeptanzkriterien:

  • ✅ Queued Conversion überlebt API-Neustart und läuft durch (persistierter PENDING-Job von frischer Worker-Instanz abgearbeitet).
  • ✅ Sidecar down → Job scheitert nach Retries (3), API bleibt gesund (healthz 200).
  • ✅ Timeout killt den Request und lässt den Job scheitern (delay-injizierender Fake-Server → converter_timeout).
  • ✅ Compose-Healthcheck deckt pandoc ab; readyz enthält Converter-Reachability (warn-level).

Tests: conversion-job.e2e.db.test.ts (Fake-Converter via neuen createTestApp-Override-Hook: enqueue→convert→poll→result, fremd/unbekannt 404, Restart-Survival, sidecar-down-fails-after-retries+API-healthy) + pandoc.converter.test.ts (success, non-200→conversion_failed, refused→converter_unavailable, delay→converter_timeout). Lokal zusätzlich gegen echtes pandoc/core:3.6 verifiziert (markdown→docx, docx→markdown-Roundtrip). api 193 (+9), shared 121, web 50 grün.

Nächste M6-Issues bauen darauf auf: #63 (Import docx/odt) + #65 (Export) enqueuen Jobs auf dieser Queue.

**#62 verifiziert & abgeschlossen** — Conversion-Job-Queue + pandoc-Sidecar (ADR 0009). **Pipeline** (`4755c18`): alle 7 Kontexte grün — CI (Lint/Typecheck/Test, i18n:check, api-db 193), Auth-e2e, Build → Deploy-Test → Smoke → Promote-Int. Der Deploy lief sauber durch, obwohl die Stages den Sidecar noch nicht hatten: die api degradiert bei fehlendem pandoc auf **warn-level** (readyz `converter: warn`, Instanz bleibt ready) — genau das zeigt der Smoke-Test-Durchlauf. **Stage-Provisioning (Test + Int, VPS 188.245.116.44):** neue `docker-compose.yml` (mit pandoc-Service) in `/home/DOCKER/dorfteich-{test,int}/` eingespielt (Backup `*.bak-pre62`), `docker compose pull pandoc && up -d`. - pandoc-Container `healthy` auf beiden Stages (Compose-Healthcheck `wget /version`). - readyz `converter: ok` auf Test **und** Int (vorher `warn (fetch failed)` → Beweis für die warn-level-Degradation). - Interner Pfad live geprüft: `dorfteich-test-api-1` → `http://pandoc:3030/` konvertiert markdown→html (korrektes HTML) **und** markdown→docx (gültige PK/OOXML-Bytes, 9925 B) — derselbe Pfad, den der Worker nutzt. **Umfang:** - Sidecar `pandoc/core:3.6` (server mode) im internen Netz, gepinnt, Healthcheck; api `depends_on` pandoc + neue Env `PANDOC_URL` (default `http://pandoc:3030`). readyz `converter`-Check (warn-level, nie unready). - `ConversionJob`-Tabelle (Per-Request-Queue, getrennt von der name-keyed Maintenance-`Job`-Tabelle): owner, Formate, input/result-Bytes, status, attempts, lockedAt. - `PandocServerConverter`: POST `/` mit `{text,from,to,standalone}`, Binär-Input-Formate (docx/odt) base64 im `text`, 60s-Timeout (AbortController), Input/Output-Size-Caps; Fehler → `converter_unavailable`/`converter_timeout` (retryable) vs `conversion_failed` (final). - `ConversionWorker`: claimt je einen Job via `FOR UPDATE SKIP LOCKED` (safe gegen überlappende Sweeps + 2. Prozess), Stale-Lock-Recovery, retry-then-fail (3 Versuche); 2s-Sweep + wake-on-enqueue → **überlebt API-Neustart**. - `ConversionJobService.enqueue` (size-limited) + owner-scoped `GET /jobs/:id` (Poll) + `GET /jobs/:id/result` (Stream); fremd/unbekannt → 404. **Akzeptanzkriterien:** - ✅ Queued Conversion überlebt API-Neustart und läuft durch (persistierter PENDING-Job von frischer Worker-Instanz abgearbeitet). - ✅ Sidecar down → Job scheitert nach Retries (3), API bleibt gesund (healthz 200). - ✅ Timeout killt den Request und lässt den Job scheitern (delay-injizierender Fake-Server → `converter_timeout`). - ✅ Compose-Healthcheck deckt pandoc ab; readyz enthält Converter-Reachability (warn-level). **Tests:** `conversion-job.e2e.db.test.ts` (Fake-Converter via neuen `createTestApp`-Override-Hook: enqueue→convert→poll→result, fremd/unbekannt 404, Restart-Survival, sidecar-down-fails-after-retries+API-healthy) + `pandoc.converter.test.ts` (success, non-200→conversion_failed, refused→converter_unavailable, delay→converter_timeout). Lokal zusätzlich gegen echtes `pandoc/core:3.6` verifiziert (markdown→docx, docx→markdown-Roundtrip). api 193 (+9), shared 121, web 50 grün. Nächste M6-Issues bauen darauf auf: #63 (Import docx/odt) + #65 (Export) enqueuen Jobs auf dieser Queue.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels   
area:auth

   
area:docs

   
area:export

   
area:ops

   
area:storage

   
area:supply-chain

   
auth

Authentication, permissions, quotas

  
backend

API server (apps/api)

  
blocked

Blocked on external dependency/infrastructure

  
collab

Collaboration server (apps/collab)

  
deployment

Docker, CI/CD, hosting

  
docs

Documentation

  
effort:L

   
effort:M

   
effort:S

   
frontend

SPA (apps/web)

  
plugins

Plugin system and SDK

  
qa

Tests, fixtures, e2e

  
vs-nfd

   
vs-nfd:blocker
No Label
area:auth
area:docs
area:export
area:ops
area:storage
area:supply-chain
auth
backend
blocked
collab
deployment
docs
effort:L
effort:M
effort:S
frontend
plugins
qa
vs-nfd
vs-nfd:blocker
Milestone
Clear milestone
No items
No Milestone
M6 — Import, export & attachments
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: stwaidele/dorfteich#62
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?