dorfteich/apps/api/src/import-export/conversion-job.service.ts
Claude Opus 4.8 4755c18ef5
All checks were successful
CD / Build and push images (push) Successful in 4m9s
CI / Lint, typecheck, test (push) Successful in 2m50s
CI / Auth e2e pack (push) Successful in 3m44s
CI / Build container images (push) Has been skipped
CD / Deploy to Test (push) Successful in 8s
CD / Smoke tests against Test (push) Successful in 1m15s
CD / Promote to Int (push) Successful in 11s
Add conversion job queue and pandoc sidecar integration (#62)
Import/export conversions run asynchronously against an internal pandoc-server
sidecar with limits and graceful failure (ADR 0009). This is the plumbing;
the import (#63) and export (#65) features enqueue jobs onto it.

Sidecar & config:
- pandoc/core:3.6 in HTTP server mode added to the Compose stack, internal
  network only, with a wget healthcheck on /version; the api depends on it
  healthy and reaches it via the new PANDOC_URL env (default http://pandoc:3030).
- readyz gains a warning-level `converter` check: an unreachable sidecar
  degrades import/export but never flips the instance to unready (new `warn`
  status on ReadinessCheck).

Conversion flow (apps/api/src/import-export/):
- ConversionJob table (per-request work queue, distinct from the name-keyed
  maintenance Job table): owner, formats, input/result bytes, status, attempts,
  lockedAt. Migration + owner cascade.
- PandocConverter (abstract) + PandocServerConverter: POST / with
  {text,from,to,standalone}; binary input formats (docx/odt/…) are base64-encoded
  in `text`; 60 s AbortController timeout; input/output size caps. Failures map
  to distinct localized codes — converter_unavailable / converter_timeout
  (retryable) and conversion_failed (final).
- ConversionWorker: claims one job at a time with `FOR UPDATE SKIP LOCKED`
  (safe against overlapping sweeps and a second process), recovers a stale
  RUNNING lock, retries transient failures up to 3 attempts then fails. A 2 s
  sweep plus wake-on-enqueue means a queued job survives an API restart.
- ConversionJobService.enqueue (size-limited) + owner-scoped GET /jobs/:id
  (poll) and GET /jobs/:id/result (stream the output); a foreign/unknown id is
  404. ConversionJobView in @dorfteich/shared.

Tests:
- conversion-job.e2e.db.test.ts (fake converter injected via a new createTestApp
  override hook): enqueue→convert→poll→result; foreign/unknown job 404; a
  persisted PENDING job picked up by a fresh app's worker (restart survival);
  sidecar-down fails after 3 retries while the API stays healthy.
- pandoc.converter.test.ts: success, non-200→conversion_failed, refused→
  converter_unavailable, and a delay-injecting server→converter_timeout.
- Verified locally against a real pandoc/core:3.6 container: markdown→html,
  markdown→docx (valid PK/OOXML bytes), and a docx→markdown round-trip.

Local: typecheck, lint, i18n:check, build all green; api 193 tests
(9 new), shared 121, web 50.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EwZ4jR4KFAPvpjWevfUGX1
2026-07-10 04:06:27 +02:00

111 lines
3.6 KiB
TypeScript

import { Injectable, NotFoundException, PayloadTooLargeException } from '@nestjs/common';
import { ConversionJob, ConversionJobStatus as PrismaStatus } from '@prisma/client';
import { ConversionJobStatus, ConversionJobView } from '@dorfteich/shared';
import { PinoLogger } from 'nestjs-pino';
import { PrismaService } from '../prisma/prisma.service';
import { ConversionWorker } from './conversion-worker.service';
import { MAX_CONVERSION_INPUT_BYTES } from './pandoc.converter';
export interface EnqueueConversion {
ownerId: string;
kind: string;
from: string;
to: string;
input: Buffer;
standalone?: boolean;
}
export interface ConversionResultPayload {
bytes: Buffer;
mimeType: string;
}
const STATUS_VIEW: Record<PrismaStatus, ConversionJobStatus> = {
PENDING: 'pending',
RUNNING: 'running',
SUCCEEDED: 'succeeded',
FAILED: 'failed',
};
/**
* Enqueues import/export conversions and answers owner-scoped polling
* (ADR 0009, issue #62). Enqueue persists the job (so it survives a restart)
* and wakes the worker for interactive latency; the worker
* ({@link ConversionWorker}) does the sidecar call out of band.
*/
@Injectable()
export class ConversionJobService {
constructor(
private readonly prisma: PrismaService,
private readonly worker: ConversionWorker,
private readonly logger: PinoLogger,
) {
this.logger.setContext(ConversionJobService.name);
}
async enqueue(request: EnqueueConversion): Promise<ConversionJob> {
if (request.input.byteLength > MAX_CONVERSION_INPUT_BYTES) {
throw new PayloadTooLargeException({
code: 'file_too_large',
details: { limitBytes: MAX_CONVERSION_INPUT_BYTES },
});
}
const job = await this.prisma.conversionJob.create({
data: {
ownerId: request.ownerId,
kind: request.kind,
sourceFormat: request.from,
targetFormat: request.to,
standalone: request.standalone ?? true,
// Prisma's Bytes maps to Uint8Array<ArrayBuffer>; a Node Buffer's
// backing store is ArrayBufferLike, so copy into a plain Uint8Array.
input: new Uint8Array(request.input),
},
});
this.logger.info(
{ jobId: job.id, kind: job.kind, ownerId: request.ownerId },
'audit: conversion enqueued',
);
this.worker.wake();
return job;
}
/** The job if it belongs to `userId`, else 404 — existence stays hidden from
* anyone but its owner (permissions.md §non-page objects). */
async getForOwner(id: string, userId: string): Promise<ConversionJobView> {
return this.viewOf(await this.ownedJob(id, userId));
}
/** The finished output bytes, or 404 while the job is not yet succeeded (so a
* caller cannot distinguish "still running" from "never existed"). */
async resultForOwner(id: string, userId: string): Promise<ConversionResultPayload> {
const job = await this.ownedJob(id, userId);
if (job.status !== 'SUCCEEDED' || !job.result) throw new NotFoundException();
return {
bytes: Buffer.from(job.result),
mimeType: job.resultMimeType ?? 'application/octet-stream',
};
}
private async ownedJob(id: string, userId: string): Promise<ConversionJob> {
const job = await this.prisma.conversionJob.findFirst({ where: { id, ownerId: userId } });
if (!job) throw new NotFoundException();
return job;
}
viewOf(job: ConversionJob): ConversionJobView {
return {
id: job.id,
status: STATUS_VIEW[job.status],
kind: job.kind,
sourceFormat: job.sourceFormat,
targetFormat: job.targetFormat,
errorCode: job.errorCode,
createdAt: job.createdAt.toISOString(),
updatedAt: job.updatedAt.toISOString(),
};
}
}