All checks were successful
CD / Build and push images (push) Successful in 4m9s
CI / Lint, typecheck, test (push) Successful in 2m50s
CI / Auth e2e pack (push) Successful in 3m44s
CI / Build container images (push) Has been skipped
CD / Deploy to Test (push) Successful in 8s
CD / Smoke tests against Test (push) Successful in 1m15s
CD / Promote to Int (push) Successful in 11s
Import/export conversions run asynchronously against an internal pandoc-server sidecar with limits and graceful failure (ADR 0009). This is the plumbing; the import (#63) and export (#65) features enqueue jobs onto it. Sidecar & config: - pandoc/core:3.6 in HTTP server mode added to the Compose stack, internal network only, with a wget healthcheck on /version; the api depends on it healthy and reaches it via the new PANDOC_URL env (default http://pandoc:3030). - readyz gains a warning-level `converter` check: an unreachable sidecar degrades import/export but never flips the instance to unready (new `warn` status on ReadinessCheck). Conversion flow (apps/api/src/import-export/): - ConversionJob table (per-request work queue, distinct from the name-keyed maintenance Job table): owner, formats, input/result bytes, status, attempts, lockedAt. Migration + owner cascade. - PandocConverter (abstract) + PandocServerConverter: POST / with {text,from,to,standalone}; binary input formats (docx/odt/…) are base64-encoded in `text`; 60 s AbortController timeout; input/output size caps. Failures map to distinct localized codes — converter_unavailable / converter_timeout (retryable) and conversion_failed (final). - ConversionWorker: claims one job at a time with `FOR UPDATE SKIP LOCKED` (safe against overlapping sweeps and a second process), recovers a stale RUNNING lock, retries transient failures up to 3 attempts then fails. A 2 s sweep plus wake-on-enqueue means a queued job survives an API restart. - ConversionJobService.enqueue (size-limited) + owner-scoped GET /jobs/:id (poll) and GET /jobs/:id/result (stream the output); a foreign/unknown id is 404. ConversionJobView in @dorfteich/shared. Tests: - conversion-job.e2e.db.test.ts (fake converter injected via a new createTestApp override hook): enqueue→convert→poll→result; foreign/unknown job 404; a persisted PENDING job picked up by a fresh app's worker (restart survival); sidecar-down fails after 3 retries while the API stays healthy. - pandoc.converter.test.ts: success, non-200→conversion_failed, refused→ converter_unavailable, and a delay-injecting server→converter_timeout. - Verified locally against a real pandoc/core:3.6 container: markdown→html, markdown→docx (valid PK/OOXML bytes), and a docx→markdown round-trip. Local: typecheck, lint, i18n:check, build all green; api 193 tests (9 new), shared 121, web 50. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EwZ4jR4KFAPvpjWevfUGX1
111 lines
3.6 KiB
TypeScript
111 lines
3.6 KiB
TypeScript
import { Injectable, NotFoundException, PayloadTooLargeException } from '@nestjs/common';
|
|
import { ConversionJob, ConversionJobStatus as PrismaStatus } from '@prisma/client';
|
|
import { ConversionJobStatus, ConversionJobView } from '@dorfteich/shared';
|
|
import { PinoLogger } from 'nestjs-pino';
|
|
|
|
import { PrismaService } from '../prisma/prisma.service';
|
|
|
|
import { ConversionWorker } from './conversion-worker.service';
|
|
import { MAX_CONVERSION_INPUT_BYTES } from './pandoc.converter';
|
|
|
|
export interface EnqueueConversion {
|
|
ownerId: string;
|
|
kind: string;
|
|
from: string;
|
|
to: string;
|
|
input: Buffer;
|
|
standalone?: boolean;
|
|
}
|
|
|
|
export interface ConversionResultPayload {
|
|
bytes: Buffer;
|
|
mimeType: string;
|
|
}
|
|
|
|
const STATUS_VIEW: Record<PrismaStatus, ConversionJobStatus> = {
|
|
PENDING: 'pending',
|
|
RUNNING: 'running',
|
|
SUCCEEDED: 'succeeded',
|
|
FAILED: 'failed',
|
|
};
|
|
|
|
/**
|
|
* Enqueues import/export conversions and answers owner-scoped polling
|
|
* (ADR 0009, issue #62). Enqueue persists the job (so it survives a restart)
|
|
* and wakes the worker for interactive latency; the worker
|
|
* ({@link ConversionWorker}) does the sidecar call out of band.
|
|
*/
|
|
@Injectable()
|
|
export class ConversionJobService {
|
|
constructor(
|
|
private readonly prisma: PrismaService,
|
|
private readonly worker: ConversionWorker,
|
|
private readonly logger: PinoLogger,
|
|
) {
|
|
this.logger.setContext(ConversionJobService.name);
|
|
}
|
|
|
|
async enqueue(request: EnqueueConversion): Promise<ConversionJob> {
|
|
if (request.input.byteLength > MAX_CONVERSION_INPUT_BYTES) {
|
|
throw new PayloadTooLargeException({
|
|
code: 'file_too_large',
|
|
details: { limitBytes: MAX_CONVERSION_INPUT_BYTES },
|
|
});
|
|
}
|
|
const job = await this.prisma.conversionJob.create({
|
|
data: {
|
|
ownerId: request.ownerId,
|
|
kind: request.kind,
|
|
sourceFormat: request.from,
|
|
targetFormat: request.to,
|
|
standalone: request.standalone ?? true,
|
|
// Prisma's Bytes maps to Uint8Array<ArrayBuffer>; a Node Buffer's
|
|
// backing store is ArrayBufferLike, so copy into a plain Uint8Array.
|
|
input: new Uint8Array(request.input),
|
|
},
|
|
});
|
|
this.logger.info(
|
|
{ jobId: job.id, kind: job.kind, ownerId: request.ownerId },
|
|
'audit: conversion enqueued',
|
|
);
|
|
this.worker.wake();
|
|
return job;
|
|
}
|
|
|
|
/** The job if it belongs to `userId`, else 404 — existence stays hidden from
|
|
* anyone but its owner (permissions.md §non-page objects). */
|
|
async getForOwner(id: string, userId: string): Promise<ConversionJobView> {
|
|
return this.viewOf(await this.ownedJob(id, userId));
|
|
}
|
|
|
|
/** The finished output bytes, or 404 while the job is not yet succeeded (so a
|
|
* caller cannot distinguish "still running" from "never existed"). */
|
|
async resultForOwner(id: string, userId: string): Promise<ConversionResultPayload> {
|
|
const job = await this.ownedJob(id, userId);
|
|
if (job.status !== 'SUCCEEDED' || !job.result) throw new NotFoundException();
|
|
return {
|
|
bytes: Buffer.from(job.result),
|
|
mimeType: job.resultMimeType ?? 'application/octet-stream',
|
|
};
|
|
}
|
|
|
|
private async ownedJob(id: string, userId: string): Promise<ConversionJob> {
|
|
const job = await this.prisma.conversionJob.findFirst({ where: { id, ownerId: userId } });
|
|
if (!job) throw new NotFoundException();
|
|
return job;
|
|
}
|
|
|
|
viewOf(job: ConversionJob): ConversionJobView {
|
|
return {
|
|
id: job.id,
|
|
status: STATUS_VIEW[job.status],
|
|
kind: job.kind,
|
|
sourceFormat: job.sourceFormat,
|
|
targetFormat: job.targetFormat,
|
|
errorCode: job.errorCode,
|
|
createdAt: job.createdAt.toISOString(),
|
|
updatedAt: job.updatedAt.toISOString(),
|
|
};
|
|
}
|
|
}
|