[VS-NfD] Prune conversion job payloads for every job kind #233

New Issue

fable-5 · 2026-07-30T01:44:00+02:00

fable-5 commented

2026-07-30 01:44:00 +02:00

Plan reference: n/a — docs/vs-nfd/10-ist-aufnahme.md -> I-22
ADR: n/a
Effort: M (2 AT)
Depends on: —

Context

The raw bytes of every import and export survive indefinitely in the
database. A deleted classified page therefore lives on inside its last
export — the kind of residue a deletion concept cannot leave unexplained.

Current state

ConversionJob.input / .result are the raw document bytes
(apps/api/prisma/schema.prisma:746–784). The schema calls them
"transient, not the durable copy an Attachment is" and refers to
"a later maintenance job" for pruning.
expiresAt is documented as set only for data-export jobs (#68) and
"Null for every other job kind, whose result never expires".
Of the five registered scheduled jobs — version-thinning,
page-compaction, trash-purge, data-export-purge,
notification-digest — only data-export-purge touches conversion job
rows (apps/api/src/import-export/import-export.module.ts:62).

Acceptance criteria

Payloads of all job kinds are pruned after a configurable period
with a documented default; the row may survive for status/audit
purposes, the bytes must not.
A backfill clears payloads of already-finished jobs.
Pruning does not break an in-flight job (lockedAt recovery path
respected) — asserted by test.
Test: finished export job's result is null after the period; a
pending job is untouched.
docs/architecture/operations.md documents the job; #229 lists it
under deletion and destruction.

Out of scope

Changing where conversion happens, and the data-export purge that already
works.

**Plan reference:** n/a — `docs/vs-nfd/10-ist-aufnahme.md` -> I-22 **ADR:** n/a **Effort:** M (2 AT) **Depends on:** — ## Context The raw bytes of every import and export survive indefinitely in the database. A deleted classified page therefore lives on inside its last export — the kind of residue a deletion concept cannot leave unexplained. ## Current state - `ConversionJob.input` / `.result` are the raw document bytes (`apps/api/prisma/schema.prisma:746–784`). The schema calls them "transient, not the durable copy an Attachment is" and refers to "a later maintenance job" for pruning. - `expiresAt` is documented as set only for data-export jobs (#68) and "Null for every other job kind, whose result never expires". - Of the five registered scheduled jobs — `version-thinning`, `page-compaction`, `trash-purge`, `data-export-purge`, `notification-digest` — only `data-export-purge` touches conversion job rows (`apps/api/src/import-export/import-export.module.ts:62`). ## Acceptance criteria - [ ] Payloads of **all** job kinds are pruned after a configurable period with a documented default; the row may survive for status/audit purposes, the bytes must not. - [ ] A backfill clears payloads of already-finished jobs. - [ ] Pruning does not break an in-flight job (`lockedAt` recovery path respected) — asserted by test. - [ ] Test: finished export job's `result` is null after the period; a pending job is untouched. - [ ] `docs/architecture/operations.md` documents the job; #229 lists it under deletion and destruction. ## Out of scope Changing where conversion happens, and the data-export purge that already works.

fable-5 added this to the M24 — VS-NfD: security quick wins milestone 2026-07-30 01:44:00 +02:00

fable-5 added the

area:storage

effort:M

vs-nfd

labels 2026-07-30 01:44:00 +02:00

fable-5 referenced this issue from a commit

2026-07-30 22:12:34 +02:00

#233: prune conversion job payloads for every job kind

fable-5 referenced this issue

2026-07-30 22:12:45 +02:00

#233: prune conversion job payloads for every job kind #254

fable-5 referenced this issue

2026-07-30 22:17:45 +02:00

#234: retention for mail_outbox #255

fable-5 referenced this issue

2026-07-30 22:18:49 +02:00

#235: keep page_links rows pointing at purged pages (recorded decision) #256

fable-5 referenced this issue

2026-07-30 22:20:56 +02:00

[VS-NfD] Write the operations manual (installation, update, backup/restore, deletion, role separation) #229

fable-5 commented

2026-07-30 22:35:25 +02:00

Implemented in PR #254 (commit ff505bc, CI run 510 green, fast-forward merged on Stefan's standing go-ahead for the M24 merges).

Evidence against the acceptance criteria:

All job kinds pruned: daily job conversion-payload-prune nulls input/result/result_mime_type of SUCCEEDED and FAILED jobs older than conversion.payloadRetentionDays (new instance setting, default 30, documented in operations.md); rows survive for status/audit.
Backfill: migration 20260730170000_prune_conversion_job_payloads makes input nullable and clears payloads of jobs already finished past the default period.
In-flight jobs safe: PENDING and RUNNING rows (incl. stale-lock recovery) are never matched - asserted in conversion-payload-prune.e2e.db.test.ts; a hand-requeued pruned job fails finally via conversionInputOf instead of crashing the worker.
Tests: finished job's payload null after the period, pending/stale-RUNNING/fresh untouched, no-op run. Full api suite green (471) on a fresh DB; migrate deploy verified on a second fresh DB.
operations.md documents the job (table + paragraph); addition for the deletion chapter recorded as a comment on #229.

CD verification follows on the main-push run.

Implemented in PR #254 (commit ff505bc, CI run 510 green, fast-forward merged on Stefan's standing go-ahead for the M24 merges). Evidence against the acceptance criteria: - All job kinds pruned: daily job conversion-payload-prune nulls input/result/result_mime_type of SUCCEEDED and FAILED jobs older than conversion.payloadRetentionDays (new instance setting, default 30, documented in operations.md); rows survive for status/audit. - Backfill: migration 20260730170000_prune_conversion_job_payloads makes input nullable and clears payloads of jobs already finished past the default period. - In-flight jobs safe: PENDING and RUNNING rows (incl. stale-lock recovery) are never matched - asserted in conversion-payload-prune.e2e.db.test.ts; a hand-requeued pruned job fails finally via conversionInputOf instead of crashing the worker. - Tests: finished job's payload null after the period, pending/stale-RUNNING/fresh untouched, no-op run. Full api suite green (471) on a fresh DB; migrate deploy verified on a second fresh DB. - operations.md documents the job (table + paragraph); addition for the deletion chapter recorded as a comment on #229. CD verification follows on the main-push run.

fable-5 closed this issue

2026-07-30 22:35:25 +02:00

Sign in to join this conversation.