Vault import job: endpoint, orchestration, rollback #117

New Issue

fable-5 · 2026-07-14T16:36:41+02:00

fable-5 commented

2026-07-14 16:36:41 +02:00

Context

Wire the transform module into the conversion-job queue and the page/file/label services: one job imports the whole vault under the chosen parent, with the chosen labels, all-or-nothing.

Scope

Prisma migration: nullable options Json? on conversion_jobs ({parentPageId, labelIds, frontmatterMode} for vault jobs; existing kinds unaffected). ConversionJobService.enqueue gains options and an optional maxInputBytes — the 25 MiB default protects the pandoc sidecar, which a vault job never touches; vault jobs pass MAX_UPLOAD_PARSE_BYTES (64 MiB).
POST /ponds/:pondId/import/vault (multipart file = .zip, body fields parentPageId?, labelIds[]?, frontmatterMode), gated @RequiresPondRole('pond_admin') — the single-file import stays editor-gated. Kind import_vault rides the existing isImportKind worker routing.
Orchestration in ImportService: transform (#116) → batch slug reservation via a new public PagesService.reserveSlugs(pondId, titles) helper (shares the suffix mechanic with generateUniqueSlugInPond; insertPage accepts a pre-reserved slug) → container pages top-down (empty state, parentId) → note pages → image assets via files.upload + linkAttachmentsToPage (image nodes carry the fileId), other instance-allowed types as attachments, disallowed dropped with a log line → labels find-or-create (nested a/b → label hierarchy, default color) + assign in-process (no HTTP round-trips), plus the dialog labels.
All-or-nothing: track created page ids and stored file ids; on any failure hard-delete pages deepest-first and remove files (quota restored) — the rollbackMedia spirit; makes the worker's 3-attempt retry safe. Error codes import_vault_invalid_zip, import_vault_too_large, quota_exceeded, conversion_failed (translated de+en). On success resultPageId = the mount parent (null = pond root).

Acceptance criteria

DB test pack: full fixture import (tree, final slugs, links resolve — page_links rows point at the imported pages, labels incl. nested, images linked), collision with pre-existing pond slugs, depth merge under a deep parent, quota exhaustion mid-import rolls back completely (no pages, no files, quota unchanged), non-admin editor gets 403, oversized zip / unpacked ceiling rejected.
A failed attempt leaves nothing behind; a retry after a transient failure creates no duplicates.

Technical notes

apps/api/src/import-export/{import.controller.ts,import.service.ts,conversion-job.service.ts,import.constants.ts}, apps/api/src/pages/pages.service.ts, apps/api/src/labels/labels.service.ts, apps/api/src/files/files.service.ts, apps/api/prisma/schema.prisma.

Dependencies

Requires #116.

Size

~2 days.

## Context Wire the transform module into the conversion-job queue and the page/file/label services: one job imports the whole vault under the chosen parent, with the chosen labels, all-or-nothing. ## Scope - Prisma migration: nullable `options Json?` on `conversion_jobs` (`{parentPageId, labelIds, frontmatterMode}` for vault jobs; existing kinds unaffected). `ConversionJobService.enqueue` gains `options` and an optional `maxInputBytes` — the 25 MiB default protects the pandoc sidecar, which a vault job never touches; vault jobs pass `MAX_UPLOAD_PARSE_BYTES` (64 MiB). - `POST /ponds/:pondId/import/vault` (multipart `file` = `.zip`, body fields `parentPageId?`, `labelIds[]?`, `frontmatterMode`), gated `@RequiresPondRole('pond_admin')` — the single-file import stays editor-gated. Kind `import_vault` rides the existing `isImportKind` worker routing. - Orchestration in `ImportService`: transform (#116) → batch slug reservation via a new public `PagesService.reserveSlugs(pondId, titles)` helper (shares the suffix mechanic with `generateUniqueSlugInPond`; `insertPage` accepts a pre-reserved slug) → container pages top-down (empty state, `parentId`) → note pages → image assets via `files.upload` + `linkAttachmentsToPage` (image nodes carry the fileId), other instance-allowed types as attachments, disallowed dropped with a log line → labels find-or-create (nested `a/b` → label hierarchy, default color) + assign in-process (no HTTP round-trips), plus the dialog labels. - All-or-nothing: track created page ids and stored file ids; on any failure hard-delete pages deepest-first and remove files (quota restored) — the `rollbackMedia` spirit; makes the worker's 3-attempt retry safe. Error codes `import_vault_invalid_zip`, `import_vault_too_large`, `quota_exceeded`, `conversion_failed` (translated de+en). On success `resultPageId` = the mount parent (null = pond root). ## Acceptance criteria - [ ] DB test pack: full fixture import (tree, final slugs, links resolve — `page_links` rows point at the imported pages, labels incl. nested, images linked), collision with pre-existing pond slugs, depth merge under a deep parent, quota exhaustion mid-import rolls back completely (no pages, no files, quota unchanged), non-admin editor gets 403, oversized zip / unpacked ceiling rejected. - [ ] A failed attempt leaves nothing behind; a retry after a transient failure creates no duplicates. ## Technical notes `apps/api/src/import-export/{import.controller.ts,import.service.ts,conversion-job.service.ts,import.constants.ts}`, `apps/api/src/pages/pages.service.ts`, `apps/api/src/labels/labels.service.ts`, `apps/api/src/files/files.service.ts`, `apps/api/prisma/schema.prisma`. ## Dependencies Requires #116. ## Size ~2 days.

fable-5 added this to the M13 — Create-from-link & Obsidian import milestone 2026-07-14 16:36:41 +02:00

fable-5 added the

backend

label 2026-07-14 16:36:41 +02:00

fable-5 referenced this issue

2026-07-14 16:36:41 +02:00

Vault import dialog in the pond settings #118

fable-5 referenced this issue

2026-07-14 16:36:41 +02:00

QA: vault-import e2e pack and documentation #119

fable-5 referenced this issue from a commit

2026-07-14 16:51:32 +02:00

Obsidian vault transform module (#116)

fable-5 referenced this issue from a commit

2026-07-14 17:13:13 +02:00

Obsidian vault import: endpoint, job orchestration, rollback (#117)

fable-5 referenced this issue from a commit

2026-07-14 18:26:28 +02:00

Vault import dialog in the pond settings (#118)

fable-5 commented

2026-07-14 18:40:37 +02:00

Implemented in 8ae0102. POST /ponds/:pondId/import/vault (pond-admin-gated) takes the ZIP plus a JSON options field; the archive is parsed at enqueue for fast 400s and the job (import_vault, riding the existing worker routing) runs the #116 transform, then creates containers top-down, notes with their uploaded images/attachments, and labels from tags (nested tags build a label hierarchy) plus the dialog labels. All-or-nothing: any failure hard-deletes the created pages and removes the stored files (quota restored), which also makes the worker's retry safe. Supporting: conversion_jobs gained a nullable options jsonb + a maxInputBytes override (the 25 MiB default only guards the pandoc sidecar); insertPage accepts a pre-reserved slug; and pages born with content now seed their outgoing page_links rows — imported pages would otherwise stay invisible to backlinks and the graph until their first collab save. import-vault.e2e.db.test.ts covers gating, the full fixture import, quota rollback, and a clean re-import.

Implemented in 8ae0102. `POST /ponds/:pondId/import/vault` (pond-admin-gated) takes the ZIP plus a JSON options field; the archive is parsed at enqueue for fast 400s and the job (`import_vault`, riding the existing worker routing) runs the #116 transform, then creates containers top-down, notes with their uploaded images/attachments, and labels from tags (nested tags build a label hierarchy) plus the dialog labels. All-or-nothing: any failure hard-deletes the created pages and removes the stored files (quota restored), which also makes the worker's retry safe. Supporting: `conversion_jobs` gained a nullable `options` jsonb + a `maxInputBytes` override (the 25 MiB default only guards the pandoc sidecar); `insertPage` accepts a pre-reserved slug; and pages born with content now seed their outgoing `page_links` rows — imported pages would otherwise stay invisible to backlinks and the graph until their first collab save. `import-vault.e2e.db.test.ts` covers gating, the full fixture import, quota rollback, and a clean re-import.

fable-5 closed this issue

2026-07-14 18:40:37 +02:00

Sign in to join this conversation.