Obsidian vault transform module (pure functions + fixture vault) #116

New Issue

fable-5 · 2026-07-14T16:36:40+02:00

fable-5 commented

2026-07-14 16:36:40 +02:00

Context

The vault import (see the milestone) needs a deterministic, DB-free transform from "uploaded ZIP" to "per-note rewritten Markdown plus metadata" that can be unit-tested in isolation. Key constraint: Dorfteich wikilinks resolve by SLUG within a pond, while Obsidian links reference note NAMES — and the shared wikilinkRule parses [[target]] verbatim, so all rewriting must happen as a text pre-pass before markdownToDoc.

Scope

New server-only module apps/api/src/import-export/obsidian-vault.ts with pure functions:

Safe unzip: fflate unzipSync with the plugin-package.service.ts protections (safeRelativePath against zip-slip, incremental unpacked-size ceiling MAX_VAULT_UNPACKED_BYTES = 256 MiB against zip bombs).
Vault model: folders / notes (.md) / assets (everything else), case-insensitive name index.
Folder depth merge: mount-parent depth + folders + note must stay within MAX_PAGE_DEPTH (6); merge the DEEPEST folder levels, merged container titles read like a/b.
Frontmatter: detect a leading --- … --- block; extract tags: (scalar, inline array, and list forms — hand-rolled, no yaml dependency); mode strip drops the block, mode preserve re-emits it as a fenced ```yaml code block at the top.
Inline tags: extract #tag / #nested/tag occurrences (skipping code fences, inline code, and headings' leading #) and remove them from the text.
Link rewrite (fence-aware text pass): [[Name]], [[Name|Display]], [[Name#Heading]] (strip the heading, keep display), [[folder/Name]] (full-path match wins over basename); ![[image.ext]] → standard image markdown against the resolved vault asset; ![[note]] → plain wikilink; unresolvable targets → [[slugify(target)]] (a phantom link).
Slug planning: name→FINAL-slug map for every container and note, -2/-3 suffixing against existing pond slugs ∪ batch; ambiguous basenames resolve to the lexicographically first vault path (documented rule).

Fixture vault under fixtures/import/obsidian-vault/ covering folders, umlauts, duplicate basenames, nested tags, both frontmatter modes, ![[img]], [[a/Note]], heading links; colocated obsidian-vault.test.ts.

Acceptance criteria

Every function unit-tested against the fixture: all link forms, umlaut slugs via the shared slugify, duplicate basenames, nested tags, both frontmatter modes, depth merge beyond 6, zip-slip and unpacked-ceiling rejection.
No DB, no DI, no NestJS imports — plain functions.

Technical notes

packages/shared/src/ponds.ts (slugify), packages/shared/src/editor-schema/markdown.ts (wikilinkRule constraints: no nesting/newlines, verbatim target), apps/api/src/plugins/plugin-package.service.ts (unzip safety model).

Dependencies

None (parallel to the 404 issue).

Size

~2 days.

## Context The vault import (see the milestone) needs a deterministic, DB-free transform from "uploaded ZIP" to "per-note rewritten Markdown plus metadata" that can be unit-tested in isolation. Key constraint: Dorfteich wikilinks resolve by SLUG within a pond, while Obsidian links reference note NAMES — and the shared `wikilinkRule` parses `[[target]]` verbatim, so all rewriting must happen as a text pre-pass before `markdownToDoc`. ## Scope New server-only module `apps/api/src/import-export/obsidian-vault.ts` with pure functions: - Safe unzip: fflate `unzipSync` with the `plugin-package.service.ts` protections (`safeRelativePath` against zip-slip, incremental unpacked-size ceiling `MAX_VAULT_UNPACKED_BYTES = 256 MiB` against zip bombs). - Vault model: folders / notes (`.md`) / assets (everything else), case-insensitive name index. - Folder depth merge: mount-parent depth + folders + note must stay within `MAX_PAGE_DEPTH` (6); merge the DEEPEST folder levels, merged container titles read like `a/b`. - Frontmatter: detect a leading `--- … ---` block; extract `tags:` (scalar, inline array, and list forms — hand-rolled, no yaml dependency); mode `strip` drops the block, mode `preserve` re-emits it as a fenced ```yaml code block at the top. - Inline tags: extract `#tag` / `#nested/tag` occurrences (skipping code fences, inline code, and headings' leading `#`) and remove them from the text. - Link rewrite (fence-aware text pass): `[[Name]]`, `[[Name|Display]]`, `[[Name#Heading]]` (strip the heading, keep display), `[[folder/Name]]` (full-path match wins over basename); `![[image.ext]]` → standard image markdown against the resolved vault asset; `![[note]]` → plain wikilink; unresolvable targets → `[[slugify(target)]]` (a phantom link). - Slug planning: name→FINAL-slug map for every container and note, `-2/-3` suffixing against `existing pond slugs ∪ batch`; ambiguous basenames resolve to the lexicographically first vault path (documented rule). Fixture vault under `fixtures/import/obsidian-vault/` covering folders, umlauts, duplicate basenames, nested tags, both frontmatter modes, `![[img]]`, `[[a/Note]]`, heading links; colocated `obsidian-vault.test.ts`. ## Acceptance criteria - [ ] Every function unit-tested against the fixture: all link forms, umlaut slugs via the shared `slugify`, duplicate basenames, nested tags, both frontmatter modes, depth merge beyond 6, zip-slip and unpacked-ceiling rejection. - [ ] No DB, no DI, no NestJS imports — plain functions. ## Technical notes `packages/shared/src/ponds.ts` (`slugify`), `packages/shared/src/editor-schema/markdown.ts` (`wikilinkRule` constraints: no nesting/newlines, verbatim target), `apps/api/src/plugins/plugin-package.service.ts` (unzip safety model). ## Dependencies None (parallel to the 404 issue). ## Size ~2 days.

fable-5 added this to the M13 — Create-from-link & Obsidian import milestone 2026-07-14 16:36:40 +02:00

fable-5 added the

backend

label 2026-07-14 16:36:40 +02:00

fable-5 referenced this issue

2026-07-14 16:36:41 +02:00

Vault import job: endpoint, orchestration, rollback #117

fable-5 referenced this issue from a commit

2026-07-14 16:51:32 +02:00

Obsidian vault transform module (#116)

fable-5 referenced this issue from a commit

2026-07-14 16:53:14 +02:00

Fix the gates the #116 commit skipped past

fable-5 referenced this issue from a commit

2026-07-14 17:13:13 +02:00

Obsidian vault import: endpoint, job orchestration, rollback (#117)

fable-5 commented

2026-07-14 18:40:37 +02:00

Implemented in 09abda3 (+ 269b36c fixing the gates that commit skipped past). apps/api/src/import-export/obsidian-vault.ts holds the whole transform as pure functions: safe unzip (zip-slip + 256 MiB unpacked ceiling, .obsidian/ skipped), frontmatter handling (strip / preserve as a yaml code block, tags: in all three Obsidian forms), fence-aware inline #tag extraction and removal, the link rewrite ([[Name]], [[Name|Display]], [[Name#Heading]], [[folder/Name]] → the FINAL slug; unresolvable → slugified phantom; ![[img]] and relative ![](path) → asset placeholders; note embeds degrade to links), folder depth-merge against MAX_PAGE_DEPTH, and batch slug planning (-n suffixes against pond ∪ batch, duplicate basenames resolved to the lexicographically first vault path). Fixture vault under fixtures/import/obsidian-vault/ with 13 unit tests.

Implemented in 09abda3 (+ 269b36c fixing the gates that commit skipped past). `apps/api/src/import-export/obsidian-vault.ts` holds the whole transform as pure functions: safe unzip (zip-slip + 256 MiB unpacked ceiling, `.obsidian/` skipped), frontmatter handling (strip / preserve as a yaml code block, `tags:` in all three Obsidian forms), fence-aware inline `#tag` extraction and removal, the link rewrite (`[[Name]]`, `[[Name|Display]]`, `[[Name#Heading]]`, `[[folder/Name]]` → the FINAL slug; unresolvable → slugified phantom; `![[img]]` and relative `![](path)` → asset placeholders; note embeds degrade to links), folder depth-merge against `MAX_PAGE_DEPTH`, and batch slug planning (`-n` suffixes against pond ∪ batch, duplicate basenames resolved to the lexicographically first vault path). Fixture vault under `fixtures/import/obsidian-vault/` with 13 unit tests.

fable-5 closed this issue

2026-07-14 18:40:37 +02:00

fable-5 referenced this issue

2026-07-14 18:40:37 +02:00

Vault import job: endpoint, orchestration, rollback #117

Sign in to join this conversation.