dorfteich/apps/api/src/pages/yjs-content.ts
Claude Fable 5 8ae010218e
Some checks failed
CD / Build and push images (push) Successful in 4m5s
CD / Deploy to Test (push) Successful in 9s
CI / Lint, typecheck, test (push) Failing after 4m21s
CI / Auth e2e pack (push) Has been skipped
CI / Import/export fidelity gate (push) Has been skipped
CI / Build container images (push) Has been skipped
CD / Smoke tests against Test (push) Successful in 1m22s
CD / Promote to Int (push) Successful in 11s
Obsidian vault import: endpoint, job orchestration, rollback (#117)
POST /ponds/:pondId/import/vault (pond-admin-gated; a vault import
creates a subtree, uploads files, and creates labels — administration,
not everyday editing) takes the ZIP plus a JSON options field
{parentPageId?, labelIds?, frontmatterMode}. The archive is parsed at
enqueue for fast 400s; the job (new kind import_vault, riding the
existing isImportKind worker routing) re-parses and runs the #116
transform, then: containers top-down → notes (asset placeholders →
uploaded pond files; non-images become page attachments) → tags to
labels (nested tags build a label hierarchy via LabelsService, so
locking and cache invalidation apply) plus the dialog labels.

All-or-nothing: any failure hard-deletes the created pages (children
first) and removes the stored files (quota restored), then surfaces as
import_vault_invalid_zip / import_vault_too_large / quota_exceeded /
conversion_failed — and makes the worker's retry policy safe.

Supporting changes:
- conversion_jobs gains a nullable options jsonb column; enqueue takes
  kind-specific options and a maxInputBytes override (the 25 MiB
  default protects the pandoc sidecar, which a vault never touches —
  vaults use the 64 MiB upload limit).
- insertPage accepts a pre-reserved slug (the batch reserves all slugs
  up front against pond ∪ batch).
- NEW: pages born with content seed their outgoing page_links rows
  (deriveContent now returns wikilinkSlugs) — imported pages would
  otherwise stay invisible to backlinks and the graph until their
  first collab save. Collab still rewrites the rows on every save, and
  the existing phantom resolution heals batch creation order.

import-vault.e2e.db.test.ts (4 tests, real worker drained): gating +
input rejection, the full fixture import (tree under a mount page,
collision suffixes, link rows incl. phantom, nested tag labels, extra
label everywhere, frontmatter stripped, image embedded + PDF attached),
complete quota rollback, and a clean re-import with fresh suffixes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 17:13:11 +02:00

104 lines
3.6 KiB
TypeScript

import {
docToHtml,
docToMarkdown,
docToPlainText,
editorSchema,
extractOutline,
OutlineEntry,
extractWikilinkSlugs,
} from '@dorfteich/shared';
import { Node } from 'prosemirror-model';
import { prosemirrorJSONToYXmlFragment, yXmlFragmentToProseMirrorRootNode } from 'y-prosemirror';
import * as Y from 'yjs';
/**
* The Yjs XmlFragment name the editor binds to (TipTap's collaboration
* extension defaults to "default", #25) — api, web, and collab (#35) must
* all agree on this or Yjs states become unreadable across them.
*/
const FRAGMENT_NAME = 'default';
/** Thrown for state bytes that are not a well-formed Yjs update for this schema. */
export class InvalidPageStateError extends Error {}
function docFromState(state: Uint8Array): Node {
const ydoc = new Y.Doc();
try {
Y.applyUpdate(ydoc, state);
return yXmlFragmentToProseMirrorRootNode(ydoc.getXmlFragment(FRAGMENT_NAME), editorSchema);
} catch (error) {
throw new InvalidPageStateError(error instanceof Error ? error.message : 'invalid Yjs state');
} finally {
ydoc.destroy();
}
}
/**
* Encode a ProseMirror document as the initial Yjs state a page is created
* with. Used both for a fresh empty page and for importing a converted
* document (#63) as a page's starting content — the editor binds to the same
* {@link FRAGMENT_NAME}, so an opening client sees exactly this document.
*/
export function docToState(doc: Node): Uint8Array<ArrayBuffer> {
const ydoc = new Y.Doc();
try {
const fragment = ydoc.getXmlFragment(FRAGMENT_NAME);
prosemirrorJSONToYXmlFragment(editorSchema, doc.toJSON(), fragment);
// Copy into a plain ArrayBuffer-backed view — yjs's own return type is
// the wider `Uint8Array<ArrayBufferLike>`, which Prisma's Bytes input
// (`Uint8Array<ArrayBuffer>`) does not accept directly.
return new Uint8Array(Y.encodeStateAsUpdate(ydoc));
} finally {
ydoc.destroy();
}
}
/** A fresh Yjs state containing a single empty paragraph. */
export function emptyPageState(): Uint8Array<ArrayBuffer> {
return docToState(editorSchema.node('doc', null, [editorSchema.node('paragraph')]));
}
export interface DerivedPageContent {
plainText: string;
markdown: string;
html: string;
outline: OutlineEntry[];
/** fileIds of every `image` node currently embedded in the document
* (issue #31) — `PagesService.saveState` uses this to keep
* `Attachment.pageId` pointed at whichever page's content actually
* embeds the file, which is what the trash-purge job uses to find a
* purged page's files. */
imageFileIds: string[];
/** Outgoing wikilink target slugs (issue #117): pages created through the
* api (imports, phantom-create) seed their `page_links` rows from this —
* collab, the content writer, rewrites them on every later save. */
wikilinkSlugs: string[];
}
function imageFileIdsOf(doc: Node): string[] {
const ids: string[] = [];
doc.descendants((node) => {
if (node.type.name === 'image' && typeof node.attrs.fileId === 'string') {
ids.push(node.attrs.fileId);
}
});
return ids;
}
/**
* Decodes a page's Yjs state into the derived representations stored in
* `page_content_cache` (issue #23). The collab server (#35) will decode
* the same way and call the same shared derivation functions (#24).
*/
export function deriveContent(state: Uint8Array): DerivedPageContent {
const doc = docFromState(state);
return {
plainText: docToPlainText(doc),
markdown: docToMarkdown(doc),
html: docToHtml(doc),
outline: extractOutline(doc),
imageFileIds: imageFileIdsOf(doc),
wikilinkSlugs: extractWikilinkSlugs(doc),
};
}