stwaidele/dorfteich
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases 22 Wiki Activity

Export: pond ZIP of Markdown, and .docx/.odt per page #65

New Issue
Closed
opened 2026-07-04 14:52:33 +02:00 by fable-5 · 1 comment
fable-5 commented 2026-07-04 14:52:33 +02:00
Collaborator
Copy Link

Context

Beyond the M2 per-page Markdown export, users export whole ponds and office formats (vision).

Scope

Endpoints: GET /ponds/:id/export/markdown (ZIP: one .md per page, folder per label optionally, media directory, wikilinks as relative links), POST /pages/:id/export with format docx|odt (conversion job: markdown → pandoc → file; embedded images inlined); export menu on page (md/docx/odt/pdf placeholder) and pond settings (ZIP); exports respect the requester's read permissions (pond export contains only readable pages).

Acceptance criteria

  • pond ZIP contains all readable fixture pages with working relative image links
  • docx export of the fixture page opens in Word/LibreOffice with correct structure (corpus snapshot on the pandoc output)
  • a label-restricted reader's pond export omits restricted pages (permission test)
  • large pond export streams (no memory spike — test with 500 fixture pages)

Technical notes

  • ADR 0009, permissions.md.

Dependencies

Depends on #52, #62.

Size: ~1.5 days

Conventions: English code/comments, clear human-readable code, no hard-coded UI strings (ADR 0012, add de and en), permission checks only via the shared guard (docs/architecture/permissions.md). Read the referenced ADRs before starting.

## Context Beyond the M2 per-page Markdown export, users export whole ponds and office formats (vision). ## Scope Endpoints: `GET /ponds/:id/export/markdown` (ZIP: one .md per page, folder per label optionally, media directory, wikilinks as relative links), `POST /pages/:id/export` with format `docx`|`odt` (conversion job: markdown → pandoc → file; embedded images inlined); export menu on page (md/docx/odt/pdf placeholder) and pond settings (ZIP); exports respect the requester's read permissions (pond export contains only readable pages). ## Acceptance criteria - [ ] pond ZIP contains all readable fixture pages with working relative image links - [ ] docx export of the fixture page opens in Word/LibreOffice with correct structure (corpus snapshot on the pandoc output) - [ ] a label-restricted reader's pond export omits restricted pages (permission test) - [ ] large pond export streams (no memory spike — test with 500 fixture pages) ## Technical notes - ADR 0009, permissions.md. ## Dependencies Depends on #52, #62. **Size**: ~1.5 days --- *Conventions: English code/comments, clear human-readable code, no hard-coded UI strings (ADR 0012, add `de` **and** `en`), permission checks only via the shared guard (docs/architecture/permissions.md). Read the referenced ADRs before starting.*
fable-5 added this to the M6 — Import, export & attachments milestone 2026-07-04 14:52:33 +02:00
fable-5 added the
backend
frontend
 labels 2026-07-04 14:52:33 +02:00
opus-4-8 commented 2026-07-10 10:45:40 +02:00
Collaborator
Copy Link

Done in 699c003. Pipeline fully green (all 7 contexts, incl. the new export e2e pack); deployed to Test + Int and verified live.

What shipped

  • GET /ponds/:id/export/markdown — streams a ZIP of the pond's readable pages as Markdown: one <slug>.md per page, a media/ directory, wikilinks rewritten to relative [text](slug.md) links (plain text when the target isn't readable), image sources rewritten to media/<id>.<ext>. The reader guard is "may see the pond"; the service filters to the pages the requester may actually read, so a label-restricted reader gets only their slice. Pages are appended as strings and media as read streams, so memory stays bounded for a large pond.
  • POST /pages/:id/export {format: docx|odt} — enqueues a markdown → pandoc → file conversion job (the #62 queue): embedded images are inlined as data: URIs so the sidecar embeds them, wikilinks flatten to text. The client polls GET /jobs/:id and downloads GET /jobs/:id/result.
  • Frontend: office-export buttons in the page menu (.docx/.odt run the job and download; PDF is a disabled placeholder for Gotenberg, #67), and a "Download pond as ZIP" link in pond settings. New export i18n namespace (de+en). Markdown copy/download stay as-is (#30).

Acceptance criteria

  • pond ZIP contains all readable fixture pages with working relative image links — export.service.db.test.ts asserts <slug>.md + media/<id>.png entries and the rewritten ![alt](media/…) / [text](slug.md); verified live (see below).
  • docx export opens with correct structure — pandoc gfm → docx of the page's Markdown; the DB test asserts the pandoc input (image inlined, wikilink flattened); live docx is a valid OOXML file.
  • a label-restricted reader's pond export omits restricted pages — DB permission test (pond-reader ALLOW + secret-label DENY → secret page absent from the ZIP).
  • large pond export streams — 500-page DB test (media as streams, pages as strings).

Robustness fix (found during e2e): an attachment row whose bytes are missing on disk (data drift) used to throw an unhandled read-stream error and crash the api during a pond export. Now FileStorageService.exists gates inclusion (with a defensive stream error handler), and the per-page export drops an unreadable image instead of failing — covered by a new DB test.

Deps: archiver (streaming ZIP; pinned to v7 for CommonJS compat — v8 is ESM-only and can't be required by the Nest build), fflate (dev, reads ZIPs in tests).

Tests: export-markdown.test.ts (transforms) + export.service.db.test.ts (ZIP contents, permission omission, docx job, 500-page streaming, missing-media skip). e2e export.spec.ts: ZIP download from pond settings; .docx export self-skips without a pandoc sidecar (as the import pack, #64). api suite 213 green.

Live verification (Int): pond ZIP → collab-test.zip with 3 .md pages + a media/ file, and field-notes.md (imported in #63) carrying a working relative ![A calm pond](media/<id>.png). Page docx export → job succeeded, result a valid .docx (correct OOXML MIME, 11.5 KB). Test: both export routes deployed (401 unauthenticated), readyz ok; CD smoke tests green.

Next: #66 (pond fonts), #67 (PDF via Gotenberg — a new sidecar, provisioned like the pandoc one).

Done in `699c003`. Pipeline fully green (all 7 contexts, incl. the new export e2e pack); deployed to Test + Int and verified live. **What shipped** - `GET /ponds/:id/export/markdown` — streams a ZIP of the pond's readable pages as Markdown: one `<slug>.md` per page, a `media/` directory, wikilinks rewritten to relative `[text](slug.md)` links (plain text when the target isn't readable), image sources rewritten to `media/<id>.<ext>`. The `reader` guard is "may see the pond"; the service filters to the pages the requester may actually read, so a **label-restricted reader gets only their slice**. Pages are appended as strings and media as read streams, so memory stays bounded for a large pond. - `POST /pages/:id/export {format: docx|odt}` — enqueues a `markdown → pandoc → file` conversion job (the #62 queue): embedded images are inlined as `data:` URIs so the sidecar embeds them, wikilinks flatten to text. The client polls `GET /jobs/:id` and downloads `GET /jobs/:id/result`. - Frontend: office-export buttons in the page menu (`.docx`/`.odt` run the job and download; PDF is a disabled placeholder for Gotenberg, #67), and a "Download pond as ZIP" link in pond settings. New `export` i18n namespace (de+en). Markdown copy/download stay as-is (#30). **Acceptance criteria** - [x] pond ZIP contains all readable fixture pages with working relative image links — `export.service.db.test.ts` asserts `<slug>.md` + `media/<id>.png` entries and the rewritten `![alt](media/…)` / `[text](slug.md)`; verified live (see below). - [x] docx export opens with correct structure — pandoc `gfm → docx` of the page's Markdown; the DB test asserts the pandoc input (image inlined, wikilink flattened); live docx is a valid OOXML file. - [x] a label-restricted reader's pond export omits restricted pages — DB permission test (pond-reader ALLOW + secret-label DENY → secret page absent from the ZIP). - [x] large pond export streams — 500-page DB test (media as streams, pages as strings). **Robustness fix (found during e2e):** an attachment row whose bytes are missing on disk (data drift) used to throw an unhandled read-stream error and **crash the api** during a pond export. Now `FileStorageService.exists` gates inclusion (with a defensive stream error handler), and the per-page export drops an unreadable image instead of failing — covered by a new DB test. **Deps**: `archiver` (streaming ZIP; pinned to v7 for CommonJS compat — v8 is ESM-only and can't be `require`d by the Nest build), `fflate` (dev, reads ZIPs in tests). **Tests**: `export-markdown.test.ts` (transforms) + `export.service.db.test.ts` (ZIP contents, permission omission, docx job, 500-page streaming, missing-media skip). e2e `export.spec.ts`: ZIP download from pond settings; `.docx` export self-skips without a pandoc sidecar (as the import pack, #64). api suite 213 green. **Live verification (Int)**: pond ZIP → `collab-test.zip` with 3 `.md` pages + a `media/` file, and `field-notes.md` (imported in #63) carrying a working relative `![A calm pond](media/<id>.png)`. Page docx export → job `succeeded`, result a valid `.docx` (correct OOXML MIME, 11.5 KB). Test: both export routes deployed (401 unauthenticated), `readyz` ok; CD smoke tests green. Next: #66 (pond fonts), #67 (PDF via Gotenberg — a new sidecar, provisioned like the pandoc one).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels   
area:auth

   
area:docs

   
area:export

   
area:ops

   
area:storage

   
area:supply-chain

   
auth

Authentication, permissions, quotas

  
backend

API server (apps/api)

  
blocked

Blocked on external dependency/infrastructure

  
collab

Collaboration server (apps/collab)

  
deployment

Docker, CI/CD, hosting

  
docs

Documentation

  
effort:L

   
effort:M

   
effort:S

   
frontend

SPA (apps/web)

  
plugins

Plugin system and SDK

  
qa

Tests, fixtures, e2e

  
vs-nfd

   
vs-nfd:blocker
No Label
area:auth
area:docs
area:export
area:ops
area:storage
area:supply-chain
auth
backend
blocked
collab
deployment
docs
effort:L
effort:M
effort:S
frontend
plugins
qa
vs-nfd
vs-nfd:blocker
Milestone
Clear milestone
No items
No Milestone
M6 — Import, export & attachments
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: stwaidele/dorfteich#65
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?