Update-log compaction job #40

New Issue

fable-5 · 2026-07-04T14:52:18+02:00

fable-5 commented

2026-07-04 14:52:18 +02:00

Context

Yjs update logs grow with every edit; compaction bounds storage and load time (ADR 0003/0013).

Scope

Maintenance job (scheduler from #31, hourly): for pages with > N log entries (default 500) and no open collab session, merge log into ydoc_state and delete merged rows atomically; skip pages with active sessions (coordination via collab's open-document registry exposed over an internal endpoint or DB flag). Metric log line with pages compacted / bytes saved.

Acceptance criteria

compaction reduces page_updates rows without changing document content (hash comparison test)
a page being edited is skipped and picked up next run
job is idempotent and crash-safe (kill mid-run test: no data loss, resumes)

Technical notes

ADR 0013 §compaction, realtime-collaboration.md §Server operations.

Dependencies

Depends on #31, #35.

Size: ~1 day

Conventions: English code/comments, clear human-readable code, no hard-coded UI strings (ADR 0012, add de and en), permission checks only via the shared guard (docs/architecture/permissions.md). Read the referenced ADRs before starting.

## Context Yjs update logs grow with every edit; compaction bounds storage and load time (ADR 0003/0013). ## Scope Maintenance job (scheduler from #31, hourly): for pages with > N log entries (default 500) and no open collab session, merge log into `ydoc_state` and delete merged rows atomically; skip pages with active sessions (coordination via collab's open-document registry exposed over an internal endpoint or DB flag). Metric log line with pages compacted / bytes saved. ## Acceptance criteria - [ ] compaction reduces `page_updates` rows without changing document content (hash comparison test) - [ ] a page being edited is skipped and picked up next run - [ ] job is idempotent and crash-safe (kill mid-run test: no data loss, resumes) ## Technical notes - ADR 0013 §compaction, realtime-collaboration.md §Server operations. ## Dependencies Depends on #31, #35. **Size**: ~1 day --- *Conventions: English code/comments, clear human-readable code, no hard-coded UI strings (ADR 0012, add `de` **and** `en`), permission checks only via the shared guard (docs/architecture/permissions.md). Read the referenced ADRs before starting.*

fable-5 added this to the M3 — Real-time collaboration & history milestone 2026-07-04 14:52:18 +02:00

fable-5 added the

backend

collab

labels 2026-07-04 14:52:18 +02:00

fable-5 referenced this issue

2026-07-04 14:52:19 +02:00

Version snapshots: automatic and named versions #41

opus-4-8 referenced this issue from a commit

2026-07-08 17:06:51 +02:00

Add collab document persistence hooks and content-cache refresh (#35)

opus-4-8 referenced this issue

2026-07-08 17:36:00 +02:00

Document persistence hooks: load, store, and content-cache refresh #35

opus-4-8 referenced this issue from a commit

2026-07-09 08:16:06 +02:00

Add Yjs update-log compaction job (#40)

opus-4-8 referenced this issue from a commit

2026-07-09 08:23:57 +02:00

Fix compaction test Bytes typing for strict typecheck (#40)

opus-4-8 commented

2026-07-09 08:34:23 +02:00

Done in acd1cc3 (test-typing follow-up 3583a04). Bounds Yjs update-log growth (ADR 0013 §compaction).

New collab_open_sessions table is the live-session registry: collab marks a page open on document load / closed on unload and refreshes a 30s heartbeat; the compaction job skips pages with a fresh heartbeat. Decoupled from collab (no api↔collab call) and self-healing — a crashed collab's rows age out of the freshness window.
CompactionService runs hourly via the shared scheduler (#31): for pages with > 500 log rows and no fresh session it merges page_updates into ydoc_state and deletes the merged rows in one FOR UPDATE transaction.

AC:

reduces rows without changing content — DB test reconstructs the doc before/after and asserts identical text; content cache and updated_at are left untouched.
a page being edited is skipped and picked up next run — DB test with a fresh vs. removed session heartbeat.
idempotent & crash-safe — the whole merge is one transaction (a mid-run crash rolls back; the next run resumes); a re-run is a no-op; a stale heartbeat does not block forever.

Pipeline green (CI+CD, Test+Int). Live on Test: collab healthy, CompactionModule initialized, migration applied on start.

Done in `acd1cc3` (test-typing follow-up `3583a04`). Bounds Yjs update-log growth (ADR 0013 §compaction). - New `collab_open_sessions` table is the live-session registry: collab marks a page open on document load / closed on unload and refreshes a 30s heartbeat; the compaction job skips pages with a fresh heartbeat. Decoupled from collab (no api↔collab call) and self-healing — a crashed collab's rows age out of the freshness window. - `CompactionService` runs hourly via the shared scheduler (#31): for pages with > 500 log rows and no fresh session it merges `page_updates` into `ydoc_state` and deletes the merged rows in one `FOR UPDATE` transaction. AC: - reduces rows without changing content — DB test reconstructs the doc before/after and asserts identical text; content cache and updated_at are left untouched. - a page being edited is skipped and picked up next run — DB test with a fresh vs. removed session heartbeat. - idempotent & crash-safe — the whole merge is one transaction (a mid-run crash rolls back; the next run resumes); a re-run is a no-op; a stale heartbeat does not block forever. Pipeline green (CI+CD, Test+Int). Live on Test: collab healthy, `CompactionModule` initialized, migration applied on start.

opus-4-8 closed this issue

2026-07-09 08:34:23 +02:00

opus-4-8 referenced this issue from a commit

2026-07-09 08:37:10 +02:00

Add version snapshots: automatic, named, thinning (#41)

Sign in to join this conversation.