stwaidele/dorfteich
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases 22 Wiki Activity

#224: read-trail storage - partitioning, retention, admin query path #280

Closed
fable-5 wants to merge 0 commits from issue-224-read-trail-storage into main
pull from: issue-224-read-trail-storage
merge into: stwaidele:main
Conversation 1 Commits - Files Changed -
fable-5 commented 2026-07-31 12:25:53 +02:00
Collaborator
Copy Link

Monthly RANGE partitions on occurred_at with a DEFAULT safety-net partition (a lagging job must never turn hard-failure semantics into an outage). Dedup unique pair moves to per-partition indexes; month-boundary duplicate documented. New daily job read-trail-maintenance (job fence 9->10) creates months ahead and prunes under readTrail.retentionDays (default 365), whole months by DROP, audited as read_trail.pruned (catalogue v1.2). Site-Admin query path GET /admin/system/read-events (API-only by design). Growth measured: ~1 MB per 1000 events (data-model.md). Fresh-DB test runs the real migration chain and proves partition shape, per-partition P2002 and DROP pruning. Stacked on #223. Refs #224

Monthly RANGE partitions on occurred_at with a DEFAULT safety-net partition (a lagging job must never turn hard-failure semantics into an outage). Dedup unique pair moves to per-partition indexes; month-boundary duplicate documented. New daily job read-trail-maintenance (job fence 9->10) creates months ahead and prunes under readTrail.retentionDays (default 365), whole months by DROP, audited as read_trail.pruned (catalogue v1.2). Site-Admin query path GET /admin/system/read-events (API-only by design). Growth measured: ~1 MB per 1000 events (data-model.md). Fresh-DB test runs the real migration chain and proves partition shape, per-partition P2002 and DROP pruning. Stacked on #223. Refs #224
fable-5 added 3 commits 2026-07-31 12:25:53 +02:00
#222: read-access trail for classified pages
Some checks failed
CI / Lint, typecheck, test (pull_request) Successful in 6m25s
Details
CI / Build container images (pull_request) Successful in 2m58s
Details
CI / Auth e2e pack (pull_request) Successful in 8m35s
Details
CI / Import/export fidelity gate (pull_request) Successful in 1m7s
Details
CD / Deploy to Test (push) Blocked by required conditions
Details
CD / Smoke tests against Test (push) Blocked by required conditions
Details
CD / Promote to Int (push) Blocked by required conditions
Details
CI / Auth e2e pack (push) Blocked by required conditions
Details
CI / Import/export fidelity gate (push) Blocked by required conditions
Details
CI / Build container images (push) Blocked by required conditions
Details
CI / Lint, typecheck, test (push) Has been cancelled
Details
CD / Build and push images (push) Has been cancelled
Details
05a979bac3 
Instrument every full-content read channel for pages with
classification = vs_nfd (ADR 0023, variant A): SPA state fetch and read
rendering, public JSON content, no-JS shell, expanded embeds, public API
GET (incl. the MCP read_page path and write echoes), attachment download
under the #212 effective classification, all export shapes (markdown,
pond ZIP, account data export, queued docx/odt/pdf at enqueue), and
collab-token issuance as the api-side proxy for the WS join.

Events land in the new read_events table (no FKs — evidence survives
page purges and hard user deletions) with actor, session key
(session:/token:/job:/anon), page, pond, channel and the classification
at read time. Recording failures are NOT swallowed: a failed write
aborts the read (hard failure, the deliberate contrast to AuditService —
decision recorded in ADR 0023 and security.md §Logging, together with
the recorded residuals: content fragments and feeds).

One e2e test per channel proves both the event and its absence for
unclassified pages, plus the hard-failure semantics.

Refs #222.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUtYMxwTCMHG9mVHnwbFg8
#223: dedup window for the read trail
Some checks failed
CI / Lint, typecheck, test (pull_request) Successful in 6m29s
Details
CI / Auth e2e pack (pull_request) Failing after 5s
Details
CI / Import/export fidelity gate (pull_request) Has been skipped
Details
CI / Build container images (pull_request) Successful in 2m55s
Details
fd4fd60c99 
One event per (session, page, channel) within an aligned window of
readTrail.dedupWindowMinutes (default 5): buckets are
floor(epoch / windowSeconds), and a unique (dedup_key, window_bucket)
pair collapses concurrent duplicates race-free at insert time — the
first access in a window is always recorded, a later duplicate lands on
the unique violation and is skipped quietly (a skipped duplicate is not
a gap; only real write failures still abort the read). Each row carries
windowSeconds, so the evidence states it represents a window, never a
request count.

Reconnects within a window stay one event; a new session records again
even for the same user; channels never collapse into each other; the
page-less attachment key uses the documented `-` placeholder. Load
evidence: 30 collab-token renewals inside one window produce exactly one
event (test), bounding a live editing session at ~12 events/hour/page.

Window semantics documented in ADR 0023, the VS-NfD security
documentation (#228) and as a hardening-guide line for the new setting
(care rule: same PR).

Refs #223.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUtYMxwTCMHG9mVHnwbFg8
#224: read-trail storage — partitioning, retention, admin query path
Some checks failed
CI / Lint, typecheck, test (pull_request) Successful in 6m30s
Details
CI / Auth e2e pack (pull_request) Failing after 5s
Details
CI / Import/export fidelity gate (pull_request) Has been skipped
Details
CI / Build container images (pull_request) Failing after 2s
Details
2bdb0ec2cf 
Convert read_events to monthly RANGE partitions on occurred_at, with a
DEFAULT partition as safety net: a lagging maintenance job must never
turn the trail's hard-failure semantics into an outage for classified
reads. The dedup unique pair (#223) moves to per-partition indexes
(PostgreSQL cannot carry it on the parent); a bucket spanning a month
boundary may record one duplicate — over-recording is acceptable, gaps
are not.

New daily job read-trail-maintenance (job-count fence 9 -> 10) creates
months ahead — each with its dedup index — and applies the trail's own
retention readTrail.retentionDays (default 365, deliberately independent
of audit.retentionDays): whole expired months are DROPped without
scanning, remainders deleted by range, every run audited as
read_trail.pruned (catalogue v1.2; the fence regex now admits an
underscore namespace).

Site-Admin query path GET /admin/system/read-events answers "who read
page X" and "what did user Y read" within a period — API-only by
design, documented. Growth measured and documented in data-model.md:
~1 MB per 1000 events including indexes.

Tests: retention pruning + audited deletion + admin queries on the
shared database; the partitioned shape, per-partition P2002 dedup,
months-ahead creation and DROP-based pruning against a fresh database
built by the real migration chain.

Refs #224.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AUtYMxwTCMHG9mVHnwbFg8
fable-5 commented 2026-07-31 12:53:59 +02:00
Author
Collaborator
Copy Link

Contained in the fast-forward merge of PR #281 (runs 591/596 green; this PR own runs 592/593 failed only on runner infrastructure: docker address-pool exhaustion, since cleaned up). Commits are on main - closed like #268/#275.

Contained in the fast-forward merge of PR #281 (runs 591/596 green; this PR own runs 592/593 failed only on runner infrastructure: docker address-pool exhaustion, since cleaned up). Commits are on main - closed like #268/#275.
fable-5 closed this pull request 2026-07-31 12:53:59 +02:00
Some checks failed
CI / Lint, typecheck, test (pull_request) Successful in 6m30s
Required
Details
CI / Auth e2e pack (pull_request) Failing after 5s
Details
CI / Import/export fidelity gate (pull_request) Has been skipped
Details
CI / Build container images (pull_request) Failing after 2s
Required
Details

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
area:auth

   
area:docs

   
area:export

   
area:ops

   
area:storage

   
area:supply-chain

   
auth

Authentication, permissions, quotas

  
backend

API server (apps/api)

  
blocked

Blocked on external dependency/infrastructure

  
collab

Collaboration server (apps/collab)

  
deployment

Docker, CI/CD, hosting

  
docs

Documentation

  
effort:L

   
effort:M

   
effort:S

   
frontend

SPA (apps/web)

  
plugins

Plugin system and SDK

  
qa

Tests, fixtures, e2e

  
vs-nfd

   
vs-nfd:blocker
No Label
area:auth
area:docs
area:export
area:ops
area:storage
area:supply-chain
auth
backend
blocked
collab
deployment
docs
effort:L
effort:M
effort:S
frontend
plugins
qa
vs-nfd
vs-nfd:blocker
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: stwaidele/dorfteich#280
No description provided.
Delete Branch "issue-224-read-trail-storage"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?