stwaidele/dorfteich
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases 22 Wiki Activity

Readiness checks, backup freshness, and uptime monitors #85

New Issue
Closed
opened 2026-07-04 14:52:46 +02:00 by fable-5 · 1 comment
fable-5 commented 2026-07-04 14:52:46 +02:00
Collaborator
Copy Link

Context

Pragmatic monitoring (kickoff): the readyz endpoint is the single integration point for external uptime checks (operations.md).

Scope

Extend /readyz per operations.md: DB, migrations, converter reachability (warning), backup freshness from status.json (< 26 h, degraded state distinct from hard failure); add collab /healthz depth (DB ping); define the Uptime-Kuma monitor set (web, api readyz, collab WebSocket connect) as documented configuration in deploy/monitoring.md and set the monitors up for Test/Int/Prod with the repo owner; degraded-vs-down semantics documented (degraded alerts, but does not restart containers — compose healthcheck uses liveness, not readyz).

Acceptance criteria

  • readyz output enumerates each check with status (JSON) and returns 503 only on hard failures
  • stale backup status flips readyz to degraded (test with old status.json)
  • docker healthchecks keep using liveness endpoints (a degraded instance is not restart-looped — verify config)
  • monitor definitions documented; Prod monitor alerting verified once live (checklist ref #89)

Technical notes

  • operations.md §Health, ADR 0015.

Dependencies

Depends on #83.

Size: ~1 day

Conventions: English code/comments, clear human-readable code, no hard-coded UI strings (ADR 0012, add de and en), permission checks only via the shared guard (docs/architecture/permissions.md). Read the referenced ADRs before starting.

## Context Pragmatic monitoring (kickoff): the readyz endpoint is the single integration point for external uptime checks (operations.md). ## Scope Extend `/readyz` per operations.md: DB, migrations, converter reachability (warning), backup freshness from status.json (< 26 h, degraded state distinct from hard failure); add collab `/healthz` depth (DB ping); define the Uptime-Kuma monitor set (web, api readyz, collab WebSocket connect) as documented configuration in `deploy/monitoring.md` and set the monitors up for Test/Int/Prod with the repo owner; degraded-vs-down semantics documented (degraded alerts, but does not restart containers — compose healthcheck uses liveness, not readyz). ## Acceptance criteria - [ ] readyz output enumerates each check with status (JSON) and returns 503 only on hard failures - [ ] stale backup status flips readyz to degraded (test with old status.json) - [ ] docker healthchecks keep using liveness endpoints (a degraded instance is not restart-looped — verify config) - [ ] monitor definitions documented; Prod monitor alerting verified once live (checklist ref #89) ## Technical notes - operations.md §Health, ADR 0015. ## Dependencies Depends on #83. **Size**: ~1 day --- *Conventions: English code/comments, clear human-readable code, no hard-coded UI strings (ADR 0012, add `de` **and** `en`), permission checks only via the shared guard (docs/architecture/permissions.md). Read the referenced ADRs before starting.*
fable-5 added this to the M8 — Self-hosting & operations milestone 2026-07-04 14:52:46 +02:00
fable-5 added the
backend
deployment
 labels 2026-07-04 14:52:46 +02:00
fable-5 commented 2026-07-11 19:13:58 +02:00
Author
Collaborator
Copy Link

Implemented in 0ef9614 (pipeline green, 8/8 contexts; live on Test + Int).

readyz with a degraded level (api)

  • GET /api/v1/readyz now enumerates five checks: database and migrations are the hard failures (HTTP 503, overall unready); converter, renderer, and the new backup check are warning-level and turn the overall status to degraded while staying HTTP 200 — users are served, monitors alert on the body.
  • The backup check reads the #83 sidecar's status.json from a new read-only backups mount in the api (BACKUPS_DIR, baked into the image like the other data dirs) and warns when the last successful backup is older than 26 h (BACKUP_FRESH_MAX_AGE_HOURS), when no backup ever succeeded (with the last run's error), or when the file is missing/unreadable. A fresh success with a newer failed run stays ok but surfaces the error in the detail.
  • The status.json shape moved to @dorfteich/shared (backup-status.ts) as the explicit contract between the sidecar (writer) and the api (#85) / admin panel (#86) as readers.

collab /healthz depth — already carried the DB probe since #33; verified live through the proxy (/collab/healthz, the documented fallback monitor target).

Docker healthchecks = liveness only (verified config) — api image HEALTHCHECK → /api/v1/healthz, web → static /healthz, collab → /healthz; none touch readyz. Compose's restart: unless-stopped restarts on process exit only, so a degraded (or even unhealthy) instance is never restart-looped — documented in monitoring.md.

Monitor definitions — new deploy/monitoring.md: the down-vs-degraded table, and a four-monitor set per stage for Uptime-Kuma (web healthz; api readyz for down; a keyword monitor on "status":"ok" that catches degraded despite HTTP 200; collab WebSocket with the /collab/healthz fallback). Prod alerting verification is referenced as a #89 go-live checklist item as the AC intends.

Acceptance criteria

  • readyz enumerates each check, 503 only on hard failures — controller returns 503 solely for unready; verified by tests and live (degraded → 200 observed locally with dead sidecars).
  • stale backup status flips readyz to degraded — unit tests (missing/torn/never-succeeded/30 h-old fixtures) plus a live boot with a 40 h-old status.json → {"status":"degraded", …"backup":"warn","detail":"last successful backup … is 40 h old (max 26 h)"}.
  • docker healthchecks keep using liveness endpoints — config audited (see above), documented in monitoring.md.
  • monitor definitions documented — deploy/monitoring.md; setting the monitors up in Uptime-Kuma is the operator step that remains open (definitions ready to click through; Prod at go-live per #89).

Robustness fixes that fell out of review: the api image now pre-creates /data/backups node-owned, so the shared backups volume stays writable for the sidecar no matter which container initializes it on a fresh stack; and the sidecar's scheduler survives a run that cannot even write status.json (logs instead of crash-looping — the missed run then shows up through this freshness check).

Stages: composes updated (*.bak-pre85) with the api's ro-mount; both stages now report readyz "ok" including the backup check end-to-end (fed by real #83 backups).

Implemented in `0ef9614` (pipeline green, 8/8 contexts; live on Test + Int). **readyz with a degraded level (api)** - `GET /api/v1/readyz` now enumerates five checks: `database` and `migrations` are the hard failures (HTTP 503, overall `unready`); `converter`, `renderer`, and the new **`backup`** check are warning-level and turn the overall status to **`degraded` while staying HTTP 200** — users are served, monitors alert on the body. - The `backup` check reads the #83 sidecar's `status.json` from a new **read-only `backups` mount in the api** (`BACKUPS_DIR`, baked into the image like the other data dirs) and warns when the last successful backup is older than **26 h** (`BACKUP_FRESH_MAX_AGE_HOURS`), when no backup ever succeeded (with the last run's error), or when the file is missing/unreadable. A fresh success with a newer *failed* run stays `ok` but surfaces the error in the detail. - The `status.json` shape moved to `@dorfteich/shared` (`backup-status.ts`) as the explicit contract between the sidecar (writer) and the api (#85) / admin panel (#86) as readers. **collab `/healthz` depth** — already carried the DB probe since #33; verified live through the proxy (`/collab/healthz`, the documented fallback monitor target). **Docker healthchecks = liveness only (verified config)** — api image `HEALTHCHECK` → `/api/v1/healthz`, web → static `/healthz`, collab → `/healthz`; none touch readyz. Compose's `restart: unless-stopped` restarts on process exit only, so a degraded (or even `unhealthy`) instance is never restart-looped — documented in monitoring.md. **Monitor definitions** — new `deploy/monitoring.md`: the down-vs-degraded table, and a four-monitor set per stage for Uptime-Kuma (web healthz; api readyz for down; a **keyword monitor on `"status":"ok"`** that catches degraded despite HTTP 200; collab WebSocket with the `/collab/healthz` fallback). Prod alerting verification is referenced as a #89 go-live checklist item as the AC intends. **Acceptance criteria** - *readyz enumerates each check, 503 only on hard failures* — controller returns 503 solely for `unready`; verified by tests and live (`degraded` → 200 observed locally with dead sidecars). - *stale backup status flips readyz to degraded* — unit tests (missing/torn/never-succeeded/30 h-old fixtures) plus a live boot with a 40 h-old `status.json` → `{"status":"degraded", …"backup":"warn","detail":"last successful backup … is 40 h old (max 26 h)"}`. - *docker healthchecks keep using liveness endpoints* — config audited (see above), documented in monitoring.md. - *monitor definitions documented* — `deploy/monitoring.md`; **setting the monitors up in Uptime-Kuma is the operator step that remains open** (definitions ready to click through; Prod at go-live per #89). **Robustness fixes that fell out of review:** the api image now pre-creates `/data/backups` node-owned, so the shared `backups` volume stays writable for the sidecar no matter which container initializes it on a fresh stack; and the sidecar's scheduler survives a run that cannot even write `status.json` (logs instead of crash-looping — the missed run then shows up through this freshness check). **Stages:** composes updated (`*.bak-pre85`) with the api's ro-mount; both stages now report readyz `"ok"` including the backup check end-to-end (fed by real #83 backups).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels   
area:auth

   
area:docs

   
area:export

   
area:ops

   
area:storage

   
area:supply-chain

   
auth

Authentication, permissions, quotas

  
backend

API server (apps/api)

  
blocked

Blocked on external dependency/infrastructure

  
collab

Collaboration server (apps/collab)

  
deployment

Docker, CI/CD, hosting

  
docs

Documentation

  
effort:L

   
effort:M

   
effort:S

   
frontend

SPA (apps/web)

  
plugins

Plugin system and SDK

  
qa

Tests, fixtures, e2e

  
vs-nfd

   
vs-nfd:blocker
No Label
area:auth
area:docs
area:export
area:ops
area:storage
area:supply-chain
auth
backend
blocked
collab
deployment
docs
effort:L
effort:M
effort:S
frontend
plugins
qa
vs-nfd
vs-nfd:blocker
Milestone
Clear milestone
No items
No Milestone
M8 — Self-hosting & operations
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: stwaidele/dorfteich#85
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?