Claude Fable 5
0ef96147e0
readyz now enumerates database/migrations (hard failures, HTTP 503), converter/renderer, and a new backup check that reads the sidecar's status.json from the read-only backups mount and warns when the last successful backup is older than 26 h. Warning-level checks surface as overall status "degraded" while staying HTTP 200 — monitors alert on the body keyword, Docker healthchecks keep using the liveness endpoints so a degraded instance is never restart-looped. The status.json shape moved to @dorfteich/shared as the contract between the sidecar and its readers (#85/#86); deploy/monitoring.md defines the Uptime-Kuma monitor set per stage. The api image also pre-creates /data/backups node-owned so the shared backups volume stays writable for the sidecar regardless of which container initializes it, and the sidecar's scheduler survives runs that cannot even record their status. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EwZ4jR4KFAPvpjWevfUGX1
3.4 KiB
Uptime monitoring (issue #85)
Pragmatic monitoring per operations.md: no metrics stack — the health endpoints are the single integration point, watched by the operator's existing Uptime-Kuma instance.
Semantics: down vs. degraded
GET /api/v1/readyz returns:
| HTTP | status |
Meaning | Reaction |
|---|---|---|---|
| 200 | ok |
all checks green | — |
| 200 | degraded |
a warning-level check: converter/renderer down, backup stale (26 h) | alert, fix without urgency — users are served |
| 503 | unready |
hard failure: database unreachable or migrations pending | page — the instance cannot serve |
Checks enumerated in the body: database, migrations (hard),
converter, renderer, backup (warning-level; backup reads the
sidecar's status.json and warns when the last successful backup is
older than 26 h — ADR 0015).
Degraded never restarts containers: the Docker healthchecks use the
liveness endpoints only (api /api/v1/healthz, web /healthz, collab
/healthz), never readyz. The collab /healthz includes its database
probe (issue #33) — Compose marks the container unhealthy then, but
restart: unless-stopped only restarts on process exit, so an unhealthy
container is visible in docker compose ps, not restart-looped.
Monitor set (per stage)
Four monitors per stage in Uptime-Kuma; suggested interval 60 s, retries 2.
| # | Monitor | Type | Target (Test example) | Alert condition |
|---|---|---|---|---|
| 1 | <stage> web |
HTTP(s) | https://test.dorfteich.cloud/healthz |
non-2xx |
| 2 | <stage> api ready |
HTTP(s) | https://test.dorfteich.cloud/api/v1/readyz |
non-2xx (= unready/down) |
| 3 | <stage> api degraded |
HTTP(s) – Keyword | same URL, keyword "status":"ok" must be present |
body says degraded while HTTP is 200 |
| 4 | <stage> collab |
WebSocket | wss://test.dorfteich.cloud/collab |
connect failure |
Monitor 3 is what catches degraded (stale backup, dead sidecars) —
monitor 2 alone only sees hard failures. If the Kuma version has no
WebSocket monitor type, an HTTP monitor on
https://<stage>/collab/healthz (200, includes the DB probe) is the
fallback for monitor 4.
Stages: test.dorfteich.cloud, int.dorfteich.cloud (web-check-only is
acceptable per operations.md — at minimum monitors 1–2, no paging),
dorfteich.online (Prod, full set + notification; set up and verified at
go-live, checklist item in #89).
Setting up the monitors is an operator action in Uptime-Kuma (owner: Stefan); this file is the definition they follow.
Related
- operations.md §Health & monitoring (endpoint semantics)
- ADR 0015 (backup freshness feeds the
backupcheck) - deploy/backup/restore.sh (what to do when the backup check warns)