Claude Fable 5
0ef96147e0
All checks were successful
CI / Lint, typecheck, test (push) Successful in 3m11s
CI / Build container images (push) Has been skipped
CD / Build and push images (push) Successful in 3m44s
CD / Deploy to Test (push) Successful in 9s
CD / Smoke tests against Test (push) Successful in 1m10s
CD / Promote to Int (push) Successful in 10s
CI / Auth e2e pack (push) Successful in 5m13s
CI / Import/export fidelity gate (push) Successful in 46s
readyz now enumerates database/migrations (hard failures, HTTP 503), converter/renderer, and a new backup check that reads the sidecar's status.json from the read-only backups mount and warns when the last successful backup is older than 26 h. Warning-level checks surface as overall status "degraded" while staying HTTP 200 — monitors alert on the body keyword, Docker healthchecks keep using the liveness endpoints so a degraded instance is never restart-looped. The status.json shape moved to @dorfteich/shared as the contract between the sidecar and its readers (#85/#86); deploy/monitoring.md defines the Uptime-Kuma monitor set per stage. The api image also pre-creates /data/backups node-owned so the shared backups volume stays writable for the sidecar regardless of which container initializes it, and the sidecar's scheduler survives runs that cannot even record their status. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EwZ4jR4KFAPvpjWevfUGX1
59 lines
3.4 KiB
Markdown
59 lines
3.4 KiB
Markdown
# Uptime monitoring (issue #85)
|
||
|
||
Pragmatic monitoring per operations.md: no metrics stack — the health
|
||
endpoints are the single integration point, watched by the operator's
|
||
existing **Uptime-Kuma** instance.
|
||
|
||
## Semantics: down vs. degraded
|
||
|
||
`GET /api/v1/readyz` returns:
|
||
|
||
| HTTP | `status` | Meaning | Reaction |
|
||
| ---- | ---------- | ------------------------------------------------------------------- | --------------------------------------------- |
|
||
| 200 | `ok` | all checks green | — |
|
||
| 200 | `degraded` | a warning-level check: converter/renderer down, backup stale (26 h) | alert, fix without urgency — users are served |
|
||
| 503 | `unready` | hard failure: database unreachable or migrations pending | page — the instance cannot serve |
|
||
|
||
Checks enumerated in the body: `database`, `migrations` (hard),
|
||
`converter`, `renderer`, `backup` (warning-level; `backup` reads the
|
||
sidecar's `status.json` and warns when the last successful backup is
|
||
older than 26 h — ADR 0015).
|
||
|
||
**Degraded never restarts containers**: the Docker healthchecks use the
|
||
liveness endpoints only (api `/api/v1/healthz`, web `/healthz`, collab
|
||
`/healthz`), never `readyz`. The collab `/healthz` includes its database
|
||
probe (issue #33) — Compose marks the container `unhealthy` then, but
|
||
`restart: unless-stopped` only restarts on process exit, so an unhealthy
|
||
container is visible in `docker compose ps`, not restart-looped.
|
||
|
||
## Monitor set (per stage)
|
||
|
||
Four monitors per stage in Uptime-Kuma; suggested interval 60 s, retries 2.
|
||
|
||
| # | Monitor | Type | Target (Test example) | Alert condition |
|
||
| --- | ---------------------- | ----------------- | ------------------------------------------------- | -------------------------------------- |
|
||
| 1 | `<stage> web` | HTTP(s) | `https://test.dorfteich.cloud/healthz` | non-2xx |
|
||
| 2 | `<stage> api ready` | HTTP(s) | `https://test.dorfteich.cloud/api/v1/readyz` | non-2xx (= unready/down) |
|
||
| 3 | `<stage> api degraded` | HTTP(s) – Keyword | same URL, keyword `"status":"ok"` must be present | body says `degraded` while HTTP is 200 |
|
||
| 4 | `<stage> collab` | WebSocket | `wss://test.dorfteich.cloud/collab` | connect failure |
|
||
|
||
Monitor 3 is what catches `degraded` (stale backup, dead sidecars) —
|
||
monitor 2 alone only sees hard failures. If the Kuma version has no
|
||
WebSocket monitor type, an HTTP monitor on
|
||
`https://<stage>/collab/healthz` (200, includes the DB probe) is the
|
||
fallback for monitor 4.
|
||
|
||
Stages: `test.dorfteich.cloud`, `int.dorfteich.cloud` (web-check-only is
|
||
acceptable per operations.md — at minimum monitors 1–2, no paging),
|
||
`dorfteich.online` (Prod, full set + notification; set up and verified at
|
||
go-live, checklist item in #89).
|
||
|
||
Setting up the monitors is an operator action in Uptime-Kuma (owner:
|
||
Stefan); this file is the definition they follow.
|
||
|
||
## Related
|
||
|
||
- operations.md §Health & monitoring (endpoint semantics)
|
||
- ADR 0015 (backup freshness feeds the `backup` check)
|
||
- deploy/backup/restore.sh (what to do when the backup check warns)
|