QA: self-hosting and operations e2e #90

New Issue

fable-5 · 2026-07-04T14:52:49+02:00

fable-5 commented

2026-07-04 14:52:49 +02:00

Context

M8's promises (wizard, backup, update) are operational paths — they regress silently without tests.

Scope

Automate: wizard e2e (fresh stack → configured instance), update simulation (deploy version N fixture data, upgrade to N+1 image, verify migrations + content intact), backup/restore roundtrip as a compose-level test (distinct from the drill: runs on every release candidate), degraded-readyz scenarios; wire into the release workflow as the pre-approval gate.

Acceptance criteria

release candidates run this suite before the manual gate (pipeline proof)
update simulation catches a deliberately backward-incompatible migration (negative test, then remove)
suite documented so future migrations add their own update fixtures

Technical notes

ADR 0014, deployment.md (one-release-back compatibility rule).

Dependencies

Depends on #80, #83, #89.

Size: ~1.5 days

Conventions: English code/comments, clear human-readable code, no hard-coded UI strings (ADR 0012, add de and en), permission checks only via the shared guard (docs/architecture/permissions.md). Read the referenced ADRs before starting.

## Context M8's promises (wizard, backup, update) are operational paths — they regress silently without tests. ## Scope Automate: wizard e2e (fresh stack → configured instance), update simulation (deploy version N fixture data, upgrade to N+1 image, verify migrations + content intact), backup/restore roundtrip as a compose-level test (distinct from the drill: runs on every release candidate), degraded-readyz scenarios; wire into the release workflow as the pre-approval gate. ## Acceptance criteria - [ ] release candidates run this suite before the manual gate (pipeline proof) - [ ] update simulation catches a deliberately backward-incompatible migration (negative test, then remove) - [ ] suite documented so future migrations add their own update fixtures ## Technical notes - ADR 0014, deployment.md (one-release-back compatibility rule). ## Dependencies Depends on #80, #83, #89. **Size**: ~1.5 days --- *Conventions: English code/comments, clear human-readable code, no hard-coded UI strings (ADR 0012, add `de` **and** `en`), permission checks only via the shared guard (docs/architecture/permissions.md). Read the referenced ADRs before starting.*

fable-5 added this to the M8 — Self-hosting & operations milestone 2026-07-04 14:52:49 +02:00

fable-5 added the

qa

label 2026-07-04 14:52:49 +02:00

fable-5 referenced this issue from a commit

2026-07-12 00:29:55 +02:00

Run operations QA on every release candidate before the prod gate (#90)

fable-5 commented

2026-07-12 00:40:46 +02:00

Implemented in a9e901c (pipeline green; proven on the real release v0.1.2).

deploy/release-qa.sh — the automated operations QA, wired into the release workflow as the second job (ops-qa): a release candidate is approvable only when both jobs of its release run are green — that is the documented pre-approval gate (the approval itself remains the prod-vX.Y.Z-* tag push, per #89).

What it proves, per candidate, in a throwaway plain-docker environment (unique prefix, trap teardown, no bind mounts — the CI job talks to the host daemon):

Update simulation — boots the previous release with SETUP_ADMIN_*-preseeded fixture content (pond + page through the real API), then swaps the api to the candidate against the same database: migrations auto-apply (migrations: ok asserted), login works, the fixture content is intact. The seed_fixture/assert_fixture pair is the documented update-fixture contract: a migration story extends exactly those two functions (write on N-1, read on N) — noted in the script header and deployment.md.
Degraded-readyz semantics — without converter/renderer sidecars and before the first backup, the candidate answers 200 degraded with warn-level checks, never 503.
Backup/restore roundtrip — the candidate's backup sidecar dumps the stack, the set restores into a second empty database, and the candidate api serves the fixture from it. Distinct from the monthly drill (#87): per release candidate, candidate images, never stage data.

The wizard e2e (fresh stack → configured instance) already runs on every CI build against a freshly provisioned stack since #81 — the suite deliberately does not duplicate it.

Acceptance criteria

release candidates run this suite before the manual gate (pipeline proof) — release v0.1.2: Build release images ✓ and Release-candidate operations QA ✓ (update simulation v0.1.1 → v0.1.2), then and only then the gate tag deployed Prod (now on v0.1.2, readyz ok).
update simulation catches a deliberately backward-incompatible migration — negative test executed on the host: a simulated destructive migration (ALTER TABLE pages DROP COLUMN title CASCADE between N-1 and N) made the suite fail loudly (FAILED — fixture page unreadable); the manipulation was a temporary copy, nothing committed.
suite documented for future update fixtures — script header + deployment.md §Self-hosting distribution.

Implemented in `a9e901c` (pipeline green; proven on the real release `v0.1.2`). **`deploy/release-qa.sh`** — the automated operations QA, wired into the release workflow as the second job (`ops-qa`): a release candidate is approvable only when **both** jobs of its release run are green — that is the documented pre-approval gate (the approval itself remains the `prod-vX.Y.Z-*` tag push, per #89). What it proves, per candidate, in a throwaway plain-docker environment (unique prefix, trap teardown, no bind mounts — the CI job talks to the host daemon): 1. **Update simulation** — boots the *previous* release with `SETUP_ADMIN_*`-preseeded fixture content (pond + page through the real API), then swaps the api to the *candidate* against the same database: migrations auto-apply (`migrations: ok` asserted), login works, the fixture content is intact. The `seed_fixture`/`assert_fixture` pair is the documented update-fixture contract: **a migration story extends exactly those two functions** (write on N-1, read on N) — noted in the script header and deployment.md. 2. **Degraded-readyz semantics** — without converter/renderer sidecars and before the first backup, the candidate answers 200 `degraded` with warn-level checks, never 503. 3. **Backup/restore roundtrip** — the candidate's backup sidecar dumps the stack, the set restores into a second empty database, and the candidate api serves the fixture from it. Distinct from the monthly drill (#87): per release candidate, candidate images, never stage data. The **wizard e2e** (fresh stack → configured instance) already runs on every CI build against a freshly provisioned stack since #81 — the suite deliberately does not duplicate it. **Acceptance criteria** - *release candidates run this suite before the manual gate (pipeline proof)* — release `v0.1.2`: `Build release images` ✓ and `Release-candidate operations QA` ✓ (update simulation v0.1.1 → v0.1.2), then and only then the gate tag deployed Prod (now on v0.1.2, readyz `ok`). - *update simulation catches a deliberately backward-incompatible migration* — negative test executed on the host: a simulated destructive migration (`ALTER TABLE pages DROP COLUMN title CASCADE` between N-1 and N) made the suite fail loudly (`FAILED — fixture page unreadable`); the manipulation was a temporary copy, nothing committed. - *suite documented for future update fixtures* — script header + deployment.md §Self-hosting distribution.

fable-5 closed this issue

2026-07-12 00:40:47 +02:00

Sign in to join this conversation.