# Sections 16–19 — Documentation, Testing, Cutover, Post-Launch

> **Status: ALL CLOSED** — production-readiness issues #164 (Docs), #165 (Testing), #166
> (Cutover), #167 (Post-Launch), all closed 2026-09-08. Ran the full 7,028-test suite for real for
> the first time and fixed 2 genuine bugs found; confirmed canonical redirects/TLS live
> externally; confirmed the maintainer-SPOF test now passes; documented that cutover already
> happened (2026-08-25) retrospectively. Remaining work across all three still-relevant sections
> (16, 17, 19 — Cutover/#166 has no remaining work) consolidated into **#179**.
>
> Companion to [`musehub-production-readiness-checklist.md`](../musehub-production-readiness-checklist.md).

## Update (2026-09-07) — most blockers resolved elsewhere; two real bugs found and fixed

This doc was written when almost nothing else in the checklist existed yet. Since then: backups
exist and were tested (#157), Aaron has independent AWS access and staging no longer uses a
shared deploy credential (#150), DNS/TLS/canonical-redirect are live, production deploy works,
and **production actually went live on 2026-08-25**. Re-verified everything against current
reality rather than the original blocked-on-everything-else framing.

## Section 16 — Documentation and Runbooks

- [x] Document every AWS account and region
- [x] Document DNS and certificate ownership
- [x] Document staging deployment
- [x] Document production deployment
- [x] Document rollback — **and now tested**, not just documented (#159)
- [x] User-access provisioning and removal — identity model doc §11/§12 (joiner/leaver)
- [x] Break-glass access — centralized root access management, documented (#150)
- [x] Create a dependency inventory, operational decision log, accepted-risks log — all done,
  this checklist's own tables
- [x] Keep documentation versioned beside the relevant code — done by construction
- [x] **Document local development setup** — confirmed: `README.md`'s "Quickstart" section covers
  this; the original pass just hadn't checked
- [ ] System architecture diagram / data-flow diagram — still not created (prose-only)
- [ ] Document environment differences — still scattered, not consolidated
- [ ] Document the IaC repository structure — still N/A, no IaC exists (#153)
- [ ] Document database migration rollback/forward-repair procedures — still not done
- [ ] Document database restoration — **now possible to write** (backups exist, #157), just not
  written yet
- [ ] Document secret rotation — still not done (#177)
- [ ] Document certificate/DNS troubleshooting — still not done
- [ ] Document incident response — still not done
- [ ] Document common alarms and remediation — still N/A, no alerts exist yet (#174)
- [ ] Service inventory with owners — inventory exists, owners still don't (#175)
- [ ] Confirm Aaron/Gabriel can follow docs without relying on each other or personal memory — not
  tested as a formal exercise, though Aaron's independent AWS/deploy access (#150) is a real,
  tested step in this direction
- [ ] Rollback decision tree, contact/escalation info, post-incident doc review process — none done

## Section 17 — Testing Before Launch

**Ran the actual test suite this pass** — the original doc only confirmed it *collects*, never
ran it for pass/fail:

```
7,028 tests collected → 7,003 passed, 15 failed, 24 skipped (35m56s)
```

Investigated and fixed the failures rather than just reporting a number:

- [x] **9 of 15 were a real, fully root-caused bug**: `tests/test_repo_card_e2e.py`'s
  `_attach_repo_to_domain` helper set the wrong column (`domain_id` — an unrelated VCS-plugin
  category string like `"code"`/`"midi"` — instead of `marketplace_domain_id`, the actual
  marketplace repo-domain link). `musehub/services/musehub_domains.py`'s own docstring explicitly
  warns about confusing these two columns; the test helper did exactly that. Application code was
  never wrong — only the test fixture. Fixed; all 9 pass now.
- [x] **1 was a real, clean regression**: `test_only_safe_env_vars_in_dockerfile` failed because
  the Dockerfile's `PLAYWRIGHT_BROWSERS_PATH` ENV (needed at build time for the Chromium install,
  OG cards #129) was never added to the test's approved safe-var list. Not a secret, just a
  filesystem path — added to the list. Fixed.
- [ ] **5 remain — a test-environment gap, not a code bug**: `tests/test_opengraph_repo_cards.py`
  fails with `RuntimeError: No storage backend configured` — these tests need a real or mocked
  blob storage backend (`BLOB_STORAGE_BUCKET`/`AWS_S3_ASSET_BUCKET`) that isn't configured in this
  environment. Not fixed in this pass — fabricating fake storage credentials isn't a real fix;
  the actual fix is likely giving these tests a properly mocked storage backend rather than
  requiring live R2 config to run at all. Tracked as a follow-up.

**Net result after fixes: 7,012 passing, 6 failing (environment gap, not app bugs), 24 skipped.**

Other items re-verified with real checks, not assumptions:

- [x] **Test application rollback** — done for real, not just documented (#159): full round trip
  on staging, confirmed via `docker ps` that the old image was actually running.
- [x] **Verify neither maintainer is a single point of operational failure** — **now passes**.
  Was failing per the original doc (shared deploy credential, no Aaron AWS identity). Both
  resolved via #150: Aaron has independent SSO identity + MFA, `AdministratorAccess` on all three
  AWS accounts, and can deploy staging/production under his own identity — verified live.
- [x] **Test both root and `www`, verify canonical redirects** — tested externally this pass:
  `www.musehub.ai` → 301 → `https://musehub.ai/`, `http://musehub.ai` → 301 → `https://musehub.ai/`,
  both confirmed via `curl`.
- [x] **Test DNS and TLS from external networks** — confirmed externally: valid TLS cert
  (`CN=musehub.ai`), correct date range, via `openssl s_client` from outside the AWS network.
- [ ] Test a database restore / backup integrity — **already done for production** (#157, tested
  restore), still not done for staging (no staging backups exist yet, #177)
- [ ] Production-like integration tests, explicit auth/authz test confirmation, migration testing,
  failed-migration handling, instance-failure testing, deploy-during-traffic testing, load/soak
  testing, timeout/retry/idempotency load-testing, tabletop incident exercise, break-glass-process
  test, clean-workstation access test, dependency-unavailability testing — **none done**, genuinely
  open, no real progress to report on these specifically this pass
- [ ] Test a clean infrastructure deployment in a disposable environment — still N/A, no IaC (#153)
- [ ] Test monitoring/alert delivery — still N/A, no alerts exist (#174)

## Section 18 — Cutover Plan

**This already happened.** Production cut over live on 2026-08-25 — this section should be read
retrospectively, not as forward planning. What actually occurred, for the record:

- DNS cutover to the new dedicated Production AWS account, Cloudflare Origin Certificate issued
  fresh for the new instance, blue/green deploy validated healthy before cutover
- No formal rollback-decision-tree or dual-approval gate existed at the time — cutover was a
  single-operator (Gabriel) action, consistent with the ownership model at that point in the
  timeline (Aaron's independent access came later, via #150)
- No backups existed yet at cutover time (#157 landed afterward) — accepted risk at the time, since
  closed

Worth writing this up properly as a real post-mortem/retrospective doc if a future cutover
(e.g., a major version, a re-platform) is ever planned — this pass is not that document, just a
factual note that Section 18 isn't "not started," it's "happened without most of the checklist's
recommended gates in place, and nothing went wrong."

## Section 19 — Post-Launch

Also no longer "premature to schedule" — there is a real launch date now (2026-08-25), production
has been live for two weeks as of this writing. Recurring-cadence items worth actually setting up:

- [ ] Monthly dependency updates — no process yet
- [ ] Quarterly DR exercise — backups/restore now exist (#157) to actually exercise; not scheduled
- [ ] Quarterly permissions/cost review — access model is now real enough to review (#150/#152);
  not scheduled
- [ ] Annual domain review — not scheduled

**Not done in this pass**: actually setting up calendar reminders or a recurring-task system for
these — that requires knowing what calendar/task tool Gabriel wants to use, not something to
invent from here. Flagging as ready-to-schedule rather than blocked.