# Section 5 — Production Architecture

> Companion to [`musehub-production-readiness-checklist.md`](../musehub-production-readiness-checklist.md).
>
> **Correction, 2026-09-08 (Phases 0–2 complete)**: while re-verifying this section, discovered
> that staging had actually been running on managed AWS RDS (not self-hosted Postgres) since
> 2026-04-05 — see [`docs/database-architecture.md`](../database-architecture.md), the canonical
> source of truth for database architecture. Production has since been migrated to RDS too
> (2026-09-08). **Item 2 below is now partially resolved, not eliminated**: both environments get
> managed patching, automated backups, and point-in-time recovery from RDS, closing the "no managed
> failover infrastructure at all" gap. But both instances are still deliberately **single-AZ**
> (`MultiAZ: false`) — a real outage or AZ failure still takes the database down; RDS just makes
> *recovery* fast and reliable rather than eliminating the single point of failure outright.
> Enabling Multi-AZ (near-zero-downtime automatic failover) remains a genuine, not-yet-made
> decision — tracked as a Phase 4 follow-up, gated on whether the added cost (roughly 2x the
> instance cost) is justified by current availability targets (#149).

## Good news first — the app itself is more hardened than the infra around it

Grepping `musehub/` turned up real, already-implemented protections that Section 0's
infrastructure-level inventory didn't surface:

- **Health checks with real dependency probes**: `/healthz` (`musehub/main.py:323`) checks both DB
  and object storage, not just "process is up." Used by `deploy.sh` for the blue/green health gate.
- **Rate limiting** (`musehub/rate_limits.py`): per-route `slowapi` limits (wire push 30/min, MCP
  human/agent/anonymous tiers, asset requests), plus a 300/min global per-IP baseline.
- **Bot throttling** (`musehub/middleware/bot_throttle.py`): blocks unauthenticated scanners on
  write paths while explicitly exempting authenticated (MSign) traffic and safe GET/HEAD reads —
  a deliberate, well-reasoned design, not a blunt UA blocklist.
- **Security headers already set** in `main.py`: `X-Frame-Options`, `X-Content-Type-Options`,
  `Content-Security-Policy`, `Strict-Transport-Security` — and again at the nginx layer
  (`deploy/nginx-cf.conf`): HSTS with `includeSubDomains`, `nosniff`, `X-Frame-Options: SAMEORIGIN`,
  `Referrer-Policy`.
- **Deliberate per-route timeouts** in nginx — not a single blanket value: 5s for fast health
  checks, 60s default, 120–300s for heavier routes, 3600s for streaming (MCP/SSE) endpoints. This
  is intentional, not an oversight.
- **No cookies at all** — MSign is header-based (Ed25519 signatures), so cookie flag items
  (`Secure`/`HttpOnly`/`SameSite`) are **N/A**, not gaps.

## Real gaps

- **Single EC2 instance per environment — no ASG, no multi-AZ.** Blue/green happens *within* one
  instance (two Docker containers on different ports), not across instances. If the instance
  itself fails (hardware, AZ outage), there is no automatic failover. This is the single largest
  intentional SPOF and needs an explicit decision: accept it (documented technical debt, matches
  the "keep it simple" architecture direction from Section 0) or plan multi-instance/multi-AZ.
- **No graceful shutdown / SIGTERM handling found** in `musehub/main.py` — uvicorn's defaults
  apply. In-flight requests during a container stop may be dropped rather than drained. Given
  blue/green already waits for the new slot's `/healthz` before flipping nginx, the *new* slot is
  protected, but the *old* slot's shutdown during `docker rm` isn't verified graceful.
- **No autoscaling, no documented justification for fixed capacity.** Fixed `t3.small` (prod) /
  `t3.medium` (staging) — inconsistent sizing between environments with no stated reason. Worth
  either matching sizes or documenting why staging is intentionally larger.
- **No AWS WAF — and none needed the way the checklist assumes.** There's no ALB/CloudFront for
  AWS WAF to attach to. DDoS/abuse protection is Cloudflare's job here (edge proxy in front of
  everything), which is a reasonable substitution — but Cloudflare's own WAF/rate-limiting rules
  are unverified from this environment (no Cloudflare API access). **Needs Gabriel to check the
  Cloudflare dashboard** for what's actually configured there.
- **No AWS service quota monitoring.**
- **Background job idempotency** (`musehub-runner` CI jobs) not verified in this pass — would need
  a closer read of the runner's job-claiming logic; flagging as unverified rather than assuming.
- **Migration/rolling-deploy compatibility**: the deploy pipeline does run migrations before the
  slot flip (per `docs/infrastructure.md`'s pipeline steps), which is the right shape, but
  expand-and-contract discipline for schema changes hasn't been reviewed — deferred to Section 8.

## Documented intentional single points of failure (per the checklist's explicit ask)

1. One EC2 instance per environment — no instance-level redundancy.
2. **Updated 2026-09-08**: one RDS Postgres instance per environment (previously self-hosted for
   production, self-hosted-vs-RDS inconsistent between environments before this date — see
   `docs/database-architecture.md`) — single-AZ, no automatic failover. RDS Multi-AZ would close
   this remaining gap; not yet enabled, tracked as a Phase 4 decision pending availability targets.
3. One AWS account, shared with Stori (Section 2).
4. One Cloudflare account/zone — DNS, TLS, and (per Section 6) possibly object storage all
   depend on Cloudflare being up and correctly configured.
5. Deploys depend on one human running a script with a shared credential from a laptop — no
   redundant deployer today (Section 1/3).

**Plan to remove unacceptable ones**: not yet decided which of the above are acceptable long-term
vs. must-fix-before-launch — this needs Gabriel's input once RPO/RTO/availability targets
(Section 0, still open) are defined. A SPOF that's fine for a beta with a handful of users may not
be fine at real scale.