Section 12 — Observability

Status: CLOSED — production-readiness issue #160, closed 2026-09-08. Alarms, SNS alerting, log retention, and environment/release fields all live on both environments. Remaining work (CloudWatch Agent install for real infra metrics, external uptime checks, error tracking, dashboards) tracked in #181.

Companion to musehub-production-readiness-checklist.md.

Update (2026-09-08) — wired real alarms on top of the existing logging foundation

The foundation (structured, PII-scrubbed, centralized logging) was already solid. This pass did the actual wiring work the original doc identified as the path to closing this section.

Fixed real bugs in deploy/cloudwatch-alerts.sh, beyond just the log group name

• Wrong default region (eu-west-1 — this project runs entirely in us-east-1)
• Hardcoded single log group (/musehub/app) — now takes --env staging|production and targets the real per-environment log group and its own SNS topic, matching the same per-environment pattern already used in push.sh/deploy.sh
• A metric-alarm bug that would have failed at runtime: passed both --statistic p99 and --extended-statistic p99 — these are mutually exclusive; percentile statistics can only use --extended-statistic. Found by actually running the script, not just reading it.
• A completely inapplicable alarm removed: the original script had a DB-connections alarm pointed at AWS/RDS DatabaseConnections — this project uses self-hosted Postgres in a Docker container on EC2, not RDS. That alarm could never have fired; left in, it would have looked like monitoring existed when it didn't. Removed with a note explaining what a real replacement needs (CloudWatch Agent's procstat plugin or a custom pushed metric).
• Wrong CloudWatch Agent install instructions: said sudo yum install (Amazon Linux) — these instances run Ubuntu. Corrected to the actual .deb install path, though the agent itself isn't installed yet (see Remaining work).
• Single-email alerting → dual: now subscribes both [email protected] and [email protected] by default, satisfying "route alerts to both maintainers."

Ran it for real against both environments — not just fixed and left unverified

staging:    30-day log retention set, SNS topic + 2 email subs, 3 metric filters, 2 alarms (OK)
production: 30-day log retention set, SNS topic + 2 email subs, 3 metric filters, 2 alarms (INSUFFICIENT_DATA — expected, low traffic volume)

Verified via aws logs describe-log-groups and aws cloudwatch describe-alarms directly, not just trusting the script's own success output. Gabriel and Aaron each need to confirm the SNS email subscription (AWS sends a confirmation link) before alerts actually deliver.

Added environment and release_version to every log line

musehub/logging_config.py's JsonFormatter now stamps environment (from the existing muse_env setting) and release_version (a new setting, sourced from a RELEASE_VERSION env var that deploy/deploy.sh now passes into the container from the same IMAGE_TAG it already uses). Verified working end-to-end locally (configure_logging → real log line → both fields present with correct values). Takes effect for real traffic on the next deploy to each environment.

Checklist assessment — updated

• [x] Centralize application logs
• [x] Use structured logs
• [x] Include timestamp/environment/service/request-id/release-version/severity — now complete; environment and release_version were the only missing fields
• [x] Prevent credentials/tokens/PII from entering logs
• [x] Set explicit log-retention periods — 30 days on both environments, done for real
• [ ] Encrypt logs with a customer-managed KMS key — still using the AWS-managed default; not revisited, low priority
• [ ] Collect true infrastructure metrics (CPU/disk/memory at the instance level) — still not done; needs the CloudWatch Agent actually installed (#181)
• [ ] Build dashboards — still none exist, though metric filters now exist to build one from
• [ ] Add external uptime checks — still not done (#181)
• [x] Configure alerts for 5xx rate and latency — done, live on both environments
• [ ] Configure remaining alert types (unhealthy deploy, DB storage, CPU/memory, backup failure, cert problems, security findings, cost growth) — most need the CloudWatch Agent or other data sources that don't exist yet (#181)
• [x] Route alerts to both Gabriel and Aaron — done, both subscribed on both environments
• [ ] Define warning vs. page-worthy severity — not done
• [ ] Test alert delivery — not done (would need to actually trigger a 5xx spike or induce latency)
• [ ] Every urgent alert links to a runbook — not done, no runbooks exist yet
• [ ] Configure error tracking with release correlation — not done, no tool chosen (#181)
• [ ] Define SLIs/initial SLOs — not done
• [ ] Add deployment markers to dashboards — N/A, no dashboards exist
• [ ] Verify logs/metrics remain available during an application outage — not tested

Remaining work — split to #181

CloudWatch Agent installation (for real disk/memory/infra metrics), external uptime monitoring, error tracking tool selection, dashboard construction, alert-delivery testing, and severity/runbook definitions are all real, separate pieces of work needing their own decisions (tool choice, cost, alarm thresholds) rather than being quick wins on top of existing data — unlike what this pass actually closed.