gabriel / musehub public
fix BREAKING fix/cloudwatch-alerts-and-log-fields #1 / 1
AI Agent gabriel · 1 day ago · Sep 8, 2026 · Diff

fix: real CloudWatch alerting on both envs, add environment/release_version to logs (#160)

deploy/cloudwatch-alerts.sh had more bugs than the known log-group-name issue: wrong default region (eu-west-1), a metric-alarm call that would fail at runtime (--statistic and --extended-statistic are mutually exclusive, found by actually running it), a DB-connections alarm pointed at AWS/RDS that could never fire (this project uses self-hosted Postgres on EC2, not RDS - removed rather than left silently broken), and CloudWatch Agent install instructions using yum on Ubuntu instances.

Fixed all of it and ran the script for real against both staging and production: 30-day log retention set, SNS topics with both Gabriel and Aaron subscribed, 3 metric filters and 2 alarms (5xx rate, p99 latency) live on each environment. Verified via aws logs/cloudwatch describe-* directly, not just the script's own output.

Added environment/release_version to every log line (logging_config.py + a new Settings field + deploy.sh passing RELEASE_VERSION from the existing IMAGE_TAG) - verified end-to-end locally, takes effect on next deploy.

sha256:7afc59b2ee9c70d18b748d269602c588385d6e936cc22b056fd775ab9f2fd499 sha
sha256:9c64dbfd65ef4e8a85f500e5909c06c2e6c65255a69e84188ee73b63f111cecd parent
+13 ~9 −9 symbols
6 changed · 1060 in snapshot files
sha256:e989713975e76fafaa61f56775f28d1548b0661ef37e79bff3049f61d767a727 snapshot
+13
symbols added
~9
symbols modified
−9
symbols removed
6
files changed
1060
files in snapshot
0
dead code introduced
Semantic Changes 31 symbols
+ .vscode/
+ alembic/
+ alembic/versions/
+ deploy/
+ deploy/load-tests/
+ docs/
+ docs/architecture/
+ docs/guides/
+ docs/issues/
+ docs/issues/assets/
+ docs/legal/
+ docs/mists/
+ docs/production-readiness/
+ docs/reference/
+ docs/releases/
+ docs/roadmaps/
+ musehub/
+ musehub/api/
+ musehub/api/routes/
+ musehub/api/routes/api/
+ musehub/api/routes/musehub/
+ musehub/auth/
+ musehub/core/
+ musehub/crypto/
+ musehub/db/
+ musehub/debug/
+ musehub/graph/
+ musehub/maintenance/
+ musehub/mcp/
+ musehub/mcp/tools/
+ musehub/mcp/write_tools/
+ musehub/middleware/
+ musehub/models/
+ musehub/muse_cli/
+ musehub/proposals/
+ musehub/protocol/
+ musehub/security/
+ musehub/services/
+ musehub/storage/
+ musehub/templates/
+ musehub/templates/mcp/
+ musehub/templates/musehub/
+ musehub/templates/musehub/fragments/
+ musehub/templates/musehub/macros/
+ musehub/templates/musehub/opengraph/
+ musehub/templates/musehub/pages/
+ musehub/templates/musehub/partials/
+ musehub/templates/musehub/static/
+ musehub/templates/musehub/static/vendor/
+ musehub/types/
+ scripts/
+ src/
+ src/scss/
+ src/scss/components/
+ src/scss/pages/
+ src/scss/patterns/
+ src/scss/theme/
+ src/ts/
+ src/ts/pages/
+ tests/
+ tests/unit/
+ tools/
+ workflows/
~ deploy/cloudwatch-alerts.sh .sh 4 symbols added, 5 symbols removed, 4 symbols modified
~ DB_INSTANCE
− ALERT_EMAIL variable variable ALERT_EMAIL L26–26
− ALERT_PHONE variable variable ALERT_PHONE L27–27
− INSTANCE_ID variable variable INSTANCE_ID L49–49
− THRESHOLD_DB_CONN_PCT variable variable THRESHOLD_DB_CONN_PCT L33–33
− THRESHOLD_DISK_PCT variable variable THRESHOLD_DISK_PCT L32–32
+ ALERT_EMAILS variable variable ALERT_EMAILS L32–32
+ ENV variable variable ENV L55–55
+ IFS variable variable IFS L92–92
+ email variable variable email L94–94
~ LOG_GROUP
~ SNS_TOPIC_NAME
~ aws_cmd
~ deploy/deploy.sh .sh
~ docs/production-readiness/12-observability.md .md 7 symbols added, 4 symbols removed, 1 symbol modified
− Checklist assessment section Checklist assessment L41–71
− The actual work items here section The actual work items here L71–82
− What's actually live — verified directly against CloudWatch section What's actually live — verified directly against CloudWatch L8–22
− What's missing section What's missing L22–41
+ Checklist assessment — updated section Checklist assessment — updated L54–81
+ Remaining work — split to #181 section Remaining work — split to #181 L81–88
+ Update (2026-09-08) — wired real alarms on top of the existing logging foundation section Update (2026-09-08) — wired real alarms on top of the existing logging foundation L10–54
+ Added environment and release_version to every log line section Added environment and release_version to every log line L46–54
+ sh, beyond just the log group name section Fixed real bugs in deploy/cloudwatch-alerts.sh, beyond just the log group name L15–35
+ Ran it for real against both environments — not just fixed and left unverified section Ran it for real against both environments — not just fixed and left unverified L35–46
+ code variable variable code L37–41
~ Section 12 — Observability
~ musehub/config.py .py 1 symbol modified
~ Settings
~ musehub/logging_config.py .py 2 symbols added, 3 symbols modified
+ _environment variable variable _environment L39–39
+ _release_version variable variable _release_version L40–40
~ JsonFormatter
~ format
~ configure_logging
~ musehub/main.py .py
Files Changed
~6
1060 in snapshot
~ deploy/cloudwatch-alerts.sh .sh
~ deploy/deploy.sh .sh
~ docs/production-readiness/12-observability.md .md
~ musehub/config.py .py
~ musehub/logging_config.py .py
~ musehub/main.py .py
← Older Oldest on fix/cloudwatch-alerts-and-log-fields
All commits
Newer → Latest on fix/cloudwatch-alerts-and-log-fields

0 comments

No comments yet. Be the first to start the discussion.

To add a comment, use the Muse CLI: muse hub commit comment sha256:7afc59b2ee9c70d18b748d269602c588385d6e936cc22b056fd775ab9f2fd499 --body "your comment"