MuseHub Cloud Infrastructure

Last updated: 2026-04-08

Overview

MuseHub runs on AWS EC2 (us-east-1) behind nginx with Let's Encrypt TLS. The application stack is Docker Compose: musehub (uvicorn) + postgres:16 + musehub-runner. No managed RDS, no ECS, no load balancer — intentionally minimal for this stage.

Two environments:

Shared AWS Resources

Security group inbound rules:

• TCP 80 — HTTP (Cloudflare IPs only)
• TCP 443 — HTTPS (Cloudflare IPs only)

SSH is intentionally disabled. All remote access is via AWS SSM Session Manager. Instance access requires the musehub-infra AWS credentials (default profile in ~/.aws/credentials).

Production Environment

Instance

Instance ID : i-0855d6efe7fa1a49d
Name        : musehub-prod
Elastic IP  : 98.89.99.211
App dir     : /opt/musehub

Namecheap DNS (musehub.ai)

Stack

nginx (host, ports 80/443)
  └─ proxy_pass → 127.0.0.1:10003
       └─ musehub container (uvicorn, port 10003)
            └─ depends_on → postgres container (port 5432 internal)
  musehub-runner container (polls musehub API for CI jobs)

Volumes

Environment variables (.env on instance at /opt/musehub/.env)

DEBUG=false
DATABASE_URL=postgresql+asyncpg://musehub:<DB_PASSWORD>@postgres:5432/musehub
DB_PASSWORD=<generated at provision time>
CORS_ORIGINS=["https://musehub.ai", "https://www.musehub.ai"]
WEBHOOK_SECRET_KEY=<generated Fernet key>
MUSEHUB_ALLOWED_ORIGINS=["musehub.ai", "www.musehub.ai"]
RUNNER_TOKEN=<generated at provision time>

Nginx config

Final SSL config lives at /etc/nginx/sites-available/musehub on the instance. Reference copy: deploy/nginx-ssl.conf.

Key timeouts:

• /push and /push/objects — 300 s (large repo push serialization)
• Everything else — 60 s

SSL

Let's Encrypt via Certbot. Auto-renews via cron (certbot renew). Certificate lives at /etc/letsencrypt/live/musehub.ai/.

Instance access (SSM — no SSH)

# Open an interactive shell on the prod instance
aws ssm start-session --target i-0855d6efe7fa1a49d --region us-east-1

# Run a one-off command
aws ssm send-command \
  --instance-ids i-0855d6efe7fa1a49d \
  --document-name "AWS-RunShellScript" \
  --parameters 'commands=["sudo docker ps"]' \
  --region us-east-1 \
  --query "Command.CommandId" --output text

Useful commands on the instance

# View running containers
sudo docker compose -C /opt/musehub ps

# Tail app logs
sudo docker compose -C /opt/musehub logs -f musehub

# Tail runner logs
sudo docker compose -C /opt/musehub logs -f musehub-runner

# Restart app only (preserves volumes)
sudo docker compose -C /opt/musehub restart musehub

# Full redeploy (rebuild image, rolling restart)
cd /opt/musehub && sudo docker compose pull && sudo docker compose up -d --build

# Run Alembic migrations manually
sudo docker compose -C /opt/musehub exec musehub alembic upgrade head

# Postgres shell
sudo docker compose -C /opt/musehub exec postgres psql -U musehub -d musehub

Staging Environment

Purpose

Full production mirror with a separate DB, separate object store, and separate domain. Used for smoke tests before every prod deploy. Never exposed to users.

Instance (provisioned by aws-provision-staging.sh)

Instance ID : i-07547cd20bee2dea5
Name        : musehub-staging
Elastic IP  : 23.22.27.39
App dir     : /opt/musehub
Domain      : staging.musehub.ai

Namecheap DNS (musehub.ai, Advanced DNS tab)

Provisioning (one-time, run locally)

# 1. Provision EC2 + EIP
chmod +x deploy/aws-provision-staging.sh
./deploy/aws-provision-staging.sh
# Note the instance ID and Elastic IP printed at the end.

# 2. Add staging.musehub.ai A record on Namecheap (see above).
#    Wait for propagation (~5 min with Automatic TTL):
watch -n 10 "dig staging.musehub.ai +short"

# 3. Bootstrap the instance (installs AWS CLI, verifies ECR access)
bash deploy/bootstrap-instance.sh staging

# 4. Run setup script on the instance via SSM
aws ssm send-command \
  --instance-ids <instance-id> \
  --document-name "AWS-RunShellScript" \
  --parameters 'commands=["chmod +x /opt/musehub/deploy/setup-ec2-staging.sh && /opt/musehub/deploy/setup-ec2-staging.sh"]' \
  --region us-east-1

# 5. Do the first deploy
bash deploy/push.sh staging

Ongoing code deploys to staging

# Standard — builds image locally, pushes to ECR, triggers blue-green on staging
bash deploy/push.sh staging

Instance access (SSM — no SSH)

# Interactive shell on staging
aws ssm start-session --target i-07547cd20bee2dea5 --region us-east-1

Deployment Workflow

Deploys are image-based via ECR. No SSH, no rsync, no code on the instance after provisioning. All deploy commands run from the local ~/ecosystem/musehub directory.

Deploy pipeline overview

Local machine:
  1. docker build (linux/amd64)
  2. docker push → ECR (musehub/musehub:<tag>)
  3. aws ssm send-command → instance runs deploy.sh

Instance (deploy.sh):
  4. aws ecr get-login-password | docker login
  5. docker pull <ecr>:<tag>
  6. docker run (migrations only, then exit)
  7. docker run -d (new slot — blue or green)
  8. curl /healthz until healthy
  9. nginx -s reload (zero-downtime flip)
  10. docker rm (old slot)

Standard deploy

# Deploy to staging (always first)
bash deploy/push.sh staging

# Deploy to prod after staging smoke test
bash deploy/push.sh prod

# Or both in sequence
bash deploy/push.sh staging prod

Rollback

# List recent ECR image tags
aws ecr describe-images \
  --repository-name musehub/musehub \
  --region us-east-1 \
  --query 'sort_by(imageDetails,&imagePushedAt)[-10:].imageTags[0]' \
  --output table

# Redeploy a specific tag (skips build+push)
IMAGE_TAG=<previous-tag> bash deploy/push.sh staging
IMAGE_TAG=<previous-tag> bash deploy/push.sh prod

Emergency migration rollback (on instance via SSM)

aws ssm send-command \
  --instance-ids i-0855d6efe7fa1a49d \
  --document-name "AWS-RunShellScript" \
  --parameters 'commands=["cd /opt/musehub && sudo docker run --rm --network musehub_musehub-internal --env-file .env <ecr-image>:<tag> alembic downgrade -1"]' \
  --region us-east-1

Backups

No automated backup is configured yet. Planned:

• Daily pg_dump compressed to S3 (or a second EBS snapshot)
• Volume snapshot via AWS before every production deploy
• Object store (musehub_data) is content-addressed — safe to snapshot at any time

Until automated backups are set up, take a manual snapshot before every prod deploy:

# On prod instance
sudo docker compose exec postgres pg_dump -U musehub musehub | gzip > ~/musehub-backup-$(date +%Y%m%d).sql.gz

Costs (approximate, us-east-1, 2025 pricing)

To pause staging when not needed:

aws ec2 stop-instances --region us-east-1 --instance-ids <STAGING_INSTANCE_ID>
# Start again with:
aws ec2 start-instances --region us-east-1 --instance-ids <STAGING_INSTANCE_ID>

The Elastic IP stays associated while the instance is stopped — no charge.

Secrets inventory

All secrets live in /opt/musehub/.env on each instance. Never committed to source.

Ed25519 identity keys live in ~/.muse/identity.toml on each client machine. No server-side secret is involved in MSign auth — the public key in the DB is the credential.