# MuseHub Cloud Infrastructure

> Last updated: 2026-04-08

---

## Overview

MuseHub runs on AWS EC2 (us-east-1) behind nginx with Let's Encrypt TLS.
The application stack is Docker Compose: musehub (uvicorn) + postgres:16 + musehub-runner.
No managed RDS, no ECS, no load balancer — intentionally minimal for this stage.

Two environments:

| Environment | Domain | Instance | Elastic IP |
|-------------|--------|----------|------------|
| Production  | `musehub.ai` | `i-0855d6efe7fa1a49d` (`musehub-prod`) | `98.89.99.211` |
| Staging     | `staging.musehub.ai` | `i-07547cd20bee2dea5` (`musehub-staging`) | `23.22.27.39` |

---

## Shared AWS Resources

| Resource | Value |
|----------|-------|
| Region | `us-east-1` |
| AMI | `ami-0c7217cdde317cfec` (Ubuntu 22.04 LTS) |
| Instance type | `t3.small` |
| Security group | `sg-05815872537fcfe76` (`musehub-sg`) |
| ECR registry | `992382692655.dkr.ecr.us-east-1.amazonaws.com` |
| ECR repository | `musehub/musehub` |
| IAM deploy user | `musehub-infra` (ECR push + SSM send) |
| IAM instance role | `musehub-ec2-ssm` (ECR pull + SSM receive) |

Security group inbound rules:
- TCP 80 — HTTP (Cloudflare IPs only)
- TCP 443 — HTTPS (Cloudflare IPs only)

SSH is intentionally disabled. All remote access is via AWS SSM Session Manager.
Instance access requires the `musehub-infra` AWS credentials (default profile in `~/.aws/credentials`).

---

## Production Environment

### Instance

```
Instance ID : i-0855d6efe7fa1a49d
Name        : musehub-prod
Elastic IP  : 98.89.99.211
App dir     : /opt/musehub
```

### Namecheap DNS (musehub.ai)

| Type | Host | Value | TTL |
|------|------|-------|-----|
| A Record | @ | 98.89.99.211 | Automatic |
| A Record | www | 98.89.99.211 | Automatic |

### Stack

```
nginx (host, ports 80/443)
  └─ proxy_pass → 127.0.0.1:10003
       └─ musehub container (uvicorn, port 10003)
            └─ depends_on → postgres container (port 5432 internal)
  musehub-runner container (polls musehub API for CI jobs)
```

### Volumes

| Volume | Contents |
|--------|----------|
| `musehub_data` | Object store — all pushed repo objects |
| `postgres_data` | PostgreSQL data directory |
| `runner_workspace` | CI job working directories |

### Environment variables (.env on instance at /opt/musehub/.env)

```
DEBUG=false
DATABASE_URL=postgresql+asyncpg://musehub:<DB_PASSWORD>@postgres:5432/musehub
DB_PASSWORD=<generated at provision time>
CORS_ORIGINS=["https://musehub.ai", "https://www.musehub.ai"]
WEBHOOK_SECRET_KEY=<generated Fernet key>
MUSEHUB_ALLOWED_ORIGINS=["musehub.ai", "www.musehub.ai"]
RUNNER_TOKEN=<generated at provision time>
```

### Nginx config

Final SSL config lives at `/etc/nginx/sites-available/musehub` on the instance.
Reference copy: `deploy/nginx-ssl.conf`.

Key timeouts:
- `/push` and `/push/objects` — 300 s (large repo push serialization)
- Everything else — 60 s

### SSL

Let's Encrypt via Certbot. Auto-renews via cron (`certbot renew`).
Certificate lives at `/etc/letsencrypt/live/musehub.ai/`.

### Instance access (SSM — no SSH)

```bash
# Open an interactive shell on the prod instance
aws ssm start-session --target i-0855d6efe7fa1a49d --region us-east-1

# Run a one-off command
aws ssm send-command \
  --instance-ids i-0855d6efe7fa1a49d \
  --document-name "AWS-RunShellScript" \
  --parameters 'commands=["sudo docker ps"]' \
  --region us-east-1 \
  --query "Command.CommandId" --output text
```

### Useful commands on the instance

```bash
# View running containers
sudo docker compose -C /opt/musehub ps

# Tail app logs
sudo docker compose -C /opt/musehub logs -f musehub

# Tail runner logs
sudo docker compose -C /opt/musehub logs -f musehub-runner

# Restart app only (preserves volumes)
sudo docker compose -C /opt/musehub restart musehub

# Full redeploy (rebuild image, rolling restart)
cd /opt/musehub && sudo docker compose pull && sudo docker compose up -d --build

# Run Alembic migrations manually
sudo docker compose -C /opt/musehub exec musehub alembic upgrade head

# Postgres shell
sudo docker compose -C /opt/musehub exec postgres psql -U musehub -d musehub
```

---

## Staging Environment

### Purpose

Full production mirror with a separate DB, separate object store, and separate domain.
Used for smoke tests before every prod deploy. Never exposed to users.

### Instance (provisioned by aws-provision-staging.sh)

```
Instance ID : i-07547cd20bee2dea5
Name        : musehub-staging
Elastic IP  : 23.22.27.39
App dir     : /opt/musehub
Domain      : staging.musehub.ai
```

### Namecheap DNS (musehub.ai, Advanced DNS tab)

| Type | Host | Value | TTL |
|------|------|-------|-----|
| A Record | staging | `23.22.27.39` | Automatic |

### Provisioning (one-time, run locally)

```bash
# 1. Provision EC2 + EIP
chmod +x deploy/aws-provision-staging.sh
./deploy/aws-provision-staging.sh
# Note the instance ID and Elastic IP printed at the end.

# 2. Add staging.musehub.ai A record on Namecheap (see above).
#    Wait for propagation (~5 min with Automatic TTL):
watch -n 10 "dig staging.musehub.ai +short"

# 3. Bootstrap the instance (installs AWS CLI, verifies ECR access)
bash deploy/bootstrap-instance.sh staging

# 4. Run setup script on the instance via SSM
aws ssm send-command \
  --instance-ids <instance-id> \
  --document-name "AWS-RunShellScript" \
  --parameters 'commands=["chmod +x /opt/musehub/deploy/setup-ec2-staging.sh && /opt/musehub/deploy/setup-ec2-staging.sh"]' \
  --region us-east-1

# 5. Do the first deploy
bash deploy/push.sh staging
```

### Ongoing code deploys to staging

```bash
# Standard — builds image locally, pushes to ECR, triggers blue-green on staging
bash deploy/push.sh staging
```

### Instance access (SSM — no SSH)

```bash
# Interactive shell on staging
aws ssm start-session --target i-07547cd20bee2dea5 --region us-east-1
```

---

## Deployment Workflow

Deploys are image-based via ECR. No SSH, no rsync, no code on the instance after provisioning.
All deploy commands run from the local `~/ecosystem/musehub` directory.

### Deploy pipeline overview

```
Local machine:
  1. docker build (linux/amd64)
  2. docker push → ECR (musehub/musehub:<tag>)
  3. aws ssm send-command → instance runs deploy.sh

Instance (deploy.sh):
  4. aws ecr get-login-password | docker login
  5. docker pull <ecr>:<tag>
  6. docker run (migrations only, then exit)
  7. docker run -d (new slot — blue or green)
  8. curl /healthz until healthy
  9. nginx -s reload (zero-downtime flip)
  10. docker rm (old slot)
```

### Standard deploy

```bash
# Deploy to staging (always first)
bash deploy/push.sh staging

# Deploy to prod after staging smoke test
bash deploy/push.sh prod

# Or both in sequence
bash deploy/push.sh staging prod
```

### Rollback

```bash
# List recent ECR image tags
aws ecr describe-images \
  --repository-name musehub/musehub \
  --region us-east-1 \
  --query 'sort_by(imageDetails,&imagePushedAt)[-10:].imageTags[0]' \
  --output table

# Redeploy a specific tag (skips build+push)
IMAGE_TAG=<previous-tag> bash deploy/push.sh staging
IMAGE_TAG=<previous-tag> bash deploy/push.sh prod
```

### Emergency migration rollback (on instance via SSM)

```bash
aws ssm send-command \
  --instance-ids i-0855d6efe7fa1a49d \
  --document-name "AWS-RunShellScript" \
  --parameters 'commands=["cd /opt/musehub && sudo docker run --rm --network musehub_musehub-internal --env-file .env <ecr-image>:<tag> alembic downgrade -1"]' \
  --region us-east-1
```

---

## Backups

No automated backup is configured yet. Planned:

- Daily `pg_dump` compressed to S3 (or a second EBS snapshot)
- Volume snapshot via AWS before every production deploy
- Object store (`musehub_data`) is content-addressed — safe to snapshot at any time

Until automated backups are set up, take a manual snapshot before every prod deploy:

```bash
# On prod instance
sudo docker compose exec postgres pg_dump -U musehub musehub | gzip > ~/musehub-backup-$(date +%Y%m%d).sql.gz
```

---

## Costs (approximate, us-east-1, 2025 pricing)

| Item | $/month |
|------|---------|
| t3.small (prod) | ~$15 |
| t3.small (staging) | ~$15 (stop when not in use to reduce cost) |
| Elastic IPs (2) | ~$0 while associated, $3.60/mo each if unassociated |
| EBS gp3 20 GB (each) | ~$1.60 |
| **Total (both running)** | **~$35/mo** |

To pause staging when not needed:

```bash
aws ec2 stop-instances --region us-east-1 --instance-ids <STAGING_INSTANCE_ID>
# Start again with:
aws ec2 start-instances --region us-east-1 --instance-ids <STAGING_INSTANCE_ID>
```

The Elastic IP stays associated while the instance is stopped — no charge.

---

## Secrets inventory

All secrets live in `/opt/musehub/.env` on each instance. Never committed to source.

| Secret | How generated | Rotation |
|--------|---------------|----------|
| `DB_PASSWORD` | `openssl rand -hex 16` | Manual, on compromise |
| `WEBHOOK_SECRET_KEY` | Fernet key | Manual, on compromise |
| `RUNNER_TOKEN` | `openssl rand -hex 32` | Manual, on compromise |

Ed25519 identity keys live in `~/.muse/identity.toml` on each client machine.
No server-side secret is involved in MSign auth — the public key in the DB is the credential.