# Deploy Runbook Two things live here: deploying the **MuseHub server** and publishing a new **Muse CLI build**. See [versioning.md](versioning.md) for why `pyproject.toml` and muse release tags use two different, both-correct version formats (PEP 440 vs SemVer 2.0), and for the alpha/beta/rc/nightly channel definitions referenced below. --- ## Standard release flow `muse` and `musehub` version **independently** — not lockstep. This mirrors the standard CLI/server pattern (git/GitHub, kubectl/the Kubernetes API server, Terraform CLI/Terraform Cloud): each evolves on its own cadence, and a change confined to one repo doesn't force a version bump (and empty-diff release) on the other. There's no technical coupling forcing otherwise — `install.sh`'s served version comes from scanning uploaded tarballs by mtime (`_latest_published_muse_version()` in `musehub/api/routes/musehub/install.py`), falling back to musehub's own package version only if no tarball has ever been uploaded. `musehub/pyproject.toml`'s version has no bearing on which muse CLI build gets served. Whichever repo(s) actually changed this cycle, use **PEP 440 canonical form** for the bump (e.g. `0.2.0rc16`, `0.2.0.dev1` — no hyphen, no `v` prefix; see [versioning.md](versioning.md)). Bump only what changed, deploy server (if musehub changed), publish CLI (if muse changed): ```bash # 1. Bump version in muse/pyproject.toml and/or musehub/pyproject.toml — # only the repo(s) that actually changed this cycle. # (the bump commit is a simple chore: on a short-lived branch, merged back to dev) # Use PEP 440 canonical form: 0.2.0rc17, 0.2.0.dev1, 0.2.0 — never a hyphen or "v" prefix here. # Nightly counters are sequential, starting at dev1 (never date/timestamp-based — see versioning.md). # 2. If muse changed: push muse source to hub (local + staging) muse -C ~/ecosystem/muse push local dev muse -C ~/ecosystem/muse push staging dev # 3. If musehub changed: deploy the server cd ~/ecosystem/musehub bash deploy/push.sh staging # 4. If muse changed: publish the CLI tarball and smoke-test it # (order relative to step 3 doesn't matter — no version dependency between them) bash deploy/publish_muse_release.sh # Steps: build sdist → upload S3 → SSM copy to staging → prune old → HTTP check → smoke (18 checks). # If smoke fails, the tarball URL is live but broken — investigate before announcing. # 5. If muse changed: verify the installer picks up the new version curl -fsSL https://staging.musehub.ai/install.sh | sh muse --version # should print the new version ``` --- ## 0 — Build frontend assets (required before deploying MuseHub) The Docker image copies compiled assets directly from the working tree. `app.css` and `app.js` are in `.museignore` (not committed), so they must be built locally before running `push.sh`. ```bash cd ~/ecosystem/musehub npm run build # compiles src/scss → app.css and src/ts → app.js ``` Or rebuild only what changed: ```bash npm run build:css # SCSS only (src/scss/app.scss → musehub/templates/musehub/static/app.css) npm run build:js # TS only (src/ts/app.ts → musehub/templates/musehub/static/app.js) ``` **Source files to edit** (never edit the compiled output directly): | Asset | Source | |-------|--------| | `static/app.css` | `src/scss/` — entry point `app.scss`, partials `_*.scss` | | `static/app.js` | `src/ts/` — entry point `app.ts` | | `static/embed.css` | `src/scss/embed.scss` | | `static/embed-player.js` | `src/ts/embed-player.ts` | During local development, use the watch commands to rebuild automatically on save: ```bash npm run watch:css # rebuilds app.css on every SCSS change npm run watch:js # rebuilds app.js on every TS change ``` --- ## 1 — Deploy MuseHub (server) ```bash cd ~/ecosystem/musehub bash deploy/push.sh staging # build, push to ECR, blue-green deploy bash deploy/push.sh prod # same for prod bash deploy/push.sh staging prod # staging then prod in sequence IMAGE_TAG= bash deploy/push.sh staging # rollback to a prior image ``` `push.sh` does a full blue-green deploy: builds a Docker image for `linux/amd64`, pushes it to ECR, starts the inactive slot, health-checks `/healthz`, switches nginx, and stops the old slot. The image tag is `-`. For full infrastructure details (instance IDs, slot switching, SSM recovery) see [infrastructure.md](infrastructure.md). --- ## 2 — Publish a new Muse CLI build ```bash cd ~/ecosystem/musehub bash deploy/publish_muse_release.sh ``` Must be run after step 1 (server deploy). Reads the version from `~/ecosystem/muse/pyproject.toml` automatically. **What it does:** 1. Builds `muse-.tar.gz` from `~/ecosystem/muse` using `python3 -m build` 2. Uploads the tarball to `s3://musehub-releases/` 3. SSMs into staging and copies it to `/data/releases/` on the instance volume 4. Prunes S3 and the instance to keep only the 3 most recent tarballs 5. Smoke-tests `https://staging.musehub.ai/releases/muse-.tar.gz` — exits non-zero on failure After a successful run, the new version is live and anyone running the installer gets it: ```bash curl -fsSL https://staging.musehub.ai/install.sh | sh ``` ### Prerequisites - Python 3.14 + `build` package: `pip install build` - AWS CLI configured with credentials that can write to `s3://musehub-releases` and send SSM commands to `i-07547cd20bee2dea5` ### Override version label ```bash MUSE_VERSION=0.2.1 bash deploy/publish_muse_release.sh ``` Useful if you need to re-publish a tarball under a different version label without editing `pyproject.toml`. --- ## 3 — Smoke test the installed CLI `deploy/smoke_muse.sh` runs 18 checks against the installed binary. It is called automatically at the end of `publish_muse_release.sh`, but can be run standalone to verify any staging or prod build: ```bash cd ~/ecosystem/musehub # Default: staging, version read from ~/ecosystem/muse/pyproject.toml bash deploy/smoke_muse.sh # Explicit URL (prod) bash deploy/smoke_muse.sh --url https://musehub.ai # Explicit version (re-smoke a prior build) bash deploy/smoke_muse.sh --version 0.2.0rc13 ``` Exit code 0 = all 18 checks passed. Exit code 1 = one or more checks failed. Exit code 2 = setup error (tarball not found, venv failed). The checks cover: `version`, `init`, `add`, `commit`, `status`, `log`, `read`, `ls-files`, `branch`, `diff`, `checkout -b`, `checkout`, `branch -d`, `tag add`, `tag list`, `verify`. --- ## 4 — Push benchmarking (manual, not a release gate) `tools/bench_push.py` creates real repos on a hub, builds N commits of random content, times the first push and the re-push (all objects already present), then cleans up. Use it to verify push performance after changes to the wire protocol or to reproduce timeout issues. ```bash cd ~/ecosystem/musehub # Sizes: xs=1c/5f/512B s=10c/10f/1KB m=100c/20f/2KB l=500c/50f/4KB xl=1000c/100f/8KB python tools/bench_push.py xs # local hub python tools/bench_push.py --hub https://staging.musehub.ai xs s m ``` **This is not a release gate.** It creates real `gabriel/bench-*` repos on the hub — clean them up afterwards if needed.