Deploy Runbook
Two things live here: deploying the MuseHub server and publishing a new Muse CLI build.
See versioning.md for why pyproject.toml and muse release
tags use two different, both-correct version formats (PEP 440 vs SemVer 2.0),
and for the alpha/beta/rc/nightly channel definitions referenced below.
Standard release flow
muse and musehub version independently — not lockstep. This mirrors
the standard CLI/server pattern (git/GitHub, kubectl/the Kubernetes API
server, Terraform CLI/Terraform Cloud): each evolves on its own cadence, and
a change confined to one repo doesn't force a version bump (and empty-diff
release) on the other. There's no technical coupling forcing otherwise —
install.sh's served version comes from scanning uploaded tarballs by mtime
(_latest_published_muse_version() in musehub/api/routes/musehub/install.py),
falling back to musehub's own package version only if no tarball has ever
been uploaded. musehub/pyproject.toml's version has no bearing on which
muse CLI build gets served.
Whichever repo(s) actually changed this cycle, use PEP 440 canonical
form for the bump (e.g. 0.2.0rc16, 0.2.0.dev1 — no hyphen, no v
prefix; see versioning.md). Bump only what changed, deploy
server (if musehub changed), publish CLI (if muse changed):
# 1. Bump version in muse/pyproject.toml and/or musehub/pyproject.toml —
# only the repo(s) that actually changed this cycle.
# (the bump commit is a simple chore: on a short-lived branch, merged back to dev)
# Use PEP 440 canonical form: 0.2.0rc17, 0.2.0.dev1, 0.2.0 — never a hyphen or "v" prefix here.
# Nightly counters are sequential, starting at dev1 (never date/timestamp-based — see versioning.md).
# 2. If muse changed: push muse source to hub (local + staging)
muse -C ~/ecosystem/muse push local dev
muse -C ~/ecosystem/muse push staging dev
# 3. If musehub changed: deploy the server
cd ~/ecosystem/musehub
bash deploy/push.sh staging
# 4. If muse changed: build, validate, publish, then activate the CLI tarball
# (order relative to step 3 doesn't matter — no version dependency between them).
# Publishing and activating are separate; see docs/release-runbook.md.
# python3 -m build --sdist (in ~/ecosystem/muse)
# python -m deploy.release_gate run ... (writes the gate report)
# MUSE_ARTIFACT=... MUSE_GATE_REPORT=... bash deploy/publish_muse_release.sh staging
# bash deploy/activate_muse_release.sh staging <version> <sha256> (runs the smoke checks)
# 5. If muse changed: verify the installer picks up the new version
curl -fsSL https://staging.musehub.ai/install.sh | sh
muse --version # should print the new version
0 — Build frontend assets (required before deploying MuseHub)
The Docker image copies compiled assets directly from the working tree. app.css and app.js are in .museignore (not committed), so they must be built locally before running push.sh.
cd ~/ecosystem/musehub
npm run build # compiles src/scss → app.css and src/ts → app.js
Or rebuild only what changed:
npm run build:css # SCSS only (src/scss/app.scss → musehub/templates/musehub/static/app.css)
npm run build:js # TS only (src/ts/app.ts → musehub/templates/musehub/static/app.js)
Source files to edit (never edit the compiled output directly):
| Asset | Source |
|---|---|
static/app.css |
src/scss/ — entry point app.scss, partials _*.scss |
static/app.js |
src/ts/ — entry point app.ts |
static/embed.css |
src/scss/embed.scss |
static/embed-player.js |
src/ts/embed-player.ts |
During local development, use the watch commands to rebuild automatically on save:
npm run watch:css # rebuilds app.css on every SCSS change
npm run watch:js # rebuilds app.js on every TS change
1 — Deploy MuseHub (server)
cd ~/ecosystem/musehub
bash deploy/push.sh staging # build, push to ECR, blue-green deploy
bash deploy/push.sh prod # same for prod
bash deploy/push.sh staging prod # staging then prod in sequence
IMAGE_TAG=<prev-tag> bash deploy/push.sh staging # rollback to a prior image
push.sh does a full blue-green deploy: builds a Docker image for linux/amd64, pushes it to ECR, starts the inactive slot, health-checks /healthz, switches nginx, and stops the old slot. The image tag is <short-commit-id>-<YYYYMMDDHHMMSS>.
For full infrastructure details (instance IDs, slot switching, SSM recovery) see infrastructure.md.
2 — Publish and activate a new Muse CLI build
The full procedure, authorization rules, rollback, retention and recovery are in release-runbook.md. In short:
cd ~/ecosystem/musehub
# build + validate (local, isolated): produces dist/muse-<ver>.tar.gz and a gate report bound to its SHA-256
python -m deploy.release_gate run --artifact ~/ecosystem/muse/dist/muse-<ver>.tar.gz \
--source-revision "$(muse -C ~/ecosystem/muse rev-parse HEAD)" --out gate-report.json
# publish: verifies the report first; create-only; never deletes; does not change what is served
MUSE_ARTIFACT=~/ecosystem/muse/dist/muse-<ver>.tar.gz MUSE_GATE_REPORT=gate-report.json \
bash deploy/publish_muse_release.sh staging
# activate: moves staging's pointer (active.json) and smoke-tests what is served
bash deploy/activate_muse_release.sh staging <ver> <sha256>
install.sh serves the version named by the instance's /data/releases/active.json (explicit version + SHA-256),
never "the newest file". Production is changed only by APPROVE_PRODUCTION=1 bash deploy/activate_muse_release.sh production …
and only to what staging currently serves. Old releases are removed only by release_safety cleanup, which is a dry run
unless given --execute --confirm <digest>.
Prerequisites
- Python 3.14 +
buildpackage:pip install build - AWS credentials that can write to
s3://musehub-releasesand send SSM commands to the target instance
3 — Smoke test the installed CLI
deploy/smoke_muse.sh runs 18 checks against the installed binary. It is called automatically at the end of activate_muse_release.sh, but can be run standalone to verify any staging or prod build:
cd ~/ecosystem/musehub
# Default: staging, version read from ~/ecosystem/muse/pyproject.toml
bash deploy/smoke_muse.sh
# Explicit URL (prod)
bash deploy/smoke_muse.sh --url https://musehub.ai
# Explicit version (re-smoke a prior build)
bash deploy/smoke_muse.sh --version 0.2.0rc13
Exit code 0 = all 18 checks passed. Exit code 1 = one or more checks failed. Exit code 2 = setup error (tarball not found, venv failed).
The checks cover: version, init, add, commit, status, log, read, ls-files, branch, diff, checkout -b, checkout, branch -d, tag add, tag list, verify.
4 — Push benchmarking (manual, not a release gate)
tools/bench_push.py creates real repos on a hub, builds N commits of random content, times the first push and the re-push (all objects already present), then cleans up. Use it to verify push performance after changes to the wire protocol or to reproduce timeout issues.
cd ~/ecosystem/musehub
# Sizes: xs=1c/5f/512B s=10c/10f/1KB m=100c/20f/2KB l=500c/50f/4KB xl=1000c/100f/8KB
python tools/bench_push.py xs # local hub
python tools/bench_push.py --hub https://staging.musehub.ai xs s m
This is not a release gate. It creates real gabriel/bench-* repos on the hub — clean them up afterwards if needed.