Deploy Runbook

Two things live here: deploying the MuseHub server and publishing a new Muse CLI build.

See versioning.md for why pyproject.toml and muse release tags use two different, both-correct version formats (PEP 440 vs SemVer 2.0), and for the alpha/beta/rc/nightly channel definitions referenced below.

Standard release flow

muse and musehub version independently — not lockstep. This mirrors the standard CLI/server pattern (git/GitHub, kubectl/the Kubernetes API server, Terraform CLI/Terraform Cloud): each evolves on its own cadence, and a change confined to one repo doesn't force a version bump (and empty-diff release) on the other. There's no technical coupling forcing otherwise — install.sh's served version comes from scanning uploaded tarballs by mtime (_latest_published_muse_version() in musehub/api/routes/musehub/install.py), falling back to musehub's own package version only if no tarball has ever been uploaded. musehub/pyproject.toml's version has no bearing on which muse CLI build gets served.

Whichever repo(s) actually changed this cycle, use PEP 440 canonical form for the bump (e.g. 0.2.0rc16, 0.2.0.dev1 — no hyphen, no v prefix; see versioning.md). Bump only what changed, deploy server (if musehub changed), publish CLI (if muse changed):

# 1. Bump version in muse/pyproject.toml and/or musehub/pyproject.toml —
#    only the repo(s) that actually changed this cycle.
#    (the bump commit is a simple chore: on a short-lived branch, merged back to dev)
#    Use PEP 440 canonical form: 0.2.0rc17, 0.2.0.dev1, 0.2.0 — never a hyphen or "v" prefix here.
#    Nightly counters are sequential, starting at dev1 (never date/timestamp-based — see versioning.md).

# 2. If muse changed: push muse source to hub (local + staging)
muse -C ~/ecosystem/muse push local dev
muse -C ~/ecosystem/muse push staging dev

# 3. If musehub changed: deploy the server
cd ~/ecosystem/musehub
bash deploy/push.sh staging

# 4. If muse changed: publish the CLI tarball and smoke-test it
#    (order relative to step 3 doesn't matter — no version dependency between them)
bash deploy/publish_muse_release.sh
#    Steps: build sdist → upload S3 → SSM copy to staging → prune old → HTTP check → smoke (18 checks).
#    If smoke fails, the tarball URL is live but broken — investigate before announcing.

# 5. If muse changed: verify the installer picks up the new version
curl -fsSL https://staging.musehub.ai/install.sh | sh
muse --version   # should print the new version

0 — Build frontend assets (required before deploying MuseHub)

The Docker image copies compiled assets directly from the working tree. app.css and app.js are in .museignore (not committed), so they must be built locally before running push.sh.

cd ~/ecosystem/musehub
npm run build        # compiles src/scss → app.css and src/ts → app.js

Or rebuild only what changed:

npm run build:css    # SCSS only  (src/scss/app.scss → musehub/templates/musehub/static/app.css)
npm run build:js     # TS only    (src/ts/app.ts → musehub/templates/musehub/static/app.js)

Source files to edit (never edit the compiled output directly):

During local development, use the watch commands to rebuild automatically on save:

npm run watch:css    # rebuilds app.css on every SCSS change
npm run watch:js     # rebuilds app.js on every TS change

1 — Deploy MuseHub (server)

cd ~/ecosystem/musehub
bash deploy/push.sh staging          # build, push to ECR, blue-green deploy
bash deploy/push.sh prod             # same for prod
bash deploy/push.sh staging prod     # staging then prod in sequence
IMAGE_TAG=<prev-tag> bash deploy/push.sh staging   # rollback to a prior image

push.sh does a full blue-green deploy: builds a Docker image for linux/amd64, pushes it to ECR, starts the inactive slot, health-checks /healthz, switches nginx, and stops the old slot. The image tag is <short-commit-id>-<YYYYMMDDHHMMSS>.

For full infrastructure details (instance IDs, slot switching, SSM recovery) see infrastructure.md.

2 — Publish a new Muse CLI build

cd ~/ecosystem/musehub
bash deploy/publish_muse_release.sh

Must be run after step 1 (server deploy). Reads the version from ~/ecosystem/muse/pyproject.toml automatically.

What it does:

1. Builds muse-<version>.tar.gz from ~/ecosystem/muse using python3 -m build
2. Uploads the tarball to s3://musehub-releases/
3. SSMs into staging and copies it to /data/releases/ on the instance volume
4. Prunes S3 and the instance to keep only the 3 most recent tarballs
5. Smoke-tests https://staging.musehub.ai/releases/muse-<version>.tar.gz — exits non-zero on failure

After a successful run, the new version is live and anyone running the installer gets it:

curl -fsSL https://staging.musehub.ai/install.sh | sh

Prerequisites

• Python 3.14 + build package: pip install build
• AWS CLI configured with credentials that can write to s3://musehub-releases and send SSM commands to i-07547cd20bee2dea5

Override version label

MUSE_VERSION=0.2.1 bash deploy/publish_muse_release.sh

Useful if you need to re-publish a tarball under a different version label without editing pyproject.toml.

3 — Smoke test the installed CLI

deploy/smoke_muse.sh runs 18 checks against the installed binary. It is called automatically at the end of publish_muse_release.sh, but can be run standalone to verify any staging or prod build:

cd ~/ecosystem/musehub

# Default: staging, version read from ~/ecosystem/muse/pyproject.toml
bash deploy/smoke_muse.sh

# Explicit URL (prod)
bash deploy/smoke_muse.sh --url https://musehub.ai

# Explicit version (re-smoke a prior build)
bash deploy/smoke_muse.sh --version 0.2.0rc13

Exit code 0 = all 18 checks passed. Exit code 1 = one or more checks failed. Exit code 2 = setup error (tarball not found, venv failed).

The checks cover: version, init, add, commit, status, log, read, ls-files, branch, diff, checkout -b, checkout, branch -d, tag add, tag list, verify.

4 — Push benchmarking (manual, not a release gate)

tools/bench_push.py creates real repos on a hub, builds N commits of random content, times the first push and the re-push (all objects already present), then cleans up. Use it to verify push performance after changes to the wire protocol or to reproduce timeout issues.

cd ~/ecosystem/musehub

# Sizes: xs=1c/5f/512B  s=10c/10f/1KB  m=100c/20f/2KB  l=500c/50f/4KB  xl=1000c/100f/8KB
python tools/bench_push.py xs                                # local hub
python tools/bench_push.py --hub https://staging.musehub.ai xs s m

This is not a release gate. It creates real gabriel/bench-* repos on the hub — clean them up afterwards if needed.