gabriel / musehub public
deploy.md markdown
173 lines 7.6 KB Hotspot Cold
Raw
sha256:db5120371e0ff3edd7a3920ffa9f31fe81583e4462228510f6db188972f0a31c deploy: memory headroom gate is 1200 MB in staging and production Sonnet 5.5 minor ⚠ breaking 4 days ago

Deploy Runbook

Two things live here: deploying the MuseHub server and publishing a new Muse CLI build.

See versioning.md for why pyproject.toml and muse release tags use two different, both-correct version formats (PEP 440 vs SemVer 2.0), and for the alpha/beta/rc/nightly channel definitions referenced below.


Standard release flow

muse and musehub version independently — not lockstep. This mirrors the standard CLI/server pattern (git/GitHub, kubectl/the Kubernetes API server, Terraform CLI/Terraform Cloud): each evolves on its own cadence, and a change confined to one repo doesn't force a version bump (and empty-diff release) on the other. There's no technical coupling forcing otherwise — install.sh's served version comes from scanning uploaded tarballs by mtime (_latest_published_muse_version() in musehub/api/routes/musehub/install.py), falling back to musehub's own package version only if no tarball has ever been uploaded. musehub/pyproject.toml's version has no bearing on which muse CLI build gets served.

Whichever repo(s) actually changed this cycle, use PEP 440 canonical form for the bump (e.g. 0.2.0rc16, 0.2.0.dev1 — no hyphen, no v prefix; see versioning.md). Bump only what changed, deploy server (if musehub changed), publish CLI (if muse changed):

# 1. Bump version in muse/pyproject.toml and/or musehub/pyproject.toml —
#    only the repo(s) that actually changed this cycle.
#    (the bump commit is a simple chore: on a short-lived branch, merged back to dev)
#    Use PEP 440 canonical form: 0.2.0rc17, 0.2.0.dev1, 0.2.0 — never a hyphen or "v" prefix here.
#    Nightly counters are sequential, starting at dev1 (never date/timestamp-based — see versioning.md).

# 2. If muse changed: push muse source to hub (local + staging)
muse -C ~/ecosystem/muse push local dev
muse -C ~/ecosystem/muse push staging dev

# 3. If musehub changed: deploy the server
cd ~/ecosystem/musehub
bash deploy/push.sh staging

# 4. If muse changed: build, validate, publish, then activate the CLI tarball
#    (order relative to step 3 doesn't matter — no version dependency between them).
#    Publishing and activating are separate; see docs/release-runbook.md.
#      python3 -m build --sdist                       (in ~/ecosystem/muse)
#      python -m deploy.release_gate run ...          (writes the gate report)
#      MUSE_ARTIFACT=... MUSE_GATE_REPORT=... bash deploy/publish_muse_release.sh staging
#      bash deploy/activate_muse_release.sh staging <version> <sha256>   (runs the smoke checks)

# 5. If muse changed: verify the installer picks up the new version
curl -fsSL https://staging.musehub.ai/install.sh | sh
muse --version   # should print the new version

0 — Build frontend assets (required before deploying MuseHub)

The Docker image copies compiled assets directly from the working tree. app.css and app.js are in .museignore (not committed), so they must be built locally before running push.sh.

cd ~/ecosystem/musehub
npm run build        # compiles src/scss → app.css and src/ts → app.js

Or rebuild only what changed:

npm run build:css    # SCSS only  (src/scss/app.scss → musehub/templates/musehub/static/app.css)
npm run build:js     # TS only    (src/ts/app.ts → musehub/templates/musehub/static/app.js)

Source files to edit (never edit the compiled output directly):

Asset Source
static/app.css src/scss/ — entry point app.scss, partials _*.scss
static/app.js src/ts/ — entry point app.ts
static/embed.css src/scss/embed.scss
static/embed-player.js src/ts/embed-player.ts

During local development, use the watch commands to rebuild automatically on save:

npm run watch:css    # rebuilds app.css on every SCSS change
npm run watch:js     # rebuilds app.js on every TS change

1 — Deploy MuseHub (server)

cd ~/ecosystem/musehub
bash deploy/push.sh staging          # build, push to ECR, blue-green deploy
bash deploy/push.sh prod             # same for prod
bash deploy/push.sh staging prod     # staging then prod in sequence
IMAGE_TAG=<prev-tag> bash deploy/push.sh staging   # rollback to a prior image

push.sh does a full blue-green deploy: builds a Docker image for linux/amd64, pushes it to ECR, starts the inactive slot, health-checks /healthz, switches nginx, and stops the old slot. The image tag is <short-commit-id>-<YYYYMMDDHHMMSS>.

For full infrastructure details (instance IDs, slot switching, SSM recovery) see infrastructure.md.


2 — Publish and activate a new Muse CLI build

The full procedure, authorization rules, rollback, retention and recovery are in release-runbook.md. In short:

cd ~/ecosystem/musehub
# build + validate (local, isolated): produces dist/muse-<ver>.tar.gz and a gate report bound to its SHA-256
python -m deploy.release_gate run --artifact ~/ecosystem/muse/dist/muse-<ver>.tar.gz \
    --source-revision "$(muse -C ~/ecosystem/muse rev-parse HEAD)" --out gate-report.json
# publish: verifies the report first; create-only; never deletes; does not change what is served
MUSE_ARTIFACT=~/ecosystem/muse/dist/muse-<ver>.tar.gz MUSE_GATE_REPORT=gate-report.json \
    bash deploy/publish_muse_release.sh staging
# activate: moves staging's pointer (active.json) and smoke-tests what is served
bash deploy/activate_muse_release.sh staging <ver> <sha256>

install.sh serves the version named by the instance's /data/releases/active.json (explicit version + SHA-256), never "the newest file". Production is changed only by APPROVE_PRODUCTION=1 bash deploy/activate_muse_release.sh production … and only to what staging currently serves. Old releases are removed only by release_safety cleanup, which is a dry run unless given --execute --confirm <digest>.

Prerequisites

  • Python 3.14 + build package: pip install build
  • AWS credentials that can write to s3://musehub-releases and send SSM commands to the target instance

3 — Smoke test the installed CLI

deploy/smoke_muse.sh runs 18 checks against the installed binary. It is called automatically at the end of activate_muse_release.sh, but can be run standalone to verify any staging or prod build:

cd ~/ecosystem/musehub

# Default: staging, version read from ~/ecosystem/muse/pyproject.toml
bash deploy/smoke_muse.sh

# Explicit URL (prod)
bash deploy/smoke_muse.sh --url https://musehub.ai

# Explicit version (re-smoke a prior build)
bash deploy/smoke_muse.sh --version 0.2.0rc13

Exit code 0 = all 18 checks passed. Exit code 1 = one or more checks failed. Exit code 2 = setup error (tarball not found, venv failed).

The checks cover: version, init, add, commit, status, log, read, ls-files, branch, diff, checkout -b, checkout, branch -d, tag add, tag list, verify.


4 — Push benchmarking (manual, not a release gate)

tools/bench_push.py creates real repos on a hub, builds N commits of random content, times the first push and the re-push (all objects already present), then cleans up. Use it to verify push performance after changes to the wire protocol or to reproduce timeout issues.

cd ~/ecosystem/musehub

# Sizes: xs=1c/5f/512B  s=10c/10f/1KB  m=100c/20f/2KB  l=500c/50f/4KB  xl=1000c/100f/8KB
python tools/bench_push.py xs                                # local hub
python tools/bench_push.py --hub https://staging.musehub.ai xs s m

This is not a release gate. It creates real gabriel/bench-* repos on the hub — clean them up afterwards if needed.

File History 11 commits
sha256:db5120371e0ff3edd7a3920ffa9f31fe81583e4462228510f6db188972f0a31c deploy: memory headroom gate is 1200 MB in staging and production Sonnet 5.5 minor ⚠ 4 days ago
sha256:b8818aebe51ac130d097273b5277ea0d2b2c8b795a6de123247dc4f3d0e38820 release: one-GO batch release flow; fail-closed memory head… Sonnet 5.5 patch 4 days ago
sha256:28b9e16197029996ce556a27058a9faa4eb27e088440726c5d8aaa8a811a4a64 docs(release): runbook separating build/validate/publish/ac… Opus 5.5 patch 12 days ago
sha256:d62b94b7068530fe28ffa1e09ebf27139d0461129490c9b01ae71b4467f51c51 docs: decouple muse and musehub version bumps from the rele… Sonnet 5 59 days ago
sha256:69ae50e537d929ac0661cd4da7f3e58719b13f80a1f799b072e874cd5ab6383e docs: lock in sequential auto-increment for nightly numbering Sonnet 4.6 136 days ago
sha256:43b5ad9422686d7119910384e280e85fcf78250346c551eea3b6977d2052880c docs: add versioning.md — PEP 440 vs SemVer 2.0, build chan… Sonnet 4.6 136 days ago
sha256:430d2e61b9caf34783abdf4f2907c841d6d64e6117fec1ad12704a03f127c05d fix: wire smoke_muse.sh into publish_muse_release.sh as step 6 Sonnet 4.6 minor ⚠ 150 days ago
sha256:e6ef23228f7ff2b0d0e4c67e92b2544dd03a742c42fd37454bc08fd9331c9bba chore: rename smoke_push.py → bench_push.py, document smoke… Sonnet 4.6 minor ⚠ 150 days ago
sha256:6bef8d08607d381507857f64f8f24bb6ea935aaa9eb579b014f8ee03477c50d9 docs: comprehensive deploy runbook in deploy.md + agent con… Sonnet 4.6 150 days ago
sha256:cc8a1fca8238f7b766bda367493a449e07ede92d5521fdcd1f817f579b78c1c9 feat: activity canvas — real domain data, purple-pink palet… Sonnet 4.6 minor ⚠ 198 days ago
sha256:6db26c24c4205041b2348bb26fcee0d28ee6030ec26dffe5f743830a7b54d973 docs: add deploy runbook for MuseHub server and Muse CLI pu… Sonnet 4.6 198 days ago