Muse release runbook — build, validate, publish, activate, promote, roll back, delete
MuseHub issue: muse#114 (Phase 6). Audience: whoever ships a Muse CLI release.
These seven operations are different. None implies another. Each has its own command, its own authorization, and its own failure mode. The incident that created this document: a publish to staging silently also deleted rc16 from the bucket shared with production. Publishing can no longer delete, and nothing here deletes unless you ask for deletion by name.
| Operation | Command | Mutates | Authorization |
|---|---|---|---|
| Build | python3 -m build --sdist (in the muse repo) |
local dist/ only |
none |
| Validate | python -m deploy.release_gate run … |
local report file only | none (runs in temp dirs) |
| Publish | bash deploy/publish_muse_release.sh <env> |
adds 2 new objects to the bucket + copies a file onto the instance | a passing, fresh gate report for these exact bytes; AWS write access |
| Activate | bash deploy/activate_muse_release.sh staging <ver> <sha> (--expect-absent \| --expect <ver> <sha> <activated_at>) |
one environment's pointer + that instance's active.json |
published release; hash match; the observed pointer state (compare-and-swap) |
| Promote | APPROVE_PRODUCTION=1 bash deploy/activate_muse_release.sh production <ver> <sha> --expect … |
production pointer + active.json |
explicit human approval; must equal what staging runs |
| Roll back | … activate_muse_release.sh <env> --rollback-to <ver> <sha> --expect <ver> <sha> <activated_at> |
that environment's pointer + active.json |
named target (version + hash) and the observed pointer state; production needs approval |
| Delete | python -m deploy.release_safety --bucket … cleanup |
nothing, unless --execute --confirm <digest> |
explicit; see below |
Run python -m deploy… from the musehub repo root.
Architecture (what actually exists)
- Bucket
s3://musehub-releasesis shared by staging and production and holds the publishedmuse-<version>.tar.gzartifacts and the release ledger (below). Objects are removed only by the explicit cleanup procedure in this runbook. - Each instance serves
/releases/*from its own Docker volumemusehub_data:/data/releases. That volume — not the bucket — is whatinstall.shreads. Staging and production volumes are separate. - Release selection is
/data/releases/active.json:{schema, environment, version, sha256, previous, activated_at}.install.shand the docs page use it; the tarball must exist and hash tosha256. Directory contents and mtimes are never consulted, and a missing/inconsistent pointer is an HTTP 503 (no fallback). - Ledger in the bucket:
pointers/staging.json,pointers/production.json(same schema) and, per release,muse-<ver>.release.json(version, sha256, source revision, installer revision, gate-report hash). Cleanup consults this ledger to know what each environment depends on.
1. Build
cd ~/ecosystem/muse && python3 -m build --sdist # → dist/muse-<ver>.tar.gz
The artifact is identified by filename version + SHA-256 (shasum -a 256). Rebuilding produces a
different artifact for the purposes of every later step: the gate report binds to the bytes.
2. Validate
python -m deploy.release_gate run \
--artifact ~/ecosystem/muse/dist/muse-<ver>.tar.gz \
--muse-repo ~/ecosystem/muse \
--source-revision "$(muse -C ~/ecosystem/muse rev-parse HEAD --json | jq -r .commit_id)" \
--python /absolute/path/to/python3.14 \
--out /path/to/gate-report.json
The gate is hermetic. It does not use the interpreter that launched it. --python (required, absolute, CPython,
satisfying the artifact's own Requires-Python) is used only to create a fresh venv in a new temporary directory,
into which deploy/gate_requirements.txt (exact == pins: pytest, build, the MuseHub runtime the installer harness
imports, …) and then the exact artifact with its [all] extra are installed, from a hash-checked copy. Before any
gate runs the environment is verified: every pin installed at its pinned version, installed muse == the candidate and
imported from inside the venv, pip's recorded archive hash == the candidate's, and every grammar of the full profile
imports (Swift is opt-in and not required). A half-built environment has no completion marker and is refused; an
existing directory is never reused. The environment is deleted on success, failure and interruption; global Python is
only ever asked to run -m venv; every pip is the venv's own with PIP_REQUIRE_VIRTUALENV=1; caller PYTHON*,
PIP_* and VIRTUAL_ENV are scrubbed; gates get an unreachable TEST_DATABASE_URL. Any failure to build, verify or
remove the environment means no report.
Two Pythons meet, deliberately: the venv holds tooling plus the installed candidate; the muse checkout is put on
PYTHONPATH for exactly one gate, runtime.installed_parser_parity, because it compares the development parser (the
checkout) with the installed candidate. Updating the pins is a deliberate act: build a scratch venv, pip install -r requirements.txt build packaging pytest-json-report, freeze, and prefer versions the suite is known to pass on
(fresh resolution once pulled SQLAlchemy 2.1 and broke the test conftest).
Source provenance. The revision in the report is observed, not typed: the gate runs muse rev-parse HEAD and
muse status --json on --muse-repo, refuses unless --source-revision equals what Muse reports and the tree is
clean (any modified/added/deleted/renamed/staged/untracked/conflicted file, merge or interrupted checkout), observes
again after the gates, and voids the evidence if the revision, cleanliness or the artifact changed meanwhile.
publish/verify observe the checkout again and refuse a report gated at another revision or on a now-dirty tree.
What this does not prove: that the artifact was built from that checkout. The build is not signed or recorded;
packaging.sdist_reproducible (a fresh sdist of the observed clean checkout is content-identical to the artifact) is
the strongest evidence available, and the report says so (provenance.build_relation: "unattested …"). The revision of
the MuseHub checkout that holds the gate code and tests is not yet recorded (Phase 6E).
Report schema 2 (provenance, environment blocks). Schema 1 reports carried an operator-typed revision and no
evidence, so they are refused, not upgraded; they expire after 24 h anyway — re-run the gate.
A missing dependency makes the affected gates fail (they never skip). Required gates:
| Group | Gates |
|---|---|
| packaging | packaging.sdist_reproducible (content-level rebuild from the checkout), packaging.version (filename = PKG-INFO = pyproject, canonical PEP 440), packaging.dependency_metadata, packaging.manifest_contract (engine and installer agree on installer_contract) |
| installation | install.full, install.slim, install.legacy_upgrade, install.profile_preservation, install.explicit_grammar |
| runtime | runtime.doctor, runtime.grammar_availability, runtime.installed_parser_parity |
The report records version, SHA-256, observed source revision, installer revision, the interpreter and every installed
distribution (plus hashes of the pins), platform, per-gate status and exit code, and an overall decision.
Verification trusts none of it: the decision is recomputed; a missing gate, a nonzero exit, a stale (>24 h) or
future-dated report, an unreadable report, a different version or a different hash of the same version, inconsistent
provenance, or environment evidence for another artifact all fail. An interrupted run writes no report.
Check any report yourself: python -m deploy.release_gate verify --artifact A --report R --muse-repo ~/ecosystem/muse.
3. Publish
MUSE_ARTIFACT=…/muse-<ver>.tar.gz MUSE_GATE_REPORT=/path/to/gate-report.json \
bash deploy/publish_muse_release.sh staging # MUSE_REPO defaults to ~/ecosystem/muse
Order: verify report → create-only upload of artifact then record → additive, hash-checked copy onto the instance → check the instance serves identical bytes. The gate runs before any mutation.
- Same version + same bytes: idempotent, including when publishers race: the artifact and then the record are
created with storage-level create-only writes (S3
If-None-Match: *); a publisher that loses a create compares the winner's bytes and records and succeeds (published/completed/already_published) only if they are identical. Same version + different bytes, or a malformed or contradicting record: refused, nothing overwritten or deleted. - Publishing never deletes, never prunes, never writes
active.json, never moves a pointer. - The release is published but not active. Staging users still get the previous release.
4. Activate (staging) · 5. Promote (production)
# read pointers/<env>.json first, then name what you saw:
bash deploy/activate_muse_release.sh staging <ver> <sha256> --expect-absent # first activation only
bash deploy/activate_muse_release.sh staging <ver> <sha256> --expect <seen-ver> <seen-sha> <seen-activated_at>
APPROVE_PRODUCTION=1 bash deploy/activate_muse_release.sh production <ver> <sha256> --expect …
Activation is a compare-and-swap, like rollback. Already at the target: nothing is written (a retry after a lost
response is safe). Pointer exactly as you observed it: replaced atomically — create-only (If-None-Match: *) when none
existed, otherwise a conditional write (If-Match: <ETag read with the pointer>) — and the abandoned release becomes
previous. Anything else (another operator moved it, malformed pointer, wrong environment) is refused with nothing
changed. Two operators racing from the same observation cannot both win; if the loser's request was identical to the
winner's it is treated as done. Caveat: S3 enforcement of If-Match on musehub-releases has not been exercised
live (Phase 6D made no live writes); do it on a scratch key in Phase 6E before the first live activation.
- Pointer in the bucket moves first (so the new and the previous release are protected from cleanup);
- the instance's
active.jsonis written atomically, only after its tarball hashes to the pointer's sha256; - the served
install.shis checked andsmoke_muse.shruns.
If step 2 or 3 fails the ledger is ahead of the instance — the safe direction. Re-run the same command. Production
activation additionally requires APPROVE_PRODUCTION=1 and that staging currently runs the same version+hash.
Production activation also tags the release (warning-only on failure).
6. Roll back
Rollback is not a swap. Read pointers/<env>.json first, then name both the release to restore and the pointer
state you saw:
bash deploy/activate_muse_release.sh <env> --rollback-to <good-version> <good-sha256> \
--expect <bad-version> <bad-sha256> <bad-activated_at>
- Target already active → success, nothing written. Re-running (or retrying after an interruption) can never re-activate the release you abandoned.
- Pointer is exactly the state you observed → moved to the target; the abandoned release is recorded as
previous(so cleanup still protects it). - Pointer is anywhere else (someone activated something newer) → refused as stale; re-inspect and decide again.
- Missing or malformed pointer, or a target whose record or artifact is missing or mismatched → refused, nothing changed.
Production needs APPROVE_PRODUCTION=1. Undoing a rollback is a deliberate activate, never a repeat.
7. Delete (retention and cleanup)
python -m deploy.release_safety --bucket musehub-releases cleanup --keep 3 # DRY RUN: prints exact set + digest
python -m deploy.release_safety --bucket musehub-releases cleanup --keep 3 \
--execute --confirm <digest> # only after a human reads the set
The dry run prints a --execute --confirm <digest> command only when execution would be accepted. A blocked plan
(a pointer missing or malformed, storage trouble) prints BLOCKED: … with the reason and exits 1; an empty deletion set
prints NOTHING TO DELETE; protected (active/rollback) and unverified artifacts are listed with the count.
A release is deletable only if all hold: it has a valid record whose sha256 matches the stored bytes; it is not the
active or previous release of any environment; and it is outside the newest --keep (PEP 440 order, not time).
Everything else is preserved and listed with the reason. It deletes nothing if either environment pointer is
missing, malformed or unreadable, or storage errors. --execute refuses unless --confirm equals the digest of the set
and every target is re-verified at execution time. Unrelated bucket prefixes are never candidates.
Cleanup is a human decision: nothing in the publish or activate path calls it.
Recovery
| Situation | What it looks like | Do |
|---|---|---|
| Publish interrupted between artifact and record | artifact exists, no .release.json; not activatable; cleanup preserves it |
re-run publish with the same artifact and report (reports expire after 24 h — re-run release_gate run on the same bytes if needed): reports completed |
| Publish refused: different bytes for an existing version | REFUSED: … already exists with different bytes |
do not overwrite. Bump the version and rebuild |
| Instance copy failed | publish exits non-zero after the bucket step | active release is unchanged; re-run publish (idempotent) |
| Activate failed after the pointer moved | install.sh still serves the old version |
re-run the same activate command; or an explicit --rollback-to |
install.sh returns 503 |
no verified active release: … |
the instance has no valid active.json, or the tarball was swapped/removed; restore the file or activate/--rollback-to |
| Gate report rejected | GATE FAILED -- nothing was published |
nothing changed; fix the failing gate, rebuild if code changed, re-validate |
Actions that always need explicit human approval
- Any production change (
APPROVE_PRODUCTION=1). - Any
cleanup --execute. - Migrating legacy objects into the ledger (below), enabling bucket versioning, or changing IAM.
- Deployment, ECR/ECS operations, source pushes/merges (outside this runbook).
Known gaps and migration plan (not performed)
- No ledger yet for existing releases. rc17/rc18/rc19 in the bucket have no
.release.json, and neither environment has a pointer. Until they do, cleanup preserves everything (blocked: pointer missing) andinstall.shon an instance withoutactive.jsonreturns 503. Migration, to be done under explicit approval: for each instance, verify the tarball hash against the bucket copy, writeactive.json(matching what that instance serves today), then write the ledger records and pointers. - Staging and production still share one bucket. Safety now rests on keys and the ledger, not on isolation. Preferred: a separate production bucket (or prefix + IAM) so a staging credential cannot write or delete production's objects at all.
- Bucket versioning is off. Enabling it (plus a noncurrent-version retention rule) would make any future mistaken
delete recoverable. IAM for staging credentials should drop
s3:DeleteObjectonmuse-*. - Per-instance pointer ≠ bucket pointer is reconciled by the activate script, not continuously; compare the two when in doubt.
- ECR image tags/retention were not changed or audited beyond read-only attempts (the audit credentials lack
ecr:Describe*).