gabriel / musehub public
release-runbook.md markdown
282 lines 20.4 KB
Raw
sha256:db5120371e0ff3edd7a3920ffa9f31fe81583e4462228510f6db188972f0a31c deploy: memory headroom gate is 1200 MB in staging and production Sonnet 5.5 minor ⚠ breaking 4 days ago

Muse release runbook — build, validate, publish, activate, promote, roll back, delete

MuseHub issue: muse#114 (Phase 6). Audience: whoever ships a Muse CLI release.

The workflow, gates and completion states are defined once, in the canonical Shipping-Velocity Policy. This runbook is the procedure it points to. Blanket observation windows are removed; the checks below are bounded and evidence-based.

Routine batch release — the default path

Authorization. Staging needs no GO. Production needs exactly one GO per batch; it covers every production step below (CLI promotion, server deploy, source push to the production remote, and an in-GO rollback under the criteria in step 5). Nothing outside the batch is covered.

  1. Preconditions. Every ticket in the batch is Integrated; a signed commit bumps the version. For a Muse CLI release run the identity check once: bash ~/ecosystem/muse/scripts/dev/which-muse.sh --preflight --task release-acceptance --require <ver> (definition). BLOCK stops the release; it never authorizes swapping the installed CLI.
  2. Staging. CLI: Build → Validate → Publish → Activate staging (§1–4). Server: bash deploy/push.sh staging. The smoke is run once: activate runs smoke_muse.sh itself, and the server check is python3 tests/bench_cli.py --size xs --hubs staging (every verb under its latency gate; details in deploy.md). Both passing = Staging Green. Reuse a gate report for the same artifact and revision; re-run it only when either changes or the report expires.
  3. Request the GO. One message: release identity (CLI version + sha256; server MuseHub <semver> (<revision8>-<timestamp>@<digest12>)), the tickets, the gate-report hash, and the staging smoke result.
  4. Production, after the GO. Pre-checks fail closed (stop and report, do not override): (a) memory headroom — deploy.sh refuses to start the new slot when available memory is below its threshold; (b) running jobs — a read-only look at running jobs before the worker restart; (c) production runs exactly what staging runs (CLI: enforced by Promote). Then Promote / push.sh prod, run the production smoke once, and watch 5–10 minutes. Longer telemetry is asynchronous and does not block.
  5. Rollback. CLI: the compare-and-swap --rollback-to of §6 (a pointer move is reversible by definition). Server: automatic image rollback (IMAGE_TAG=<previous> bash deploy/push.sh prod) is permitted only when the deploy is demonstrably reversible — the previous image tag is still deployable and the deploy applied no migration (deploy.sh ends with rollback-safe: yes). If it says rollback-safe: no, or you cannot tell, do not roll back automatically: stop and report.
  6. Close out. Advance main to the released dev revision, push source to the remotes, post one close comment per ticket, close them, retire the task branches.

The seven operations

These seven operations are different. None implies another. Each has its own command, its own authorization, and its own failure mode. The incident that created this document: a publish to staging silently also deleted rc16 from the bucket shared with production. Publishing can no longer delete, and nothing here deletes unless you ask for deletion by name.

Operation Command Mutates Authorization
Build python3 -m build --sdist (in the muse repo) local dist/ only none
Validate python -m deploy.release_gate run … local report file only none (runs in temp dirs)
Publish bash deploy/publish_muse_release.sh <env> adds 2 new objects to the bucket + copies a file onto the instance a passing, fresh gate report for these exact bytes; AWS write access
Activate bash deploy/activate_muse_release.sh staging <ver> <sha> (--expect-absent \| --expect <ver> <sha> <activated_at>) one environment's pointer + that instance's active.json published release; hash match; the observed pointer state (compare-and-swap)
Promote APPROVE_PRODUCTION=1 bash deploy/activate_muse_release.sh production <ver> <sha> --expect … production pointer + active.json explicit human approval; must equal what staging runs
Roll back … activate_muse_release.sh <env> --rollback-to <ver> <sha> --expect <ver> <sha> <activated_at> that environment's pointer + active.json named target (version + hash) and the observed pointer state; production needs approval
Delete python -m deploy.release_safety --bucket … cleanup nothing, unless --execute --confirm <digest> explicit; see below

Run python -m deploy… from the musehub repo root.

Architecture (what actually exists)

  • Bucket s3://musehub-releases is shared by staging and production and holds the published muse-<version>.tar.gz artifacts and the release ledger (below). Objects are removed only by the explicit cleanup procedure in this runbook.
  • Each instance serves /releases/* from its own Docker volume musehub_data:/data/releases. That volume — not the bucket — is what install.sh reads. Staging and production volumes are separate.
  • Release selection is /data/releases/active.json: {schema, environment, version, sha256, previous, activated_at}. install.sh and the docs page use it; the tarball must exist and hash to sha256. Directory contents and mtimes are never consulted, and a missing/inconsistent pointer is an HTTP 503 (no fallback).
  • Ledger in the bucket: pointers/staging.json, pointers/production.json (same schema) and, per release, muse-<ver>.release.json (version, sha256, source revision, installer revision, gate-report hash). Cleanup consults this ledger to know what each environment depends on.

1. Build

cd ~/ecosystem/muse && python3 -m build --sdist     # → dist/muse-<ver>.tar.gz

The artifact is identified by filename version + SHA-256 (shasum -a 256). Rebuilding produces a different artifact for the purposes of every later step: the gate report binds to the bytes.

2. Validate

python -m deploy.release_gate run \
  --artifact ~/ecosystem/muse/dist/muse-<ver>.tar.gz \
  --muse-repo ~/ecosystem/muse \
  --source-revision "$(muse -C ~/ecosystem/muse rev-parse HEAD --json | jq -r .commit_id)" \
  --python /absolute/path/to/python3.14 \
  --out /path/to/gate-report.json

The gate is hermetic. It does not use the interpreter that launched it. --python (required, absolute, CPython, satisfying the artifact's own Requires-Python) is used only to create a fresh venv in a new temporary directory, into which deploy/gate_requirements.txt (exact == pins: pytest, build, the MuseHub runtime the installer harness imports, …) and then the exact artifact with its [all] extra are installed, from a hash-checked copy. Before any gate runs the environment is verified: every pin installed at its pinned version, installed muse == the candidate and imported from inside the venv, pip's recorded archive hash == the candidate's, and every grammar of the full profile imports (Swift is opt-in and not required). A half-built environment has no completion marker and is refused; an existing directory is never reused. The environment is deleted on success, failure and interruption; global Python is only ever asked to run -m venv; every pip is the venv's own with PIP_REQUIRE_VIRTUALENV=1; caller PYTHON*, PIP_* and VIRTUAL_ENV are scrubbed; gates get an unreachable TEST_DATABASE_URL. Any failure to build, verify or remove the environment means no report.

Two Pythons meet, deliberately: the venv holds tooling plus the installed candidate; the muse checkout is put on PYTHONPATH for exactly one gate, runtime.installed_parser_parity, because it compares the development parser (the checkout) with the installed candidate. Updating the pins is a deliberate act: build a scratch venv, pip install -r requirements.txt build packaging pytest-json-report, freeze, and prefer versions the suite is known to pass on (fresh resolution once pulled SQLAlchemy 2.1 and broke the test conftest).

Source provenance. The revision in the report is observed, not typed: the gate runs muse rev-parse HEAD and muse status --json on --muse-repo, refuses unless --source-revision equals what Muse reports and the tree is clean (any modified/added/deleted/renamed/staged/untracked/conflicted file, merge or interrupted checkout), observes again after the gates, and voids the evidence if the revision, cleanliness or the artifact changed meanwhile. publish/verify observe the checkout again and refuse a report gated at another revision or on a now-dirty tree. What this does not prove: that the artifact was built from that checkout. The build is not signed or recorded; packaging.sdist_reproducible (a fresh sdist of the observed clean checkout is content-identical to the artifact) is the strongest evidence available, and the report says so (provenance.build_relation: "unattested …"). The revision of the MuseHub checkout that holds the gate code and tests is not yet recorded (Phase 6E).

Report schema 2 (provenance, environment blocks). Schema 1 reports carried an operator-typed revision and no evidence, so they are refused, not upgraded; they expire after 24 h anyway — re-run the gate.

A missing dependency makes the affected gates fail (they never skip). Required gates:

Group Gates
packaging packaging.sdist_reproducible (content-level rebuild from the checkout), packaging.version (filename = PKG-INFO = pyproject, canonical PEP 440), packaging.dependency_metadata, packaging.manifest_contract (engine and installer agree on installer_contract)
installation install.full, install.slim, install.legacy_upgrade, install.profile_preservation, install.explicit_grammar
runtime runtime.doctor, runtime.grammar_availability, runtime.installed_parser_parity

The report records version, SHA-256, observed source revision, installer revision, the interpreter and every installed distribution (plus hashes of the pins), platform, per-gate status and exit code, and an overall decision. Verification trusts none of it: the decision is recomputed; a missing gate, a nonzero exit, a stale (>24 h) or future-dated report, an unreadable report, a different version or a different hash of the same version, inconsistent provenance, or environment evidence for another artifact all fail. An interrupted run writes no report. Check any report yourself: python -m deploy.release_gate verify --artifact A --report R --muse-repo ~/ecosystem/muse.

3. Publish

MUSE_ARTIFACT=…/muse-<ver>.tar.gz MUSE_GATE_REPORT=/path/to/gate-report.json \
  bash deploy/publish_muse_release.sh staging      # MUSE_REPO defaults to ~/ecosystem/muse

Order: verify report → create-only upload of artifact then record → additive, hash-checked copy onto the instance → check the instance serves identical bytes. The gate runs before any mutation.

  • Same version + same bytes: idempotent, including when publishers race: the artifact and then the record are created with storage-level create-only writes (S3 If-None-Match: *); a publisher that loses a create compares the winner's bytes and records and succeeds (published / completed / already_published) only if they are identical. Same version + different bytes, or a malformed or contradicting record: refused, nothing overwritten or deleted.
  • Publishing never deletes, never prunes, never writes active.json, never moves a pointer.
  • The release is published but not active. Staging users still get the previous release.

4. Activate (staging) · 5. Promote (production)

# read pointers/<env>.json first, then name what you saw:
bash deploy/activate_muse_release.sh staging <ver> <sha256> --expect-absent                        # first activation only
bash deploy/activate_muse_release.sh staging <ver> <sha256> --expect <seen-ver> <seen-sha> <seen-activated_at>
APPROVE_PRODUCTION=1 bash deploy/activate_muse_release.sh production <ver> <sha256> --expect …

Activation is a compare-and-swap, like rollback. Already at the target: nothing is written (a retry after a lost response is safe). Pointer exactly as you observed it: replaced atomically — create-only (If-None-Match: *) when none existed, otherwise a conditional write (If-Match: <ETag read with the pointer>) — and the abandoned release becomes previous. Anything else (another operator moved it, malformed pointer, wrong environment) is refused with nothing changed. Two operators racing from the same observation cannot both win; if the loser's request was identical to the winner's it is treated as done. Caveat: S3 enforcement of If-Match on musehub-releases has not been exercised live (Phase 6D made no live writes); do it on a scratch key in Phase 6E before the first live activation.

  1. Pointer in the bucket moves first (so the new and the previous release are protected from cleanup);
  2. the instance's active.json is written atomically, only after its tarball hashes to the pointer's sha256;
  3. the served install.sh is checked and smoke_muse.sh runs.

If step 2 or 3 fails the ledger is ahead of the instance — the safe direction. Re-run the same command. Production activation additionally requires APPROVE_PRODUCTION=1 and that staging currently runs the same version+hash. Production activation also tags the release (warning-only on failure).

6. Roll back

Rollback is not a swap. Read pointers/<env>.json first, then name both the release to restore and the pointer state you saw:

bash deploy/activate_muse_release.sh <env> --rollback-to <good-version> <good-sha256> \
     --expect <bad-version> <bad-sha256> <bad-activated_at>
  • Target already active → success, nothing written. Re-running (or retrying after an interruption) can never re-activate the release you abandoned.
  • Pointer is exactly the state you observed → moved to the target; the abandoned release is recorded as previous (so cleanup still protects it).
  • Pointer is anywhere else (someone activated something newer) → refused as stale; re-inspect and decide again.
  • Missing or malformed pointer, or a target whose record or artifact is missing or mismatched → refused, nothing changed.

Production needs APPROVE_PRODUCTION=1. Undoing a rollback is a deliberate activate, never a repeat.

7. Delete (retention and cleanup)

python -m deploy.release_safety --bucket musehub-releases cleanup --keep 3            # DRY RUN: prints exact set + digest
python -m deploy.release_safety --bucket musehub-releases cleanup --keep 3 \
        --execute --confirm <digest>                                                 # only after a human reads the set

The dry run prints a --execute --confirm <digest> command only when execution would be accepted. A blocked plan (a pointer missing or malformed, storage trouble) prints BLOCKED: … with the reason and exits 1; an empty deletion set prints NOTHING TO DELETE; protected (active/rollback) and unverified artifacts are listed with the count.

A release is deletable only if all hold: it has a valid record whose sha256 matches the stored bytes; it is not the active or previous release of any environment; and it is outside the newest --keep (PEP 440 order, not time). Everything else is preserved and listed with the reason. It deletes nothing if either environment pointer is missing, malformed or unreadable, or storage errors. --execute refuses unless --confirm equals the digest of the set and every target is re-verified at execution time. Unrelated bucket prefixes are never candidates. Cleanup is a human decision: nothing in the publish or activate path calls it.

Recovery

Situation What it looks like Do
Publish interrupted between artifact and record artifact exists, no .release.json; not activatable; cleanup preserves it re-run publish with the same artifact and report (reports expire after 24 h — re-run release_gate run on the same bytes if needed): reports completed
Publish refused: different bytes for an existing version REFUSED: … already exists with different bytes do not overwrite. Bump the version and rebuild
Instance copy failed publish exits non-zero after the bucket step active release is unchanged; re-run publish (idempotent)
Activate failed after the pointer moved install.sh still serves the old version re-run the same activate command; or an explicit --rollback-to
install.sh returns 503 no verified active release: … the instance has no valid active.json, or the tarball was swapped/removed; restore the file or activate/--rollback-to
Gate report rejected GATE FAILED -- nothing was published nothing changed; fix the failing gate, rebuild if code changed, re-validate

MuseHub server releases — release identity

This runbook ships the Muse CLI. MuseHub server builds follow the same naming discipline (docs/versioning.md, MuseHub versioning); operational c-series labels are retired and appear only in that document's legacy table.

  1. Bump. A signed commit sets pyproject.toml [project].version to the next release version (0.2.3rc1, then 0.2.3rc2, …). Never rebuild a version that was ever pushed.
  2. Build and push (deploy/push.sh <env>). It derives the identity from the build context's pyproject.toml, builds with the org.opencontainers.image.version label, refuses an image whose label or runtime reports another version, refuses a SemVer tag that already exists, pushes <revision8>-<timestamp> and tags it <semver>, and writes the release manifest musehub-<semver>.release.json once ($RELEASE_MANIFEST_DIR, default ~/.musehub/releases).
  3. Deploy. deploy/deploy.sh first refuses to start the new slot if available memory is below the headroom threshold (MIN_MEM_AVAILABLE_MB; default 1200 MB, the production H8 gate and the staging pre-start gate), and switches traffic only after the new slot's GET /protocol reports the image's version and revision. It ends with rollback-safe: yes|no (yes = no migration ran in this deploy). Rollback by <revision8>-<timestamp> (IMAGE_TAG=… deploy/push.sh <env>) is unchanged.
  4. Report every build and deployment as MuseHub <semver> (<revision8>-<timestamp>@<digest12>) plus environment, and record where it runs in the deploy evidence, not in the manifest.

Authorization

  • Staging — no GO: builds, gate runs, pushes to the local and staging remotes, staging publish/activate and push.sh staging, for a batch of Integrated tickets.
  • Production — one GO per batch (APPROVE_PRODUCTION=1), as in the routine release above.
  • Always individually approved, GO or not: any cleanup --execute; migrating legacy objects into the ledger (below), enabling bucket versioning, or changing IAM; ECR/ECS or infrastructure operations outside this runbook; anything outside the batch.

Known gaps and migration plan (not performed)

  1. No ledger yet for existing releases. rc17/rc18/rc19 in the bucket have no .release.json, and neither environment has a pointer. Until they do, cleanup preserves everything (blocked: pointer missing) and install.sh on an instance without active.json returns 503. Migration, to be done under explicit approval: for each instance, verify the tarball hash against the bucket copy, write active.json (matching what that instance serves today), then write the ledger records and pointers.
  2. Staging and production still share one bucket. Safety now rests on keys and the ledger, not on isolation. Preferred: a separate production bucket (or prefix + IAM) so a staging credential cannot write or delete production's objects at all.
  3. Bucket versioning is off. Enabling it (plus a noncurrent-version retention rule) would make any future mistaken delete recoverable. IAM for staging credentials should drop s3:DeleteObject on muse-*.
  4. Per-instance pointer ≠ bucket pointer is reconciled by the activate script, not continuously; compare the two when in doubt.
  5. ECR image tags/retention were not changed or audited beyond read-only attempts (the audit credentials lack ecr:Describe*).
File History 8 commits
sha256:db5120371e0ff3edd7a3920ffa9f31fe81583e4462228510f6db188972f0a31c deploy: memory headroom gate is 1200 MB in staging and production Sonnet 5.5 minor ⚠ 4 days ago
sha256:b8818aebe51ac130d097273b5277ea0d2b2c8b795a6de123247dc4f3d0e38820 release: one-GO batch release flow; fail-closed memory head… Sonnet 5.5 patch 4 days ago
sha256:90e8ad5cadbe1c97b574605e4dcfbf7b1d1353e28861ab572de03fd9fd6e8c07 Require Muse CLI identity preflight from MuseHub entry poin… Sonnet 5.5 4 days ago
sha256:8081ececc94a2fbc5ae695dcc9ad50dac9183dfa12331c8879863ed5743f1d3b Reference the canonical Shipping-Velocity Policy from MuseH… Sonnet 5.5 4 days ago
sha256:dba57911ebca551305cf03f3bed21293ed12bb10c6ffadfd47f3929b97952022 docs(release): SemVer release identity for MuseHub; c-serie… Opus 5.5 5 days ago
sha256:112c7c11ddeb1cfaf69d56ec6b2980820b6d34a69abe89a71db520dba17250d4 docs(release): runbook for Phase 6D contracts; production-r… Opus 5.5 12 days ago
sha256:9f814406166bc157b00dfd4078fb4516ad46e9844628da6bb3823e8448eca1e9 fix(release): rollback is an explicit, idempotent, stale-pr… Opus 5.5 minor ⚠ 12 days ago
sha256:28b9e16197029996ce556a27058a9faa4eb27e088440726c5d8aaa8a811a4a64 docs(release): runbook separating build/validate/publish/ac… Opus 5.5 patch 12 days ago