# Muse Wire Protocol — Architecture **Goal**: Performance parity with Git smart HTTP for all repo sizes. **Invariants**: Every frame is content-addressed (SHA-256). No frame is ever transmitted unsigned. --- ## Comparison Grid: Git Smart HTTP vs MWP Today vs MWP Target | Capability | Git Smart HTTP (GitHub) | MWP Today | MWP Target | |---|---|---|---| | **Push flow** | `GET /info/refs` → `POST /git-receive-pack` (packfile stream) | `GET /refs` → optional `POST /presign` → `POST /push/stream` | `GET /refs` → `POST /push/stream` (all objects, no sidecar) | | **Fetch flow** | `GET /info/refs` → `POST /git-upload-pack` (packfile stream) | `GET /refs` → `POST /fetch/stream` | same — already correct | | **Object transport** | All objects in one packfile stream, single connection | Small objects in stream; large (>64 KB) via individual presigned R2 PUTs | All objects in one MWP stream, single connection | | **Large object handling** | Objects >2 GB use Git LFS (separate API); regular objects stay in pack | Presigned PUT per object >64 KB — N serial HTTP round-trips | CF Worker receives full MWP body, writes all objects to R2 via `Promise.all` (~3–8 ms/object) | | **Delta compression** | OFS_DELTA and REF_DELTA; server reuses pack deltas | `delta+zlib` encoding in O frames (implemented, used for code) | Same | | **Content addressing** | SHA-1 per object + pack checksum | SHA-256 per object (`sha256:`) | Same | | **Negotiation** | Multi-round HAVE/WANT exchange | Single `POST /negotiate` (depth-limited) | Same | | **Deduplication** | Server skips objects it already has | Server deduplicates on stream ingestion; presign path skips dedup (re-uploads everything) | Dedup before presign call is eliminated — streaming path already deduplicates | | **Parallel object writes** | Server-side pack indexing | Server-side `asyncio.gather` + semaphore (100 slots) in `wire_push_stream` | Same | | **Connection count per push** | 2 (refs + packfile) | 2 + N large objects (N = num objects >64 KB) | 2 (refs + stream) | | **Round trips per push** | 2 | 2 + N | 2 | | **Body size limit** | Unlimited on GitHub | Cloudflare 100 MB per request (blocks large MWP streams) | CF Worker receives full body — no CF proxy limit | | **HTTP version** | HTTP/1.1 (packfile is one sequential stream; HTTP/2 multiplexing adds nothing here) | HTTP/1.1 (uvicorn) | Same — HTTP/1.1 is correct | | **Auth** | HTTPS basic auth / OAuth token | MSign Ed25519 signature in Authorization header | Same | | **Streaming response** | Packfile streamed as it's generated | MWP frames streamed (`StreamingResponse`) | Same | | **Fetch streaming** | Single response stream | Single response stream, O frames dispatched on arrival | Same | --- ## Root Cause of the 5-Minute Push The `LARGE_OBJECT_THRESHOLD` is **64 KB**. Audio files, MIDI files, and soundfonts are almost always >64 KB. A typical Stori repo push has 200–400 objects above this threshold. The current push sequence for a large push: ``` 1. GET /refs ~50 ms 2. POST /presign (one request) ~100 ms — server issues N presigned URLs 3. PUT R2-presigned-url (object 1) ~100–400 ms 4. PUT R2-presigned-url (object 2) ~100–400 ms ... N+3. PUT R2-presigned-url (obj N) ~100–400 ms N+4. sleep(cooldown) 0.5–10 s — SSL session state cooldown N+5. POST /push/stream ~200 ms — commits only, objects already in R2 ``` For 328 objects: `328 × 250ms ≈ 82 seconds`. Plus cooldown. That is the 5-minute hang. Git has **zero** presigned sidecar. All 328 objects go into a single POST body. --- ## Why Presign Exists (The Constraint) Cloudflare's proxy has a **100 MB per-request body limit** on all plans. A push of 328 objects at average 200 KB each = ~65 MB — already near the limit. A large repo with MIDI + audio could easily be 500 MB. Git avoids this because GitHub terminates TLS before Cloudflare (or uses Cloudflare's Enterprise plan with custom limits). We are behind Cloudflare's shared proxy. **The presign path was the workaround.** It's the wrong long-term answer. Local hub has no Cloudflare proxy, so it never needed presign either — the local `LocalBackend.presign_batch()` returns an empty dict. --- ## The Right Answer: Cloudflare Worker Pack Receiver `cloudflare/pack-receiver/worker.js` already exists and is already wired up on the server side (`POST /{owner}/{slug}/push/object-pack` in wire.py). It is **not yet used by the CLI client**. ### How it works ``` CLI client → POST /push/object-pack → CF Worker ↓ R2.put(obj1, obj2, ... objN) [Promise.all, ~3 ms/object] ↓ POST /internal/register-objects → MuseHub EC2 ↓ 200 OK ← response to CLI ``` Latency profile: - **Current (presign)**: EC2 receives → boto3 PUT×250 serial → respond: **~875 ms/PUT** - **Target (Worker)**: Worker receives → R2 PUT×250 via `Promise.all` → notify MuseHub: **~3–8 ms/PUT** For 328 objects: `328 × 5ms = 1.6 seconds` vs `328 × 300ms = ~98 seconds`. The Worker receives the full MWP body (same format the CLI already sends), so no CLI frame-format change is needed — only routing. --- ## Phase Plan (TDD, Sequenced) ### Phase A — Server + test cleanup ✅ COMPLETE Committed `sha256:b851473` (musehub), `sha256:c6958f1` (muse): | Deleted | What it was | |---|---| | `wire_push()` (~430 lines) | Non-streaming legacy push; all security invariants exist in `wire_push_stream` | | `iter_wire_frames_grpc()` | gRPC-era frame parser — unreachable from any route | | `WirePushRequest` / `WirePushResponse` | Pydantic models that only served `wire_push` | | `application/grpc` Content-Type branch in `request_signing.py` | Dead check | | `test_grpc_wire_framing.py` | Tests for dead gRPC parser | | `grpc_frame` / `GRPC_CONTENT_TYPE` usage in `test_background_jobs.py`, `test_wire_oc_server.py` | Both files crashed at import — now fixed to raw MWP framing | | `musewire-performance.md`, `push-v2.md` | Superseded planning docs | Also fixed: `muse status --json` now sets `clean=False` when untracked files exist, matching git behaviour. `wire_push_stream` is the sole push entry point. Zero live gRPC references remain. --- ### Phase B — Delete presign from MuseHub Delete the presign sidecar from the server entirely. Local hub never needed it (LocalBackend returns empty presign dict). Production will use the CF Worker. **Delete from `musehub/api/routes/wire.py`:** - `POST /{owner}/{slug}/presign` route **Delete from `musehub/services/musehub_wire.py`:** - `presign_and_register_objects()` **Delete from `musehub/models/wire.py`:** - `WirePresignRequest`, `WirePresignResponse` **Delete:** - `tests/test_wire_presign_integrity.py` (tests for the deleted route) **Tests to update:** - Any test in `test_wire_multibatch_push.py` or `test_wire_push_stream.py` that calls `/presign` --- ### Phase C — Delete presign from CLI **Delete from `muse/cli/commands/push.py`:** - `_make_r2_client()`, `_presigned_put()`, presign call, cooldown sleep - `LARGE_OBJECT_THRESHOLD` constant **Delete from `muse/core/transport.py`:** - `presign_objects()` on both `MuseTransport` and `HttpTransport` --- ### Phase D — TDD + implement CF Worker path ✅ COMPLETE **Server side was already complete** (existed before this phase): - `wire_refs()` returns `pack_origin` from `settings.pack_worker_url` when `worker_internal_key` is set - `RemoteInfo.pack_origin: str` field in `pack.py` - `_parse_remote_info()` reads `pack_origin` from wire response - `/internal/register-objects` endpoint — Worker callback after R2 writes - `settings.pack_worker_url` + `settings.worker_internal_key` in `config.py` **CLI changes (muse repo)**: - `_post_object_pack(worker_url, objects, signing, client)` — POSTs objects to CF Worker as msgpack `{"objects": [...]}`; Worker writes to R2 via `Promise.all` and calls `/internal/register-objects` - `_push_stream(... pack_origin: str | None = None)` — when set, sends objects to Worker in batches, then sends commits+snapshots only via `push/stream` - `run_cmd_push` passes `remote_info.get("pack_origin")` to `_push_stream` - Tests: `muse/tests/test_wire_pack_post.py` — 16 tests, all green **Push flow with Worker enabled:** ``` GET /refs → server returns pack_origin = "https://pack.staging.musehub.ai" POST /push/object-pack → CF Worker (native R2 binding, ~3–8 ms/obj via Promise.all) Worker → POST /internal/register-objects → MuseHub DB POST /push/stream → commits + snapshots only (objects=[], ~200 ms) ``` **Push flow without Worker (local hub):** ``` GET /refs → no pack_origin POST /push/stream → all objects + commits in one MWP stream body ``` --- ### Phase E — Deploy CF Worker + verify ```bash cd ~/ecosystem/musehub/cloudflare/pack-receiver wrangler deploy wrangler secret put MUSEHUB_INTERNAL_KEY wrangler secret put MUSEHUB_ORIGIN # https://staging.musehub.ai ``` `wrangler.toml`: ```toml [[r2_buckets]] binding = "BUCKET" bucket_name = "musehub-objects" ``` Set in production `.env`: ``` WORKER_ENDPOINT=https://objects.musehub.ai ``` Proof of done: ```bash time muse -C ~/ecosystem/muse push staging dev # Target: <5 seconds for a repo with 300+ objects # Baseline (git): ~2 seconds for equivalent repo ``` --- ## Current Protocol State (Accurate as of 2026-04-24) | Property | Value | |---|---| | Content-Type | `application/x-muse-wire` | | Transport | HTTP/1.1 (uvicorn) | | Push endpoint | `POST /{owner}/{slug}/push/stream` — sole push entry point | | Fetch endpoint | `POST /{owner}/{slug}/fetch/stream` | | Negotiation | `POST /{owner}/{slug}/negotiate` | | Refs | `GET /{owner}/{slug}/refs` | | Frame types | H, O, OC, C, E (request) · X, P, R (response) | | Object encoding | `raw`, `zlib`, `delta+zlib` | | Object addressing | `sha256:<64-hex>` | | Auth | MSign Ed25519 in `Authorization` header | | Large objects | All objects stream via MWP; CF Worker handles R2 writes in parallel when `pack_origin` is advertised | | CF Worker | Built + wired (`cloudflare/pack-receiver/worker.js`). Deploy to activate (Phase E). | | gRPC | **Deleted.** Zero references remain. |