Muse Wire Protocol — Architecture

Goal: Performance parity with Git smart HTTP for all repo sizes. Invariants: Every frame is content-addressed (SHA-256). No frame is ever transmitted unsigned.

Comparison Grid: Git Smart HTTP vs MWP Today vs MWP Target

Root Cause of the 5-Minute Push

The LARGE_OBJECT_THRESHOLD is 64 KB. Audio files, MIDI files, and soundfonts are almost always >64 KB. A typical Stori repo push has 200–400 objects above this threshold.

The current push sequence for a large push:

1. GET  /refs                        ~50 ms
2. POST /presign  (one request)      ~100 ms  — server issues N presigned URLs
3. PUT  R2-presigned-url (object 1)  ~100–400 ms
4. PUT  R2-presigned-url (object 2)  ~100–400 ms
   ...
N+3. PUT R2-presigned-url (obj N)    ~100–400 ms
N+4. sleep(cooldown)                 0.5–10 s  — SSL session state cooldown
N+5. POST /push/stream               ~200 ms  — commits only, objects already in R2

For 328 objects: 328 × 250ms ≈ 82 seconds. Plus cooldown. That is the 5-minute hang.

Git has zero presigned sidecar. All 328 objects go into a single POST body.

Why Presign Exists (The Constraint)

Cloudflare's proxy has a 100 MB per-request body limit on all plans. A push of 328 objects at average 200 KB each = ~65 MB — already near the limit. A large repo with MIDI + audio could easily be 500 MB.

Git avoids this because GitHub terminates TLS before Cloudflare (or uses Cloudflare's Enterprise plan with custom limits). We are behind Cloudflare's shared proxy.

The presign path was the workaround. It's the wrong long-term answer. Local hub has no Cloudflare proxy, so it never needed presign either — the local LocalBackend.presign_batch() returns an empty dict.

The Right Answer: Cloudflare Worker Pack Receiver

cloudflare/pack-receiver/worker.js already exists and is already wired up on the server side (POST /{owner}/{slug}/push/object-pack in wire.py). It is not yet used by the CLI client.

How it works

CLI client  →  POST /push/object-pack  →  CF Worker
                                              ↓
                                    R2.put(obj1, obj2, ... objN)  [Promise.all, ~3 ms/object]
                                              ↓
                                    POST /internal/register-objects  →  MuseHub EC2
                                              ↓
                                    200 OK
                                    ←  response to CLI

Latency profile:

• Current (presign): EC2 receives → boto3 PUT×250 serial → respond: ~875 ms/PUT
• Target (Worker): Worker receives → R2 PUT×250 via Promise.all → notify MuseHub: ~3–8 ms/PUT

For 328 objects: 328 × 5ms = 1.6 seconds vs 328 × 300ms = ~98 seconds.

The Worker receives the full MWP body (same format the CLI already sends), so no CLI frame-format change is needed — only routing.

Phase Plan (TDD, Sequenced)

Phase A — Server + test cleanup ✅ COMPLETE

Committed sha256:b851473 (musehub), sha256:c6958f1 (muse):

Also fixed: muse status --json now sets clean=False when untracked files exist, matching git behaviour. wire_push_stream is the sole push entry point. Zero live gRPC references remain.

Phase B — Delete presign from MuseHub

Delete the presign sidecar from the server entirely. Local hub never needed it (LocalBackend returns empty presign dict). Production will use the CF Worker.

Delete from musehub/api/routes/wire.py:

• POST /{owner}/{slug}/presign route

Delete from musehub/services/musehub_wire.py:

• presign_and_register_objects()

Delete from musehub/models/wire.py:

• WirePresignRequest, WirePresignResponse

Delete:

• tests/test_wire_presign_integrity.py (tests for the deleted route)

Tests to update:

• Any test in test_wire_multibatch_push.py or test_wire_push_stream.py that calls /presign

Phase C — Delete presign from CLI

Delete from muse/cli/commands/push.py:

• _make_r2_client(), _presigned_put(), presign call, cooldown sleep
• LARGE_OBJECT_THRESHOLD constant

Delete from muse/core/transport.py:

• presign_objects() on both MuseTransport and HttpTransport

Phase D — TDD + implement CF Worker path ✅ COMPLETE

Server side was already complete (existed before this phase):

• wire_refs() returns pack_origin from settings.pack_worker_url when worker_internal_key is set
• RemoteInfo.pack_origin: str field in pack.py
• _parse_remote_info() reads pack_origin from wire response
• /internal/register-objects endpoint — Worker callback after R2 writes
• settings.pack_worker_url + settings.worker_internal_key in config.py

CLI changes (muse repo):

• _post_object_pack(worker_url, objects, signing, client) — POSTs objects to CF Worker as msgpack {"objects": [...]}; Worker writes to R2 via Promise.all and calls /internal/register-objects
• _push_stream(... pack_origin: str | None = None) — when set, sends objects to Worker in batches, then sends commits+snapshots only via push/stream
• run_cmd_push passes remote_info.get("pack_origin") to _push_stream
• Tests: muse/tests/test_wire_pack_post.py — 16 tests, all green

Push flow with Worker enabled:

GET /refs              → server returns pack_origin = "https://pack.staging.musehub.ai"
POST /push/object-pack → CF Worker (native R2 binding, ~3–8 ms/obj via Promise.all)
                         Worker → POST /internal/register-objects → MuseHub DB
POST /push/stream      → commits + snapshots only (objects=[], ~200 ms)

Push flow without Worker (local hub):

GET /refs              → no pack_origin
POST /push/stream      → all objects + commits in one MWP stream body

Phase E — Deploy CF Worker + verify

cd ~/ecosystem/musehub/cloudflare/pack-receiver
wrangler deploy
wrangler secret put MUSEHUB_INTERNAL_KEY
wrangler secret put MUSEHUB_ORIGIN  # https://staging.musehub.ai

wrangler.toml:

[[r2_buckets]]
binding = "BUCKET"
bucket_name = "musehub-objects"

Set in production .env:

WORKER_ENDPOINT=https://objects.musehub.ai

Proof of done:

time muse -C ~/ecosystem/muse push staging dev
# Target: <5 seconds for a repo with 300+ objects
# Baseline (git): ~2 seconds for equivalent repo

Current Protocol State (Accurate as of 2026-04-24)