Muse Wire Protocol — Four Verbs: Status, Bugs, and Fix Plan

Per the MWP spec at /muse/wire. Each verb maps directly to a Git smart HTTP analogue. This document is the authoritative record of what works, what breaks, and what must be fixed.

How MWP Maps to Git

Every verb is a single HTTP round trip after the refs preflight. No exotic infrastructure.

Scale Ladder (used throughout this doc)

1. Push — POST /{owner}/{slug}/push/stream

What the spec says

Client sends [H][O…][OC…][C][E]. Server processes frames as they arrive, writes objects to R2 in parallel batches, and streams [P…][R] back on the response body before the upload is complete. Equiv: git-receive-pack.

What the client does (muse/core/transport.py)

• Client batches objects in chunks of 500 (CHUNK_OBJECTS = 500)
• Each chunk is one POST to /push/stream
• Non-final chunks: objects only, no commits
• Final chunk: remaining objects + all 830 commits + all snapshots
• Auth: MSign header, empty-body hash
• HTTP: http2=False, timeout=300s
• Response parsed with msgpack.Unpacker via async with client.stream() + aiter_bytes()

What the server does (musehub/services/musehub_wire.py)

1. Auth check
2. Read H frame
3. Read O/OC frames → flush to R2 in parallel batches (100-slot semaphore)
4. Read C frame → parse commits + snapshots
5. Read E frame → count cross-check
6. Phase C: referential integrity (bulk SELECT, no per-object exists() call after fix)
7. Quota check
8. Signature check
9. bulk_upsert_snapshot_entries() → one INSERT
10. _topological_sort() → build ordered commit list
11. Bulk INSERT commits
12. Branch pointer update (SELECT FOR UPDATE)
13. Yield R frame

Known bugs / freeze points

Push P3 — current freeze: what we need to find out

Push froze with zero terminal output. That means the freeze is happening before the first P frame reaches the client. Candidates in order of likelihood:

1. Phase C integrity check is still slow — need to confirm the backend.exists() fix actually deployed to staging. Check staging server logs for the batch that froze.
2. bulk_upsert_snapshot_entries() — 830 snapshots × manifests with up to 6869 entries each. One big INSERT but could be enormous if manifests are fully expanded.
3. _topological_sort() — pure Python DAG sort of 830 nodes. Should be fast but worth timing.
4. SELECT FOR UPDATE on branch row — if a prior failed push left a lock on the branch.
5. R2 write semaphore deadlock — if a prior request is holding slots and the semaphore was never released.

Atomic tests for push (must all pass before scaling up)

All push atomic tests pass on localhost. First byte arrives in < 100ms at all scales (server streams immediately). T6 (830 commits) completes in 9.5s — well inside any proxy timeout.

2. Fetch — POST /{owner}/{slug}/fetch/stream

What the spec says

Client sends {"want": [...], "have": [...]} msgpack body. Server walks DAG from wanted commits, stops at have commits, and streams back [H][O…][C][E] — the same frame format as push but server→client. Equiv: git-upload-pack.

What the client does (muse/core/transport.py)

• Calls GET /refs to get remote branch heads
• Calls POST /fetch/stream with want/have lists
• Reads response as MWP frame stream
• Writes received objects to local store
• Writes received commits to local DB

What the server does (musehub/services/musehub_wire.py — fetch side)

1. Repo lookup
2. One bulk JOIN (commits LEFT OUTER JOIN snapshots) — single round-trip for all commit + manifest data
3. BFS in Python from want, stop at have
4. Branch heads SELECT
5. Object metadata bulk SELECT
6. Yield H frame (first byte)
7. O frames via asyncio.as_completed (32-slot R2 concurrency)
8. C frame (commits + snapshots)
9. E frame

Known bugs / freeze points — fixed

Pre-H-frame timing profile (830 commits, 500 objects, localhost)

Atomic tests for fetch

What was verified

1. ✅ fetch/stream uses StreamingResponse wrapping an async generator — correctly streams
2. ✅ No sequential per-object calls — all bulk queries
3. ✅ Client parses with MPackStreamReader (streaming Unpacker) — not buffered
4. ✅ parent_commit_id correctly preserved through push → DB → fetch → client
5. ⏳ F1–F4 incremental fetch tests — not yet run

3. Pull — muse pull <remote> <branch>

What pull does

Pull = GET /refs + POST /fetch/stream + local merge. It is fetch under the hood. Every fetch bug is a pull bug. There is no separate pull endpoint.

Client flow (muse/cli/commands/pull.py or similar)

1. GET /{owner}/{slug}/refs → get remote head for branch
2. Compare remote head to local head → compute want/have
3. POST /fetch/stream with want=remote_head, have=local_commits
4. Apply received objects and commits locally
5. Advance local branch pointer

Known bugs / freeze points

Same as Fetch above. Additionally:

• If have list is incomplete (client sends too few commits), server sends more objects than necessary → wasted bandwidth at scale
• If negotiation is skipped, have=[] and server sends the entire repo history

Atomic tests for pull

PL0  Pull into empty local repo (≡ clone)     → same as F3
PL1  Pull 1 new commit after existing history → < 3s, only 1 commit + its objects
PL2  Pull 10 new commits                      → < 5s
PL3  Pull after 400-commit divergence         → correct subset of objects only

4. Clone — muse clone <url>

What clone does

Clone = create empty local repo + pull everything. Uses POST /fetch/stream with want=[remote_branch_head] and have=[]. No negotiation possible on empty repo.

Client flow

1. Create local repo
2. GET /{owner}/{slug}/refs → get all branch heads
3. For each branch: POST /fetch/stream with want=head, have=[]
4. Write all received objects and commits locally
5. Set local branch pointers

Known bugs / freeze points

Atomic tests for clone

CL0  Clone repo with 1 commit, 1 object       → < 2s
CL1  Clone repo with 10 commits, 50 objects   → < 5s
CL2  Clone repo with 100 commits, 500 objects → < 15s
CL3  Clone gabriel/muse (830c, 6869o)         → < 120s

Immediate Debug Plan

Step 1 — Confirm push fix is on staging

Push is freezing again with no logs. Before anything else:

# Does staging have the backend.exists() fix?
# Check if the sequential loop is gone:
ssh to EC2 and grep or check docker image tag

Or run the timing script against staging:

python3 /tmp/push_timing_test.py 500 830

If it takes > 60s, the fix did not deploy. Run bash deploy/push.sh staging.

Step 2 — Add server-side timing logs to push

Every phase needs a timestamp so we know exactly where time is spent:

_t0 = time.perf_counter()
# ... phase C
logger.info("[push] phase_c: %.3fs", time.perf_counter() - _t0)
_t0 = time.perf_counter()
# ... bulk_upsert_snapshot_entries
logger.info("[push] snapshots: %.3fs", time.perf_counter() - _t0)
# ... etc

Step 3 — Audit fetch/stream server response

Check musehub/api/routes/wire.py fetch_stream handler:

• Does it use the same streaming http.response.start → http.response.body pattern as push?
• Or does it buffer the full response before sending?

Step 4 — Run atomic tests in order

Don't test the full gabriel/muse push until XS/S/M all pass cleanly. Use /tmp/push_timing_test.py for synthetic pushes without needing a real repo.

Step 5 — Fix fetch/stream streaming if needed

If fetch buffers the response (CF 524 on large repos), apply the same fix as push: stream each O/C/E frame as an HTTP body chunk with more_body=True.

What "Done" Looks Like

All four must work without CF 524, without client-side timeouts, and with progress visible on stderr throughout — matching the experience of git push origin main.