# Muse Wire Protocol — Four Verbs: Status, Bugs, and Fix Plan

Per the MWP spec at `/muse/wire`. Each verb maps directly to a Git smart HTTP analogue.
This document is the authoritative record of what works, what breaks, and what must be fixed.

---

## How MWP Maps to Git

| Git | Muse | Endpoint |
|-----|------|----------|
| `GET /info/refs?service=git-upload-pack` | refs preflight | `GET /{owner}/{slug}/refs` |
| `POST /git-receive-pack` | push | `POST /{owner}/{slug}/push/stream` |
| `POST /git-upload-pack` | fetch/pull/clone | `POST /{owner}/{slug}/fetch/stream` |
| sideband multiplexing | PROGRESS frames | interleaved on response body |

Every verb is a single HTTP round trip after the refs preflight. No exotic infrastructure.

---

## Scale Ladder (used throughout this doc)

| Label | Objects | Commits | Representative repo |
|-------|---------|---------|---------------------|
| XS | 1–10 | 1–5 | brand new repo |
| S | 10–500 | 5–100 | small project |
| M | 500–2000 | 100–400 | active project |
| L | 2000–7000 | 400–900 | gabriel/muse |
| XL | 7000–50000 | 900–5000 | large monorepo |

---

## 1. Push — `POST /{owner}/{slug}/push/stream`

### What the spec says
Client sends `[H][O…][OC…][C][E]`. Server processes frames as they arrive, writes objects
to R2 in parallel batches, and streams `[P…][R]` back on the response body **before the
upload is complete**. Equiv: `git-receive-pack`.

### What the client does (`muse/core/transport.py`)
- Client batches objects in chunks of 500 (`CHUNK_OBJECTS = 500`)
- Each chunk is one POST to `/push/stream`
- Non-final chunks: objects only, no commits
- Final chunk: remaining objects + all 830 commits + all snapshots
- Auth: MSign header, empty-body hash
- HTTP: `http2=False`, `timeout=300s`
- Response parsed with `msgpack.Unpacker` via `async with client.stream()` + `aiter_bytes()`

### What the server does (`musehub/services/musehub_wire.py`)
1. Auth check
2. Read H frame
3. Read O/OC frames → flush to R2 in parallel batches (100-slot semaphore)
4. Read C frame → parse commits + snapshots
5. Read E frame → count cross-check
6. Phase C: referential integrity (bulk SELECT, no per-object exists() call after fix)
7. Quota check
8. Signature check
9. `bulk_upsert_snapshot_entries()` → one INSERT
10. `_topological_sort()` → build ordered commit list
11. Bulk INSERT commits
12. Branch pointer update (SELECT FOR UPDATE)
13. Yield R frame

### Known bugs / freeze points

| # | Symptom | Root cause | Status |
|---|---------|-----------|--------|
| P1 | CF 524 on large repos | `_WireResponse` buffered all frames before sending first byte | **Fixed** — server now streams |
| P2 | Freeze after "parsing commits" | Phase C called `backend.exists()` per object — 6500 sequential calls = 65s silence | **Fixed** — replaced with one bulk SELECT |
| P3 | Push freezes with no logs | **Unknown — current blocker** | **Open** |
| P4 | Garbled terminal output | P frames printed with `end="\r"`, overwritten by next batch progress | Fixed in client |

### Push P3 — current freeze: what we need to find out

Push froze with zero terminal output. That means the freeze is happening **before** the
first P frame reaches the client. Candidates in order of likelihood:

1. **Phase C integrity check is still slow** — need to confirm the `backend.exists()` fix
   actually deployed to staging. Check staging server logs for the batch that froze.
2. **`bulk_upsert_snapshot_entries()`** — 830 snapshots × manifests with up to 6869
   entries each. One big INSERT but could be enormous if manifests are fully expanded.
3. **`_topological_sort()`** — pure Python DAG sort of 830 nodes. Should be fast but
   worth timing.
4. **SELECT FOR UPDATE on branch row** — if a prior failed push left a lock on the branch.
5. **R2 write semaphore deadlock** — if a prior request is holding slots and the semaphore
   was never released.

### Atomic tests for push (must all pass before scaling up)

| Test | Scale | Body | first_byte | total | Result |
|------|-------|------|-----------|-------|--------|
| T0: 1obj+0commits | XS | 1.1 KB | 100ms | 188ms | ✅ localhost |
| T1: 0obj+1commit | XS | 0.8 KB | 24ms | 98ms | ✅ localhost |
| T2: 1obj+1commit | XS | 1.6 KB | 30ms | 142ms | ✅ localhost |
| T3: 10obj+5commits | XS | 10.4 KB | 36ms | 224ms | ✅ localhost |
| T4: 500obj+0commits | S | 360 KB | 74ms | 3.2s | ✅ localhost |
| T5: 500obj+100commits | M | 418 KB | 75ms | 3.7s | ✅ localhost |
| T6: 500obj+830commits | L (final batch) | 846 KB | 70ms | 9.5s | ✅ localhost |

All push atomic tests pass on localhost. First byte arrives in < 100ms at all scales (server streams immediately). T6 (830 commits) completes in 9.5s — well inside any proxy timeout.

---

## 2. Fetch — `POST /{owner}/{slug}/fetch/stream`

### What the spec says
Client sends `{"want": [...], "have": [...]}` msgpack body. Server walks DAG from wanted
commits, stops at `have` commits, and streams back `[H][O…][C][E]` — the same frame
format as push but server→client. Equiv: `git-upload-pack`.

### What the client does (`muse/core/transport.py`)
- Calls `GET /refs` to get remote branch heads
- Calls `POST /fetch/stream` with want/have lists
- Reads response as MWP frame stream
- Writes received objects to local store
- Writes received commits to local DB

### What the server does (`musehub/services/musehub_wire.py` — fetch side)
1. Repo lookup
2. One bulk JOIN (commits LEFT OUTER JOIN snapshots) — single round-trip for all commit + manifest data
3. BFS in Python from `want`, stop at `have`
4. Branch heads SELECT
5. Object metadata bulk SELECT
6. Yield H frame (first byte)
7. O frames via `asyncio.as_completed` (32-slot R2 concurrency)
8. C frame (commits + snapshots)
9. E frame

### Known bugs / freeze points — fixed

| # | Symptom | Root cause | Status |
|---|---------|-----------|--------|
| F1 | Fetch hangs or 524s on large repos | Server buffers full response — **not true**, uses `StreamingResponse` + async generator | **Audited — streaming confirmed** |
| F2 | 3504ms first-byte on 830-commit fetch | BFS was O(depth) sequential DB queries (one per commit level) | **Fixed** — one bulk SELECT + Python BFS |
| F3 | Double-snapshot query | `get_snapshot_manifests_batch` re-queried rows already fetched | **Fixed** — decode directly from JOIN result |
| F4 | Silent parent chain loss in test script | Test sent `parent_ids: [...]` but `WireCommit` expects `parent_commit_id` (Pydantic `extra=ignore` dropped it silently) | **Fixed** — test now sends `parent_commit_id` |

### Pre-H-frame timing profile (830 commits, 500 objects, localhost)

| Phase | Time | Notes |
|-------|------|-------|
| repo lookup | 17ms | |
| commits+snapshots JOIN (830 rows each) | 281ms | single round-trip |
| BFS in Python + build dicts | 113ms | includes msgpack decode for 830 blobs |
| branch heads | 3ms | |
| object metadata SELECT (500 rows) | 129ms | |
| **total pre-H** | **545ms** | H frame = first byte to client |

### Atomic tests for fetch

| Test | Scale | first_byte | total | Result |
|------|-------|-----------|-------|--------|
| F0: clone 830c+500o (want=head, have=[]) | L | 570ms | 1347ms | ✅ localhost |

### What was verified

1. ✅ `fetch/stream` uses `StreamingResponse` wrapping an async generator — correctly streams
2. ✅ No sequential per-object calls — all bulk queries
3. ✅ Client parses with `MPackStreamReader` (streaming Unpacker) — not buffered
4. ✅ `parent_commit_id` correctly preserved through push → DB → fetch → client
5. ⏳ F1–F4 incremental fetch tests — not yet run

---

## 3. Pull — `muse pull <remote> <branch>`

### What pull does
Pull = `GET /refs` + `POST /fetch/stream` + local merge. It is fetch under the hood.
Every fetch bug is a pull bug. There is no separate pull endpoint.

### Client flow (`muse/cli/commands/pull.py` or similar)
1. `GET /{owner}/{slug}/refs` → get remote head for branch
2. Compare remote head to local head → compute want/have
3. `POST /fetch/stream` with want=remote_head, have=local_commits
4. Apply received objects and commits locally
5. Advance local branch pointer

### Known bugs / freeze points

Same as Fetch above. Additionally:
- If `have` list is incomplete (client sends too few commits), server sends more objects
  than necessary → wasted bandwidth at scale
- If negotiation is skipped, `have=[]` and server sends the entire repo history

### Atomic tests for pull

```
PL0  Pull into empty local repo (≡ clone)     → same as F3
PL1  Pull 1 new commit after existing history → < 3s, only 1 commit + its objects
PL2  Pull 10 new commits                      → < 5s
PL3  Pull after 400-commit divergence         → correct subset of objects only
```

---

## 4. Clone — `muse clone <url>`

### What clone does
Clone = create empty local repo + pull everything. Uses `POST /fetch/stream` with
`want=[remote_branch_head]` and `have=[]`. No negotiation possible on empty repo.

### Client flow
1. Create local repo
2. `GET /{owner}/{slug}/refs` → get all branch heads
3. For each branch: `POST /fetch/stream` with want=head, have=[]
4. Write all received objects and commits locally
5. Set local branch pointers

### Known bugs / freeze points

| # | Symptom | Root cause | Status |
|---|---------|-----------|--------|
| C1 | Clone hangs on large repos | Fetch streaming issue (see F1) | **Blocked on fetch fix** |
| C2 | `have=[]` sends full repo history | Expected — no negotiation possible on empty repo | Not a bug |

### Atomic tests for clone

```
CL0  Clone repo with 1 commit, 1 object       → < 2s
CL1  Clone repo with 10 commits, 50 objects   → < 5s
CL2  Clone repo with 100 commits, 500 objects → < 15s
CL3  Clone gabriel/muse (830c, 6869o)         → < 120s
```

---

## Immediate Debug Plan

### Step 1 — Confirm push fix is on staging

Push is freezing again with no logs. Before anything else:

```bash
# Does staging have the backend.exists() fix?
# Check if the sequential loop is gone:
ssh to EC2 and grep or check docker image tag
```

Or run the timing script against staging:
```bash
python3 /tmp/push_timing_test.py 500 830
```
If it takes > 60s, the fix did not deploy. Run `bash deploy/push.sh staging`.

### Step 2 — Add server-side timing logs to push

Every phase needs a timestamp so we know exactly where time is spent:

```python
_t0 = time.perf_counter()
# ... phase C
logger.info("[push] phase_c: %.3fs", time.perf_counter() - _t0)
_t0 = time.perf_counter()
# ... bulk_upsert_snapshot_entries
logger.info("[push] snapshots: %.3fs", time.perf_counter() - _t0)
# ... etc
```

### Step 3 — Audit fetch/stream server response

Check `musehub/api/routes/wire.py` `fetch_stream` handler:
- Does it use the same streaming `http.response.start` → `http.response.body` pattern as push?
- Or does it buffer the full response before sending?

### Step 4 — Run atomic tests in order

Don't test the full gabriel/muse push until XS/S/M all pass cleanly.
Use `/tmp/push_timing_test.py` for synthetic pushes without needing a real repo.

### Step 5 — Fix fetch/stream streaming if needed

If fetch buffers the response (CF 524 on large repos), apply the same fix as push:
stream each O/C/E frame as an HTTP body chunk with `more_body=True`.

---

## What "Done" Looks Like

| Verb | Success criterion |
|------|-------------------|
| push | `gabriel/muse` (830c, 6869o) pushes in < 120s with visible progress, no freezes |
| fetch | `muse fetch staging dev` after 0 local commits completes in < 120s |
| pull | Incremental pull of 10 new commits completes in < 5s |
| clone | `muse clone` of `gabriel/muse` completes in < 120s |

All four must work without CF 524, without client-side timeouts, and with progress
visible on stderr throughout — matching the experience of `git push origin main`.