# Push 422 Root Cause — Have-Anchor Object Gap

## Symptom

`muse push staging dev` returns:

```
push rejected: 824 snapshot object(s) not in push bundle or storage —
missing: sha256:00145bb1bd38, sha256:002028f94a3b, sha256:0059ccbc0d19…
```

The same 824 OIDs are missing on every push attempt.  The error is deterministic.

## Smoking Gun

The push log says:

```
[stream] walk: … (683 commits, 4789 objects)
[stream] loaded 3965 objects …
```

**4789 − 3965 = 824.**  Exactly the number of missing objects.  The `have`
computation excluded 824 objects from the wire bundle, and the server reports
exactly those 824 as absent from its DB.

## Root Cause

The push command computes `have` from the target remote's live branch heads
(GET /refs), filtered to commits that exist locally:

```python
candidate_have = list(remote_branch_heads.values())
have = [c for c in candidate_have
        if c != local_head and _is_valid_commit_id(c) and commit_exists(root, c)]
```

If staging has a `main` branch whose head commit exists in the local repo, it
becomes a `have` anchor.  The object walk stops at `main`'s head, excluding the
824 objects reachable from `main` but not from the new `dev` commits.

The integrity check on the final batch requires ALL snapshot-referenced objects
to be in the server DB.  Those 824 excluded objects are referenced by `dev`
snapshot manifests (shared with `main`), but `main` was never successfully
pushed to staging — so the server DB has no rows for them.

## Invariant

```
MB-6: A push of branch B that excludes objects via `have=[A_head]`
      SUCCEEDS only if the server already has those objects in its DB
      (from a prior push of branch A or any other path).

      If the server does NOT have those objects, the integrity check
      correctly rejects the push with 422.
```

This invariant is already enforced by the server (see MB-3).  The gap is
on the client side: the walk assumes the server has branch-A objects, but
branch A may never have been pushed to that remote.

## Immediate Fix

Push `main` to staging before pushing `dev`.  This puts the 824 shared
objects into the server DB, so the next `dev` push finds them.

```bash
muse push staging main
muse push staging dev
```

## Long-Term Fix

The `have` anchor for object exclusion should only include remote branch heads
whose objects are verifiably on the server — i.e., commits that were previously
pushed to THIS specific remote, not just commits that exist locally and whose
ID happens to match a remote branch head.

Options:
- Track per-remote "object-coverage" in local tracking refs (expensive)
- On first push to a remote or force push, widen the bundle to include all
  ancestors regardless of `have` anchors (safe default)
- Server: expose an object-presence check endpoint so the client can verify
  before excluding objects from the bundle

## Timeline

| Session | What happened |
|---------|--------------|
| Previous session | SSL BAD_RECORD_MAC on MWP POST after 216 R2 PUTs — retry loop not firing (wrong exception type: `_httpx.TransportError` vs muse `TransportError`) |
| Previous session | Fixed retry exception type; added 0.5s cooldown after R2 phase |
| This session | Scaled cooldown to `min(10s, n_large_objects * 0.05s)`; bumped retries 3→5 |
| This session | `presign_and_register_objects()` written + 4 TDD tests (PI-1–PI-4) |
| This session | MB-1–MB-5 written + passing — confirms non-final batch objects land in DB |
| This session | **Root cause identified**: 4789−3965=824; `main` never pushed to staging |

## Tests

| ID | File | What it proves |
|----|------|---------------|
| PI-1 | `test_wire_presign_integrity.py` | `presign_and_register_objects` inserts `storage_uri=pending` row |
| PI-2 | `test_wire_presign_integrity.py` | push succeeds when snapshot refs presigned (pre-registered) object |
| PI-3 | `test_wire_presign_integrity.py` | `presign_and_register_objects` is idempotent |
| PI-4 | `test_wire_presign_integrity.py` | creates `musehub_object_refs` row |
| MB-1 | `test_wire_multibatch_push.py` | non-final batch objects land in DB after request |
| MB-2 | `test_wire_multibatch_push.py` | final batch succeeds when earlier batches stored objects |
| MB-3 | `test_wire_multibatch_push.py` | final batch fails when referenced objects never sent (negative baseline) |
| MB-4 | `test_wire_multibatch_push.py` | presigned objects pass final batch integrity check |
| MB-5 | `test_wire_multibatch_push.py` | rolled-back earlier batch causes final failure (causal proof) |
| **MB-6** | `test_wire_multibatch_push.py` | **final batch fails when have-excluded objects are absent from DB** |
| **MB-7** | `test_wire_multibatch_push.py` | **final batch succeeds when have-excluded objects are pre-pushed (push main → dev)** |