fix: sha256:f367ba20336a65046aa8107faa93af2ec3141bfc74dcf682b1ac927569073d9b · gabriel/muse

fix(push): chunk C frame into <=8MB batches to prevent CF stream truncation

Wall 5 root cause: the push_stream POST body contains a single 66 MB C (COMMIT_PACK) frame. Cloudflare truncates or resets the streaming body mid-frame. The server's msgpack Unpacker then misaligns, interpreting mid-frame binary bytes as a new top-level value. A random 0xDF byte (map32 format) followed by arbitrary bytes produces a garbage map length (e.g. 2,662,457,625) that exceeds max_map_len → ValueError.

Fix — two parts:

1. Client (transport.py): _frame_generator() now chunks commits+snapshots into batches of COMMIT_CHUNK_SIZE (50) commits per C frame. Each snapshot is sent exactly once (tracked via sent_snap_ids). Result: ~14 C frames of ~5-8 MB each instead of one 66 MB frame. Added MAX_C_FRAME_BYTES (8 MB) and COMMIT_CHUNK_SIZE (50) constants.

2. Server (musehub_wire.py): C frame handler changes from replace to extend for commits_wire and snapshots_wire. A single C frame still works. Multiple C frames accumulate their commits/snapshots correctly.

Tests (T19, T20 — 6 new tests, 24 total in test_presign_upload.py): - T19: truncation of large C frame produces max_map_len error (documents the failure mode); chunked frames survive individually; each chunk is below MAX_C_FRAME_BYTES - T20: server accumulates commits across multiple C frames; single C frame still works; snapshots dedup correctly

+12

symbols added

symbols modified

dead code introduced

Semantic Changes 17 symbols

~ docs/protocol/musewire-performance.md .md 3 symbols modified

~ MuseWire Protocol — GitHub-Level Performance Checklist

~ Wall log — errors hit in order, TDD items attached

~ Wall 5 — stream read error: 2662457625 exceeds max_map_len(1073741824) (ACTIVE)

~ muse/core/transport.py .py 2 symbols added, 2 symbols modified

+ COMMIT_CHUNK_SIZE variable variable COMMIT_CHUNK_SIZE L246–246

+ MAX_C_FRAME_BYTES variable variable MAX_C_FRAME_BYTES L242–242

~ HttpTransport

~ _push_stream_httpx

~ tests/test_presign_upload.py .py 10 symbols added

+ TestT19CFrameTruncation class class TestT19CFrameTruncation L548–710

+ _make_commit_dict method method _make_commit_dict L562–591

+ _make_snapshot_dict method method _make_snapshot_dict L593–603

+ test_chunked_c_frames_survive_individually method method test_chunked_c_frames_survive_individually L637–677

+ test_chunked_frames_each_below_size_limit method method test_chunked_frames_each_below_size_limit L679–710

+ test_truncated_c_frame_produces_max_map_len_error method method test_truncated_c_frame_produces_max_map_len_error L605–635

+ TestT20ServerMultiCFrame class class TestT20ServerMultiCFrame L717–784

+ test_multiple_c_frames_accumulate_commits method method test_multiple_c_frames_accumulate_commits L728–751

+ test_single_c_frame_still_works method method test_single_c_frame_still_works L753–763

+ test_snapshots_deduplicated_across_chunks method method test_snapshots_deduplicated_across_chunks L765–784

← Older Oldest on task/wall5-chunked-c-frames

All commits

Newer → Latest on task/wall5-chunked-c-frames

0 comments

No comments yet. Be the first to start the discussion.

To add a comment, use the Muse CLI: muse hub commit comment sha256:f367ba20336a65046aa8107faa93af2ec3141bfc74dcf682b1ac927569073d9b --body "your comment"