gabriel / musehub public
feat BREAKING wire feat/streaming-push #1 / 1
gabriel · 161 days ago · Apr 16, 2026 · Diff

feat(wire): MWP v2 streaming push — 8-tier TDD tests, 43/43 green

Replace five-phase MWP v1 push (filter-objects → presign → push/objects → object-pack → confirm → push) with a single bidirectional msgpack stream over POST /{owner}/{slug}/push/stream — the direct equivalent of git-receive-pack over HTTP smart transport.

Wire protocol (application/x-muse-packstream): Client → Server: HEADER | OBJECT* | COMMIT_PACK | END Server → Client: PROGRESS* | (ERROR | RESULT)

Frame constants added to musehub/models/wire.py: SFRAME_HEADER / OBJECT / COMMIT_PACK / END / PROGRESS / ERROR / RESULT STREAM_MIME_TYPE, STREAM_MAX_OBJECTS, STREAM_MAX_COMMITS, STREAM_MAX_OBJECT_WIRE_BYTES, STREAM_OBJECT_BATCH_SIZE

Service (musehub_wire.wire_push_stream): - Auth + repo lookup before reading any OBJECT frames - Parallel R2 puts batched by STREAM_OBJECT_BATCH_SIZE (64), bounded by _R2_PUT_SEM (100 slots) - Inline SHA-256 verification and polyglot file check per object - Topological commit sort, parent-link validation, FF check - Snapshot bulk-upsert, branch pointer update, background job enqueue - Per-stream object/commit caps prevent unbounded memory use

Route (wire.push_stream): - Pre-buffers request body via await request.body() to avoid httpx ASGI transport deadlock in tests (request.stream() inside StreamingResponse is bidirectional and blocks the test transport) - Old MWP v1 endpoints deleted: /push, /filter-objects, /presign-objects, /push/objects, /push/object-pack, /push/objects/confirm

Tests (tests/test_wire_push_stream.py — 43 tests, 8 tiers): T1 Unit — frame codec helpers (pack/unpack round-trip) T2 Unit — server protocol state machine (pure dispatch, no DB/R2) T3 Comp — object validation (hash check, size limits, enc modes) T4 Comp — commit-pack validation (schema, limits, signature gate) T5 Svc — wire_push_stream() against in-memory stubs T6 Intg — service against real test DB + local storage backend T7 Route — POST /push/stream via ASGI test client T8 E2E — full push → GET /refs + fetch/objects round-trip

sha256:8fd05fc8cbe900ab2155fdbe8a2ebed9ed78f971a3ec2f8abb67334f03571fa2 sha
sha256:17031e20563ca94fc9892d70e63df3d435f53da7a868909e49e35ad4d494237f parent
+129 ~11 −17 symbols
sha256:d04a105649e4d54bfda193dba360b50b8197e68d21ec6189b08f03fcb5fb1ae3 snapshot
+129
symbols added
~11
symbols modified
−17
symbols removed
0
dead code introduced
Semantic Changes 157 symbols
~ tests/test_wire_push_stream.py .py 95 symbols added
+ TestT1FrameCodec class class TestT1FrameCodec L170–233
+ test_client_server_frame_tags_are_disjoint method method test_client_server_frame_tags_are_disjoint L183–188
+ test_commit_pack_frame_round_trips method method test_commit_pack_frame_round_trips L209–215
+ test_end_frame_round_trips method method test_end_frame_round_trips L217–219
+ test_header_frame_round_trips method method test_header_frame_round_trips L190–198
+ test_multiple_frames_concatenated_parse_correctly method method test_multiple_frames_concatenated_parse_correctly L221–228
+ test_object_frame_round_trips method method test_object_frame_round_trips L200–207
+ test_sframe_constants_are_single_chars method method test_sframe_constants_are_single_chars L176–181
+ test_stream_limits_are_positive method method test_stream_limits_are_positive L230–233
+ test_stream_mime_type method method test_stream_mime_type L173–174
+ TestT2ServerHelpers class class TestT2ServerHelpers L240–278
+ _import_helpers method method _import_helpers L243–245
+ test_err_default_code_is_400 method method test_err_default_code_is_400 L261–264
+ test_err_encodes_error_frame_with_code method method test_err_encodes_error_frame_with_code L253–259
+ test_prog_encodes_progress_frame method method test_prog_encodes_progress_frame L247–251
+ test_result_failure_encodes_correctly method method test_result_failure_encodes_correctly L275–278
+ test_result_ok_encodes_correctly method method test_result_ok_encodes_correctly L266–273
+ TestT3ObjectValidation class class TestT3ObjectValidation L285–329
+ _oid_for method method _oid_for L288–289
+ test_large_object_frame_exceeds_limit_is_detectable method method test_large_object_frame_exceeds_limit_is_detectable L318–323
+ test_object_frame_with_empty_content_is_encodable method method test_object_frame_with_empty_content_is_encodable L311–316
+ test_raw_encoding_preserves_exact_bytes method method test_raw_encoding_preserves_exact_bytes L325–329
+ test_sha256_oid_format method method test_sha256_oid_format L291–295
+ test_wire_size_limit_constant_is_reasonable method method test_wire_size_limit_constant_is_reasonable L307–309
+ test_zlib_object_frame_decompresses_correctly method method test_zlib_object_frame_decompresses_correctly L297–305
+ TestT4CommitPackValidation class class TestT4CommitPackValidation L336–374
+ test_commit_pack_frame_encodes_many_commits method method test_commit_pack_frame_encodes_many_commits L361–364
+ test_commit_pack_limit_constant method method test_commit_pack_limit_constant L358–359
+ test_commit_without_required_fields_raises method method test_commit_without_required_fields_raises L345–349
+ test_minimal_commit_passes_model_validate method method test_minimal_commit_passes_model_validate L339–343
+ test_signed_commit_has_signature_fields method method test_signed_commit_has_signature_fields L366–374
+ test_snapshot_passes_model_validate method method test_snapshot_passes_model_validate L351–356
+ TestT5ServiceStream class class TestT5ServiceStream L381–499
+ stub_backend method method stub_backend L390–399
+ stub_session method method stub_session L402–407
+ test_missing_commit_pack_yields_error method async_method test_missing_commit_pack_yields_error L448–461
+ test_missing_end_frame_yields_error method async_method test_missing_end_frame_yields_error L426–445
+ test_missing_header_yields_error method async_method test_missing_header_yields_error L410–423
+ test_object_hash_mismatch_yields_error method async_method test_object_hash_mismatch_yields_error L464–479
+ test_progress_frames_emitted_on_header method async_method test_progress_frames_emitted_on_header L482–499
+ _body_iter function async_function _body_iter L160–163
+ _collect_frames function async_function _collect_frames L152–157
+ _commit_pack_frame function function _commit_pack_frame L108–113
+ _end_frame function function _end_frame L116–117
+ _header_frame function function _header_frame L79–95
+ _make_commit function function _make_commit L120–141
+ _make_obj_bytes function function _make_obj_bytes L73–76
+ _make_repo function async_function _make_repo L530–552
+ _make_snapshot function function _make_snapshot L144–149
+ _object_frame function function _object_frame L98–105
+ _pack function function _pack L53–55
+ _sha256_oid function function _sha256_oid L65–66
+ _stub_r2_backend function function _stub_r2_backend L506–527
+ _unpack_all function function _unpack_all L58–62
+ _utc function function _utc L69–70
+ Any import import Any L25–25
+ AsyncClient import import AsyncClient L30–30
+ AsyncGenerator import import AsyncGenerator L23–23
+ AsyncIterator import import AsyncIterator L23–23
+ AsyncMock import import AsyncMock L26–26
+ AsyncSession import import AsyncSession L31–31
+ MagicMock import import MagicMock L26–26
+ SFRAME_COMMIT_PACK import import SFRAME_COMMIT_PACK L33–33
+ SFRAME_END import import SFRAME_END L33–33
+ SFRAME_ERROR import import SFRAME_ERROR L33–33
+ SFRAME_HEADER import import SFRAME_HEADER L33–33
+ SFRAME_OBJECT import import SFRAME_OBJECT L33–33
+ SFRAME_PROGRESS import import SFRAME_PROGRESS L33–33
+ SFRAME_RESULT import import SFRAME_RESULT L33–33
+ STREAM_MAX_COMMITS import import STREAM_MAX_COMMITS L33–33
+ STREAM_MAX_OBJECTS import import STREAM_MAX_OBJECTS L33–33
+ STREAM_MAX_OBJECT_WIRE_BYTES import import STREAM_MAX_OBJECT_WIRE_BYTES L33–33
+ STREAM_MIME_TYPE import import STREAM_MIME_TYPE L33–33
+ annotations import import annotations L18–18
+ asyncio import import asyncio L20–20
+ create_repo import import create_repo L46–46
+ datetime import import datetime L24–24
+ hashlib import import hashlib L21–21
+ msgpack import import msgpack L28–28
+ patch import import patch L26–26
+ pytest import import pytest L29–29
+ timezone import import timezone L24–24
+ zlib import import zlib L22–22
+ test_t6_force_push_advances_branch function async_function test_t6_force_push_advances_branch L644–676
+ test_t6_push_single_commit_no_objects function async_function test_t6_push_single_commit_no_objects L556–577
+ test_t6_push_with_object function async_function test_t6_push_with_object L581–608
+ test_t6_push_zlib_compressed_object function async_function test_t6_push_zlib_compressed_object L612–640
+ test_t7_old_push_endpoints_deleted function async_function test_t7_old_push_endpoints_deleted L771–789
+ test_t7_push_stream_404_for_missing_repo function async_function test_t7_push_stream_404_for_missing_repo L752–767
+ test_t7_push_stream_requires_auth function async_function test_t7_push_stream_requires_auth L731–748
+ test_t7_push_stream_response_contains_result_frame function async_function test_t7_push_stream_response_contains_result_frame L707–727
+ test_t7_push_stream_returns_200_with_packstream_content_type function async_function test_t7_push_stream_returns_200_with_packstream_content_type L684–703
+ test_t8_push_chain_of_commits_parent_linking function async_function test_t8_push_chain_of_commits_parent_linking L877–913
+ test_t8_push_then_refs_show_commit function async_function test_t8_push_then_refs_show_commit L797–831
+ test_t8_push_with_objects_then_fetch_objects function async_function test_t8_push_with_objects_then_fetch_objects L835–873
~ musehub/api/routes/wire.py .py 3 symbols added, 17 symbols removed, 11 symbols modified
− filter_objects function async_function filter_objects L440–465
− WireConfirmObjectsRequest import import WireConfirmObjectsRequest L54–54
− WireObjectPackRequest import import WireObjectPackRequest L54–54
− WireObjectsRequest import import WireObjectsRequest L54–54
− WirePresignRequest import import WirePresignRequest L54–54
− WirePushRequest import import WirePushRequest L54–54
− wire_confirm_objects import import wire_confirm_objects L69–69
− wire_filter_objects import import wire_filter_objects L69–69
− wire_presign import import wire_presign L69–69
− wire_push import import wire_push L69–69
− wire_push_object_pack import import wire_push_object_pack L69–69
− wire_push_objects import import wire_push_objects L69–69
− presign_objects function async_function presign_objects L474–502
− push function async_function push L769–799
− push_object_pack function async_function push_object_pack L578–621
− push_objects function async_function push_objects L547–569
− push_objects_confirm function async_function push_objects_confirm L734–760
+ STREAM_MIME_TYPE import import STREAM_MIME_TYPE L54–54
+ wire_push_stream import import wire_push_stream L65–65
+ push_stream function async_function push_stream L430–488
~ ObjectHashMismatch
~ PolyglotObjectError
~ WireFetchRequest
~ WireFilterRequest
~ WireNegotiateRequest
~ wire_fetch
~ wire_fetch_objects
~ wire_negotiate
~ wire_refs
~ wire_repair_object
~ wire_repair_snapshot
~ musehub/models/wire.py .py 12 symbols added
+ SFRAME_COMMIT_PACK variable variable SFRAME_COMMIT_PACK L573–573
+ SFRAME_END variable variable SFRAME_END L574–574
+ SFRAME_ERROR variable variable SFRAME_ERROR L578–578
+ SFRAME_HEADER variable variable SFRAME_HEADER L571–571
+ SFRAME_OBJECT variable variable SFRAME_OBJECT L572–572
+ SFRAME_PROGRESS variable variable SFRAME_PROGRESS L577–577
+ SFRAME_RESULT variable variable SFRAME_RESULT L579–579
+ STREAM_MAX_COMMITS variable variable STREAM_MAX_COMMITS L588–588
+ STREAM_MAX_OBJECTS variable variable STREAM_MAX_OBJECTS L587–587
+ STREAM_MAX_OBJECT_WIRE_BYTES variable variable STREAM_MAX_OBJECT_WIRE_BYTES L592–592
+ STREAM_MIME_TYPE variable variable STREAM_MIME_TYPE L568–568
+ STREAM_OBJECT_BATCH_SIZE variable variable STREAM_OBJECT_BATCH_SIZE L596–596
~ musehub/services/musehub_wire.py .py 19 symbols added
+ _err function function _err L1853–1855
+ _prog function function _prog L1848–1850
+ _result function function _result L1858–1860
+ _sp function function _sp L1843–1845
+ AsyncGenerator import import AsyncGenerator L1832–1832
+ AsyncIterator import import AsyncIterator L1832–1832
+ SFRAME_COMMIT_PACK import import SFRAME_COMMIT_PACK L1835–1835
+ SFRAME_END import import SFRAME_END L1835–1835
+ SFRAME_ERROR import import SFRAME_ERROR L1835–1835
+ SFRAME_HEADER import import SFRAME_HEADER L1835–1835
+ SFRAME_OBJECT import import SFRAME_OBJECT L1835–1835
+ SFRAME_PROGRESS import import SFRAME_PROGRESS L1835–1835
+ SFRAME_RESULT import import SFRAME_RESULT L1835–1835
+ STREAM_MAX_COMMITS import import STREAM_MAX_COMMITS L1835–1835
+ STREAM_MAX_OBJECTS import import STREAM_MAX_OBJECTS L1835–1835
+ STREAM_MAX_OBJECT_WIRE_BYTES import import STREAM_MAX_OBJECT_WIRE_BYTES L1835–1835
+ STREAM_OBJECT_BATCH_SIZE import import STREAM_OBJECT_BATCH_SIZE L1835–1835
+ _msgpack import import _msgpack L1833–1833
+ wire_push_stream function async_function wire_push_stream L1863–2467
← Older Oldest on feat/streaming-push
All commits
Newer → Latest on feat/streaming-push

0 comments

No comments yet. Be the first to start the discussion.

To add a comment, use the Muse CLI: muse hub commit comment sha256:8fd05fc8cbe900ab2155fdbe8a2ebed9ed78f971a3ec2f8abb67334f03571fa2 --body "your comment"