gabriel / musehub public
feat patch wire task/wire-compression-layers #1 / 1
AI Agent gabriel · 146 days ago · Apr 29, 2026 · Diff

feat(wire): add zstd + path-sort + delta compression to fetch stream

Layer 1: compress_and_encode() replaces zlib.compress() for O frames, using zstd when available with zlib fallback.

Layer 2: collect all batch results, sort by path before yielding O frames for deterministic order and better delta locality.

Layer 3: track path → (last_oid, raw_bytes) across batches; compute delta for successive file versions; send delta+zstd frame when profitable (delta smaller than full zstd frame), else fall back to full frame.

Also adds 7 batch-stream TDD tests (B1-B7) and 8 compression TDD tests (C1-C8) covering all three layers including cross-batch delta and empty fetch edge cases.

sha256:3aafd5d4ac38b8efcb6030fdefe87cd45d11922569e297d952358906be4d7700 sha
sha256:46613161da2f21e3739808952eda167629e0523f40c2d19ea35a7271a75b56e1 parent
+80 ~1 symbols
sha256:9236f8e7e87d973e843792436cabfa81d20add34c65ddc55c6ad4f093ef6b504 snapshot
+80
symbols added
~1
symbol modified
0
dead code introduced
Semantic Changes 81 symbols
~ tests/test_wire_batch_stream.py .py 42 symbols added
+ TestAllObjectsPresent class class TestAllObjectsPresent L305–349
+ test_all_unique_objects_sent_exactly_once method async_method test_all_unique_objects_sent_exactly_once L307–349
+ TestEmptyFetch class class TestEmptyFetch L396–412
+ test_empty_want_produces_header_and_end method async_method test_empty_want_produces_header_and_end L398–412
+ TestLargeSingleCommit class class TestLargeSingleCommit L419–441
+ test_single_commit_many_objects method async_method test_single_commit_many_objects L421–441
+ TestMultiBatchCompletion class class TestMultiBatchCompletion L258–298
+ test_all_commits_delivered_across_batches method async_method test_all_commits_delivered_across_batches L260–298
+ TestNoDuplicateObjects class class TestNoDuplicateObjects L228–251
+ test_no_object_sent_twice method async_method test_no_object_sent_twice L230–251
+ TestObjectsPrecedeCommits class class TestObjectsPrecedeCommits L190–221
+ test_objects_arrive_before_their_commit method async_method test_objects_arrive_before_their_commit L192–221
+ TestSharedObjectsSentOnce class class TestSharedObjectsSentOnce L356–389
+ test_shared_snapshot_objects_not_duplicated method async_method test_shared_snapshot_objects_not_duplicated L358–389
+ _c_frame_commits function function _c_frame_commits L164–171
+ _collect_frames function async_function _collect_frames L152–156
+ _make_commit function async_function _make_commit L121–142
+ _make_objects function async_function _make_objects L65–99
+ _make_snapshot function async_function _make_snapshot L102–118
+ _now function function _now L61–62
+ _o_frame_oids function function _o_frame_oids L159–161
+ _snapshot_oids function function _snapshot_oids L174–183
+ _uid function function _uid L57–58
+ _unpack_stream function function _unpack_stream L145–149
+ Any import import Any L35–35
+ AsyncSession import import AsyncSession L39–39
+ SFRAME_COMMIT_PACK import import SFRAME_COMMIT_PACK L43–43
+ SFRAME_END import import SFRAME_END L43–43
+ SFRAME_HEADER import import SFRAME_HEADER L43–43
+ SFRAME_OBJECT import import SFRAME_OBJECT L43–43
+ WireFetchRequest import import WireFetchRequest L43–43
+ annotations import import annotations L30–30
+ asyncio import import asyncio L32–32
+ blob_id import import blob_id L41–41
+ create_repo import import create_repo L50–50
+ datetime import import datetime L34–34
+ db import import db L42–42
+ fake_id import import fake_id L41–41
+ msgpack import import msgpack L37–37
+ pytest import import pytest L38–38
+ timezone import import timezone L34–34
+ zlib import import zlib L33–33
~ tests/test_wire_compression.py .py 35 symbols added
+ _collect_frames function async_function _collect_frames L133–137
+ _make_commit function async_function _make_commit L110–130
+ _make_object function async_function _make_object L59–88
+ _make_snapshot function async_function _make_snapshot L91–107
+ _now function function _now L55–56
+ _o_frames function function _o_frames L140–141
+ _realistic_source function function _realistic_source L144–169
+ _uid function function _uid L51–52
+ AsyncSession import import AsyncSession L32–32
+ SFRAME_COMMIT_PACK import import SFRAME_COMMIT_PACK L37–37
+ SFRAME_END import import SFRAME_END L37–37
+ SFRAME_HEADER import import SFRAME_HEADER L37–37
+ SFRAME_OBJECT import import SFRAME_OBJECT L37–37
+ WireFetchRequest import import WireFetchRequest L37–37
+ ZSTD_AVAILABLE import import ZSTD_AVAILABLE L34–34
+ annotations import import annotations L25–25
+ apply_delta import import apply_delta L34–34
+ blob_id import import blob_id L35–35
+ compute_delta import import compute_delta L34–34
+ create_repo import import create_repo L44–44
+ datetime import import datetime L28–28
+ db import import db L36–36
+ fake_id import import fake_id L35–35
+ msgpack import import msgpack L30–30
+ pytest import import pytest L31–31
+ timezone import import timezone L28–28
+ zlib import import zlib L27–27
+ test_c1_server_uses_zstd_encoding function async_function test_c1_server_uses_zstd_encoding L178–195
+ test_c2_zstd_frame_decompresses_correctly function async_function test_c2_zstd_frame_decompresses_correctly L204–224
+ test_c3_o_frames_sorted_by_path function async_function test_c3_o_frames_sorted_by_path L232–253
+ test_c4_successive_versions_use_delta_frame function async_function test_c4_successive_versions_use_delta_frame L261–298
+ test_c5_delta_frame_reconstructs_correctly function async_function test_c5_delta_frame_reconstructs_correctly L306–349
+ test_c6_unprofitable_delta_uses_full_frame function async_function test_c6_unprofitable_delta_uses_full_frame L357–389
+ test_c7_full_clone_correct_object_ids function async_function test_c7_full_clone_correct_object_ids L397–427
+ test_c8_cross_batch_delta function async_function test_c8_cross_batch_delta L435–481
~ musehub/services/musehub_wire.py .py 3 symbols added, 1 symbol modified
+ ZSTD_AVAILABLE import import ZSTD_AVAILABLE L70–70
+ compress_and_encode import import compress_and_encode L70–70
+ compute_delta import import compute_delta L70–70
~ wire_fetch_stream
← Older Oldest on task/wire-compression-layers
All commits
Newer → Latest on task/wire-compression-layers

0 comments

No comments yet. Be the first to start the discussion.

To add a comment, use the Muse CLI: muse hub commit comment sha256:3aafd5d4ac38b8efcb6030fdefe87cd45d11922569e297d952358906be4d7700 --body "your comment"