gabriel / musehub public
fix patch phase1 task/phase1-zstd-decompress-fix #1 / 1
AI Agent gabriel · 126 days ago · May 17, 2026 · Diff

fix(phase1): decompress zstd objects from covering mpack before serving

_extract_from_mpack was reading raw bytes from the push mpack without checking the 'encoding' field. Push mpacks store objects with encoding='zstd' and compressed content, but object IDs are sha256 of *decompressed* bytes. Serving compressed bytes caused the muse client's integrity check (sha256(content) == oid) to fail, all objects to be skipped, and muse apply_mpack to leave HEAD unadvanced — resulting in 0 commits visible after every l/xl clone.

Fix: check encoding field in _extract_from_mpack; decompress zstd content before placing in mpack_hits.

Also adds: - FN-7 regression test: zstd-compressed objects in covering mpack must be decompressed before appearing in the assembled fetch mpack - _make_mpack_bytes_zstd() test helper that builds realistic push mpacks - Verbose diagnostic logging in _walk_commit_delta and wire_fetch_mpack to expose gen_range, graph hits, snap_ids, and presigned URL on every fetch — confirms server assembles 1000 commits/snapshots correctly - Restore _wait_indexed clone_dir variable (was shadowed by inline name)

sha256:56cbc136427d1e5cd1a711151b9bde9c8f7dbc65985a34761b4171a5d3ac4722 sha
sha256:5aed54e2001fe3ba0230b494791d9b38b3ff4c9b3f60ef2a66e4adf51f7927e6 parent
+39 ~4 symbols
sha256:fb71859eb956ec8aca49675f171bffd033cf433907af03b5e3a103e315cc4f7d snapshot
+39
symbols added
~4
symbols modified
0
dead code introduced
Semantic Changes 43 symbols
~ tests/test_fetch_mpack_native_phase1.py .py 39 symbols added
+ _FakeBackend class class _FakeBackend L87–127
+ __init__ method method __init__ L94–98
+ delete method async_method delete L118–119
+ exists method async_method exists L115–116
+ get method async_method get L104–106
+ get_mpack method async_method get_mpack L108–110
+ presign_get method async_method presign_get L121–122
+ put method async_method put L100–102
+ put_mpack method async_method put_mpack L112–113
+ uri_for method method uri_for L124–125
+ _index_object_in_mpack function async_function _index_object_in_mpack L183–204
+ _make_commit function async_function _make_commit L130–180
+ _make_mpack_bytes function function _make_mpack_bytes L43–53
+ _make_mpack_bytes_zstd function function _make_mpack_bytes_zstd L56–80
+ _mpack_key function function _mpack_key L83–84
+ _now function function _now L39–40
+ AsyncMock import import AsyncMock L24–24
+ AsyncSession import import AsyncSession L29–29
+ FetchNotIndexedError import import FetchNotIndexedError L33–33
+ annotations import import annotations L20–20
+ blob_id import import blob_id L31–31
+ call import import call L24–24
+ create_repo import import create_repo L34–34
+ datetime import import datetime L23–23
+ db import import db L32–32
+ fake_id import import fake_id L31–31
+ hashlib import import hashlib L22–22
+ msgpack import import msgpack L26–26
+ pg_insert import import pg_insert L28–28
+ pytest import import pytest L27–27
+ timezone import import timezone L23–23
+ wire_fetch_mpack import import wire_fetch_mpack L33–33
+ test_fn1_content_cache_served_without_mpack_get function async_function test_fn1_content_cache_served_without_mpack_get L210–253
+ test_fn2_object_served_from_mpack_no_per_object_key function async_function test_fn2_object_served_from_mpack_no_per_object_key L259–302
+ test_fn3_two_mpacks_each_fetched_once function async_function test_fn3_two_mpacks_each_fetched_once L308–353
+ test_fn4_missing_mpack_falls_back_to_per_object_get function async_function test_fn4_missing_mpack_falls_back_to_per_object_get L359–392
+ test_fn5_unindexed_object_raises_fetch_not_indexed_error function async_function test_fn5_unindexed_object_raises_fetch_not_indexed_error L398–429
+ test_fn6_mixed_sources_all_merged function async_function test_fn6_mixed_sources_all_merged L435–516
+ test_fn7_zstd_compressed_mpack_objects_decompressed function async_function test_fn7_zstd_compressed_mpack_objects_decompressed L522–576
~ musehub/services/musehub_wire.py .py 2 symbols modified
~ _walk_commit_delta
~ wire_fetch_mpack
~ tests/bench_cli.py .py 1 symbol modified
~ _wait_indexed
~ tests/test_ltier_fetch_regression.py .py 1 symbol modified
~ _stub_backend
← Older Oldest on task/phase1-zstd-decompress-fix
All commits
Newer → Latest on task/phase1-zstd-decompress-fix

0 comments

No comments yet. Be the first to start the discussion.

To add a comment, use the Muse CLI: muse hub commit comment sha256:56cbc136427d1e5cd1a711151b9bde9c8f7dbc65985a34761b4171a5d3ac4722 --body "your comment"