gabriel / musehub public
fix patch task/wire-clone-fix #1 / 1
gabriel · 163 days ago · Apr 14, 2026 · Diff

fix: wire clone — all snapshots + bounded R2 concurrency

Two fixes for muse clone correctness and speed:

1. wire_fetch returns ALL snapshots (not just tip) - The 'lazy fetch' comment was aspirational — never implemented - muse verify walks all commits and fails on every missing snapshot - Fix: collect all snapshot IDs from all returned commits and batch-load - Snapshot manifests are small (path→hash) so sending all is safe

2. _pack_objects_response bounded to 32 concurrent R2 GETs - Unbounded asyncio.gather on 665 objects floods R2 → rate limits + OOM - Fix: asyncio.Semaphore(32) caps concurrent backend.get() calls - All objects still returned; peak memory and R2 load stay bounded

TDD: 8 tests covering both behaviours, all green.

sha256:cc96d33afd14450354ddf00bad3f1996c588aee01c3fc7c5f2d359562afc9c2d sha
sha256:7180baefd59438921a889763ed9277301f75f05249e9ea49963f3919fff5a1fe parent
+42 ~2 symbols
sha256:6a0e77075c7f475bb524897a6496d3391725e98e41612ce8168948449337d798 snapshot
+42
symbols added
~2
symbols modified
0
dead code introduced
Semantic Changes 44 symbols
~ tests/test_wire_fetch_all_snapshots.py .py 18 symbols added
+ _mp function function _mp L37–38
+ _push_commit_chain function async_function _push_commit_chain L49–107
+ _sha256_oid function function _sha256_oid L45–46
+ _utc function function _utc L41–42
+ AsyncClient import import AsyncClient L26–26
+ AsyncSession import import AsyncSession L27–27
+ StrDict import import StrDict L30–30
+ annotations import import annotations L18–18
+ datetime import import datetime L22–22
+ factory_create_repo import import factory_create_repo L29–29
+ hashlib import import hashlib L20–20
+ msgpack import import msgpack L24–24
+ pytest import import pytest L25–25
+ timezone import import timezone L22–22
+ uuid import import uuid L21–21
+ test_clone_then_verify_all_snapshots_present function async_function test_clone_then_verify_all_snapshots_present L200–243
+ test_wire_fetch_have_excludes_snapshots_for_known_commits function async_function test_wire_fetch_have_excludes_snapshots_for_known_commits L156–196
+ test_wire_fetch_returns_all_snapshots_not_just_tip function async_function test_wire_fetch_returns_all_snapshots_not_just_tip L116–152
~ tests/test_wire_fetch_concurrency.py .py 23 symbols added
+ _CONCURRENCY_LIMIT variable variable _CONCURRENCY_LIMIT L37–37
+ _get_backend function function _get_backend L52–54
+ _mp function function _mp L40–41
+ _push_and_commit_to_r2 function async_function _push_and_commit_to_r2 L57–98
+ _sha256_oid function function _sha256_oid L48–49
+ _utc function function _utc L44–45
+ AsyncClient import import AsyncClient L29–29
+ AsyncSession import import AsyncSession L30–30
+ StrDict import import StrDict L35–35
+ annotations import import annotations L19–19
+ asyncio import import asyncio L21–21
+ datetime import import datetime L25–25
+ db import import db L34–34
+ factory_create_repo import import factory_create_repo L33–33
+ hashlib import import hashlib L22–22
+ msgpack import import msgpack L27–27
+ pytest import import pytest L28–28
+ stat import import stat L23–23
+ timezone import import timezone L25–25
+ update import import update L31–31
+ uuid import import uuid L24–24
+ test_fetch_objects_all_correct_with_bounded_concurrency function async_function test_fetch_objects_all_correct_with_bounded_concurrency L164–194
+ test_fetch_objects_bounded_concurrency function async_function test_fetch_objects_bounded_concurrency L102–160
~ musehub/api/routes/wire.py .py 1 symbol added, 1 symbol modified
+ _R2_CONCURRENCY_LIMIT variable variable _R2_CONCURRENCY_LIMIT L156–156
~ _pack_objects_response
~ musehub/services/musehub_wire.py .py 1 symbol modified
~ wire_fetch
← Older Oldest on task/wire-clone-fix
All commits
Newer → Latest on task/wire-clone-fix

0 comments

No comments yet. Be the first to start the discussion.

To add a comment, use the Muse CLI: muse hub commit comment sha256:cc96d33afd14450354ddf00bad3f1996c588aee01c3fc7c5f2d359562afc9c2d --body "your comment"