gabriel / musehub public
perf patch task/mpack-phase2-no-per-object-puts #1 / 1
AI Agent gabriel · 120 days ago · May 17, 2026 · Diff

perf: remove per-object MinIO PUTs from mpack.index background job (Phase 2 issue #69)

Objects are now served from the covering mpack via MPackIndex (Phase 1). The parallel PUT loop was the ~25s bottleneck on XL jobs. storage_uri is now set to mpack://{mpack_key} instead of a per-object MinIO key.

Adds TDD tests P2-1, P2-2, P2-3 verifying zero put() calls and correct mpack URI storage after job completion.

sha256:469672af9a6ebe85d6427a36c60abef318cad2be30ba91a070a474eef7969701 sha
sha256:56cbc136427d1e5cd1a711151b9bde9c8f7dbc65985a34761b4171a5d3ac4722 parent
+38 ~1 symbols
sha256:941e8feb21841c89a96f253d669be64ba40ab2e343da846e731842efec124a9c snapshot
+38
symbols added
~1
symbol modified
0
dead code introduced
Semantic Changes 39 symbols
~ tests/test_mpack_phase2.py .py 38 symbols added
+ _FakeBackend class class _FakeBackend L62–99
+ __init__ method method __init__ L65–68
+ delete method async_method delete L87–88
+ exists method async_method exists L84–85
+ get method async_method get L75–76
+ get_mpack method async_method get_mpack L78–79
+ presign_get method async_method presign_get L90–91
+ put method async_method put L70–73
+ put_mpack method async_method put_mpack L81–82
+ quarantine_mpack method async_method quarantine_mpack L93–94
+ uri_for method method uri_for L96–97
+ _build_push_mpack function function _build_push_mpack L49–59
+ _make_job function async_function _make_job L102–129
+ _mpack_id function function _mpack_id L45–46
+ _now function function _now L41–42
+ AsyncMock import import AsyncMock L22–22
+ AsyncSession import import AsyncSession L28–28
+ MagicMock import import MagicMock L22–22
+ MusehubBackgroundJob import import MusehubBackgroundJob L33–33
+ annotations import import annotations L18–18
+ blob_id import import blob_id L31–31
+ call import import call L22–22
+ create_repo import import create_repo L38–38
+ datetime import import datetime L21–21
+ db import import db L32–32
+ fake_id import import fake_id L31–31
+ hashlib import import hashlib L20–20
+ msgpack import import msgpack L24–24
+ pg_insert import import pg_insert L27–27
+ process_mpack_index_job import import process_mpack_index_job L34–34
+ pytest import import pytest L25–25
+ select import import select L29–29
+ timezone import import timezone L21–21
+ wire_fetch_mpack import import wire_fetch_mpack L34–34
+ zstandard import import zstandard L26–26
+ test_p2_1_no_per_object_put_calls function async_function test_p2_1_no_per_object_put_calls L135–166
+ test_p2_2_storage_uri_is_mpack_uri function async_function test_p2_2_storage_uri_is_mpack_uri L172–204
+ test_p2_3_full_roundtrip_no_per_object_keys function async_function test_p2_3_full_roundtrip_no_per_object_keys L210–292
~ musehub/services/musehub_wire.py .py 1 symbol modified
~ process_mpack_index_job
← Older Oldest on task/mpack-phase2-no-per-object-puts
All commits
Newer → Latest on task/mpack-phase2-no-per-object-puts

0 comments

No comments yet. Be the first to start the discussion.

To add a comment, use the Muse CLI: muse hub commit comment sha256:469672af9a6ebe85d6427a36c60abef318cad2be30ba91a070a474eef7969701 --body "your comment"