gabriel / musehub public
perf patch intel task/8-intel-hardening #17 / 17
AI Agent gabriel · 4 days ago · Nov 13, 2026 · Diff

perf(intel): share bounded, immutable source-object reads across providers; reject oversize objects before and during decompression (#8 R2)

Six snapshot providers each re-read every object of the analysed revision (626 objects x 5 providers at 100 commits = ~14 s of read cost). A process-wide byte-bounded LRU (32 MiB, items over a quarter of the budget not stored, only successful reads stored) now serves repeats; the object row is always consulted so a deleted row stays unreadable. The recorded size_bytes rejects an oversize object before any read, and decompress_bounded caps the expansion because the recorded size is not trusted alone. Oversize files are still counted (files_oversize), unreadable ones still fail the job.

sha256:bdc0664ea517f0343fee86e1998df5d9ee2d833bb76e4ce43f85d9f8fb8dd621 sha
+48 ~2 symbols
sha256:d0c21a06e674191ce984d90d0dbd7afb2a8eb84dfd504fad8690c0df2797d22d snapshot
+48
symbols added
~2
symbols modified
0
dead code introduced
Semantic Changes 50 symbols
~ tests/test_intel_source_cache.py .py 35 symbols added
+ StorageSpy class class StorageSpy L48–59
+ __init__ method method __init__ L51–59
+ add_object function async_function add_object L62–67
+ fresh_cache function function fresh_cache L42–45
+ A_V1 import import A_V1 L33–33
+ B_SRC import import B_SRC L33–33
+ DecompressedTooLarge import import DecompressedTooLarge L31–31
+ MusehubIntelType import import MusehubIntelType L28–28
+ MusehubObject import import MusehubObject L29–29
+ P import import P L30–30
+ World import import World L33–33
+ annotations import import annotations L19–19
+ create_repo import import create_repo L32–32
+ decompress_bounded import import decompress_bounded L31–31
+ hashlib import import hashlib L21–21
+ pytest import import pytest L24–24
+ pytest_asyncio import import pytest_asyncio L25–25
+ select import import select L26–26
+ world_with_side_branch import import world_with_side_branch L33–33
+ zlib import import zlib L22–22
+ session_factory function async_function session_factory L37–38
+ test_s1_a_second_read_of_an_object_does_not_touch_storage function async_function test_s1_a_second_read_of_an_object_does_not_touch_storage L73–82
+ test_s1_snapshot_providers_share_one_read_per_object function async_function test_s1_snapshot_providers_share_one_read_per_object L86–93
+ test_s2_an_object_larger_than_a_quarter_of_the_budget_is_not_cached function async_function test_s2_an_object_larger_than_a_quarter_of_the_budget_is_not_cached L119–128
+ test_s2_the_cache_evicts_least_recently_used_and_never_exceeds_its_budget function async_function test_s2_the_cache_evicts_least_recently_used_and_never_exceeds_its_budget L99–115
+ test_s3_a_failed_read_is_not_cached_and_the_next_attempt_reads_again function async_function test_s3_a_failed_read_is_not_cached_and_the_next_attempt_reads_again L134–147
+ test_s3_an_unreadable_object_still_fails_the_provider_with_a_warm_cache function async_function test_s3_an_unreadable_object_still_fails_the_provider_with_a_warm_cache L151–165
+ test_s4_a_zero_byte_file_is_a_cache_hit_not_unreadable function async_function test_s4_a_zero_byte_file_is_a_cache_hit_not_unreadable L171–178
+ test_s5_a_cached_object_whose_row_is_gone_is_unreadable function async_function test_s5_a_cached_object_whose_row_is_gone_is_unreadable L184–190
+ test_s6_a_decompression_bomb_with_a_small_recorded_size_is_rejected_bounded function async_function test_s6_a_decompression_bomb_with_a_small_recorded_size_is_rejected_bounded L206–213
+ test_s6_an_oversize_file_is_counted_not_silently_dropped_and_the_job_still_completes function async_function test_s6_an_oversize_file_is_counted_not_silently_dropped_and_the_job_still_completes L236–251
+ test_s6_decompress_bounded_handles_zstd_and_a_corrupt_stream_is_treated_as_plain_bytes function function test_s6_decompress_bounded_handles_zstd_and_a_corrupt_stream_is_treated_as_plain_bytes L226–232
+ test_s6_decompress_bounded_stops_at_the_limit_for_zlib_and_passes_small_and_plain_data function function test_s6_decompress_bounded_stops_at_the_limit_for_zlib_and_passes_small_and_plain_data L216–223
+ test_s6_the_recorded_size_rejects_an_oversize_object_before_any_read function async_function test_s6_the_recorded_size_rejects_an_oversize_object_before_any_read L196–202
+ test_s7_a_warm_cache_never_shows_a_provider_another_revisions_content function async_function test_s7_a_warm_cache_never_shows_a_provider_another_revisions_content L257–274
~ musehub/services/musehub_intel_providers.py .py 10 symbols added, 2 symbols modified
+ OversizeObject class class OversizeObject L1448–1453
+ _SOURCE_CACHE variable variable _SOURCE_CACHE L1495–1495
+ _SOURCE_CACHE_BYTES variable variable _SOURCE_CACHE_BYTES L1494–1494
+ _SourceCache class class _SourceCache L1456–1490
+ __init__ method method __init__ L1465–1471
+ get method method get L1473–1480
+ put method method put L1482–1490
+ clear_source_cache function function clear_source_cache L1504–1506
+ OrderedDict import import OrderedDict L27–27
+ source_cache_stats function function source_cache_stats L1498–1501
~ musehub/types/compression.py .py 2 symbols added
+ DecompressedTooLarge class class DecompressedTooLarge L30–31
+ decompress_bounded function function decompress_bounded L34–69
~ tests/conftest.py .py 1 symbol added
+ _fresh_intel_source_cache function function _fresh_intel_source_cache L66–69

0 comments

No comments yet. Be the first to start the discussion.

To add a comment, use the Muse CLI: muse hub commit comment sha256:bdc0664ea517f0343fee86e1998df5d9ee2d833bb76e4ce43f85d9f8fb8dd621 --body "your comment"