gabriel / musehub public
refactor BREAKING dev
AI Agent gabriel · 156 days ago · Apr 21, 2026 · Diff

refactor: domain-agnostic intel system + MWP wire protocol cleanup

Intelligence system: - Replace MusehubSymbolIndex + MusehubFileIntelCache with single musehub_intel_results table (namespaced intel_type keys) - Add IntelProvider protocol + CodeProvider, StructuralProvider, MidiProvider - build_symbol_index() now returns list[tuple[str, dict]] instead of DB row - All readers (load_symbol_history, load_intel_snapshot, etc.) query intel_results - enqueue_push_intel() dispatches correct job types per domain - enqueue_job() dedup fixed (was claiming ON CONFLICT but not doing it) - reclaim_stale_jobs() resets stuck running jobs after 10 minutes - worker.py uses provider registry for extensible job dispatch - compute_intel_result_id() added to genesis.py

Wire protocol cleanup: - Delete wire_push_objects(), wire_fetch_objects(), wire_filter_objects(), wire_presign(), wire_confirm_objects(), wire_push_object_pack() — all unreachable (no routes); MWP streaming push/fetch is the only path - Delete ~580 lines of dead service code + 10 dead Pydantic models - Delete test_object_refs.py, test_perf_push_pipeline.py, test_wire_compression.py (tested dead code) - Restore _fetch_object_meta() as internal helper for wire_fetch_stream - Restore WireNegotiateRequest/Response models (accidentally deleted) - Remove unused imports (WireFilterRequest, wire_fetch_objects) from routes

Schema: - Alembic migration intel_results_table.py: drop old tables, create new - Alembic migration 0022_composite_indexes.py: all high-traffic composite indexes - factories.py: compute repo_id/branch_id as genesis hashes (not random UUIDs) - test_wire_push_stream.py: _make_repo uses genesis IDs - test_background_jobs.py: push test uses /push/stream MWP format

sha256:0c53e64d8f7d1cd3bc244c843734393417f9a0d35aa86f705b705fd088d4e9e4 sha
+78 ~75 −174 symbols
sha256:860d3e6f2908c36ad147e4ed31696ac652458c4a8d038ca4de530720c4376aea snapshot
+78
symbols added
~75
symbols modified
−174
symbols removed
0
dead code introduced
Semantic Changes 327 symbols
+ branch_labels variable variable branch_labels L19–19
+ depends_on variable variable depends_on L20–20
+ down_revision variable variable down_revision L18–18
+ downgrade function function downgrade L71–79
+ Union import import Union L12–12
+ annotations import import annotations L10–10
+ op import import op L15–15
+ sa import import sa L14–14
+ revision variable variable revision L17–17
+ upgrade function function upgrade L23–68
+ branch_labels variable variable branch_labels L21–21
+ depends_on variable variable depends_on L22–22
+ down_revision variable variable down_revision L20–20
+ downgrade function function downgrade L83–130
+ Union import import Union L14–14
+ annotations import import annotations L12–12
+ op import import op L17–17
+ sa import import sa L16–16 ← moved from tests/test_object_refs.py
+ revision variable variable revision L19–19
+ upgrade function function upgrade L25–80
+ CodeProvider class class CodeProvider L181–204
+ compute method async_method compute L196–204
+ IntelProvider class class IntelProvider L52–68
+ compute method async_method compute L55–68
+ IntelResults variable variable IntelResults L48–48
+ MidiProvider class class MidiProvider L211–229
+ compute method async_method compute L222–229
+ StructuralProvider class class StructuralProvider L120–174
+ compute method async_method compute L131–174
+ _PROVIDER_REGISTRY variable variable _PROVIDER_REGISTRY L238–242
+ get_provider function function get_provider L245–247
+ AsyncSession import import AsyncSession L36–36 ← moved from tests/test_wire_compression.py
+ JSONObject import import JSONObject L40–40 ← moved from musehub/worker.py
+ Protocol import import Protocol L32–32
+ annotations import import annotations L27–27 ← moved from tests/test_wire_compression.py
+ compute_intel_result_id import import compute_intel_result_id L38–38
+ datetime import import datetime L31–31
+ db import import db L39–39 ← moved from tests/test_perf_push_pipeline.py
+ json import import json L29–29
+ logging import import logging L30–30
+ pg_insert import import pg_insert L35–35 ← moved from musehub/services/musehub_jobs.py
+ runtime_checkable import import runtime_checkable L32–32
+ select import import select L34–34
+ timedelta import import timedelta L31–31
+ timezone import import timezone L31–31
+ job_types_for_push function function job_types_for_push L250–263
+ logger variable variable logger L42–42
+ persist_intel_results function async_function persist_intel_results L75–113
~ tests/test_object_refs.py .py 28 symbols removed
− _mp function function _mp L44–45
− _object_count function async_function _object_count L73–80
− _ref_count function async_function _ref_count L63–70
− _ref_exists function async_function _ref_exists L52–60
− _sha256_oid function function _sha256_oid L40–41
− _utc_now function function _utc_now L33–34
− _zlib function function _zlib L48–49
− AsyncClient import import AsyncClient L25–25
− AsyncSession import import AsyncSession L26–26
− JSONValue import import JSONValue L29–29
− StrDict import import StrDict L29–29
− annotations import import annotations L15–15
− datetime import import datetime L20–20
− db import import db L28–28
− factory_create_repo import import factory_create_repo L30–30
− hashlib import import hashlib L17–17
− msgpack import import msgpack L22–22
− pytest import import pytest L23–23
− sa import import sa L24–24 → moved to alembic/versions/intel_results_table.py
− timezone import import timezone L20–20
− uuid import import uuid L18–18
− zlib import import zlib L19–19
− test_confirm_objects_writes_refs function async_function test_confirm_objects_writes_refs L242–268
− test_idempotent_push_does_not_duplicate_refs function async_function test_idempotent_push_does_not_duplicate_refs L276–297
− test_pack_push_writes_ref_for_deduplicated_object function async_function test_pack_push_writes_ref_for_deduplicated_object L117–158
− test_pack_push_writes_ref_for_new_object function async_function test_pack_push_writes_ref_for_new_object L88–109
− test_push_objects_endpoint_writes_refs function async_function test_push_objects_endpoint_writes_refs L166–185
− test_wire_push_bundle_objects_writes_refs function async_function test_wire_push_bundle_objects_writes_refs L193–234
~ tests/test_perf_push_pipeline.py .py 30 symbols removed
− _MsgpackInput variable variable _MsgpackInput L40–40
− _make_snapshots function function _make_snapshots L57–71
− _make_wire_objects function function _make_wire_objects L74–81
− _mp function function _mp L84–85
− _obj_id function function _obj_id L53–54
− _snap_id function function _snap_id L49–50
− _utc_now function function _utc_now L45–46
− AsyncClient import import AsyncClient L30–30
− AsyncSession import import AsyncSession L31–31
− StrDict import import StrDict L37–37
− WireObject import import WireObject L34–34
− WireObjectsRequest import import WireObjectsRequest L34–34
− annotations import import annotations L20–20
− asyncio import import asyncio L22–22
− bulk_upsert_snapshot_entries import import bulk_upsert_snapshot_entries L35–35
− datetime import import datetime L26–26
− db import import db L33–33 → moved to musehub/services/musehub_intel_providers.py
− factory_create_repo import import factory_create_repo L38–38
− hashlib import import hashlib L23–23
− msgpack import import msgpack L28–28
− pytest import import pytest L29–29
− time import import time L24–24
− timezone import import timezone L26–26
− uuid import import uuid L25–25
− wire_push_objects import import wire_push_objects L36–36
− test_push_pack_1000_commits_100_files_under_5s function async_function test_push_pack_1000_commits_100_files_under_5s L114–132
− test_push_pack_100_commits_500_files_under_3s function async_function test_push_pack_100_commits_500_files_under_3s L91–110
− test_symbol_indexer_large_push_does_not_crash function async_function test_symbol_indexer_large_push_does_not_crash L205–277
− test_wire_push_objects_500_existing_objects_under_200ms function async_function test_wire_push_objects_500_existing_objects_under_200ms L165–199
− test_wire_push_objects_500_new_objects_under_1s function async_function test_wire_push_objects_500_new_objects_under_1s L138–161
~ tests/test_wire_compression.py .py 35 symbols removed
− _DeltaObjectPayload class class _DeltaObjectPayload L50–55
− _MsgpackInput variable variable _MsgpackInput L33–33
− _RawObjectPayload class class _RawObjectPayload L36–40
− _ZlibObjectPayload class class _ZlibObjectPayload L43–47
− _compress_zlib function function _compress_zlib L65–67
− _compute_delta function function _compute_delta L70–84
− _make_delta_object function function _make_delta_object L107–121
− _make_raw_object function function _make_raw_object L92–94
− _make_zlib_object function function _make_zlib_object L97–104
− _mp function function _mp L87–89
− _sha256_oid function function _sha256_oid L60–62
− AsyncClient import import AsyncClient L25–25
− AsyncSession import import AsyncSession L26–26 → moved to musehub/services/musehub_intel_providers.py
− StrDict import import StrDict L31–31
− TypedDict import import TypedDict L28–28
− annotations import import annotations L17–17 → moved to musehub/services/musehub_intel_providers.py
− factory_create_repo import import factory_create_repo L30–30
− hashlib import import hashlib L19–19
− msgpack import import msgpack L23–23
− pytest import import pytest L24–24
− struct import import struct L20–20
− zlib import import zlib L21–21
− test_filter_objects_hint_for_different_path_no_base function async_function test_filter_objects_hint_for_different_path_no_base L360–384
− test_filter_objects_no_hints_returns_empty_bases function async_function test_filter_objects_no_hints_returns_empty_bases L251–271
− test_filter_objects_present_object_not_in_bases function async_function test_filter_objects_present_object_not_in_bases L324–356
− test_filter_objects_returns_bases_for_known_path function async_function test_filter_objects_returns_bases_for_known_path L275–320
− test_push_object_pack_delta_hash_mismatch_rejected function async_function test_push_object_pack_delta_hash_mismatch_rejected L506–545
− test_push_object_pack_delta_missing_base_id_rejected function async_function test_push_object_pack_delta_missing_base_id_rejected L444–471
− test_push_object_pack_delta_nonexistent_base_rejected function async_function test_push_object_pack_delta_nonexistent_base_rejected L475–502
− test_push_object_pack_delta_zlib_stored function async_function test_push_object_pack_delta_zlib_stored L391–440
− test_push_object_pack_mixed_encodings function async_function test_push_object_pack_mixed_encodings L549–589
− test_push_object_pack_raw_encoding_accepted function async_function test_push_object_pack_raw_encoding_accepted L128–148
− test_push_object_pack_zlib_encoding_stored function async_function test_push_object_pack_zlib_encoding_stored L152–186
− test_push_object_pack_zlib_hash_mismatch_rejected function async_function test_push_object_pack_zlib_hash_mismatch_rejected L190–219
− test_push_object_pack_zlib_idempotent function async_function test_push_object_pack_zlib_idempotent L223–244
~ docs/db-query-runbook.md .md 2 symbols added, 2 symbols removed, 2 symbols modified
− Q7 — Latest symbol index (intel endpoints) section Q7 — Latest symbol index (intel endpoints) L93–104
− code[sql]@L94 variable variable code[sql]@L94 L94–100
+ Q7 — Intel result lookup (intel endpoints) section Q7 — Intel result lookup (intel endpoints) L93–104
+ code[sql]@L94 variable variable code[sql]@L94 L94–98
~ musehub/api/routes/wire.py .py 2 symbols removed, 11 symbols modified
~ musehub/core/genesis.py .py 1 symbol added
+ compute_intel_result_id function function compute_intel_result_id L282–294
~ musehub/db/musehub_models.py .py 1 symbol added, 2 symbols removed, 1 symbol modified
− MusehubFileIntelCache class class MusehubFileIntelCache L1056–1088
− MusehubSymbolIndex class class MusehubSymbolIndex L1010–1053
+ MusehubIntelResult class class MusehubIntelResult L1011–1061
~ musehub/models/wire.py .py 17 symbols removed, 2 symbols modified
− PACK_MAX_BYTES variable variable PACK_MAX_BYTES L443–443
− PACK_MAX_OBJECTS variable variable PACK_MAX_OBJECTS L442–442
− WireConfirmObjectsRequest class class WireConfirmObjectsRequest L375–397
− _validate_ids method method _validate_ids L396–397
− WireConfirmObjectsResponse class class WireConfirmObjectsResponse L400–404
− WireFilterRequest class class WireFilterRequest L300–323
− _validate_ids method method _validate_ids L322–323
− WireFilterResponse class class WireFilterResponse L326–330
− WireObjectPackRequest class class WireObjectPackRequest L446–496
− _check_total_size method method _check_total_size L482–496
− _validate_object_ids method method _validate_object_ids L476–479
− WireObjectPackResponse class class WireObjectPackResponse L499–507
− WireObjectsRequest class class WireObjectsRequest L278–287
− WireObjectsResponse class class WireObjectsResponse L290–294
− WirePresignRequest class class WirePresignRequest L336–360
− _validate_ids method method _validate_ids L359–360
− WirePresignResponse class class WirePresignResponse L363–369
~ musehub/services/musehub_jobs.py .py 4 symbols added, 1 symbol removed, 3 symbols modified
− pg_insert import import pg_insert L28–28 → moved to musehub/services/musehub_intel_providers.py
+ _STALE_CLAIM_MINUTES variable variable _STALE_CLAIM_MINUTES L46–46
+ enqueue_push_intel function async_function enqueue_push_intel L101–120
+ timedelta import import timedelta L34–34
+ reclaim_stale_jobs function async_function reclaim_stale_jobs L151–177
~ musehub/services/musehub_symbol_indexer.py .py 2 symbols added, 3 symbols removed, 10 symbols modified
− _get_index_row function async_function _get_index_row L617–627
− msgpack import import msgpack L28–28
− uuid import import uuid L23–23
+ _get_intel_result function async_function _get_intel_result L487–497
+ _load_existing function async_function _load_existing L299–342
~ musehub/services/musehub_wire.py .py 1 symbol added, 16 symbols removed, 14 symbols modified
− WireConfirmObjectsRequest import import WireConfirmObjectsRequest L56–56
− WireConfirmObjectsResponse import import WireConfirmObjectsResponse L56–56
− WireFilterRequest import import WireFilterRequest L56–56
− WireFilterResponse import import WireFilterResponse L56–56
− WireObjectPackRequest import import WireObjectPackRequest L56–56
− WireObjectPackResponse import import WireObjectPackResponse L56–56
− WireObjectsRequest import import WireObjectsRequest L56–56
− WireObjectsResponse import import WireObjectsResponse L56–56
− WirePresignRequest import import WirePresignRequest L56–56
− WirePresignResponse import import WirePresignResponse L56–56
− wire_confirm_objects function async_function wire_confirm_objects L1619–1707
− wire_fetch_objects function async_function wire_fetch_objects L1442–1481
− wire_filter_objects function async_function wire_filter_objects L1484–1546
− wire_presign function async_function wire_presign L1549–1616
− wire_push_object_pack function async_function wire_push_object_pack L891–1074
− wire_push_objects function async_function wire_push_objects L727–846
+ _fetch_object_meta function async_function _fetch_object_meta L932–950
~ musehub/worker.py .py 8 symbols removed, 2 symbols modified
− _HANDLERS variable variable _HANDLERS L76–79
− _Handler variable variable _Handler L29–29
− _HandlerMap variable variable _HandlerMap L30–30
− _handle_symbol_index function async_function _handle_symbol_index L47–59
− Callable import import Callable L20–20
− Coroutine import import Coroutine L20–20
− JSONObject import import JSONObject L24–24 → moved to musehub/services/musehub_intel_providers.py
− TYPE_CHECKING import import TYPE_CHECKING L21–21
~ tests/factories.py .py 2 symbols modified
~ tests/test_api_performance.py .py 1 symbol added, 1 symbol removed
− test_wire_push_triggers_symbol_indexer function function test_wire_push_triggers_symbol_indexer L193–201
+ test_wire_push_triggers_intel_indexer function function test_wire_push_triggers_intel_indexer L193–198
~ tests/test_background_jobs.py .py 1 symbol added, 3 symbols removed
− _MsgpackInput variable variable _MsgpackInput L32–32
− uuid import import uuid L13–13
− test_push_enqueues_symbol_index_and_gc_jobs function async_function test_push_enqueues_symbol_index_and_gc_jobs L166–226
+ test_push_enqueues_intel_jobs function async_function test_push_enqueues_intel_jobs L163–239
~ tests/test_database.py .py 1 symbol added, 1 symbol removed, 2 symbols modified
− test_musehub_symbol_index_has_repo_built_at_composite function function test_musehub_symbol_index_has_repo_built_at_composite L85–90
+ test_musehub_intel_results_has_repo_type_composite function function test_musehub_intel_results_has_repo_type_composite L85–90
~ tests/test_snapshot_symbol_indexer.py .py 16 symbols added, 25 symbols removed, 20 symbols modified
− test_creates_index_row_for_structured_delta method async_method test_creates_index_row_for_structured_delta L265–279
− test_rebuild_invalidates_file_intel_cache method async_method test_rebuild_invalidates_file_intel_cache L353–379
− test_rebuild_prunes_old_index_row method async_method test_rebuild_prunes_old_index_row L330–350
− test_returns_none_when_no_structured_delta method async_method test_returns_none_when_no_structured_delta L252–262
− test_symbol_history_blob_contains_correct_entries method async_method test_symbol_history_blob_contains_correct_entries L282–303
− test_only_one_index_row_per_repo_after_multiple_builds method async_method test_only_one_index_row_per_repo_after_multiple_builds L699–717
− TestMsgpackRoundTrip class class TestMsgpackRoundTrip L224–243
− test_bytes_values_survive_round_trip method method test_bytes_values_survive_round_trip L239–243
− test_schema_version_preserved method method test_schema_version_preserved L227–237
− TestPerSymbolIntelBackfill class class TestPerSymbolIntelBackfill L808–1039
− test_author_count_reflects_unique_authors method async_method test_author_count_reflects_unique_authors L987–1016
− test_early_return_fires_when_intel_populated method async_method test_early_return_fires_when_intel_populated L823–850
− test_lookup_symbol_intel_returns_empty_when_per_symbol_intel_null method async_method test_lookup_symbol_intel_returns_empty_when_per_symbol_intel_null L919–940
− test_lookup_symbol_intel_returns_matching_addresses method async_method test_lookup_symbol_intel_returns_matching_addresses L1019–1039
− test_per_symbol_intel_contains_expected_fields method async_method test_per_symbol_intel_contains_expected_fields L963–984
− test_per_symbol_intel_populated_on_first_build method async_method test_per_symbol_intel_populated_on_first_build L943–960
− test_stale_index_missing_per_symbol_intel_is_backfilled method async_method test_stale_index_missing_per_symbol_intel_is_backfilled L853–916
− TestPerformance class class TestPerformance L1141–1215
− test_batch_manifest_50_snapshots_under_300ms method async_method test_batch_manifest_50_snapshots_under_300ms L1192–1215
− test_build_symbol_index_100_commits_under_3s method async_method test_build_symbol_index_100_commits_under_3s L1159–1177
− test_extract_ops_1000_calls_under_100ms method method test_extract_ops_1000_calls_under_100ms L1142–1156
− test_upsert_1000_files_under_500ms method async_method test_upsert_1000_files_under_500ms L1180–1189
− test_build_with_unknown_head_commit_returns_none method async_method test_build_with_unknown_head_commit_returns_none L770–780
− test_corrupt_msgpack_returns_empty_not_exception method async_method test_corrupt_msgpack_returns_empty_not_exception L747–767
− msgpack import import msgpack L44–44
+ test_rebuild_upserts_one_row_per_intel_type method async_method test_rebuild_upserts_one_row_per_intel_type L322–345
+ test_returns_empty_when_no_structured_delta method async_method test_returns_empty_when_no_structured_delta L247–257
+ test_returns_results_for_structured_delta method async_method test_returns_results_for_structured_delta L260–275
+ test_symbol_history_contains_correct_entries method async_method test_symbol_history_contains_correct_entries L278–297
+ test_only_one_result_per_intel_type_after_multiple_builds method async_method test_only_one_result_per_intel_type_after_multiple_builds L656–676
+ TestPerSymbolIntel class class TestPerSymbolIntel L778–891
+ test_author_count_reflects_unique_authors method async_method test_author_count_reflects_unique_authors L837–859
+ test_early_return_when_already_current method async_method test_early_return_when_already_current L780–800
+ test_lookup_symbol_intel_returns_empty_when_no_index method async_method test_lookup_symbol_intel_returns_empty_when_no_index L884–891
+ test_lookup_symbol_intel_returns_matching_addresses method async_method test_lookup_symbol_intel_returns_matching_addresses L862–881
+ test_per_symbol_intel_contains_expected_fields method async_method test_per_symbol_intel_contains_expected_fields L818–834
+ test_per_symbol_intel_populated_on_first_build method async_method test_per_symbol_intel_populated_on_first_build L803–815
+ test_build_with_unknown_head_commit_returns_empty method async_method test_build_with_unknown_head_commit_returns_empty L734–744
+ test_corrupt_json_returns_empty_not_exception method async_method test_corrupt_json_returns_empty_not_exception L705–731
+ _build_and_persist function async_function _build_and_persist L90–101
+ _get_result_data function function _get_result_data L104–109
← Older Oldest on dev
All commits
Newer → Latest on dev

0 comments

No comments yet. Be the first to start the discussion.

To add a comment, use the Muse CLI: muse hub commit comment sha256:0c53e64d8f7d1cd3bc244c843734393417f9a0d35aa86f705b705fd088d4e9e4 --body "your comment"