gabriel / musehub public
feat BREAKING dev
AI Agent gabriel · 140 days ago · Apr 29, 2026 · Diff

feat: normalize symbol intel into relational tables

Replace three unbounded JSON blobs (code.symbol_history, code.per_symbol_intel, code.hash_occurrence) in musehub_intel_results with normalized relational tables:

musehub_symbol_history_entries — one row per (repo_id, address, commit_id) musehub_symbol_intel — one row per (repo_id, address) musehub_hash_occurrence_entries — one row per (content_id, repo_id, address)

Alembic migration a1b2c3d4e5f6 creates the tables and deletes the stale blobs.

build_symbol_index now upserts into normalized rows. lookup_symbol_intel, load_symbol_history, and load_hash_occurrence query the new tables directly — O(1) point lookups regardless of repo commit history depth.

intel_summary and intel_snapshot remain as small aggregate blobs.

TDD: 93 tests green across test_intel_normalized_schema.py (40) and test_snapshot_symbol_indexer.py (53).

sha256:c43ecbccbea7b2b0d9e9fe6a1ab0797867f57e88ff621423e8741e816cd55449 sha
sha256:a683100531e38c525912e4ac4d664061f4fd2d60a56cc4af59d21b3a9994574d parent
+88 ~24 −4 symbols
sha256:34b3e629391670c52e4a9421037b46ff5891451117aecfb0beb1910415907499 snapshot
+88
symbols added
~24
symbols modified
−4
symbols removed
0
dead code introduced
Semantic Changes 116 symbols
~ alembic/versions/20260429_2045_a1b2c3d4e5f6_normalized_symbol_intel_tables.py .py 11 symbols added
+ branch_labels variable variable branch_labels L27–27
+ depends_on variable variable depends_on L28–28
+ down_revision variable variable down_revision L26–26
+ downgrade function function downgrade L84–87
+ Sequence import import Sequence L20–20
+ Union import import Union L20–20
+ annotations import import annotations L18–18
+ op import import op L23–23
+ sa import import sa L22–22
+ revision variable variable revision L25–25
+ upgrade function function upgrade L31–81
~ tests/test_intel_normalized_schema.py .py 67 symbols added
+ TestAggregatesStillWork class class TestAggregatesStillWork L790–839
+ test_SI39_intel_summary_fields_correct method async_method test_SI39_intel_summary_fields_correct L794–815
+ test_SI40_rebuild_updates_summary_symbol_count method async_method test_SI40_rebuild_updates_summary_symbol_count L818–839
+ TestBuildWritesNormalizedRows class class TestBuildWritesNormalizedRows L163–335
+ test_SI10_history_entry_commit_id_stored method async_method test_SI10_history_entry_commit_id_stored L189–206
+ test_SI11_single_commit_writes_symbol_intel_rows method async_method test_SI11_single_commit_writes_symbol_intel_rows L209–226
+ test_SI12_hash_occurrence_rows_written method async_method test_SI12_hash_occurrence_rows_written L229–249
+ test_SI13_intel_summary_still_written_to_intel_results method async_method test_SI13_intel_summary_still_written_to_intel_results L252–270
+ test_SI14_intel_snapshot_still_written_to_intel_results method async_method test_SI14_intel_snapshot_still_written_to_intel_results L273–289
+ test_SI15_blob_types_not_written_to_intel_results method async_method test_SI15_blob_types_not_written_to_intel_results L292–314
+ test_SI16_author_stored_in_history_entry method async_method test_SI16_author_stored_in_history_entry L317–335
+ test_SI9_single_commit_writes_history_entry_rows method async_method test_SI9_single_commit_writes_history_entry_rows L168–186
+ TestDataIntegrity class class TestDataIntegrity L589–640
+ test_SI30_empty_repo_returns_empty_history method async_method test_SI30_empty_repo_returns_empty_history L593–598
+ test_SI31_unknown_head_commit_returns_empty_results method async_method test_SI31_unknown_head_commit_returns_empty_results L601–607
+ test_SI32_commit_with_no_structured_delta_skipped method async_method test_SI32_commit_with_no_structured_delta_skipped L610–631
+ test_SI33_lookup_symbol_intel_empty_address_list method async_method test_SI33_lookup_symbol_intel_empty_address_list L634–640
+ TestIncrementalUpdates class class TestIncrementalUpdates L468–582
+ test_SI25_second_push_adds_new_history_entries method async_method test_SI25_second_push_adds_new_history_entries L472–496
+ test_SI26_second_push_does_not_duplicate_existing_entries method async_method test_SI26_second_push_does_not_duplicate_existing_entries L499–518
+ test_SI27_modify_op_updates_symbol_intel_churn method async_method test_SI27_modify_op_updates_symbol_intel_churn L521–540
+ test_SI28_second_push_history_has_both_commit_ids method async_method test_SI28_second_push_history_has_both_commit_ids L543–566
+ test_SI29_intel_summary_ref_advances_after_second_push method async_method test_SI29_intel_summary_ref_advances_after_second_push L569–582
+ TestNormalizedSchemaModels class class TestNormalizedSchemaModels L113–156
+ test_SI1_symbol_history_entry_model_importable method method test_SI1_symbol_history_entry_model_importable L117–119
+ test_SI2_symbol_history_entry_columns method method test_SI2_symbol_history_entry_columns L121–125
+ test_SI3_symbol_history_entry_pk method method test_SI3_symbol_history_entry_pk L127–130
+ test_SI4_symbol_intel_model_importable method method test_SI4_symbol_intel_model_importable L132–134
+ test_SI5_symbol_intel_columns method method test_SI5_symbol_intel_columns L136–142
+ test_SI6_symbol_intel_pk method method test_SI6_symbol_intel_pk L144–147
+ test_SI7_hash_occurrence_entry_model_importable method method test_SI7_hash_occurrence_entry_model_importable L149–151
+ test_SI8_hash_occurrence_entry_pk method method test_SI8_hash_occurrence_entry_pk L153–156
+ TestPerformance class class TestPerformance L647–715
+ test_SI34_single_symbol_lookup_under_50ms method async_method test_SI34_single_symbol_lookup_under_50ms L652–675
+ test_SI35_file_scoped_history_lookup_under_50ms method async_method test_SI35_file_scoped_history_lookup_under_50ms L678–699
+ test_SI36_load_symbol_history_no_file_filter_returns_all method async_method test_SI36_load_symbol_history_no_file_filter_returns_all L702–715
+ TestReadHelpers class class TestReadHelpers L342–461
+ test_SI17_load_symbol_history_returns_entries_for_address method async_method test_SI17_load_symbol_history_returns_entries_for_address L346–359
+ test_SI18_load_symbol_history_file_path_filter method async_method test_SI18_load_symbol_history_file_path_filter L362–377
+ test_SI19_load_symbol_history_empty_when_no_index method async_method test_SI19_load_symbol_history_empty_when_no_index L380–386
+ test_SI20_lookup_symbol_intel_returns_metrics method async_method test_SI20_lookup_symbol_intel_returns_metrics L389–406
+ test_SI21_lookup_symbol_intel_missing_address_excluded method async_method test_SI21_lookup_symbol_intel_missing_address_excluded L409–415
+ test_SI22_load_hash_occurrence_returns_clone_pairs method async_method test_SI22_load_hash_occurrence_returns_clone_pairs L418–432
+ test_SI23_load_intel_snapshot_still_works method async_method test_SI23_load_intel_snapshot_still_works L435–446
+ test_SI24_get_index_meta_returns_correct_ref method async_method test_SI24_get_index_meta_returns_correct_ref L449–461
+ TestStress class class TestStress L722–783
+ test_SI37_index_500_symbols_across_10_commits method async_method test_SI37_index_500_symbols_across_10_commits L726–752
+ test_SI38_same_symbol_modified_50_times method async_method test_SI38_same_symbol_modified_50_times L755–783
+ _build_and_persist function async_function _build_and_persist L96–106
+ _cid function function _cid L57–58
+ _commit_with_delta function async_function _commit_with_delta L73–93
+ _insert_op function function _insert_op L61–66
+ _now function function _now L49–50
+ _patch_op function function _patch_op L69–70
+ _uid function function _uid L53–54
+ AsyncSession import import AsyncSession L39–39
+ JSONObject import import JSONObject L42–42
+ annotations import import annotations L30–30
+ create_repo import import create_repo L41–41
+ datetime import import datetime L35–35
+ func function import func L38–38
+ json import import json L32–32 ← moved from musehub/services/musehub_symbol_indexer.py
+ pytest import import pytest L37–37
+ select import import select L38–38
+ time import import time L33–33
+ timezone import import timezone L35–35
+ uuid import import uuid L34–34
~ musehub/db/musehub_models.py .py 3 symbols added
+ MusehubHashOccurrenceEntry class class MusehubHashOccurrenceEntry L1131–1150
+ MusehubSymbolHistoryEntry class class MusehubSymbolHistoryEntry L1070–1094
+ MusehubSymbolIntel class class MusehubSymbolIntel L1097–1128
~ musehub/services/musehub_symbol_indexer.py .py 7 symbols added, 4 symbols removed, 12 symbols modified
− _compute_per_symbol_intel function function _compute_per_symbol_intel L124–241
− _get_intel_result function async_function _get_intel_result L487–497
− _load_existing function async_function _load_existing L299–342
− json import import json L36–36 → moved to tests/test_intel_normalized_schema.py
+ _compute_symbol_intel function function _compute_symbol_intel L118–231
+ _get_prior_ref function async_function _get_prior_ref L288–296
+ _load_full_history_from_db function async_function _load_full_history_from_db L538–560
+ _upsert_hash_occurrences function async_function _upsert_hash_occurrences L396–415
+ _upsert_history_entries function async_function _upsert_history_entries L303–337
+ _upsert_symbol_intel function async_function _upsert_symbol_intel L340–393
+ pg_insert import import pg_insert L41–41
~ SymbolIntel
~ _extract_ops
~ _op_to_muse_op
~ _walk_new_commits
~ build_symbol_index
~ get_index_meta
~ JSONObject
~ JSONValue
~ load_hash_occurrence
~ load_intel_snapshot
~ load_symbol_history
~ lookup_symbol_intel
~ tests/test_snapshot_symbol_indexer.py .py 12 symbols modified
~ TestBuildSymbolIndex
~ test_hash_occurrence_tracks_shared_content
~ test_rebuild_upserts_one_row_per_intel_type
~ test_returns_results_for_structured_delta
~ test_symbol_history_contains_correct_entries
~ TestDataIntegrity
~ test_only_one_result_per_intel_type_after_multiple_builds
~ test_symbol_history_includes_commit_id_and_timestamp
~ TestPerSymbolIntel
~ test_author_count_reflects_unique_authors
~ test_per_symbol_intel_contains_expected_fields
~ test_per_symbol_intel_populated_on_first_build
← Older Oldest on dev
All commits
Newer → Latest on dev

0 comments

No comments yet. Be the first to start the discussion.

To add a comment, use the Muse CLI: muse hub commit comment sha256:c43ecbccbea7b2b0d9e9fe6a1ab0797867f57e88ff621423e8741e816cd55449 --body "your comment"