gabriel / musehub public
perf patch dev
AI Agent gabriel · 149 days ago · Apr 29, 2026 · Diff

perf: batch manifest fetch + recover dev-branch work from sparse snapshots

- get_last_commit_for_file: replace N+1 loop with get_snapshot_manifests_batch - _fetch_file_history: O(n) queries → O(1) with batch SELECT - blob_page: run phases 2/3/4 concurrently with asyncio.gather - add musehub_repository.py service module - recover alembic migration, docs, and test files from dev branch that were lost when dev was rebuilt from main

sha256:29414592f1fb31f0db86360710a45891bb4ef848e3247fb83ad6e0ff5bc0d6cb sha
sha256:864d59986b43001d6ccc58c432a4c7d104e722e11f334072f6ed6e9fe67edc39 parent
+300 ~4 symbols
sha256:c926ee4a11115f6f2f543c89aed79be0ec29804943c5cc15298be4161058df99 snapshot
+300
symbols added
~4
symbols modified
0
dead code introduced
Semantic Changes 304 symbols
~ alembic/versions/20260428_1506_49eb70491119_add_musehub_bridge_mirrors.py .py 12 symbols added
+ branch_labels variable variable branch_labels L19–19
+ depends_on variable variable depends_on L20–20
+ down_revision variable variable down_revision L18–18
+ downgrade function function downgrade L102–159
+ Sequence import import Sequence L10–10
+ Union import import Union L10–10
+ annotations import import annotations L8–8
+ op import import op L12–12
+ postgresql import import postgresql L14–14
+ sa import import sa L13–13
+ revision variable variable revision L17–17
+ upgrade function function upgrade L23–98
~ docs/push-timeout-diagnosis.md .md 35 symbols added
+ Push Timeout Diagnosis section Push Timeout Diagnosis L1–258
+ Atomic Tests (Bottom-Up) section Atomic Tests (Bottom-Up) L141–195
+ Layer 0 — Baseline: can we push at all? section Layer 0 — Baseline: can we push at all? L145–153
+ table@L147 section table@L147 L147–152
+ Layer 1 — R2 PUT latency section Layer 1 — R2 PUT latency L153–164
+ table@L155 section table@L155 L155–161
+ Layer 2 — One batch, end-to-end timing section Layer 2 — One batch, end-to-end timing L164–175
+ table@L166 section table@L166 L166–172
+ Layer 3 — Heartbeat delivery section Layer 3 — Heartbeat delivery L175–185
+ table@L177 section table@L177 L177–182
+ Layer 4 — Full push section Layer 4 — Full push L185–195
+ table@L187 section table@L187 L187–192
+ Code Locations section Code Locations L246–258
+ table@L248 section table@L248 L248–258
+ Everything We Have Tried section Everything We Have Tried L64–102
+ Attempt 1 — Fresh push after delete+recreate (failed: CF 524) section Attempt 1 — Fresh push after delete+recreate (failed: CF 524) L66–72
+ Attempt 2 — Repair script for ghost objects (worked, but wrong problem) section Attempt 2 — Repair script for ghost objects (worked, but wrong problem) L72–78
+ Attempt 3 — Stream _WireResponse response frames (froze) section Attempt 3 — Stream _WireResponse response frames (froze) L78–102
+ table@L90 section table@L90 L90–97
+ Next Actions (in order) section Next Actions (in order) L215–224
+ The Architecture We Need to Match section The Architecture We Need to Match L112–130
+ The Experiment to Run First section The Experiment to Run First L195–215
+ The Goldilocks Zone (What "Fixed" Looks Like) section The Goldilocks Zone (What "Fixed" Looks Like) L130–141
+ The Repo Under Test section The Repo Under Test L9–21
+ table@L13 section table@L13 L13–18
+ What We Are Trying To Do section What We Are Trying To Do L3–9
+ What We Do NOT Know (Unknowns to Resolve) section What We Do NOT Know (Unknowns to Resolve) L102–112
+ What We Know About the Push Path section What We Know About the Push Path L21–64
+ Client-side batching (no CF Worker configured for staging) section Client-side batching (no CF Worker configured for staging) L23–38
+ code@L27 variable variable code@L27 L27–30
+ Cloudflare proxy in between section Cloudflare proxy in between L58–64
+ Server-side per batch section Server-side per batch L38–58
+ code@L40 variable variable code@L40 L40–57
+ What We Now Know About the Push Path (corrected) section What We Now Know About the Push Path (corrected) L224–246
+ code[python]@L228 variable variable code[python]@L228 L228–232
~ docs/ssl.md .md 22 symbols added
+ TLS / SSL Configuration section TLS / SSL Configuration L1–195
+ Local Development section Local Development L12–78
+ Certificate renewal section Certificate renewal L70–78
+ Files section Files L59–70
+ code@L61 variable variable code@L61 L61–66
+ First-time setup section First-time setup L23–59
+ code[bash]@L30 variable variable code[bash]@L30 L30–45
+ code[bash]@L48 variable variable code[bash]@L48 L48–51
+ code[bash]@L54 variable variable code[bash]@L54 L54–58
+ How it works section How it works L14–23
+ Staging and Production section Staging and Production L78–186
+ Architecture section Architecture L80–107
+ code@L82 variable variable code@L82 L82–94
+ Certificate renewal section Certificate renewal L178–186
+ EC2 security group section EC2 security group L165–178
+ Generating a Cloudflare Origin Certificate section Generating a Cloudflare Origin Certificate L116–145
+ code[bash]@L128 variable variable code[bash]@L128 L128–141
+ Why not Let's Encrypt? section Why not Let's Encrypt? L107–116
+ nginx configuration section nginx configuration L145–165
+ code[nginx]@L153 variable variable code[nginx]@L153 L153–161
+ TLS version policy section TLS version policy L186–195
+ table@L5 section table@L5 L5–9
~ docs/wire-verbs-status.md .md 49 symbols added
+ Muse Wire Protocol — Four Verbs: Status, Bugs, and Fix Plan section Muse Wire Protocol — Four Verbs: Status, Bugs, and Fix Plan L1–288
+ Push — POST /{owner}/{slug}/push/stream section 1. Push — POST /{owner}/{slug}/push/stream L33–104
+ Atomic tests for push (must all pass before scaling up) section Atomic tests for push (must all pass before scaling up) L88–104
+ table@L90 section table@L90 L90–99
+ Known bugs / freeze points section Known bugs / freeze points L64–73
+ table@L66 section table@L66 L66–72
+ Push P3 — current freeze: what we need to find out section Push P3 — current freeze: what we need to find out L73–88
+ py) section What the client does (muse/core/transport.py) L40–49
+ py) section What the server does (musehub/services/musehub_wire.py) L49–64
+ What the spec says section What the spec says L35–40
+ Fetch — POST /{owner}/{slug}/fetch/stream section 2. Fetch — POST /{owner}/{slug}/fetch/stream L104–165
+ Atomic tests for fetch section Atomic tests for fetch L149–155
+ table@L151 section table@L151 L151–154
+ Known bugs / freeze points — fixed section Known bugs / freeze points — fixed L129–138
+ table@L131 section table@L131 L131–137
+ Pre-H-frame timing profile (830 commits, 500 objects, localhost) section Pre-H-frame timing profile (830 commits, 500 objects, localhost) L138–149
+ table@L140 section table@L140 L140–148
+ py) section What the client does (muse/core/transport.py) L111–118
+ py — fetch side) section What the server does (musehub/services/musehub_wire.py — fetch side) L118–129
+ What the spec says section What the spec says L106–111
+ What was verified section What was verified L155–165
+ Pull — muse pull <remote> <branch> section 3. Pull — muse pull <remote> <branch> L165–196
+ Atomic tests for pull section Atomic tests for pull L185–196
+ code@L187 variable variable code@L187 L187–193
+ py or similar) section Client flow (muse/cli/commands/pull.py or similar) L171–178
+ Known bugs / freeze points section Known bugs / freeze points L178–185
+ What pull does section What pull does L167–171
+ Clone — muse clone <url> section 4. Clone — muse clone <url> L196–227
+ Atomic tests for clone section Atomic tests for clone L216–227
+ code@L218 variable variable code@L218 L218–224
+ Client flow section Client flow L202–209
+ Known bugs / freeze points section Known bugs / freeze points L209–216
+ table@L211 section table@L211 L211–215
+ What clone does section What clone does L198–202
+ How MWP Maps to Git section How MWP Maps to Git L8–21
+ table@L10 section table@L10 L10–16
+ Immediate Debug Plan section Immediate Debug Plan L227–277
+ Step 1 — Confirm push fix is on staging section Step 1 — Confirm push fix is on staging L229–245
+ code[bash]@L233 variable variable code[bash]@L233 L233–238
+ code[bash]@L240 variable variable code[bash]@L240 L240–243
+ Step 2 — Add server-side timing logs to push section Step 2 — Add server-side timing logs to push L245–259
+ code[python]@L249 variable variable code[python]@L249 L249–258
+ Step 3 — Audit fetch/stream server response section Step 3 — Audit fetch/stream server response L259–265
+ Step 4 — Run atomic tests in order section Step 4 — Run atomic tests in order L265–270
+ Step 5 — Fix fetch/stream streaming if needed section Step 5 — Fix fetch/stream streaming if needed L270–277
+ Scale Ladder (used throughout this doc) section Scale Ladder (used throughout this doc) L21–33
+ table@L23 section table@L23 L23–30
+ What "Done" Looks Like section What "Done" Looks Like L277–288
+ table@L279 section table@L279 L279–285
~ tests/test_file_history_performance.py .py 34 symbols added
+ _FILE variable variable _FILE L44–44
+ _OTHER_FILE variable variable _OTHER_FILE L45–45
+ _OWNER_ID variable variable _OWNER_ID L43–43
+ _add_commit function async_function _add_commit L111–133
+ _add_snapshot function async_function _add_snapshot L86–96
+ _fresh_session function async_function _fresh_session L100–108
+ _make_repo function async_function _make_repo L69–83
+ _manifest_blob function function _manifest_blob L65–66
+ _obj_id function function _obj_id L61–62
+ _repo_id function function _repo_id L52–54
+ _snap_id function function _snap_id L57–58
+ _uid function function _uid L48–49
+ AsyncGenerator import import AsyncGenerator L28–28
+ AsyncSession import import AsyncSession L32–32
+ _database import import _database L37–37
+ _fetch_file_history import import _fetch_file_history L34–34
+ annotations import import annotations L23–23
+ asynccontextmanager import import asynccontextmanager L27–27
+ compute_identity_id import import compute_identity_id L35–35
+ compute_repo_id import import compute_repo_id L35–35
+ datetime import import datetime L26–26
+ db import import db L36–36
+ msgpack import import msgpack L30–30
+ pytest import import pytest L31–31
+ timedelta import import timedelta L26–26
+ timezone import import timezone L26–26
+ uuid import import uuid L25–25
+ test_file_history_calls_batch_fetch_once function async_function test_file_history_calls_batch_fetch_once L173–207
+ test_file_history_does_not_call_per_commit_manifest_fetch function async_function test_file_history_does_not_call_per_commit_manifest_fetch L142–169
+ test_file_history_respects_limit function async_function test_file_history_respects_limit L281–300
+ test_file_history_returns_empty_when_file_not_in_head function async_function test_file_history_returns_empty_when_file_not_in_head L253–264
+ test_file_history_returns_empty_when_no_commits function async_function test_file_history_returns_empty_when_no_commits L268–277
+ test_file_history_returns_only_commits_where_file_changed function async_function test_file_history_returns_only_commits_where_file_changed L216–249
+ test_file_history_unchanged_file_returns_one_entry function async_function test_file_history_unchanged_file_returns_one_entry L304–321
~ tests/test_last_commit_for_file_performance.py .py 35 symbols added
+ _FILE variable variable _FILE L48–48
+ _OTHER variable variable _OTHER L49–49
+ _OWNER_ID variable variable _OWNER_ID L47–47
+ _blob function function _blob L71–72
+ _commit function async_function _commit L97–113
+ _fresh_session function async_function _fresh_session L117–119
+ _make_repo function async_function _make_repo L75–84
+ _obj function function _obj L67–68
+ _repo_id function function _repo_id L56–60
+ _snap function async_function _snap L87–94
+ _snap_id function function _snap_id L63–64
+ _uid function function _uid L52–53
+ AsyncGenerator import import AsyncGenerator L32–32
+ AsyncSession import import AsyncSession L36–36
+ _database import import _database L40–40
+ annotations import import annotations L26–26
+ asynccontextmanager import import asynccontextmanager L30–30
+ asyncio import import asyncio L28–28
+ compute_identity_id import import compute_identity_id L38–38
+ compute_repo_id import import compute_repo_id L38–38
+ datetime import import datetime L31–31
+ db import import db L39–39
+ get_last_commit_for_file import import get_last_commit_for_file L41–41
+ msgpack import import msgpack L34–34
+ pytest import import pytest L35–35
+ timedelta import import timedelta L31–31
+ timezone import import timezone L31–31
+ uuid import import uuid L29–29
+ test_blob_page_phases_run_concurrently function async_function test_blob_page_phases_run_concurrently L294–357
+ test_last_commit_does_not_call_per_commit_manifest_fetch function async_function test_last_commit_does_not_call_per_commit_manifest_fetch L128–153
+ test_last_commit_returns_commit_that_introduced_current_version function async_function test_last_commit_returns_commit_that_introduced_current_version L197–222
+ test_last_commit_returns_none_when_commit_not_found function async_function test_last_commit_returns_none_when_commit_not_found L275–285
+ test_last_commit_returns_none_when_file_missing_from_head function async_function test_last_commit_returns_none_when_file_missing_from_head L259–271
+ test_last_commit_returns_oldest_unbroken_run function async_function test_last_commit_returns_oldest_unbroken_run L226–255
+ test_last_commit_uses_batch_fetch function async_function test_last_commit_uses_batch_fetch L157–188
~ tests/test_push_object_integrity.py .py 35 symbols added
+ _commit_pack_frame function function _commit_pack_frame L88–96
+ _end_frame function function _end_frame L99–104
+ _fw variable variable _fw L45–45
+ _header_frame function function _header_frame L64–73
+ _make_commit function function _make_commit L107–121
+ _make_snapshot function function _make_snapshot L124–125
+ _object_frame function function _object_frame L76–85
+ _oid function function _oid L60–61
+ _pack function function _pack L52–53
+ _wrap function function _wrap L56–57
+ AsyncClient import import AsyncClient L29–29
+ AsyncMock import import AsyncMock L31–31
+ AsyncSession import import AsyncSession L30–30
+ JSONObject import import JSONObject L42–42
+ JSONValue import import JSONValue L42–42
+ MagicMock import import MagicMock L31–31
+ MuseWireFrameWriter import import MuseWireFrameWriter L34–34
+ SFRAME_COMMIT_PACK import import SFRAME_COMMIT_PACK L35–35
+ SFRAME_END import import SFRAME_END L35–35
+ SFRAME_HEADER import import SFRAME_HEADER L35–35
+ SFRAME_OBJECT import import SFRAME_OBJECT L35–35
+ SFRAME_RESULT import import SFRAME_RESULT L35–35
+ StrDict import import StrDict L42–42
+ WIRE_CONTENT_TYPE import import WIRE_CONTENT_TYPE L34–34
+ annotations import import annotations L24–24
+ blob_id import import blob_id L33–33
+ create_repo import import create_repo L43–43
+ inspect import import inspect L26–26
+ msgpack import import msgpack L27–27
+ patch import import patch L31–31
+ pytest import import pytest L28–28
+ test_i1_local_backend_put_is_durable function function test_i1_local_backend_put_is_durable L132–147
+ test_i2_external_ref_in_db_but_missing_from_storage_is_rejected function async_function test_i2_external_ref_in_db_but_missing_from_storage_is_rejected L155–225
+ test_i3_successful_push_all_objects_retrievable function async_function test_i3_successful_push_all_objects_retrievable L234–283
+ test_i4_integrity_check_uses_backend_exists function function test_i4_integrity_check_uses_backend_exists L290–307
~ tests/test_push_timeout_fix.py .py 35 symbols added
+ _commit_pack_frame function function _commit_pack_frame L88–93
+ _end_frame function function _end_frame L96–97
+ _fw variable variable _fw L46–46
+ _header_frame function function _header_frame L65–74
+ _make_commit function function _make_commit L100–118
+ _make_snapshot function function _make_snapshot L121–122
+ _object_frame function function _object_frame L77–85
+ _oid function function _oid L61–62
+ _pack function function _pack L53–54
+ _parse_response_frames function function _parse_response_frames L125–129
+ _wrap function function _wrap L57–58
+ AsyncClient import import AsyncClient L29–29
+ AsyncSession import import AsyncSession L30–30
+ JSONObject import import JSONObject L43–43
+ JSONValue import import JSONValue L43–43
+ MuseWireFrameWriter import import MuseWireFrameWriter L33–33
+ SFRAME_COMMIT_PACK import import SFRAME_COMMIT_PACK L34–34
+ SFRAME_END import import SFRAME_END L34–34
+ SFRAME_ERROR import import SFRAME_ERROR L34–34
+ SFRAME_HEADER import import SFRAME_HEADER L34–34
+ SFRAME_OBJECT import import SFRAME_OBJECT L34–34
+ SFRAME_PROGRESS import import SFRAME_PROGRESS L34–34
+ SFRAME_RESULT import import SFRAME_RESULT L34–34
+ StrDict import import StrDict L43–43
+ WIRE_CONTENT_TYPE import import WIRE_CONTENT_TYPE L33–33
+ annotations import import annotations L22–22
+ blob_id import import blob_id L32–32
+ create_repo import import create_repo L44–44
+ inspect import import inspect L24–24
+ msgpack import import msgpack L27–27
+ pytest import import pytest L28–28
+ textwrap import import textwrap L25–25
+ test_t1_wire_response_sends_headers_before_consuming_full_stream function function test_t1_wire_response_sends_headers_before_consuming_full_stream L136–173
+ test_t2_push_response_is_frame_stream function async_function test_t2_push_response_is_frame_stream L181–231
+ test_t3_wire_push_stream_emits_progress_during_commit_loop function function test_t3_wire_push_stream_emits_progress_during_commit_loop L238–274
~ tests/test_raw_endpoint.py .py 39 symbols added
+ TestRawEndpoint class class TestRawEndpoint L169–313
+ test_404_manifest_miss_and_storage_miss_have_distinct_messages method async_method test_404_manifest_miss_and_storage_miss_have_distinct_messages L227–259
+ test_binary_file_served_as_attachment method async_method test_binary_file_served_as_attachment L295–313
+ test_returns_200_for_file_in_manifest_and_storage method async_method test_returns_200_for_file_in_manifest_and_storage L171–189
+ test_returns_404_for_unknown_ref method async_method test_returns_404_for_unknown_ref L261–274
+ test_returns_404_when_file_not_in_manifest method async_method test_returns_404_when_file_not_in_manifest L191–205
+ test_returns_404_when_object_missing_from_storage method async_method test_returns_404_when_object_missing_from_storage L207–225
+ test_text_file_served_as_text_plain method async_method test_text_file_served_as_text_plain L276–293
+ TestStorageBackendExistsInterface class class TestStorageBackendExistsInterface L124–162
+ test_local_backend_exists_returns_true_after_put method async_method test_local_backend_exists_returns_true_after_put L134–138
+ test_local_backend_exists_single_arg method async_method test_local_backend_exists_single_arg L127–132
+ test_local_backend_exists_two_args_raises method async_method test_local_backend_exists_two_args_raises L140–144
+ test_s3_backend_exists_single_arg method async_method test_s3_backend_exists_single_arg L146–156
+ test_s3_backend_exists_two_args_raises method async_method test_s3_backend_exists_two_args_raises L158–162
+ _make_branch_at_commit function async_function _make_branch_at_commit L87–117
+ _make_repo function async_function _make_repo L47–67
+ _make_snapshot function async_function _make_snapshot L70–84
+ _uid function function _uid L41–42
+ AsyncClient import import AsyncClient L27–27
+ AsyncSession import import AsyncSession L28–28
+ LocalBackend import import LocalBackend L38–38
+ MagicMock import import MagicMock L23–23
+ MusehubBranch import import MusehubBranch L32–32
+ MusehubCommit import import MusehubCommit L32–32
+ MusehubRepo import import MusehubRepo L32–32
+ MusehubSnapshot import import MusehubSnapshot L32–32
+ Path import import Path L22–22
+ S3Backend import import S3Backend L38–38
+ annotations import import annotations L18–18
+ compute_branch_id import import compute_branch_id L31–31
+ compute_identity_id import import compute_identity_id L31–31
+ compute_repo_id import import compute_repo_id L31–31
+ datetime import import datetime L21–21
+ fake_id import import fake_id L30–30
+ msgpack import import msgpack L25–25
+ patch import import patch L23–23
+ pytest import import pytest L26–26
+ timezone import import timezone L21–21
+ uuid import import uuid L20–20
~ musehub/api/routes/musehub/ui_blob.py .py 3 symbols added, 3 symbols modified
+ _empty_list function async_function _empty_list L380–381
+ asyncio import import asyncio L11–11
+ get_snapshot_manifests_batch import import get_snapshot_manifests_batch L29–29
~ _fetch_file_history
~ blob_page
~ get_snapshot_manifest
~ musehub/services/musehub_repository.py .py 1 symbol added, 1 symbol modified
+ get_snapshot_manifests_batch import import get_snapshot_manifests_batch L23–23
~ get_last_commit_for_file
← Older Oldest on dev
All commits
Newer → Latest on dev

0 comments

No comments yet. Be the first to start the discussion.

To add a comment, use the Muse CLI: muse hub commit comment sha256:29414592f1fb31f0db86360710a45891bb4ef848e3247fb83ad6e0ff5bc0d6cb --body "your comment"