gabriel / musehub public
perf BREAKING fix/mwp-db-only-filter #1 / 1
gabriel · 168 days ago · Apr 12, 2026 · Diff

perf: MWP filter/push use single SQL IN — no per-object S3 HEAD requests

wire_filter_objects: one SELECT IN query, O(1) round-trips. wire_push_objects: one SELECT IN query for dedup, O(1) round-trips.

This matches how git's smart HTTP works: the server answers 'what refs do you have?' — it does not verify every object on every push. The push invariant (backend.put succeeds → DB row inserted) ensures that a DB record always means bytes exist in storage. No per-push storage verification needed.

The previous approach (parallel S3 HEAD requests in wire_filter_objects) turned every push into N × TLS round-trips to Cloudflare, making a 534-object push take 60+ seconds instead of <1s.

deploy/repair_objects.py handles the one-time staging data-loss repair: checks all DB records against actual storage, deletes orphaned rows so a normal push re-uploads the missing bytes.

sha256:3c9077aa4718d0762aab92978ad0edec924e97e3e8f3ea989026bfe88e37d44a sha
sha256:4f4456f0b9645143fe9d9096958f9b479e4453f583bad56f626c2637beba851d parent
+13 ~2 −1 symbols
sha256:158e50bbd17a19eed9af014fb530a181ba37eead6f40127eb3519eb2fd971d21 snapshot
+13
symbols added
~2
symbols modified
−1
symbol removed
0
dead code introduced
Semantic Changes 16 symbols
~ deploy/repair_objects.py .py 13 symbols added
+ AsyncSession import import AsyncSession L31–31
+ annotations import import annotations L24–24
+ argparse import import argparse L26–26
+ asyncio import import asyncio L27–27
+ create_async_engine import import create_async_engine L31–31
+ db import import db L35–35
+ get_backend import import get_backend L36–36
+ sa import import sa L30–30
+ sessionmaker import import sessionmaker L32–32
+ settings import import settings L34–34
+ sys import import sys L28–28
+ main function function main L124–132
+ repair function async_function repair L39–121
~ musehub/services/musehub_wire.py .py 2 symbols modified
~ wire_filter_objects
~ wire_push_objects
~ tests/test_wire_protocol.py .py 1 symbol removed
− test_filter_objects_storage_is_truth_not_db function async_function test_filter_objects_storage_is_truth_not_db L519–577
← Older Oldest on fix/mwp-db-only-filter
All commits
Newer → Latest on fix/mwp-db-only-filter

0 comments

No comments yet. Be the first to start the discussion.

To add a comment, use the Muse CLI: muse hub commit comment sha256:3c9077aa4718d0762aab92978ad0edec924e97e3e8f3ea989026bfe88e37d44a --body "your comment"