gabriel / musehub public
fix patch task/proposal-models-v2 #11 / 19
AI Agent gabriel · 127 days ago · May 8, 2026 · Diff

fix: eliminate ghost objects — confirm guard, decompress bugs, repair script

Root cause investigation 2026-05-08: three bugs created phantom DB rows whose bytes were absent from storage, causing push/stream Phase-C rejections.

Layer 1 — decompress_objects.py (deploy/decompress_objects.py): Bug 1: bare_oid was only assigned inside the hash-mismatch branch but used on the success path → NameError silently caught → no object ever actually decompressed. Bug 2: even if defined, bare_oid is bare hex, so backend.put(bare_oid, …) stored at the wrong R2 key (objects/<hex> vs objects/sha256:<hex>). Fix: split_id(oid) moved above the hash check; backend.put uses canonical oid.

Layer 2 — wire_push_confirm (musehub/services/musehub_wire.py): Bug: confirm registered DB rows without verifying the client's R2 PUT succeeded. A silent PUT failure (expired presigned URL, network blip) left a DB row with no bytes → ghost. Fix: backend.exists(oid, repo_root=…) gate added; absent objects are rejected (logged + counted); returns confirmed/rejected counts.

Layer 3 — Phase-C distinction (confirmed correct, tested): P3.1 truly-missing → 'not in push bundle or storage' P3.2 ghost (DB row, no bytes) → 'found in DB but missing from storage' P3.3 inline bundle object → passes without DB pre-existence

Layer 4 — repair_objects.py (deploy/repair_objects.py): Added repo join (MusehubObjectRef → MusehubRepo) so LocalBackend gets owner/slug for repo_root resolution; no-op for S3/R2. Operational: docker exec musehub-blue python3 /app/deploy/repair_objects.py --dry-run

22 new tests across three files: D1-D7, C1-C7, P3.1-P3.3, R4.1-R4.5.

sha256:f1534b4412ad996d45cc3fbbc13fe61e6595f511ef02a9ccfa38ea9adb9c9bc2 sha
sha256:ded4068f3e65f31aed312006b87d6b91781d3fcfa8e0acfb8cbdcdc72782efd3 parent
+147 ~3 symbols
sha256:1259a9c3d1f2e0ebc505e70a6a274d866b22c70b9985ea9d96b105dd7b5d5ccd snapshot
+147
symbols added
~3
symbols modified
0
dead code introduced
Semantic Changes 150 symbols
~ GHOST_OBJECTS_INVESTIGATION.md .md 43 symbols added
+ Ghost Objects — Investigation & TDD Roadmap section Ghost Objects — Investigation & TDD Roadmap L1–318
+ Atomic Test Plan — Bottom-Up section Atomic Test Plan — Bottom-Up L161–257
+ Layer 0 — Terminology section Layer 0 — Terminology L163–166
+ py bug (isolated, no DB/R2 needed) section Layer 1 — decompress_objects.py bug (isolated, no DB/R2 needed) L166–191
+ code[python]@L169 variable variable code[python]@L169 L169–175
+ code[python]@L177 variable variable code[python]@L177 L177–183
+ code[python]@L185 variable variable code[python]@L185 L185–190
+ Layer 2 — wire_push_confirm missing existence check section Layer 2 — wire_push_confirm missing existence check L191–215
+ code[python]@L194 variable variable code[python]@L194 L194–200
+ code[python]@L202 variable variable code[python]@L202 L202–207
+ code[python]@L209 variable variable code[python]@L209 L209–214
+ Layer 3 — Phase C ghost check wiring section Layer 3 — Phase C ghost check wiring L215–241
+ code[python]@L218 variable variable code[python]@L218 L218–224
+ code[python]@L226 variable variable code[python]@L226 L226–232
+ code[python]@L234 variable variable code[python]@L234 L234–240
+ py correctness section Layer 4 — repair_objects.py correctness L241–257
+ code[python]@L244 variable variable code[python]@L244 L244–248
+ code[python]@L250 variable variable code[python]@L250 L250–254
+ Execution Order section Execution Order L308–318
+ Fix Plan (once root cause confirmed) section Fix Plan (once root cause confirmed) L257–308
+ py: define bare_oid on correct path section Fix 1 — decompress_objects.py: define bare_oid on correct path L259–279
+ code[python]@L261 variable variable code[python]@L261 L261–274
+ Fix 2 — wire_push_confirm: verify R2 existence before inserting DB row section Fix 2 — wire_push_confirm: verify R2 existence before inserting DB row L279–295
+ code[python]@L281 variable variable code[python]@L281 L281–294
+ py on staging to purge ghost rows section Fix 3 (operational) — Run repair_objects.py on staging to purge ghost rows L295–308
+ code[bash]@L297 variable variable code[bash]@L297 L297–302
+ Hypotheses (ranked by likelihood) section Hypotheses (ranked by likelihood) L136–161
+ H1 — push/confirm with unverified R2 PUTs (presigned path) section H1 — push/confirm with unverified R2 PUTs (presigned path) L138–144
+ H2 — R2 key migration orphaned DB rows section H2 — R2 key migration orphaned DB rows L144–153
+ py NameError left objects inaccessible section H3 — decompress_objects.py NameError left objects inaccessible L153–161
+ The Symptom section The Symptom L7–19
+ code@L9 variable variable code@L9 L9–13
+ What We Don't Know section What We Don't Know L110–136
+ What We Know for Certain section What We Know for Certain L19–110
+ How the ghost check works (Phase C) section How the ghost check works (Phase C) L21–36
+ code@L23 variable variable code@L23 L23–33
+ Presigned path (push/presign → R2 PUT → push/confirm) section Presigned path (push/presign → R2 PUT → push/confirm) L70–80
+ R2 key history — three formats existed section R2 key history — three formats existed L43–54
+ table@L45 section table@L45 L45–50
+ Streaming push path (inline objects — enc=raw/zlib/zstd/delta+zlib) section Streaming push path (inline objects — enc=raw/zlib/zstd/delta+zlib) L54–70
+ exists() checks section What backend.exists() checks L36–43
+ py bug (confirmed) section decompress_objects.py bug (confirmed) L80–110
+ code[python]@L82 variable variable code[python]@L82 L82–94
~ tests/test_decompress_objects.py .py 20 symbols added
+ _bare function function _bare L58–60
+ _buggy_process function async_function _buggy_process L68–132
+ _fixed_process function async_function _fixed_process L174–230
+ _oid function function _oid L54–55
+ _zlib_compress function function _zlib_compress L50–51
+ AsyncMock import import AsyncMock L39–39
+ MagicMock import import MagicMock L39–39
+ annotations import import annotations L35–35
+ asyncio import import asyncio L37–37
+ blob_id import import blob_id L43–43
+ pytest import import pytest L41–41
+ split_id import import split_id L43–43
+ zlib import import zlib L38–38
+ test_D1_bare_oid_name_error_on_success_path function async_function test_D1_bare_oid_name_error_on_success_path L140–167
+ test_D2_fixed_process_decompresses_valid_zlib function async_function test_D2_fixed_process_decompresses_valid_zlib L238–256
+ test_D3_fixed_process_puts_with_canonical_oid function async_function test_D3_fixed_process_puts_with_canonical_oid L264–286
+ test_D4_hash_mismatch_skipped function async_function test_D4_hash_mismatch_skipped L294–312
+ test_D5_plain_object_skipped function async_function test_D5_plain_object_skipped L320–337
+ test_D6_missing_from_r2_treated_as_plain function async_function test_D6_missing_from_r2_treated_as_plain L345–361
+ test_D7_dry_run_does_not_put function async_function test_D7_dry_run_does_not_put L369–387
~ tests/test_ghost_repair.py .py 45 symbols added
+ SFRAME_ERROR variable variable SFRAME_ERROR L44–44
+ _TEST_DB_URL variable variable _TEST_DB_URL L46–46
+ _body_iter function async_function _body_iter L93–94
+ _collect_frames function async_function _collect_frames L97–104
+ _commit_pack_frame function function _commit_pack_frame L80–83
+ _end_frame function function _end_frame L76–77
+ _fw variable variable _fw L43–43
+ _header_frame function function _header_frame L69–73
+ _insert_ghost_row function async_function _insert_ghost_row L133–143
+ _now function function _now L53–54
+ _o_frame function function _o_frame L86–90
+ _pack function function _pack L61–62
+ _push_stream_raw function function _push_stream_raw L107–130
+ _run_repair function async_function _run_repair L228–236
+ _uid function function _uid L57–58
+ _wrap function function _wrap L65–66
+ AsyncSession import import AsyncSession L29–29
+ MuseWireFrameWriter import import MuseWireFrameWriter L32–32
+ SFRAME_COMMIT_PACK import import SFRAME_COMMIT_PACK L35–35
+ SFRAME_END import import SFRAME_END L35–35
+ SFRAME_HEADER import import SFRAME_HEADER L35–35
+ SFRAME_OBJECT import import SFRAME_OBJECT L35–35
+ annotations import import annotations L20–20
+ blob_id import import blob_id L33–33
+ create_async_engine import import create_async_engine L29–29
+ create_repo import import create_repo L41–41
+ datetime import import datetime L23–23
+ db import import db L34–34
+ fake_id import import fake_id L33–33
+ func function import func L27–27
+ msgpack import import msgpack L25–25
+ pg_insert import import pg_insert L28–28
+ pytest import import pytest L26–26
+ select import import select L27–27
+ sessionmaker import import sessionmaker L30–30
+ timezone import import timezone L23–23
+ zlib import import zlib L22–22
+ test_P3_1_truly_missing_rejected_with_not_in_bundle function async_function test_P3_1_truly_missing_rejected_with_not_in_bundle L151–171
+ test_P3_2_ghost_rejected_with_ghost_message function async_function test_P3_2_ghost_rejected_with_ghost_message L175–198
+ test_P3_3_object_in_bundle_passes_phase_c function async_function test_P3_3_object_in_bundle_passes_phase_c L202–221
+ test_R4_1_repair_deletes_ghost_rows function async_function test_R4_1_repair_deletes_ghost_rows L240–264
+ test_R4_2_repair_preserves_present_objects function async_function test_R4_2_repair_preserves_present_objects L268–293
+ test_R4_3_repair_dry_run_does_not_delete function async_function test_R4_3_repair_dry_run_does_not_delete L297–317
+ test_R4_4_repair_then_push_with_objects_succeeds function async_function test_R4_4_repair_then_push_with_objects_succeeds L321–354
+ test_R4_5_repair_with_no_orphans_is_a_noop function async_function test_R4_5_repair_with_no_orphans_is_a_noop L358–384
~ tests/test_push_confirm_ghost_guard.py .py 39 symbols added
+ SFRAME_ERROR variable variable SFRAME_ERROR L57–57
+ _body_iter function async_function _body_iter L100–101
+ _collect_frames function async_function _collect_frames L104–111
+ _commit_pack_frame function function _commit_pack_frame L94–97
+ _end_frame function function _end_frame L90–91
+ _fw variable variable _fw L60–60
+ _header_frame function function _header_frame L83–87
+ _now function function _now L67–68
+ _pack function function _pack L75–76
+ _push_stream_raw function function _push_stream_raw L114–128
+ _uid function function _uid L71–72
+ _wrap function function _wrap L79–80
+ AsyncMock import import AsyncMock L41–41
+ AsyncSession import import AsyncSession L46–46
+ MagicMock import import MagicMock L41–41
+ MuseWireFrameWriter import import MuseWireFrameWriter L48–48
+ SFRAME_COMMIT_PACK import import SFRAME_COMMIT_PACK L51–51
+ SFRAME_END import import SFRAME_END L51–51
+ SFRAME_HEADER import import SFRAME_HEADER L51–51
+ annotations import import annotations L37–37
+ blob_id import import blob_id L49–49
+ create_repo import import create_repo L58–58
+ datetime import import datetime L40–40
+ db import import db L50–50
+ fake_id import import fake_id L49–49
+ func function import func L45–45
+ msgpack import import msgpack L43–43
+ patch import import patch L41–41
+ pytest import import pytest L44–44
+ select import import select L45–45
+ timezone import import timezone L40–40
+ zlib import import zlib L39–39
+ test_C1_confirm_rejects_absent_object_ghost_guard function async_function test_C1_confirm_rejects_absent_object_ghost_guard L136–164
+ test_C2_fixed_confirm_rejects_absent_object function async_function test_C2_fixed_confirm_rejects_absent_object L172–201
+ test_C3_fixed_confirm_accepts_present_object function async_function test_C3_fixed_confirm_accepts_present_object L209–237
+ test_C4_fixed_confirm_idempotent_for_existing_row function async_function test_C4_fixed_confirm_idempotent_for_existing_row L245–268
+ test_C5_fixed_confirm_mixed_batch function async_function test_C5_fixed_confirm_mixed_batch L276–315
+ test_C6_phase_c_passes_after_valid_confirm function async_function test_C6_phase_c_passes_after_valid_confirm L323–349
+ test_C7_phase_c_rejects_ghost_from_bad_confirm function async_function test_C7_phase_c_rejects_ghost_from_bad_confirm L357–393
~ deploy/decompress_objects.py .py 1 symbol modified
~ backfill
~ deploy/repair_objects.py .py 1 symbol modified
~ repair
~ musehub/services/musehub_wire.py .py 1 symbol modified
~ wire_push_confirm
Older feat(proposals): Phase 7 — UI/template rendering for strategy, t… sha256:ded4068f3e65f31aed312006b87d6b91781d3fcfa8e0acfb8cbdcdc72782efd3 All commits Newer fix: add ghost guard to wire_push_presign (Layer 5) sha256:4feba6d72278319d9831dd3d18b610e5976562d1e474b8630d55bdf2de1ba04b

0 comments

No comments yet. Be the first to start the discussion.

To add a comment, use the Muse CLI: muse hub commit comment sha256:f1534b4412ad996d45cc3fbbc13fe61e6595f511ef02a9ccfa38ea9adb9c9bc2 --body "your comment"