gabriel / musehub public
fix BREAKING fix/phase8-blob-put-30s #1 / 1
AI Agent gabriel · 126 days ago · May 11, 2026 · Diff

fix: resolve 30s blob PUT in wire_push_stream phase 8

Root cause: Python 3.14's http.client incorrectly parses the HTTP exchange when botocore uses Expect: 100-continue and MinIO sends 100 Continue + 200 OK in the same TCP segment. The 200 OK bytes appear as unparsed_data after the 100 Continue headers, triggering MissingHeaderBodySeparatorDefect in urllib3. urllib3 catches the defect but pools the connection without draining the response body (dirty connection). Phase 8's next PUT reuses this dirty connection; MinIO reads the leftover bytes as a new request, becomes confused, and closes the connection after its 30s keep-alive timeout.

Fix: register a before-send.s3.PutObject event handler in BlobBackend._get_client() that removes the Expect header before each PUT. Without Expect: 100-continue, botocore sends headers and body in a single shot. MinIO responds with a plain 200 OK that Python 3.14's http.client parses cleanly. No dirty connection.

Also adds Config(request_checksum_calculation='when_required', response_checksum_validation='when_required') to suppress optional CRC32 headers that add response overhead.

Tests added: tests/test_phase8_atomic.py T1-T9 TDD regression suite tests/test_diag_phase8.py event-loop alive + timing diagnostics tests/bench_phase8_repro.py minimal INSERT step benchmark tests/bench_phases.py layered push pipeline benchmark

Result: phase 8 blob_put drops from 30.266s to 0.005s.

sha256:345db3b22e487edaa82b2b4056956ef2f74267db2bf059d9e7974e994e1d094e sha
sha256:6fadea36d754a8a64a5fda40f88d06e5a117f814d5617a6b98132f9148b49280 parent
+134 ~3 symbols
sha256:9f7057b00b18c78e7317744e9d397ab050693d6904c7d41c2890383f8042b989 snapshot
+134
symbols added
~3
symbols modified
0
dead code introduced
Semantic Changes 137 symbols
~ tests/bench_phase8_repro.py .py 29 symbols added
+ _DB_URL variable variable _DB_URL L42–45
+ _MINIO variable variable _MINIO L46–52
+ _UPSERT_SET variable variable _UPSERT_SET L54–59
+ _commit_row function function _commit_row L96–111
+ _hr function function _hr L70–71
+ _insert_stmt function function _insert_stmt L114–122
+ _ms function function _ms L62–63
+ _row function function _row L66–67
+ _seed_repo function async_function _seed_repo L74–93
+ Base import import Base L38–38
+ BlobBackend import import BlobBackend L39–39
+ NullPool import import NullPool L33–33
+ Path import import Path L26–26
+ annotations import import annotations L18–18
+ async_sessionmaker import import async_sessionmaker L31–31
+ asyncio import import asyncio L20–20
+ blob_id import import blob_id L40–40
+ create_async_engine import import create_async_engine L31–31
+ datetime import import datetime L21–21
+ db import import db L37–37
+ muse_cli_models import import musehub.db.muse_cli_models L36–36
+ os import import os L22–22
+ pg_insert import import pg_insert L32–32
+ secrets import import secrets L23–23
+ select import import select L34–34
+ sys import import sys L24–24
+ text import import text L34–34
+ time import import time L25–25
+ main function async_function main L125–207
~ tests/bench_phases.py .py 41 symbols added
+ _DB_URL variable variable _DB_URL L40–43
+ _MINIO_BUCKET variable variable _MINIO_BUCKET L45–45
+ _MINIO_ENDPOINT variable variable _MINIO_ENDPOINT L44–44
+ _MINIO_KEY variable variable _MINIO_KEY L46–46
+ _MINIO_REGION variable variable _MINIO_REGION L48–48
+ _MINIO_SECRET variable variable _MINIO_SECRET L47–47
+ _backend function function _backend L104–111
+ _fw variable variable _fw L50–50
+ _hr function function _hr L114–115
+ _make_chain function function _make_chain L63–86
+ _pack function function _pack L55–56
+ _push_body function function _push_body L89–101
+ _row function function _row L118–120
+ _wrap function function _wrap L59–60
+ bench_l0_direct function function bench_l0_direct L125–143
+ bench_l1_blob_backend function async_function bench_l1_blob_backend L148–156
+ bench_l2_db_snapshot function async_function bench_l2_db_snapshot L161–183
+ bench_l3_wire function async_function bench_l3_wire L188–234
+ Base import import Base L31–31
+ BlobBackend import import BlobBackend L36–36
+ MuseWireFrameWriter import import MuseWireFrameWriter L29–29
+ NullPool import import NullPool L27–27
+ Path import import Path L20–20
+ SFRAME_COMMIT_PACK import import SFRAME_COMMIT_PACK L33–33
+ SFRAME_END import import SFRAME_END L33–33
+ SFRAME_HEADER import import SFRAME_HEADER L33–33
+ SFRAME_PROGRESS import import SFRAME_PROGRESS L33–33
+ SFRAME_RESULT import import SFRAME_RESULT L33–33
+ annotations import import annotations L13–13
+ async_sessionmaker import import async_sessionmaker L26–26
+ asyncio import import asyncio L15–15
+ blob_id import import blob_id L30–30
+ create_async_engine import import create_async_engine L26–26
+ msgpack import import msgpack L25–25
+ muse_cli_models import import musehub.db.muse_cli_models L32–32
+ now_utc_iso import import now_utc_iso L30–30
+ os import import os L16–16
+ secrets import import secrets L17–17
+ sys import import sys L18–18
+ time import import time L19–19
+ main function async_function main L239–256
~ tests/test_diag_phase8.py .py 26 symbols added
+ _BLOB_PUT_SLOW_THRESHOLD_S variable variable _BLOB_PUT_SLOW_THRESHOLD_S L36–36
+ _body_iter function async_function _body_iter L85–86
+ _fw variable variable _fw L35–35
+ _make_1commit_body function function _make_1commit_body L47–82
+ _pack function function _pack L39–40
+ _run_wire_push function async_function _run_wire_push L89–100
+ _wrap function function _wrap L43–44
+ AsyncSession import import AsyncSession L22–22
+ MuseWireFrameWriter import import MuseWireFrameWriter L24–24
+ SFRAME_COMMIT_PACK import import SFRAME_COMMIT_PACK L26–26
+ SFRAME_END import import SFRAME_END L26–26
+ SFRAME_HEADER import import SFRAME_HEADER L26–26
+ SFRAME_PROGRESS import import SFRAME_PROGRESS L26–26
+ SFRAME_RESULT import import SFRAME_RESULT L26–26
+ annotations import import annotations L13–13
+ asyncio import import asyncio L15–15
+ blob_id import import blob_id L25–25
+ create_repo import import create_repo L33–33
+ datetime import import datetime L16–16
+ msgpack import import msgpack L20–20
+ now_utc_iso import import now_utc_iso L25–25
+ pytest import import pytest L21–21
+ secrets import import secrets L17–17
+ time import import time L18–18
+ test_diag_phase8_blob_put_timing function async_function test_diag_phase8_blob_put_timing L163–200
+ test_diag_phase8_event_loop_alive_during_blob_put function async_function test_diag_phase8_event_loop_alive_during_blob_put L104–159
~ tests/test_phase8_atomic.py .py 38 symbols added
+ _BACKEND variable variable _BACKEND L82–88
+ _THRESHOLD_MS variable variable _THRESHOLD_MS L89–89
+ _collect function async_function _collect L102–106
+ _fw_wire variable variable _fw_wire L54–54
+ _gen_t1 function async_function _gen_t1 L111–113
+ _gen_t2 function async_function _gen_t2 L129–132
+ _gen_t3 function async_function _gen_t3 L148–153
+ _gen_t5 function async_function _gen_t5 L208–234
+ _gen_t6 function async_function _gen_t6 L279–306
+ _make_1commit_body function function _make_1commit_body L57–80
+ _put_timed function async_function _put_timed L94–99
+ _put_timed_backend function async_function _put_timed_backend L272–276
+ AsyncSession import import AsyncSession L45–45
+ BlobBackend import import BlobBackend L48–48
+ MuseWireFrameWriter import import MuseWireFrameWriter L51–51
+ SFRAME_COMMIT_PACK import import SFRAME_COMMIT_PACK L52–52
+ SFRAME_END import import SFRAME_END L52–52
+ SFRAME_HEADER import import SFRAME_HEADER L52–52
+ _storage_backends import import _storage_backends L49–49
+ annotations import import annotations L36–36
+ asyncio import import asyncio L38–38
+ blob_id import import blob_id L47–47
+ msgpack import import msgpack L50–50
+ now_utc_iso import import now_utc_iso L47–47
+ os import import os L39–39
+ pytest import import pytest L43–43
+ secrets import import secrets L40–40
+ text import import text L44–44
+ time import import time L41–41
+ test_T1_put_after_yield_no_db function async_function test_T1_put_after_yield_no_db L117–124
+ test_T2_put_after_yield_open_tx function async_function test_T2_put_after_yield_open_tx L136–143
+ test_T3_put_after_yield_then_db_execute function async_function test_T3_put_after_yield_then_db_execute L157–164
+ test_T4_nullpool_reconnect_does_not_set_socket_timeout function async_function test_T4_nullpool_reconnect_does_not_set_socket_timeout L170–203
+ test_T5_nullpool_reconnect_3yields_then_put_fresh_backend function async_function test_T5_nullpool_reconnect_3yields_then_put_fresh_backend L238–267
+ test_T6_warmed_backend_after_p7_then_p8_put function async_function test_T6_warmed_backend_after_p7_then_p8_put L310–337
+ test_T7_many_nullpool_cycles_then_warmed_backend_put function async_function test_T7_many_nullpool_cycles_then_warmed_backend_put L343–381
+ test_T8_s3put_sub_timing_in_wire_push_phase8 function async_function test_T8_s3put_sub_timing_in_wire_push_phase8 L387–478
+ test_T9_wire_push_phase8_blob_put_under_threshold function async_function test_T9_wire_push_phase8_blob_put_under_threshold L484–550
~ musehub/services/musehub_wire.py .py 1 symbol modified
~ wire_push_stream
~ musehub/storage/backends.py .py 2 symbols modified
~ BlobBackend
~ _get_client
← Older Oldest on fix/phase8-blob-put-30s
All commits
Newer → Latest on fix/phase8-blob-put-30s

0 comments

No comments yet. Be the first to start the discussion.

To add a comment, use the Muse CLI: muse hub commit comment sha256:345db3b22e487edaa82b2b4056956ef2f74267db2bf059d9e7974e994e1d094e --body "your comment"