perf: eliminate head_object, raise S3 pool+thread-pool to 100, global R2 semaphore
Three bottlenecks eliminated in the pack upload hot path:
1. Remove redundant head_object from S3Backend._s3_put - DB filter-objects endpoint already ensures only missing objects reach _s3_put - head_object + put_object = 2 R2 round-trips per object (156 ms each) - put_object alone = 1 round-trip (78 ms) — 2x throughput improvement - R2 PutObject is idempotent, so the double-write race is safe
2. Raise boto3 connection pool from 10 → 100 (S3_POOL_CONNECTIONS) - Default 10 connections serialized 64 concurrent asyncio.to_thread puts through a 10-slot queue, negating the asyncio.gather concurrency - 100 connections lets every in-flight thread make immediate progress
3. Dedicated S3 thread pool with 100 workers (S3_THREAD_POOL_SIZE) - asyncio.to_thread uses the default executor: min(32, cpu+4) = 8 threads on a 4-CPU server — only 8 concurrent R2 puts regardless of semaphore - loop.run_in_executor(_get_s3_thread_pool()) uses a 100-worker pool, matching the boto3 pool and the asyncio semaphore
4. Move asyncio.Semaphore to module level (_get_r2_put_sem) - Per-request semaphore meant 8 parallel packs × 64 slots = 512 concurrent R2 attempts against a 10-connection boto3 pool - Shared module-level semaphore(100) caps total concurrent R2 puts at 100 across all simultaneous pack requests — matches connection pool exactly
Expected result: 2000-file push — from 78s serial to ~1.5s parallel. All 107 wire protocol tests pass.
0 comments
muse hub commit comment sha256:431c69227922aa440200a17981d4be20967155cc5434eadf3ab10d58af3ff768 --body "your comment"
No comments yet. Be the first to start the discussion.