gabriel / musehub public
perf BREAKING feat/async-r2-upload-and-cf-worker #1 / 1
AI Agent gabriel · 166 days ago · Apr 12, 2026 · Diff

perf: async R2 upload + Cloudflare Worker for native R2 binding

- wire_push_object_pack now returns 200 immediately after a single DB write (storage_uri='pending', content_cache=raw_bytes). R2 upload happens in a background asyncio task, eliminating EC2→R2 network latency from the client's critical path entirely (~875ms → ~5ms).

- Fetch path handles pending objects: _pack_objects_response serves content_cache when storage_uri=='pending', so pulls never block on the background upload task.

- Startup recovery in lifespan re-queues any objects left pending from a prior crash — content_cache bytes survive in Postgres.

- Alembic migration 0042 adds content_cache BYTEA (nullable) to musehub_objects.

- cloudflare/pack-receiver/worker.js: Cloudflare Worker that receives MWP packs, stores objects directly to R2 via native R2 binding (~1ms/PUT, no EC2 hop), then calls /internal/register-objects to register IDs in Postgres.

- /internal/register-objects route + WORKER_INTERNAL_KEY setting for the Worker→MuseHub callback leg.

107/107 wire protocol tests pass.

sha256:38d96d4059b3ebc86e30f7fe0d6df03d245d82ebc3675453e5dbb6a481972b4a sha
sha256:431c69227922aa440200a17981d4be20967155cc5434eadf3ab10d58af3ff768 parent
+40 ~10 symbols
sha256:ddd698388937667bd382a398a0372f42f89e9569535fe8b3ca742b8e65ec925e snapshot
+40
symbols added
~10
symbols modified
0
dead code introduced
Semantic Changes 50 symbols
+ cloudflare
+ cloudflare/pack-receiver
~ alembic/versions/0042_add_object_content_cache.py .py 9 symbols added
+ branch_labels variable variable branch_labels L20–20
+ depends_on variable variable depends_on L21–21
+ down_revision variable variable down_revision L19–19
+ downgrade function function downgrade L31–32
+ annotations import import annotations L13–13
+ op import import op L15–15
+ sa import import sa L16–16
+ revision variable variable revision L18–18
+ upgrade function function upgrade L24–28
~ cloudflare/pack-receiver/worker.js .js 8 symbols added
+ enc function function enc L116–162
+ fetch method async_method fetch L182–283
+ msgpackDecode function function msgpackDecode L30–110
+ msgpackEncode function function msgpackEncode L114–169
+ r2Key function function r2Key L173–177
+ read function function read L46–107
+ readBin function function readBin L40–44
+ readStr function function readStr L34–38
~ cloudflare/pack-receiver/wrangler.toml .toml 17 symbols added
+ compatibility_date variable variable compatibility_date L0–0
+ env section env L0–0
+ production section production L0–0
+ name variable variable name L0–0
+ vars section vars L0–0
+ MUSEHUB_ORIGIN variable variable MUSEHUB_ORIGIN L0–0
+ staging section staging L0–0
+ name variable variable name L0–0
+ vars section vars L0–0
+ MUSEHUB_ORIGIN variable variable MUSEHUB_ORIGIN L0–0
+ main variable variable main L0–0
+ name variable variable name L0–0
+ r2_buckets[0] section r2_buckets[0] L0–0
+ binding variable variable binding L0–0
+ bucket_name variable variable bucket_name L0–0
+ vars section vars L0–0
+ MUSEHUB_ORIGIN variable variable MUSEHUB_ORIGIN L0–0
~ musehub/api/routes/wire.py .py 4 symbols added, 2 symbols modified
+ _WorkerObjectEntry class class _WorkerObjectEntry L873–880
+ _WorkerRegisterRequest class class _WorkerRegisterRequest L883–890
+ pydantic import import pydantic L40–40
+ worker_register_objects function async_function worker_register_objects L898–964
~ ObjMapDict
~ _pack_objects_response
~ musehub/config.py .py 1 symbol modified
~ Settings
~ musehub/db/musehub_models.py .py 1 symbol modified
~ MusehubObject
~ musehub/main.py .py 1 symbol modified
~ lifespan
~ musehub/services/musehub_wire.py .py 2 symbols added, 5 symbols modified
+ _upload_objects_to_r2 function async_function _upload_objects_to_r2 L769–808
+ update import import update L52–52
~ _ObjFetchMap
~ func
~ select
~ wire_fetch_objects
~ wire_push_object_pack
← Older Oldest on feat/async-r2-upload-and-cf-worker
All commits
Newer → Latest on feat/async-r2-upload-and-cf-worker

0 comments

No comments yet. Be the first to start the discussion.

To add a comment, use the Muse CLI: muse hub commit comment sha256:38d96d4059b3ebc86e30f7fe0d6df03d245d82ebc3675453e5dbb6a481972b4a --body "your comment"