gabriel / musehub public
feat BREAKING feat/background-job-queue #1 / 1
gabriel · 172 days ago · Apr 9, 2026 · Diff

feat: replace asyncio.create_task with durable DB-backed job queue

Symbol indexing and GC now run in a separate worker process instead of fire-and-forget asyncio tasks inside uvicorn. This is the correct fix for the OOM crash that occurred when indexing 906 commits × 4000 files.

Architecture: - musehub_background_jobs table (migration 0032): pending/running/done/failed lifecycle with retry (up to 3 attempts), SKIP LOCKED for multi-worker safety - musehub/services/musehub_jobs.py: enqueue_job, claim_next_job, complete_job, fail_job — all thin DB operations, no business logic - musehub/worker.py: standalone Python process, polls every 1s, handles symbol_index and gc job types, crashes without affecting the web server - docker-compose.yml: new 'worker' service, 2 GiB memory limit (web server stays at 512 MiB), read_only filesystem, restarts on failure

Push handler change: Before: asyncio.create_task(_build_symbol_index_async(...)) After: await enqueue_job(session, repo_id, 'symbol_index', {'head': ...})

The push handler returns immediately — zero latency impact. The worker picks up and processes jobs asynchronously in its own memory space.

Tests: 7 new tests cover enqueue/claim/complete/fail/retry lifecycle and verify the push route calls enqueue_job (not asyncio.create_task) via spy.

sha256:1b4d7cb810e752df1b1aec17e17dba03249d3499201f92ca284c0475d875bb05 sha
sha256:e649c062850aae26b3d40684a22d5bda0443feca8f348bb6f37ddf6046e965e7 parent
+65 ~2 −4 symbols
sha256:012389adc2e342fcb4affdba284ee90b09e7265d846a2584c4a120fde8e791e7 snapshot
+65
symbols added
~2
symbols modified
−4
symbols removed
0
dead code introduced
Semantic Changes 71 symbols
~ alembic/versions/0032_background_jobs.py .py 9 symbols added
+ branch_labels variable variable branch_labels L18–18
+ depends_on variable variable depends_on L19–19
+ down_revision variable variable down_revision L17–17
+ downgrade function function downgrade L54–56
+ annotations import import annotations L11–11
+ op import import op L14–14
+ sa import import sa L13–13
+ revision variable variable revision L16–16
+ upgrade function function upgrade L22–51
~ musehub/services/musehub_jobs.py .py 17 symbols added
+ _MAX_ATTEMPTS variable variable _MAX_ATTEMPTS L35–35
+ _test_enqueued_calls variable variable _test_enqueued_calls L38–38
+ _utc_now function function _utc_now L41–42
+ claim_next_job function async_function claim_next_job L77–102
+ complete_job function async_function complete_job L105–111
+ enqueue_job function async_function enqueue_job L45–74
+ fail_job function async_function fail_job L114–141
+ AsyncSession import import AsyncSession L29–29
+ annotations import import annotations L22–22
+ datetime import import datetime L25–25
+ db import import db L31–31
+ logging import import logging L24–24
+ pg_insert import import pg_insert L28–28
+ select import import select L27–27
+ timezone import import timezone L25–25
+ update import import update L27–27
+ logger variable variable logger L33–33
~ musehub/worker.py .py 13 symbols added
+ _HANDLERS variable variable _HANDLERS L69–72
+ _MAX_JOBS variable variable _MAX_JOBS L29–29
+ _POLL_INTERVAL variable variable _POLL_INTERVAL L28–28
+ _handle_gc function async_function _handle_gc L53–66
+ _handle_symbol_index function async_function _handle_symbol_index L36–50
+ _process_one function async_function _process_one L79–115
+ annotations import import annotations L14–14
+ asyncio import import asyncio L16–16 ← moved from musehub/api/routes/wire.py
+ logging import import logging L17–17
+ os import import os L18–18
+ traceback import import traceback L19–19
+ logger variable variable logger L26–26
+ run function async_function run L118–143
~ tests/test_background_jobs.py .py 25 symbols added
+ _mp function function _mp L36–37
+ _utc_now function function _utc_now L32–33
+ AsyncClient import import AsyncClient L18–18
+ AsyncSession import import AsyncSession L20–20
+ annotations import import annotations L10–10
+ claim_next_job import import claim_next_job L23–23
+ complete_job import import complete_job L23–23
+ datetime import import datetime L14–14
+ db import import db L22–22
+ enqueue_job import import enqueue_job L23–23
+ factory_create_repo import import factory_create_repo L29–29
+ fail_job import import fail_job L23–23
+ hashlib import import hashlib L12–12
+ msgpack import import msgpack L16–16
+ pytest import import pytest L17–17
+ select import import select L19–19
+ timezone import import timezone L14–14
+ uuid import import uuid L13–13
+ test_claim_next_job_returns_none_when_queue_empty function async_function test_claim_next_job_returns_none_when_queue_empty L82–85
+ test_claim_next_job_returns_oldest_pending function async_function test_claim_next_job_returns_oldest_pending L62–78
+ test_complete_job_sets_done_status function async_function test_complete_job_sets_done_status L91–108
+ test_enqueue_job_creates_pending_row function async_function test_enqueue_job_creates_pending_row L43–56
+ test_fail_job_permanently_fails_after_max_attempts function async_function test_fail_job_permanently_fails_after_max_attempts L136–157
+ test_fail_job_retries_on_first_failure function async_function test_fail_job_retries_on_first_failure L114–132
+ test_push_enqueues_symbol_index_and_gc_jobs function async_function test_push_enqueues_symbol_index_and_gc_jobs L163–223
~ docker-compose.override.yml .yml
~ docker-compose.yml .yml
~ musehub/api/routes/wire.py .py 4 symbols removed, 1 symbol modified
− _MAX_INDEX_COMMITS_PER_PUSH variable variable _MAX_INDEX_COMMITS_PER_PUSH L794–794
− _build_symbol_index_async function async_function _build_symbol_index_async L797–832
− _run_gc_async function async_function _run_gc_async L835–850
− asyncio import import asyncio L35–35 → moved to musehub/worker.py
~ push
~ musehub/db/musehub_models.py .py 1 symbol added
+ MusehubBackgroundJob class class MusehubBackgroundJob L982–1023
~ tests/conftest.py .py 1 symbol modified
~ _stub_push_background_tasks
← Older Oldest on feat/background-job-queue
All commits
Newer → Latest on feat/background-job-queue

0 comments

No comments yet. Be the first to start the discussion.

To add a comment, use the Muse CLI: muse hub commit comment sha256:1b4d7cb810e752df1b1aec17e17dba03249d3499201f92ca284c0475d875bb05 --body "your comment"