gabriel / musehub public
fix BREAKING intel task/intel-hardening-8b #1 / 1
AI Agent gabriel · 3 days ago · Nov 13, 2026 · Diff

fix(intel): bounded failure recording, terminal resource-limit failures, no green zero without a published result

Stage 1 of musehub#8 hardening (real-repo localhost run: PostgreSQL OOM mid-ingest).

Failure persistence: the worker's error handler called fail_job once; with PostgreSQL recovering it raised, the row stayed 'running' and the stale sweep would have re-run the 1.7k-commit index ~10 min later. _record_failure now retries the fenced write (5 tries, 1/2/4/8 s backoff, 30 s per try), logs job/type/repo/attempt/error when it gives up, never raises, never marks the job done; a retry after reclaim+re-claim is still fenced by attempt. Retry classification (intel.* only): memory ceiling, CPU budget, wall clock, lease cap, a dead child and a DB-connection traceback from the child are terminal (fail_job retryable=False). A PostgreSQL backend OOM reaches the worker only as an IntelChildError traceback, so that case is classified by traceback text (documented limitation); other errors keep the bounded _MAX_ATTEMPTS retry. Non-intel jobs unchanged. Stale sweep: a stale intel.code lease is failed, not re-run (resource-intensive); other job types still reclaimed; exhausted attempts still fail.

Provider status: provider_state mapped any finished job that published nothing for the head to 'empty', which counted as current, so the Alerts strip drew a green 0 for providers that had run against an empty index after intel.code died. 'empty' is removed: only a published result for the head is current; a zero is a published result with count 0.

Tests: tests/test_intel_failure_hardening.py (16), +2 alerts reconciliation, health-status updated; F7 and B1 now expect a terminal failure for an intel lease cap.

sha256:f9cda821d29cd59f4b95c1cf9cd667e1af3a8e40de55f505bcf50fdd0dc52b1d sha
+48 ~10 −1 symbols
sha256:22d49dc65f4e7fd36005325fd37a7fa6aa8b8d49ef3695f4c16b5efe40201229 snapshot
+48
symbols added
~10
symbols modified
−1
symbol removed
0
dead code introduced
Semantic Changes 59 symbols
~ tests/test_intel_failure_hardening.py .py 40 symbols added
+ RaisingChild class class RaisingChild L46–53
+ __init__ method method __init__ L47–49
+ run method async_method run L51–53
+ _outage function function _outage L56–57
+ _run_intel function async_function _run_intel L60–68
+ _stale function async_function _stale L223–228
+ AsyncSession import import AsyncSession L29–29
+ IntelChildCpuExceeded import import IntelChildCpuExceeded L34–34
+ IntelChildDied import import IntelChildDied L34–34
+ IntelChildError import import IntelChildError L34–34
+ IntelChildMemoryExceeded import import IntelChildMemoryExceeded L34–34
+ IntelChildTimeout import import IntelChildTimeout L34–34
+ J import import J L41–41
+ MusehubBackgroundJob import import MusehubBackgroundJob L33–33
+ OperationalError import import OperationalError L28–28
+ add_job import import add_job L43–43
+ annotations import import annotations L20–20
+ asyncio import import asyncio L22–22
+ create_repo import import create_repo L42–42
+ database import import database L32–32
+ datetime import import datetime L24–24
+ job import import job L43–43
+ logging import import logging L23–23
+ pytest import import pytest L26–26
+ select import import select L27–27
+ timedelta import import timedelta L24–24
+ timezone import import timezone L24–24
+ update import import update L27–27
+ worker import import worker L31–31
+ worker_db import import worker_db L43–43
+ test_c1_c4_a_resource_limit_failure_is_terminal_and_never_re_run function async_function test_c1_c4_a_resource_limit_failure_is_terminal_and_never_re_run L80–85
+ test_c2_a_database_connection_failure_in_the_child_is_terminal function async_function test_c2_a_database_connection_failure_in_the_child_is_terminal L97–101
+ test_c3_a_non_intel_job_failure_is_unchanged function async_function test_c3_a_non_intel_job_failure_is_unchanged L118–129
+ test_c3_an_ordinary_child_error_is_still_retried_within_the_attempt_bound function async_function test_c3_an_ordinary_child_error_is_still_retried_within_the_attempt_bound L107–114
+ test_c3_the_lease_cap_of_an_intel_job_is_terminal function async_function test_c3_the_lease_cap_of_an_intel_job_is_terminal L133–148
+ test_p1_fail_job_rides_out_a_short_outage function async_function test_p1_fail_job_rides_out_a_short_outage L154–167
+ test_p2_a_database_that_never_recovers_is_bounded_logged_and_never_marks_the_job_done function async_function test_p2_a_database_that_never_recovers_is_bounded_logged_and_never_marks_the_job_done L171–185
+ test_p3_a_hung_fail_job_is_bounded_by_a_per_try_timeout function async_function test_p3_a_hung_fail_job_is_bounded_by_a_per_try_timeout L189–197
+ test_p4_a_late_persistence_retry_cannot_touch_a_job_another_execution_now_owns function async_function test_p4_a_late_persistence_retry_cannot_touch_a_job_another_execution_now_owns L201–218
+ test_s1_a_stale_intel_code_claim_is_failed_not_re_run function async_function test_s1_a_stale_intel_code_claim_is_failed_not_re_run L232–249
~ musehub/worker.py .py 5 symbols added, 1 symbol modified
+ _DB_CONNECTION_MARKERS variable variable _DB_CONNECTION_MARKERS L320–324
+ _FAIL_PERSIST_DELAYS variable variable _FAIL_PERSIST_DELAYS L314–314
+ _FAIL_PERSIST_TRY_TIMEOUT variable variable _FAIL_PERSIST_TRY_TIMEOUT L316–316
+ _failure_retryable function function _failure_retryable L327–342
+ _record_failure function async_function _record_failure L345–379
+ test_a_done_job_for_the_head_does_not_hide_an_older_result_as_current function async_function test_a_done_job_for_the_head_does_not_hide_an_older_result_as_current L206–212
+ test_a_finished_job_without_a_published_result_is_never_a_green_zero function async_function test_a_finished_job_without_a_published_result_is_never_a_green_zero L189–202
~ tests/test_intel_lease_fencing.py .py 1 symbol added, 1 symbol removed
− test_f7_the_lease_cap_cancels_the_execution_and_records_a_retryable_failure function async_function test_f7_the_lease_cap_cancels_the_execution_and_records_a_retryable_failure L171–182
+ test_f7_the_lease_cap_cancels_the_execution_and_records_a_terminal_failure function async_function test_f7_the_lease_cap_cancels_the_execution_and_records_a_terminal_failure L171–182
← Older Oldest on task/intel-hardening-8b
All commits
Newer → Latest on task/intel-hardening-8b

0 comments

No comments yet. Be the first to start the discussion.

To add a comment, use the Muse CLI: muse hub commit comment sha256:f9cda821d29cd59f4b95c1cf9cd667e1af3a8e40de55f505bcf50fdd0dc52b1d --body "your comment"