fix(intel): bounded failure recording, terminal resource-limit failures, no green zero without a published result
Stage 1 of musehub#8 hardening (real-repo localhost run: PostgreSQL OOM mid-ingest).
Failure persistence: the worker's error handler called fail_job once; with PostgreSQL recovering it raised, the row stayed 'running' and the stale sweep would have re-run the 1.7k-commit index ~10 min later. _record_failure now retries the fenced write (5 tries, 1/2/4/8 s backoff, 30 s per try), logs job/type/repo/attempt/error when it gives up, never raises, never marks the job done; a retry after reclaim+re-claim is still fenced by attempt. Retry classification (intel.* only): memory ceiling, CPU budget, wall clock, lease cap, a dead child and a DB-connection traceback from the child are terminal (fail_job retryable=False). A PostgreSQL backend OOM reaches the worker only as an IntelChildError traceback, so that case is classified by traceback text (documented limitation); other errors keep the bounded _MAX_ATTEMPTS retry. Non-intel jobs unchanged. Stale sweep: a stale intel.code lease is failed, not re-run (resource-intensive); other job types still reclaimed; exhausted attempts still fail.
Provider status: provider_state mapped any finished job that published nothing for the head to 'empty', which counted as current, so the Alerts strip drew a green 0 for providers that had run against an empty index after intel.code died. 'empty' is removed: only a published result for the head is current; a zero is a published result with count 0.
Tests: tests/test_intel_failure_hardening.py (16), +2 alerts reconciliation, health-status updated; F7 and B1 now expect a terminal failure for an intel lease cap.
Semantic Changes
59 symbols
0 comments
muse hub commit comment sha256:f9cda821d29cd59f4b95c1cf9cd667e1af3a8e40de55f505bcf50fdd0dc52b1d --body "your comment"
No comments yet. Be the first to start the discussion.