Skip to content

Capacity Guard & Account Failover

Every autonomous surface in vita runs on a Claude account with usage limits. When an account exhausts, the surfaces that depend on it die — silently, and usually overnight. The capacity guard's job is stated plainly:

bio-zack stops being the failover mechanism.

Capacity exists during nearly every outage. It just wasn't being selected.

Status: phases 1–3 shipped 2026-07-26 (dd07d598e, 0cbb34343), from plans/proposals/2026-07-06-capacity-kill-guard-and-spine-failover.md — ranked #1 [94] across 11 sleep revisions and unbuilt the whole time.

Why

Five user-facing exhibits in three weeks:

Date What happened
Jul-6 /sleep dark all night while another account sat at 7d:0% usage
Jul-8 a live user turn was dropped; /sleep dead at reflect, exec loop dead 5–11am, telegram crashed
Jul-10 account-2 hit max — bio-zack manually failed over
Jul-18 manual failover again
Jul-26 his first action home from a family trip was hand-switching accounts after a weekly limit killed morning_kickoff, deferred_prompt, mastermind_ideas, garmin_review, and board_meeting

The canonical exhibit is the Jul-8 blackout (executive cycle c404), where the credit-exhaustion and wedged-stepwise signatures were deliberately kept unflattened in the record because they are different failures with different fixes.

Architecture

   a surface fails
        │
        ▼
  classify_failure(text)
        │
        ├── "cancelled"  ──▶ not a capacity problem, don't failover
        ├── "error"      ──▶ ordinary failure, normal retry
        └── "capacity_kill"
                │
                ▼
        parse_reset(text) ──▶ CapacityKill(reset_at)
                │
                ▼
        record_block(account, reset_at)     data/capacity/blocks.json
        (confirmed exhaustion OUTRANKS stale telemetry; expires at reset)
                │
                ▼
        pick_failover_target(states, exclude=blocked)
                │  3-tier availability model
                │  (fresh telemetry > aged telemetry > unknown)
                ▼
        switch_account()  ──▶ verify with `claude auth status`
                │
                ├── success ──▶ record_event(action="failover")  ──▶ alert
                └── no target ─▶ defer_run(until=reset_at)       ──▶ alert

Reading the real error vocabulary

The guard does not pattern-match optimistically. classify_failure() works against the actual Anthropic capacity-kill strings recorded from live failures, in both forms they appear, and distinguishes three classes:

  • capacity_kill — a genuine limit
  • cancelled — a turn that was interrupted (never triggers failover)
  • error — anything else

Two guards keep it honest. _NOT_CAPACITY_SIGNS and a _USAGE_CONTEXT requirement prevent false positives on messages that merely mention "limit". And a loud parse_miss antibody fires when a message looks like a capacity kill but the reset time can't be parsed — rather than silently guessing a reset and re-arming into a still-dead window.

The blocks ledger

data/capacity/blocks.json records confirmed exhaustion, and it outranks telemetry.

This matters because the usage telemetry in ~/.claude/accounts/usage.json can be stale — it reflects the last time a client wrote it, not the current truth. An account that just returned a capacity kill is definitively blocked, whatever the telemetry says. Blocks expire automatically at the parsed reset time, so a stale block can't strand an account that has since recovered.

Account selection uses a three-tier availability model that prefers fresh telemetry (5-hour window under ~90 min old, 7-day window under ~24 h) and degrades explicitly to unknown rather than assuming availability. The tier that produced a choice is recorded on the event, so "the guard switched onto an account that was also exhausted" is diagnosable rather than mysterious.

Wiring

The guard is not a standalone daemon — it sits in the paths that actually die:

  • _sleep_loop (api) — a capacity kill now stops the 5/15/30-minute retry burst instead of exhausting all three attempts inside the same dead window.
  • _run_sleep preflight — checks before starting rather than discovering mid-run.
  • prompt_executor — a capacity-killed deferred prompt is re-armed past the reset instead of being destroyed. This is the leg that turns a dropped user request into a late one.

Alerts that can't be silenced

Part of the original failure was that the alerts about capacity kills were themselves being dropped by admission control. notify_sleep_failure now uses on_reject="raise", and both it and notify_capacity_event are exempt from admission rate limits. An alert about the spine being down is not a candidate for rate-limiting.

Honest-failure ledger: data/capacity/events.jsonl. Kill switch: data/capacity/auto_failover_off (create the file to disable automatic failover entirely).

Phase 3: verification legs

Shipped the same night (0cbb34343):

  • Morning-after verifier — checks that what the guard claimed it did overnight actually happened, closing the loop on the guard's own reporting.
  • Kill sweeper — sweeps stepwise jobs killed by capacity exhaustion so they surface as capacity events rather than mysterious wedges.
  • Real error vocabulary — the classify_failure / parse_reset work described above.

A companion leg, _run_services_liveness() in flows/sleep/data_source_health.py, gives every ./run service a nightly liveness row — the service-monitor included, since an unwatched watcher was its own gap (issue f576911c).

Honest gaps

  • The guard has never fired in anger. Everything is verified against real recorded error strings and live telemetry/queue/ledger paths — not against a genuine kill. That is the single biggest caveat on this page.
  • Failover is still manual for some surfaces. The wiring covers the sleep loop, sleep preflight, and deferred prompts; other surfaces still die.
  • Service liveness is nightly detection only — no real-time alert, no auto-recovery.
  • Phase 4 (exec-loop event labeling, degraded-/sleep mode, model right-sizing) is unbuilt.

Success and atrophy signals

Working: the next real capacity kill produces an events.jsonl row, a failover (or a deferred queue.db row), a delivered alert, and a completed run — with zero bio-zack involvement.

Failing: a kill happens and the ledger is empty, or the guard repeatedly switches onto accounts that are also exhausted (check the recorded signal tier).

Atrophy: if six months pass with no firing and no manual failovers either, the fleet grew enough headroom and this can be retired.