Consolidation Loop

Continuous micro-processing of conversation exhaust, scratch pad synthesis, and coaching map updates during waking hours. Instead of batching all data processing into the nightly /sleep cycle, the consolidation loop runs sub-tasks at staggered intervals throughout the day, keeping downstream systems (daily scratch, identity observations, coaching maps) incrementally up to date.

Architecture

MicroConsolidationLoop (60s tick, 7am-11pm PT)
       |
       +-----> Triage Captures (5 min)
       |         stream-captures/YYYY-MM-DD.jsonl --> classify --> route
       |
       +-----> CLI Session Scan (15 min)
       |         ~/.claude/projects/.../*.jsonl --> extract tags --> persist
       |
       +-----> Coaching Refresh (15 min, conditional)
       |         new observations --> match to active goals --> update map
       |
       +-----> Scratch Synthesis (4 hours)
                 daily.md --> LLM summary --> ## Running Summary section

The loop runs as a scheduler task: _consolidation_loop → _run_consolidation_service → MicroConsolidationLoop.start() (api/src/scheduler.py). It manages its own 60-second tick internally. Each tick checks whether enough time has elapsed since the last run of each sub-task and only executes those that are due (the _maybe_* methods debounce against seconds_since(last_ts)).

Sub-Tasks

Stream Capture Triage (every 5 minutes)

Reads new entries from data/stream-captures/YYYY-MM-DD.jsonl since the last processed offset and routes them by tag type.

Tag	Classification	Destination
`<file-this>`	LLM classifies into schedule/people/decision/event/preference/health/project	`data/scratch/daily.md` under appropriate section
`<observation>`	LLM validates as durable identity trait; deduped by content hash	`data/identity/observations.jsonl`
`<follow-up>`	LLM classifies as reminder/commitment/note	Reminder: proactive queue. Commitment: `data/assistant/commitments.json`. Note: `data/scratch/daily.md`
`<note-to-self>`	No LLM needed	`data/scratch/daily.md` under "Session Notes"
`<ask-next-session>`	Skipped (already handled by telegram_integration)	--

Processing is capped at 50 entries per tick to bound cost and time. New observations trigger a pending coaching refresh on the next eligible tick.

CLI Session Scan (every 15 minutes)

Scans Claude Code session files (~/.claude/projects/<project-path>/<uuid>.jsonl) for markup tags that weren't processed via the Telegram real-time path. Delegates to scripts/stream_capture/cli_scanner.py, which tracks its own scan state at data/stream-captures/.cli-scan-state.json.

This replaces the old model where CLI session captures were only processed during /sleep. Now they're picked up within 15 minutes.

Coaching Refresh (every 15 minutes, conditional)

Runs only when the triage sub-task has produced new identity observations. Each new observation is checked against active goals (from goals/active/*.json) via LLM to determine relevance, then appended to the goal-identity coaching map.

The coaching map (data/coaching/goal-identity-map.json) caches which observations are relevant to which goals, so the coaching system doesn't need to re-evaluate the full observation set on every use.

Scratch Synthesis (every 4 hours)

Generates a running summary paragraph of the current data/scratch/daily.md and writes it under a ## Running Summary section. The summary captures key events, decisions, open items, and the day's overall theme.

The synthesis strips any existing Running Summary section before generating to avoid recursive summarization. The summary is replaced in-place on each run rather than appended.

Scheduling

The loop is active during waking hours only (7am-11pm PT). Outside this window, ticks are no-ops.

Sub-Task	Interval	Uses LLM	Notes
Capture triage	5 min	Yes (per capture)	Capped at 50 captures/tick
CLI scan	15 min	No	Delegates to cli_scanner
Coaching refresh	15 min	Yes (per observation)	Only runs when new observations exist
Scratch synthesis	4 hours	Yes (1 call)	Replaces existing summary

Daily inference cap: MAX_DAILY_INFERENCE_CALLS = 200 across all sub-tasks. The counter (_daily_inference_calls) resets at midnight PT inside the start() loop. When the cap is reached, LLM-dependent sub-tasks (triage, coaching refresh, scratch synthesis) are skipped.

Note: The counter lives only in memory on the running loop instance — it is not persisted to state.json. A process restart resets the daily budget to 0 mid-day, so a restart can effectively grant a fresh 200-call budget.

Note: The cap is approximate, not exact. _maybe_triage_captures increments the counter by the raw processed count (the code comment calls it a "Rough estimate"), but some captures consume no LLM call — <note-to-self> routes with no inference and <ask-next-session> is skipped entirely. So the running counter over-counts real inference, and the 200 cap is a conservative upper bound rather than a 1:1 call tracker. All consolidation LLM calls are tagged task_type="micro_consolidation", which is how their cost is attributed in the inference/usage tracking.

State Management

All state is persisted to data/consolidation/state.json via ConsolidationState, which uses atomic_write() for crash safety.

{
  "version": 1,
  "last_triage": {
    "file": "2026-02-24.jsonl",
    "offset": 3,
    "ts": "2026-02-24T15:00:36.929574+00:00"
  },
  "last_cli_scan": "2026-02-24T23:41:39.726540+00:00",
  "last_scratch_synthesis": "2026-02-24T23:03:32.225560+00:00",
  "last_coaching_refresh": "2026-02-22T17:24:19.489940+00:00",
  "stats": {
    "captures_triaged": 227,
    "observations_added": 16,
    "commitments_extracted": 0,
    "reminders_created": 0,
    "coaching_refreshes": 15
  }
}

Triage offset tracking: The triage sub-task tracks which JSONL file and line offset it last processed. When the date rolls over (new file), the offset resets to 0. This prevents re-processing old entries and allows the loop to pick up exactly where it left off after restarts.

Interval debouncing: Each sub-task checks seconds_since(last_ts) against its interval constant. If the timestamp is empty (first run or missing), seconds_since returns infinity, ensuring immediate execution.

Integration with Other Systems

Stream Capture

The consolidation loop is the primary consumer of stream captures. Captures are produced by two paths: 1. Telegram real-time -- tags extracted inline during Claude responses 2. CLI scanner -- tags extracted from Claude Code session files

Both paths write to data/stream-captures/YYYY-MM-DD.jsonl. The triage sub-task processes these files incrementally.

Scratch Pads

Triage routes classified captures to data/scratch/daily.md under section headers (Events, Decisions, People, Projects, Notes, Follow-ups, Session Notes). The scratch synthesis sub-task then periodically generates a Running Summary of the accumulated content.

This feeds into the nightly rollup (00:05 PT) which incorporates daily.md into weekly.md.

Coaching System

When triage produces new observations, the coaching refresh sub-task maps them to active goals. The resulting data/coaching/goal-identity-map.json is consumed by the coaching system to personalize coaching nudges and recommendations based on identity traits relevant to each goal.

Identity System

Validated observations are appended to data/identity/observations.jsonl with source: "micro_consolidation" and needs_review: true. After storing, the triage sub-task calls mark_views_stale("identity") to trigger rebuild of identity-related memory views.

Dedup mechanism: Each triage run calls _load_existing_obs_hashes() once, building a set of content hashes from every existing observation's obs text. A hash is the first 16 hex chars of sha256(normalized_text), where normalization lowercases and collapses whitespace. Each incoming <observation> is hashed and dropped if its hash is already in the set; surviving observations add their hash back to the set so duplicates within the same run also collapse.

Malformed JSONL recovery: capture and observation readers validate each decoded line at the deserialization boundary. They recover the historical double-encoded-JSON case by decoding a string-valued outer record once more, and skip blank, malformed, or non-object records. A defense-in-depth guard also counts an unexpected non-object capture as processed so the offset advances instead of wedging the entire loop on the same line. Triage exceptions include tracebacks in service logs.

Proactive Outreach

Follow-up captures classified as reminders are routed directly to the proactive message queue via enqueue_message(), creating time-triggered Telegram reminders.

/sleep Cycle

The consolidation loop handles incremental processing during the day, but two related modules run exclusively during /sleep:

Implicit check-in (scripts/consolidation/implicit_checkin.py): Synthesizes a daily health/energy snapshot from garmin data, scratch pad, stream captures, and activities when no explicit check-in exists. Stores in data/checkins/ with depth: "implicit".
Commitment triage (scripts/consolidation/commitment_triage.py): Categorizes pending commitments by urgency (critical/high/moderate/today/upcoming) and generates suggested actions. Output flows into the VDB morning brief via gather_context() and into /sleep phase 2c for overnight review.

VDB Morning Brief

Commitment triage results are formatted by format_triage_for_vdb() and included in the VDB gather context as the commitment_triage key, surfacing overdue items and suggested actions in the morning brief.

Manual Trigger

The scheduler exposes a generic trigger endpoint (api/src/routers/scheduler.py, prefix /api/v1/scheduler):

# Via scheduler API
POST /api/v1/scheduler/trigger/consolidation

The endpoint maps consolidation to scheduler.run_consolidation_now, which calls self._consolidation_ref._tick() on the live loop instance. Because _tick() runs the same interval-debounced _maybe_* sub-tasks as a normal tick, the API trigger respects each sub-task's interval timer — it only runs the sub-tasks that are actually due. It is a "run a tick now" button, not a "force everything" button. (If the loop hasn't been initialized yet, the call logs a warning and does nothing.)

To force every sub-task regardless of intervals, call run_once() directly — but this is only reachable programmatically; no API or CLI surface invokes it:

# Programmatic — ignores all interval timers
from consolidation import MicroConsolidationLoop
loop = MicroConsolidationLoop(data_dir=Path("data"))
results = await loop.run_once()

File Locations

Path	Purpose
`scripts/consolidation/__init__.py`	`MicroConsolidationLoop` class -- main loop and sub-task orchestration
`scripts/consolidation/state.py`	`ConsolidationState` -- crash-safe state tracking
`scripts/consolidation/triage.py`	Stream capture classification and routing
`scripts/consolidation/scratch_synthesis.py`	Incremental daily.md summarization
`scripts/consolidation/coaching_refresh.py`	Goal-identity map cache refresh
`scripts/consolidation/commitments.py`	Commitment extraction from captures (used by triage)
`scripts/consolidation/implicit_checkin.py`	Implicit check-in generation (used by /sleep)
`scripts/consolidation/commitment_triage.py`	Commitment urgency triage (used by /sleep and VDB)
`data/consolidation/state.json`	Processing state (offsets, timestamps, stats)
`data/coaching/goal-identity-map.json`	Cached observation-to-goal mappings
`data/stream-captures/YYYY-MM-DD.jsonl`	Input: daily stream capture files
`data/scratch/daily.md`	Output: classified captures and running summary
`data/identity/observations.jsonl`	Output: validated identity observations
`data/assistant/commitments.json`	Output: extracted commitments and follow-ups

Known Limitations

Inference cost -- Each <file-this> and <observation> capture triggers an LLM call for classification/validation. High-capture days can consume the 200-call daily budget. The cap prevents runaway cost but means late-day captures may be deferred.
Coaching refresh is reactive -- Only fires when triage produces new observations. If observations are added by other paths (e.g., manual /interview), the coaching map won't update until the next /sleep or manual refresh.
No backfill -- If the loop is stopped for hours, triage processes only today's capture file from the last offset. Captures from previous days that were missed are not automatically reprocessed.
Scratch synthesis overwrites -- Each synthesis run replaces the Running Summary section entirely rather than appending. If the summary quality degrades on a particular run, the previous summary is lost.

Where to Go Next

Stream Capture -- the markup-tag producers (Telegram real-time + CLI scanner) that feed triage
Scratch Pads -- the daily.md/weekly.md layer triage writes into and scratch synthesis summarizes
Coaching -- consumer of the goal-identity map refreshed here
Identity -- where validated <observation> captures land
Capture & Commitments -- the commitment store follow-up captures route into
Check-in -- implicit check-ins generated during /sleep alongside this loop's daytime work
Scheduling -- the scheduler task framework this loop runs under