Sentinel Loop

A config-driven event detection and response system. Sentinels watch for changes across the vita ecosystem -- file drops, email arrivals, health anomalies, stale data -- and fire configurable responses: telegram messages, Claude prompts, or shell scripts. The loop runs continuously inside the scheduler, ticking every 30 seconds, with each sentinel governed by its own polling interval, cooldown, and active hours.

Architecture

                              SENTINEL LOOP (30s tick)
                              ========================

    +-------------------+     +-------------------+     +-------------------+
    | sentinels.json    |     | state.json        |     | log.jsonl         |
    | (definitions)     |     | (runtime state)   |     | (execution log)   |
    +--------+----------+     +--------+----------+     +--------+----------+
             |                         |                         ^
             v                         v                         |
    +--------------------------------------------------------------+
    |                      SentinelLoop._run_tick()                 |
    |                                                              |
    |   1. Hot-reload sentinels.json (mtime cache)                 |
    |   2. Check poll sentinels (interval + active_hours gating)   |
    |   3. Drain filesystem events (watchdog queue)                |
    |   4. Drain IMAP IDLE events (thread queue)                   |
    |   5. Drain webhook events (async queue)                      |
    +------+---------------------------+---------------------------+
           |                           |
           v                           v
    +-------------+             +--------------+
    | DETECTORS   |             | RESPONDERS   |
    |             |             |              |
    | poll_script | ----+       | queue_message|----> Telegram
    | poll_agent  |     |       | invoke_claude|----> Claude prompt
    | filesystem  |     +-----> | run_script   |----> Shell command
    | imap_idle   |  triggered  +--------------+
    | webhook     |  (cooldown       ^
    +-------------+   check)         |
                                     |
                              event data passed
                              via $template vars

How It Works

The sentinel loop is an asyncio task managed by the vita scheduler. On startup it:

Loads all sentinel definitions from data/sentinel/sentinels.json
Starts persistent watchers for filesystem and imap_idle sentinels (background threads with event queues)
Enters the main poll loop (30-second tick interval)

Each tick:

Hot reload -- Invalidates the registry cache and reloads sentinels.json. New sentinels get watchers started; removed sentinels get watchers stopped. This enables live editing of sentinel config without restarts.
Poll sentinels -- For each enabled poll_script or poll_agent sentinel, checks if interval_seconds has elapsed since last_checked and whether the current time falls within active_hours. If both pass, runs detection.
Filesystem events -- Drains the watchdog event queue. Each event carries its sentinel_id and triggers the corresponding responder.
IMAP IDLE events -- Drains the IMAP thread event queue. New-email events trigger the corresponding responder.
Webhook events -- Drains the async webhook queue (populated by the API endpoint). Each event triggers its sentinel's responder.

Before firing a responder, the loop checks the sentinel's cooldown_seconds. If the sentinel was triggered too recently, the event is logged as skipped_cooldown and dropped.

Error handling: Consecutive errors are tracked per sentinel. After 5 consecutive errors, the sentinel is auto-disabled and an auto_disabled event is logged.

Detectors

Detectors determine whether a sentinel should fire. Each detector implements the BaseDetector protocol:

class BaseDetector(Protocol):
    async def detect(self, sentinel_config: dict, state: dict) -> DetectionResult

@dataclass
class DetectionResult:
    triggered: bool
    event: dict = field(default_factory=dict)
    error: bool = False

Mode	Class	Mechanism	When to Use
`poll_script`	`PollScriptDetector`	Runs a shell command, applies a change detection strategy to its stdout	Threshold checks, data freshness, any scriptable condition
`poll_agent`	`PollAgentDetector`	Sends context files + prompt to an LLM (via the inference factory), parses `{"triggered": bool, "summary": str}` response	Semantic analysis, cross-source correlation, fuzzy pattern detection
`filesystem`	`FilesystemDetector`	Uses watchdog/inotify to watch a directory for file events (create, modify, delete, move) with glob pattern filtering	File drops, auto-ingest pipelines
`imap_idle`	`IMAPIdleDetector`	Maintains a persistent IMAP IDLE connection in a daemon thread, detects new mail via `EXISTS` responses	Near-instant email detection
`webhook`	`WebhookDetector`	Receives HTTP POST payloads via the API, extracts fields using dot-path mapping. Always triggers on receipt.	External system integrations, GitHub webhooks

poll_script Config

{
  "command": "python scripts/sentinel/checks/some_check.py",
  "timeout_seconds": 30,
  "change_strategy": "hash"
}

The script's stdout is compared against the previous output using the configured change strategy. Non-zero exit codes (except for exit_code strategy) are treated as detection errors, not triggers.

Two implementation details matter for reimplementers:

PYTHONPATH injection. Before running the command, the detector prepends scripts/ to PYTHONPATH and sets cwd to the project root. This is why check scripts can import vita modules directly (e.g. from utils import safe_read_json) regardless of where they're invoked from.
Empty-output suppression (hash strategy only). Empty stdout never triggers under the hash strategy. This prevents a false alert when a check transitions from "had anomalies" (non-empty output) to "clean" (empty output) — the emptying-out is logged as a check, not a trigger.

poll_agent Config

{
  "prompt": "Review the data below. If anything warrants attention...",
  "context_files": ["memory/views/health-snapshot.json"],
  "model": "minimax/minimax-m2.7",
  "timeout_seconds": 45,
  "calendar_guard": true
}

Context files support glob patterns (resolved with glob.glob, sorted, concatenated). Total context is capped at 50KB (MAX_CONTEXT_BYTES = 50_000); the last file is truncated and the rest skipped once the budget is hit. The LLM must respond with a JSON object containing triggered and summary fields; markdown code fences are stripped before parsing, and an unparseable response is treated as "not triggered" (logged, never crashes the loop).

The call goes through the vita inference factory (scripts.inference.factory.complete), not a hardcoded provider — the model string is just a routing key. The default model is minimax/minimax-m2.7 (DEFAULT_MODEL in detectors/poll_agent.py); MiniMax happens to be served via OpenRouter, but the detector knows nothing about that.

The optional calendar_guard / calendar_guard_view keys enable the calendar-attribution guard — a deterministic post-detection layer described below.

filesystem Config

{
  "path": "/home/<user>/Downloads/vita",
  "events": ["created"],
  "pattern": "*",
  "recursive": false
}

Events are filtered by file glob pattern and event type. Directory events are ignored. File size is included in the event payload.

imap_idle Config

{
  "host": "imap.example.com",
  "port": 993,
  "username": "user@example.com",
  "credentials_file": "email_credentials.json",
  "credentials_key": "imap_password",
  "mailbox": "INBOX",
  "idle_timeout": 300,
  "reconnect_delay": 30
}

Credentials are loaded from ~/.config/vita/{credentials_file}. The IDLE cycle re-issues every 5 minutes (well under RFC 2177's 29-minute limit). Automatic reconnection with configurable delay on connection errors.

webhook Config

{
  "extract": {
    "number": "issue.number",
    "action": "action"
  }
}

If extract is omitted, the full payload is passed through. Dot-path notation navigates nested objects.

Calendar-attribution guard

poll_agent detectors that set calendar_guard: true run a deterministic post-detection layer (scripts/sentinel/calendar_guard.py) before the responder fires. It exists because a prompt instruction ("cross-reference the calendar before attributing a cause") is soft — the LLM can ignore it and emit "investigate causes" boilerplate for an anomaly whose cause (travel) is plainly on the calendar.

The guard runs only when a sentinel has already triggered, and only reshapes the summary:

Load the travel-context view (memory/views/calendar-travel-context.json by default, overridable via calendar_guard_view).
Find is_travel events overlapping the recent anomaly window (7 days back, matching the health snapshot's rolling averages). Future events and non-travel all-day events are ignored.
If a triggered summary is "calendar-blind" (doesn't already name a travel cause) during such a window, downgrade it: strip the "investigate / may indicate / recommend / unknown cause" sentences, keep the metrics, and append the explanatory event names with travel-attributed framing.

Note: The guard downgrades, not suppresses. A genuine mid-trip anomaly still fires — deterministic code can't distinguish a travel-explained deficit from a travel-coincident one, so retaining the corrected signal is safer than dropping it. It is also fail-open: a missing, stale (>48h old), or unreadable view leaves the alert untouched with a logged warning. The result event records the guard's action (pass/downgrade), reason, and (on downgrade) the original_summary and matched events.

The garmin-health-anomaly sentinel uses this guard.

Responders

Responders execute the action when a sentinel triggers. Each implements the BaseResponder protocol:

class BaseResponder(Protocol):
    async def respond(self, sentinel_config: dict, event: dict) -> ResponseResult

@dataclass
class ResponseResult:
    status: str   # "ok", "error", "skipped"
    summary: str

Mode	Class	Action	Template Variables
`queue_message`	`MessageResponder`	Enqueues a telegram message via the proactive queue	`$sentinel_id` + all event fields
`invoke_claude`	`ClaudeResponder`	Enqueues a Claude prompt (full LLM session) via the proactive queue	`$sentinel_id` + all event fields
`run_script`	`ScriptResponder`	Runs a shell command with `SENTINEL_EVENT` (JSON) and `SENTINEL_ID` env vars	N/A (uses env vars)

All responders use Python string.Template for variable substitution in templates. Event data keys become template variables (e.g., $output, $summary, $filename).

queue_message Config

{
  "template": "health alert: $summary"
}

invoke_claude Config

{
  "prompt_template": "New file detected: $filename ($size_human). Process it."
}

run_script Config

{
  "command": "python scripts/mail/fastmail_sentinel.py respond",
  "timeout_seconds": 60
}

The event payload is available as the SENTINEL_EVENT environment variable (JSON string).

Change Detection Strategies

Used by poll_script detectors to compare the current script output against the previous output stored in state.

Strategy	Trigger Condition	Use Case
`hash`	SHA256 of output changed	General-purpose; any output change triggers
`exit_code`	Non-zero exit code	Binary checks (something exists or doesn't)
`json_diff`	JSON keys added, removed, or changed	Structured data comparisons
`regex_match`	Pattern matches in output (format: `pattern\\|\\|\\|output`)	Pattern detection in text
`numeric_threshold`	Value `>=` threshold (format: `threshold\\|\\|\\|value`)	Metric threshold crossing

These five are the only strategies implemented in state.py (detect_change raises ValueError for anything else).

Warning: The (currently disabled) openrouter-credits sentinel in data/sentinel/sentinels.json references change_strategy: "json_field" with a change_field key — a strategy that does not exist in detect_change(). If that sentinel were enabled, the detector would raise ValueError. This is a live config/code mismatch; either implement the strategy or fix the config before enabling it.

Check Scripts

Reusable detection scripts in scripts/sentinel/checks/. Each script is a standalone Python module that exits 0 (no trigger) or 1 (trigger), or prints structured output for hash/json_diff strategies.

Script	Sentinel	What It Checks
`pending_submissions.py`	`user-submission-processor`	Pending URL submissions exist
`process_submissions.py`	`user-submission-processor` (responder)	Processes up to 3 pending submissions, sends telegram notifications
`stagnating_goals.py`	`coaching-nudge`	Goals with no progress in 7+ days
`run_coaching.py`	`coaching-nudge` (responder)	Runs proactive coaching engine, queues nudges
`sleep_gate_status.py`	`sleep-gate-transition`	Outputs sleep gate status + avg hours for hash comparison
`sleep_cycle_check.py`	`sleep-cycle-monitor`	Checks if `/sleep` cycle ran in the last 36 hours
`context_freshness.py`	`context-staleness`	Checks garmin sync recency, view staleness, daily scratch activity
`alert_staleness.py`	`context-staleness` (responder)	Formats and sends data freshness alert via telegram
`unnamed_faces_check.py`	`unnamed-faces`	Queries faces.db for unnamed clusters with 10+ detections
`garmin_realtime.py`	`garmin-realtime`	Real-time HR/HRV check: RHR spike, HRV drop, sleep score, body battery, stress, ACWR, weight change against 7-day baselines
`garmin_sleep_landed.py`	`boot-sequence-orient`	Detects a fresh Garmin sleep sync and enqueues the morning boot-sequence orient block
`email_pattern.py`	`email-pattern`	VIP email buildup, pending overflow, high-score email aging
`training_load.py`	`training-load`	ACWR overreach/detraining, load imbalance, weekly target risk
`training_window.py`	`training-window`	Emits a message when training conditions are open but no session logged (empty output = no trigger)
`cgm_anomaly.py`	`cgm-anomaly`	CGM anomaly check: hypoglycemia, glucose spikes, sustained lows, rapid drops
`cross_source.py`	`cross-source-synthesis` (context)	Gathers cross-source data for LLM correlation analysis
`overdue_commitments.py`	`commitment-followup`	Overdue commitments (with dead-trigger suppression)
`alert_commitments.py`	`commitment-followup` (responder)	Formats and sends overdue commitment alert
`pending_drafts.py`	`pending-email-draft`	Exits 1 if any email draft in `pending_drafts/` is older than 2 hours
`inbox_meeting_stale.py`	`inbox-meeting-ingest-stale`	Fires when a meeting in `data/meetings/inbox/` sits unprocessed past the staleness threshold (companion to the `meeting-transcript-ingest` watcher)
`scheduler_health.py`	`scheduler-health`	Detects a stalled proactive scheduler loop (past-due messages piling up during active hours)
`dwell_board.py`	`dwell-board-monitor`	New items on a tracked GitHub project board

Configuration

Each sentinel is a JSON object in data/sentinel/sentinels.json:

{
  "id": "garmin-realtime",
  "name": "Garmin realtime health thresholds",
  "enabled": true,
  "detection": {
    "mode": "poll_script",
    "config": {
      "command": "python scripts/sentinel/checks/garmin_realtime.py",
      "timeout_seconds": 15,
      "change_strategy": "hash"
    }
  },
  "response": {
    "mode": "queue_message",
    "config": {
      "template": "health alert: $output"
    }
  },
  "schedule": {
    "cooldown_seconds": 14400,
    "interval_seconds": 900,
    "active_hours": {
      "tz": "America/Los_Angeles",
      "start": 6,
      "end": 23
    }
  },
  "tags": ["health", "garmin"],
  "created_at": "2026-02-12T00:00:00+00:00"
}

Schedule Fields

Field	Default	Description
`interval_seconds`	300	Minimum time between poll checks (poll modes only)
`cooldown_seconds`	60	Minimum time between trigger firings (all modes, min 10)
`active_hours.tz`	UTC	Timezone for active hours window
`active_hours.start`	0	Start hour (inclusive)
`active_hours.end`	24	End hour (exclusive)

Validation Rules

id is required and must be unique
detection.mode is required (one of: poll_script, poll_agent, filesystem, imap_idle, webhook)
response.mode is required (one of: queue_message, invoke_claude, run_script)
cooldown_seconds must be >= 10

API Endpoints

All endpoints are under /api/v1/sentinel.

Method	Endpoint	Purpose
GET	`/list`	List all sentinels with merged runtime state
GET	`/{sentinel_id}`	Get a single sentinel with state
POST	`/create`	Create a new sentinel (409 if duplicate)
PUT	`/{sentinel_id}`	Update an existing sentinel
DELETE	`/{sentinel_id}`	Delete a sentinel and its state
POST	`/{sentinel_id}/toggle`	Toggle enabled/disabled
POST	`/{sentinel_id}/fire`	Manually fire a sentinel (pushes webhook event, runs tick)
GET	`/log`	Get execution log (filter by `sentinel_id`, `limit`, `since`)
POST	`/hook/{sentinel_id}`	Webhook receiver (returns 200 immediately)

Data Structures

Sentinel Definition

Stored in data/sentinel/sentinels.json as a JSON array.

{
  "id": "string",
  "name": "string",
  "enabled": true,
  "detection": {
    "mode": "poll_script | poll_agent | filesystem | imap_idle | webhook",
    "config": {}
  },
  "response": {
    "mode": "queue_message | invoke_claude | run_script",
    "config": {}
  },
  "schedule": {
    "cooldown_seconds": 60,
    "interval_seconds": 300,
    "active_hours": {"tz": "America/Los_Angeles", "start": 8, "end": 22}
  },
  "tags": ["string"],
  "created_at": "ISO-8601"
}

Runtime State

Stored in data/sentinel/state.json as a dict keyed by sentinel ID.

{
  "sentinel-id": {
    "last_checked": "ISO-8601 | null",
    "last_triggered": "ISO-8601 | null",
    "trigger_count": 0,
    "consecutive_errors": 0,
    "last_output_hash": "string | null",
    "last_output": "string | null"
  }
}

Log Entry

Appended to data/sentinel/log.jsonl (one JSON object per line, newest at bottom).

{
  "id": "uuid",
  "timestamp": "ISO-8601",
  "sentinel_id": "string",
  "event_type": "triggered | checked | error | skipped_cooldown | auto_disabled",
  "detection": {"event": {}},
  "response": {"mode": "string", "result": "string", "summary": "string"},
  "result": {"status": "string", "summary": "string"},
  "duration_ms": 123.4,
  "v": 1
}

Event Types

Type	Meaning
`triggered`	Detection fired and responder executed successfully
`checked`	Poll ran but detection did not trigger
`error`	Detection or response failed
`skipped_cooldown`	Detection triggered but cooldown period hasn't elapsed
`auto_disabled`	Sentinel disabled after 5 consecutive errors

File Locations

Path	Contents
`scripts/sentinel/__init__.py`	`SentinelLoop` class -- main orchestrator
`scripts/sentinel/registry.py`	CRUD for sentinel definitions (mtime-cached)
`scripts/sentinel/state.py`	Runtime state management + change detection strategies
`scripts/sentinel/log.py`	Execution log read/write (JSONL)
`scripts/sentinel/detectors/__init__.py`	`DetectionResult`, `BaseDetector` protocol, `detector_for()` factory
`scripts/sentinel/detectors/poll_script.py`	Shell command detector with output diffing
`scripts/sentinel/detectors/poll_agent.py`	LLM-based semantic detector (inference factory)
`scripts/sentinel/calendar_guard.py`	Deterministic travel-attribution guard for `poll_agent` alerts
`scripts/sentinel/detectors/filesystem.py`	Watchdog/inotify file event detector
`scripts/sentinel/detectors/imap_idle.py`	Persistent IMAP IDLE email detector
`scripts/sentinel/detectors/webhook.py`	Webhook payload extractor
`scripts/sentinel/responders/__init__.py`	`ResponseResult`, `BaseResponder` protocol, `responder_for()` factory
`scripts/sentinel/responders/message.py`	Telegram message via proactive queue
`scripts/sentinel/responders/claude.py`	Claude prompt via proactive queue
`scripts/sentinel/responders/script.py`	Shell command with event env vars
`scripts/sentinel/checks/*.py`	Reusable detection and response scripts
`api/src/routers/sentinel.py`	FastAPI router (CRUD, toggle, fire, webhook, log)
`api/src/services/scheduler_sentinel.py`	Loop lifecycle helpers (`create_sentinel_loop`, `start_sentinel_loop`, `stop_sentinel_loop`, `run_sentinel_tick`)
`api/src/scheduler.py`	Scheduler integration — the `_sentinel_loop` task delegates create/start/stop/tick to the service module above
`data/sentinel/sentinels.json`	Sentinel definitions (the config file)
`data/sentinel/state.json`	Runtime state (last_checked, trigger_count, errors)
`data/sentinel/log.jsonl`	Execution history

Scripts

Script	Purpose
`scripts/sentinel/__init__.py`	`SentinelLoop` -- tick loop, watcher management, cooldown enforcement, auto-disable
`scripts/sentinel/registry.py`	Sentinel CRUD with mtime-based caching and validation
`scripts/sentinel/state.py`	State persistence + 5 change detection strategies (hash, json_diff, exit_code, regex_match, numeric_threshold)
`scripts/sentinel/log.py`	Append-only JSONL logging with filtered reads (by sentinel, time, limit)
`scripts/sentinel/detectors/poll_script.py`	Async subprocess runner with timeout, output capture, strategy dispatch
`scripts/sentinel/detectors/poll_agent.py`	LLM classifier via the inference factory, with glob-based context gathering (50KB cap) and optional calendar guard
`scripts/sentinel/detectors/filesystem.py`	Watchdog observer pool with glob filtering and thread-safe event queue
`scripts/sentinel/detectors/imap_idle.py`	Persistent IMAP connection with auto-reconnect and IDLE keepalive cycling
`scripts/sentinel/detectors/webhook.py`	Dot-path field extraction from webhook payloads
`scripts/sentinel/responders/message.py`	Template rendering + telegram enqueue
`scripts/sentinel/responders/claude.py`	Template rendering + Claude prompt enqueue with dedup
`scripts/sentinel/responders/script.py`	Async subprocess with event data as `SENTINEL_EVENT` env var

Active Sentinels

26 sentinels are defined (19 enabled, 7 disabled). This is a point-in-time snapshot — sentinels are enabled/disabled via the /toggle endpoint and edited live in data/sentinel/sentinels.json, so the live set is authoritative. Several health sentinels (garmin-realtime, cgm-anomaly) are currently disabled in favor of the LLM-based garmin-health-anomaly poll_agent.

Sentinel	Detection	Response	Interval	Cooldown	Enabled
`downloads-vita-watcher`	filesystem	invoke_claude	--	10s	✓
`fastmail-inbox`	imap_idle	run_script	--	60s	✓
`sleep-cycle-monitor`	poll_script (exit_code)	queue_message	1h	12h	✓
`unnamed-faces`	poll_script (exit_code)	queue_message	24h	7d	✓
`user-submission-processor`	poll_script (exit_code)	run_script	60s	15s	✓
`coaching-nudge`	poll_script (hash)	run_script	4h	4h	✗
`sleep-gate-transition`	poll_script (hash)	queue_message	2h	12h	✓
`goal-deadline-pressure`	poll_agent	queue_message	6h	12h	✗
`garmin-health-anomaly`	poll_agent	queue_message	4h	8h	✓
`context-staleness`	poll_script (exit_code)	run_script	4h	12h	✓
`commitment-followup`	poll_script (exit_code)	run_script	6h	24h	✗
`garmin-realtime`	poll_script (hash)	queue_message	15m	4h	✗
`email-pattern`	poll_script (exit_code)	queue_message	1h	12h	✓
`training-load`	poll_script (exit_code)	queue_message	2h	12h	✓
`cgm-anomaly`	poll_script (hash)	queue_message	15m	4h	✗
`cross-source-synthesis`	poll_agent	queue_message	6h	24h	✓
`meeting-transcript-ingest`	filesystem	run_script	--	30s	✓
`dwell-board-monitor`	poll_script (hash)	queue_message	3h	3h	✓
`openrouter-credits`	poll_script	queue_message	--	24h	✗
`training-window`	poll_script	queue_message	4h	24h	✗
`pending-email-draft`	poll_script	queue_message	--	2h	✓
`scheduler-health`	poll_script	queue_message	--	1h	✓
`reports-watcher`	filesystem	run_script	--	10s	✓
`inbox-meeting-ingest-stale`	poll_script	run_script	30m	10m	✓
`boot-sequence-orient`	poll_script	queue_message	5m	24h	✓
`test-sentinel`	poll_script	queue_message	5m	60s	✓

Where to Go Next

Executive Loop — the OODA decision loop that consumes some sentinel signals.
Scheduling — the scheduler that hosts the sentinel loop alongside other recurring tasks.
Proactive — the message/prompt queue that queue_message and invoke_claude responders enqueue into.
Sleep Guardian — a more specialized predictive watcher; sentinels are the general-purpose engine.
Model Guidance — the inference-factory routing that backs poll_agent.