Skip to content

Screen Journal

The screen journal is vita's attention ledger: a Dayflow-style pipeline that turns continuous screen capture into a browsable, searchable timeline of what bio-Zack actually did at the computer. Body (Garmin/CGM), comms (email/telegram), and output (git) were already instrumented — this closes the last gap, where screen attention actually goes.

Derived from Dayflow (MIT, macOS/Swift); rebuilt for Linux/Hyprland as ~2k lines of Python in vita idiom (forge idea 065, built 2026-07-17/18). The pattern was ported, not the app: two-stage LLM analysis (transcription → cards), sliding-window card merge, honest gap handling. Vita's version adds two things Dayflow doesn't have: per-frame window metadata from the compositor, and ground-truth input presence from evdev.

Architecture

capture (10s tick)                analysis (10-min scheduler tick)
┌─────────────────────┐           ┌──────────────────────────────────┐
│ grim (focused       │           │ batch: group frames into 15-min  │
│  monitor, 2560x720) │  frames   │  batches, split on >2min gaps    │
│ hyprctl activewindow│──┬──────▶│  idle = no input in window       │
│  (class/title/ws)   │  │ meta   │        │                         │
│ evdev presence      │  │ JSONL  │        ▼ pending batches         │
│  (input_idle_s)     │  │        │ transcribe: haiku CLI (frames +  │
│ blocklist/redaction │  │        │  metadata timeline → timestamped │
│ frame-diff metric   │  │        │  observations)                   │
└─────────────────────┘  │        │  fallback: gemini flash-lite     │
   data/screen/frames/   │        │        │                         │
   (3-day retention)     │        │        ▼                         │
                         │        │ cards: 45-min lookback, existing │
   data/screen/meta/     │        │  cards + new observations →      │
   (JSONL, kept forever) │        │  merged timeline cards           │
                         │        │  (replace-in-range + coverage    │
                         ▼        │   guard)                         │
                  journal.db ◀────┴──────────────────────────────────┘
                 (batches, observations, cards)
                         │
        ┌────────────────┼─────────────────┬──────────────────┐
        ▼                ▼                 ▼                  ▼
  workshop page    search index     daily rollup +     data-source-health
  (timeline UI)    (screenjournal   /sleep staging     liveness row
                    source)         (screen-time §)

Capture layer

scripts/screenjournal/capture.py, run as ./run screen-journal.

Every 10 seconds (configurable):

  • Frame: grim screenshots the focused monitor at 0.5 scale (5120x1440 → 2560x720 JPEG q80, ~240KB) into data/screen/frames/YYYY-MM-DD/<epoch>.jpg.
  • Metadata: hyprctl activewindow attaches window class, title (300 chars), and workspace to every frame — this carries most of the semantic signal and is why cheap models stay viable. One JSONL row per tick in data/screen/meta/YYYY-MM-DD.jsonl.
  • Presence (presence.py): a daemon thread reads /dev/input/event* (zack is in the input group; evdev fds are per-reader copies, so nothing is stolen from the compositor) and tracks seconds since the last EV_KEY / EV_REL / EV_ABS event. Each row carries input_idle_s. Device chatter (switches, LEDs) is ignored; hotplug is handled by periodic rescan.
  • Frame-diff: cheap mean-abs-diff vs the previous frame (64x36 grayscale) — retained as the legacy idle signal and as context for the models.
  • Privacy: window classes/title patterns in the blocklist (1Password, KeePassXC, Bitwarden, private browsing, vault) suppress the image — a redacted metadata row is still written so gaps stay honest. Screen-off (DPMS) and hyprlock produce no rows at all.
  • Retention: raw frames pruned after 3 days, timelapses after 14; metadata, observations, and cards are text and kept forever.

Analysis pipeline

scripts/screenjournal/analyze.py, invoked by the screen_journal VitaScheduler loop every 10 minutes (registered in api/src/loop_engine/registry.py).

Batching (batch.py): frames group into ~15-minute batches, splitting on gaps >2 minutes. Classification:

  • idle — no keyboard/mouse input landed inside the batch window (a row at time t with input_idle_s = I means the last input was at t−I). Idle batches skip the LLM entirely — free. Rows predating the presence field fall back to the frame-diff rule (≥90% static + ≤1 distinct window). The input rule fixes both failure modes of pixel-diff: a terminal scrolling while away is idle; reading a static page is not.
  • redacted — all rows blocklisted.
  • pending — real activity; a 1fps h264 timelapse is built (ffmpeg) and the batch queues for transcription.

Stage 1 — transcription (providers.py): pending batches go to the configured lane with a metadata timeline (one line per frame: time, class, title, diff, AWAY Nm markers when input_idle_s > 300) plus subsampled frames. Output: strict-JSON timestamped observations stored first-class in journal.db. Seven instrumented lanes exist (cost_test.py compares them on identical windows); production default is claude CLI haiku with automatic gemini flash-lite fallback via OpenRouter if the CLI fails — the capacity-exhaustion class cannot kill the pipeline.

Stage 2 — cards (cards.py): observations in a 45-minute lookback window, plus the existing cards they overlap, are re-emitted by the model as merged timeline cards (category, title, summary, detailed summary, distractions, app sites). Cards replace-in-range so a continuing activity stays one card instead of fragmenting. A coverage guard (Dayflow's validateTimeCoverage equivalent) re-inserts any existing card <50% covered by the new set — the model cannot silently delete recorded time.

Observations are the durable substrate; cards are derived and can be regenerated from them at any time (prompt iteration, lane changes, repair).

Quality & cost

Decided empirically 2026-07-18 on a rich real window (head-to-head, same frames): haiku and gemini-2.5-pro read actual pixels (telegram unread counts, doc highlights, article titles); flash-lite mostly echoes window metadata; sonnet fabricated on a vision edge case. Pro would cost ~$29/mo (over the $20 cap); haiku is the same tier at $0 cash on the Max account (~25–50 light calls/day). Local lanes (qwen2.5vl:7b via ollama) work but are ~6.5 min/batch and 17GB unified RAM while loaded — kept for backfill only.

Net: ≈$0.10/mo cash (fallback only) + light Max usage. ~78% of batches classify idle and cost nothing. Full analysis: data/reports/2026-07-18-screen-journal-cost-analysis.md.

Presence metrics

Two derived numbers, everywhere the journal surfaces:

  • at computer (present_min) — ticks with input within the last 5 min; includes reading.
  • hands-on (interacting_min) — ticks with input within 15s; actively typing/mousing.

Days before 2026-07-18 lack the field and render unknown, never 0; partial days carry a coverage fraction.

Web UI

/workshop/pages/screen-journal/ on the workshop (port 33800/33801) — built on the capability layer (vita.screenjournal.*, read-only):

  • day timeline with category-colored cards and honest away/idle gaps
  • click a card → detailed summary, distractions, raw observations (evidence), and a frame scrubber: drag across the card's range to see the actual screen at that moment (3-day frame window)
  • stats strip (active time, cards, top category, at-computer/hands-on), category donut, pipeline strip (per-batch status dots; failed batches clickable for the error)
  • live capture pill + 60s auto-refresh when viewing today

Capabilities: vita.screenjournal.timeline · .observations · .frames · .days (docs: docs/api/capabilities/vita.screenjournal.md).

Integrations

  • Search (scripts/search/extractors/screenjournal.py): cards and 30-min observation windows are indexed chunks (source_type screenjournal, 14-day recency half-life) — screen activity is part of silicon-zack's long-term memory.
  • Daily rollup (scripts/screenjournal_export.py + daily_rollup.py): a "screen time (auto)" section (total active, at-computer, per-category minutes, top cards) is appended to the day's notes before the nightly fold.
  • /sleep staging: data/sleep/staging/screenjournal.json — day summary + pipeline health for the nightly reflection.
  • Data-source-health: a screen_journal.capture liveness row (process alive + captured-in-24h + no wedged batches; tolerant of screen-off nights).
  • Agenda kickoff (scripts/proactive/agenda_kickoff.py, 2026-07-19): the presence stream drives a morning trigger — a 2-min scheduler loop (5am–noon) watches for the day's first sustained input (≥6 rows with input_idle_s ≤ 20 spanning ≥ 2 min; debounced against single-bump idle tails and past-midnight sessions) and enqueues a deferred agenda-setting telegram prompt, once per day. Requested by bio-zack so mornings start with organizing, not X.

Operations

./run screen-journal [start|stop|restart|status|log]   # capture service
PYTHONPATH=scripts uv run python -m screenjournal.analyze   # manual tick
PYTHONPATH=scripts uv run python -m screenjournal.cost_test \
  --start <epoch> --end <epoch> --lanes openrouter_flashlite,claude_cli_haiku

Config: data/screen/config.json (hot-reloaded) — transcribe_lane, transcribe_fallback_lane, cards_lane, capture_interval_s, frame_retention_days, blocklists, thresholds. Defaults in scripts/screenjournal/config.py.

Troubleshooting:

  • capture pill red on the page → ./run screen-journal restart
  • failed batches (red dots) → click for the error; re-tick with python -m screenjournal.analyze; both lanes' errors are recorded
  • cards look wrong for a window → regenerate: from screenjournal.cards import generate_cards; generate_cards(now=<epoch>) (observations survive; cards are derived)
  • scheduler-loop status shows last_run: None — known cosmetic gap in the subprocess-loop pattern (shared with ooda_watchdog); real liveness is the data-source-health row, which reads primary sources.

Privacy model

Local-first: frames, metadata, and the DB live in data/screen/ (gitignored, like all personal data). The cloud fallback lane ships frames to OpenRouter→Google only when the claude CLI fails; the default lane keeps pixels inside the Anthropic boundary vita already uses everywhere. Sensitive apps are blocklisted at capture. Raw pixels age out in 3 days; what persists is text.