Stream Dock & IoT Peripherals

vita drives a small fleet of physical desk hardware: a USB stream dock with programmable, dynamically-rendered buttons; an ESP32 "todo console" with OLED screens and hardware buttons that bridges to the task system; and a registry of WiFi IoT devices that lets scripts resolve a device's current address by ID instead of hardcoding IPs. This page documents the wiring, protocols, control flow, and where state lives for each, at a level you could reimplement.

Scope. The e-paper display surfaces (the creative-canvas EPD47 and the coach training dashboard) live on the E-ink Displays page. The Precedent page covers the task/habit system; this page covers the hardware that surfaces those tasks. The dock's side-strip pulls from the calendar (see Voice & Location for the ESP32 voice device that also lives in the registry).

The fleet at a glance

Peripheral	Bus	What it is	Status
Stream dock (Ajazz AKP153E)	USB HID (hidraw)	15 pressable buttons + 2 side-strip slots, dynamic icons	Active
Todo console	WiFi (HTTP)	ESP32-S3, 4 OLED rows + buttons, bridges to Precedent	Active (integration wired; physical unit may be offline)
EPD47 e-paper	WiFi (HTTP)	4.7" e-paper; creative canvas + coach dashboard	Active (canvas); see E-ink Displays
Voice ESP32	WiFi (HTTP)	`silicon-zack` device, push-to-talk	See Voice & Location

All WiFi devices are tracked in a single device registry. The stream dock is the only USB device and is managed entirely inside the API process.

Stream Dock (Ajazz AKP153E)

A USB stream dock on the desk. The display surface is a 3×5 grid of 15 pressable LCD keys plus a narrow side strip. Each key is a small color LCD that vita pushes a JPEG to; pressing a key sends an HID event back. vita renders icons dynamically (task counts, calendar events, currently-clocked-in task) and dispatches a configured action per key.

Three-layer design

                ┌─────────────────────────────────────────┐
   hardware     │  Ajazz AKP153E   (USB, VID 0x0300/PID 0x3010) │
                └───────────────────────┬─────────────────┘
                                        │ hidraw read/write
                ┌───────────────────────┴─────────────────┐
   driver       │  scripts/streamdock/driver.py            │
                │  StreamDock: HID packets, JPEG encode,   │
                │  key<->grid mapping, button-event parse  │
                └───────────────────────┬─────────────────┘
                ┌───────────────────────┴─────────────────┐
   listener     │  scripts/streamdock/listener.py          │
                │  config load, icon push, poll thread,    │
                │  action dispatch (shell|http|vita)       │
                └───────────────────────┬─────────────────┘
                ┌───────────────────────┴─────────────────┐
   manager      │  scripts/streamdock/manager.py           │
                │  dynamic icons, project-focus state,     │
                │  overlays, data fetch, complex actions   │
                └──────────────────────────────────────────┘

Layer	File	Responsibility
Driver	`scripts/streamdock/driver.py`	Raw hidraw I/O. Finds the device, encodes 126×126 JPEGs, sends them to keys, parses button-press ACK packets. No vita logic.
Listener	`scripts/streamdock/listener.py`	Owns the `StreamDock` instance + config (`data/streamdock/config.json`). Pushes icons, runs the blocking button-poll thread, dispatches actions.
Manager	`scripts/streamdock/manager.py`	Higher-level state: project-focus, overlay modes, dynamic-icon refresh every 60s, multi-step actions (full sync, fire agent).
Icon gen	`scripts/streamdock/icon_gen.py`	Builds dynamic icons with Pillow (task-count badges, calendar strips, sync/recording state).
Icon gen (static)	`scripts/streamdock/icons.py`	Generates static icons from a text prompt via Google Imagen 4, with a Pillow text fallback.
API router	`api/src/routers/streamdock.py`	HTTP surface for status/refresh/brightness/button config and all manager actions.
Service helpers	`api/src/services/streamdock_service.py`	Singleton accessors + health/connect/reconnect wrappers used by the router and scheduler.

Driver protocol (hidraw)

The driver speaks the device's framed HID protocol directly — no vendor SDK, only Pillow for JPEG encoding.

Constant	Value	Meaning
`VID` / `PID`	`0x0300` / `0x3010`	USB identifiers; used to locate the hidraw node
`PACKET_SIZE`	`1024`	Each write is a 1025-byte buffer (a `0x00` HID report-ID prefix + 1024 payload)
`HEADER`	`b"CRT\x00\x00"`	Command preamble
`IMAGE_SIZE`	`126`	Icons are resized to 126×126 px
`JPEG_QUALITY`	`90`	JPEG encode quality
`ROTATION_DEG`	`90` (CCW)	Images are rotated before send to match panel orientation

Device discovery (find_device()) globs /dev/hidraw*, reads each node's HIDIOCGRAWINFO, and keeps the two interfaces matching the VID/PID. The interface with the smaller HID descriptor is the data interface (images + button events); the larger one is the HID keyboard (unused). Reading /dev/hidraw* requires permissions, so a host udev rule granting access to the VID/PID node is a deployment prerequisite.

Key grid mapping. Keys are numbered right-to-left in column-major order:

# driver.py
for col in range(1, 6):
    for row in range(1, 4):
        key_id = (5 - col) * 3 + row   # 1-indexed (row, col)
KEY_GRID[16] = (1, 6)   # side strip top
KEY_GRID[17] = (2, 6)   # side strip middle
# key 18 (3,6) is dead/unaddressable on this unit

pos_to_key(row, col) / key_to_pos(key_id) convert between the two. The side-strip keys (16, 17) are physically narrower; the driver insets content by a SIDE_STRIP_LEFT_MARGIN (35 px) so the housing doesn't clip it, and applies a small post-rotation pixel shift to compensate for the panel clipping its bottom-left corner.

Sending an image (set_key_image): resize → rotate 90° → JPEG-encode → send a BAT announce packet (length + key id) → stream the JPEG in 1024-byte chunks → flush with STP.

Reading a press (read_button_event): poll() the fd, read up to 1024 bytes, validate the ACK/OK header, then read key_id at byte 9 and the pressed flag at byte 10. The device only reports button events after at least one image has been written — so the listener pushes icons before it starts polling.

Listener: config, icons, and action dispatch

The listener loads data/streamdock/config.json and pushes an icon per configured button. A button whose icon is the literal string "dynamic" is skipped at load time — the manager owns those and pushes live data.

Config shape (data/streamdock/config.json):

{
  "brightness": 80,
  "buttons": {
    "1,1": {
      "label": "Workspace 1",
      "icon": "workspace-1.jpg",
      "action": { "type": "shell", "cmd": "hyprctl dispatch workspace 1" }
    },
    "2,3": {
      "label": "Todo Filter",
      "icon": "wrench.jpg",
      "action": { "type": "http", "method": "POST",
                  "url": "http://localhost:33800/todo-console/refresh" }
    },
    "1,6": {
      "label": "Status", "icon": "status.jpg",
      "action": { "type": "vita", "endpoint": "/todo-console/status" }
    }
  }
}

Three action types (listener._execute_action):

`type`	Fields	Behavior
`shell`	`cmd`	Runs a shell command (10s timeout)
`http`	`method`, `url`, `body`, `headers`	Arbitrary HTTP request (TLS verify off)
`vita`	`endpoint`, `method`, `body`	Request against the local API; prefixed with `http://localhost:33800/api/v1`

A daemon thread runs the blocking poll loop. On press it looks up the button config and schedules the action on the asyncio loop via call_soon_threadsafe. The loop self-heals: too many consecutive read errors, an OSError (device unplugged), or a crash flips _running false so the health check can detect it and trigger a reconnect.

Manager: dynamic content & overlays

The manager runs as an async task once the listener connects. Every 60s it refreshes status buttons, task-view icons, calendar side-strips, and the calisthenics counter; project data refreshes every ~5 min. State (focused_project, auto_mode, overlay_mode) persists to data/streamdock/state.json.

Dynamic surfaces it renders:

Status buttons (row 1): an analytics-tool launcher with a live user-count badge, a meeting-recording toggle showing elapsed time, a sync button (animated while syncing), an agent-fire button, and the currently-clocked-in task.
Task-view buttons (row 3): in-progress / due / scheduled / next-action counts pulled from /todo-console/task-counts, plus a "projects" overlay entry. Today/Weekend tag-focus tiles exist too.
Side strips (keys 16, 17): the next two upcoming (non-all-day) calendar events.
Project focus row: the three most-recently-active projects, scored by latest task create/complete/clock timestamp. Pressing one tells the todo console to focus on that project (POST /todo-console/focus/{path}).
Overlays: a projects overlay fills all 15 keys with a scrollable project list; a calisthenics overlay turns the grid into pushup/pullup increment buttons that log counts and debounce a coach e-ink refresh. Each overlay rewrites _button_config in place and restores the normal layout on "Back".

Notable composite actions (manager.py):

Full sync (action_full_sync): runs Peloton→Garmin, Garmin, RWGPS (sync + track download + analysis), CGM, the training auto-logger, rebuilds stale memory views, refreshes the e-ink display, and re-triggers the auto-curator — all sequentially with per-step timeouts.
Fire agent (action_fire_agent): enqueues a deferred proactive prompt so silicon-zack reaches out on Telegram (see Proactive Outreach).
Open URL / run host command: delegated to the host-side cadence daemon over a Unix socket (open_url / run_command), since the API may run inside a container.

Stream dock API

Router prefix: /api/v1/streamdock (mounted in api/src/main.py). All paths below are relative to http://localhost:33800 (see Architecture for ports).

Method & path	Purpose
`GET /api/v1/streamdock/status`	Connection state, device path, current config, brightness
`GET /api/v1/streamdock/health`	`healthy` / `connected` / `running` / `poll_thread_alive`
`POST /api/v1/streamdock/connect`	Connect + load config
`POST /api/v1/streamdock/reconnect`	Reset and reconnect
`POST /api/v1/streamdock/refresh`	Reload config, push all static icons
`POST /api/v1/streamdock/brightness`	Body `{"percent": 80}`
`POST /api/v1/streamdock/button/{row}/{col}`	Configure a button (label/action/icon; `icon_prompt` generates one)
`DELETE /api/v1/streamdock/button/{row}/{col}`	Remove a button
`POST /api/v1/streamdock/clear`	Blank all keys
`GET /api/v1/streamdock/manager-state`	Focus/auto/overlay/project state
`POST /api/v1/streamdock/action/{name}`	Trigger a manager action (see below)
`POST /api/v1/streamdock/action/refresh-dynamic`	Force-refresh all dynamic icons

Action endpoints under /action/ include full-sync, agent-fire, eink-refresh, meeting-record, cadence/rwgps/notion/amplitude/vita-terminal (host launchers), project-press/{col}, unfocus-project, auto-mode, focus-tag/{tag}, focus-mode/{mode}, projects-overlay / project-overlay-select/{idx} / projects-back, and calisthenics-overlay / calisthenics-log/{kind}/{delta} / calisthenics-back.

# health
curl http://localhost:33800/api/v1/streamdock/status
curl http://localhost:33800/api/v1/streamdock/health

# set brightness
curl -X POST http://localhost:33800/api/v1/streamdock/brightness \
     -H 'content-type: application/json' -d '{"percent": 80}'

# kick off the full data sync (same as pressing the sync button)
curl -X POST http://localhost:33800/api/v1/streamdock/action/full-sync

Lifecycle: it lives in the scheduler, not `./run`

The dock is not a separate ./run service. The scheduler runs a _streamdock_loop (registered as the streamdock operations loop) that:

waits 10s after startup for USB enumeration,
connects (listener + manager + poll thread),
health-checks every STREAMDOCK_RECONNECT_INTERVAL (30s) and reconnects on failure with exponential backoff up to 5 min.

After the API starts, ./run (and the refresh_streamdock helper) polls /api/v1/streamdock/health for "healthy":true and then POSTs action/refresh-dynamic. A --no-streamdock-refresh flag skips that step in dev/test startup.

Tip: to drive a fresh stream dock from scratch, implement the driver protocol above, write a config.json, and POST connect then refresh. Everything dynamic is optional — a static-icon config with shell/http actions is a complete stream-deck controller on its own.

Device registry & check-in API

WiFi peripherals don't have stable IPs, so vita keeps a registry (data/devices/registry.json) keyed by device ID. Devices POST their current address periodically; scripts resolve a URL by ID instead of hardcoding an IP.

File	Role
`api/src/routers/devices.py`	HTTP check-in + listing endpoints
`scripts/devices/registry.py`	Read/write registry, URL resolution, online/offline logic
`data/devices/registry.json`	Persisted registry (atomic writes)

Registry entry shape (illustrative — IPs are placeholders):

{
  "todo-console": {
    "ip": "192.0.2.10",
    "port": 80,
    "last_seen": "2026-06-21T12:00:00+00:00",
    "firmware": "1.4.2",
    "endpoints": ["/display"]
  }
}

Check-in flow

  device  ──POST /api/v1/devices/checkin {device_id, port, ip?, firmware?}──▶  API
                                                                               │
                                                  register_device() updates    │
                                                  registry.json + last_seen ◀───┘

The check-in handler prefers the self-reported ip in the body over the request's source IP — this is the key trick for resolving a device behind Docker NAT, where the request IP would be the container gateway rather than the device. The todo console also performs a passive check-in: every button-press callback re-registers its source IP, so it stays fresh without a dedicated heartbeat.

Method & path	Purpose
`POST /api/v1/devices/checkin`	Device self-registration
`GET /api/v1/devices`	All devices with `online` + `age_minutes`
`GET /api/v1/devices/{device_id}`	One device's record

curl http://localhost:33800/api/v1/devices
curl -X POST http://localhost:33800/api/v1/devices/checkin \
     -H 'content-type: application/json' \
     -d '{"device_id": "todo-console", "port": 80}'

Resolution & staleness

scripts/devices/registry.py is the library side:

Function	Behavior
`register_device(id, ip, port, **meta)`	Upsert; merges firmware/endpoints metadata
`get_device_url(id, endpoint)`	Build URL; raises `DeviceOfflineError` if `last_seen` older than `STALE_MINUTES` (15)
`get_device_url_with_fallback(id, fallback, endpoint)`	Prefer the last-known registry IP even if stale (the IP usually still works); only fall back to a configured base URL if the device never registered
`list_devices()` / `is_online(id)`	Tag each device `online` (age ≤ 15 min) and report `age_minutes`

The 15-minute staleness threshold is the single definition of "online" across the whole system. Three devices are currently tracked: epd47 (creative canvas), todo-console, and silicon-zack (the voice ESP32, documented under Voice & Location).

Peripheral health snapshot

One endpoint rolls up all desk hardware into a single payload for monitoring.

File	Role
`api/src/routers/peripherals.py`	`GET /api/v1/peripherals/health`
`api/src/services/peripheral_health.py`	Combines registry counts + stream dock health

get_peripheral_health_snapshot() calls list_devices() (online/offline counts) and get_streamdock_health() (connection + poll-thread state), and returns:

{
  "generated_at": "...",
  "devices": [ /* full per-device list with online flags */ ],
  "device_summary": { "total": 3, "online": 2, "offline": 1 },
  "streamdock": { "status": "ok", "healthy": true, "connected": true,
                  "running": true, "poll_thread_alive": true }
}

If the stream dock service raises, the streamdock block degrades gracefully to status: "error" with the exception text rather than failing the whole call.

curl http://localhost:33800/api/v1/peripherals/health

Todo Console (ESP32 OLED + buttons)

A physical ESP32-S3 on the desk with 4 OLED rows (128×32 each) and hardware buttons. It shows tasks tagged console from the Precedent task system and lets you clock in/out, complete, pin, and page through tasks without a keyboard. The API router is the entire bridge between hardware and tasks.

File	Role
`api/src/routers/todo_console.py`	The bridge: serves rows, handles callbacks, owns display state
`api/src/services/todo_console_device.py`	URL resolution + `push_display_row` HTTP to the device
`api/src/services/todo_console_tasks.py`	Bucketing/ordering of console/auto/project/tag/focus task views
`api/src/services/todo_console_paging.py`	Page-offset math
`api/src/services/todo_console_mutations.py`	Tag add/remove (pin/unpin/add-to-console)
`api/src/services/todo_console_callbacks.py` / `_actions.py` / `_format.py`	Button-press resolution, display mapping, row text formatting

Note: this is distinct from api/src/routers/console.py, an unrelated software "Activity Console" timeline. Only todo_console drives the ESP32.

Tags & display priority

Three Precedent tags drive what shows up:

Tag	Constant	Meaning
`console`	`CONSOLE_TAG`	Task is eligible for the console
`next`	`PRIORITY_TAG`	Pinned to the top (after any clocked-in task)
`auto`	`AUTO_TAG`	Curated by silicon-zack for the "Auto" view

With MAX_DISPLAYS = 4, the normal ordering is: row 0 = the globally clocked-in task (injected regardless of mode), rows 1–3 = pinned (next) tasks, then other console tasks by recency. A clocked-in task renders inverted on its OLED. Beyond 4 tasks, row 3 gets a (N+) more-indicator and long-press bottom-left pages forward.

Button callbacks

The device POSTs a form-encoded callback per press:

POST /todo-console/callback
  row=0..3   button=0(left)|1(right)   type=short|long   oled_text=<current text>

resolve_button_action(button, type) maps these to actions:

Press	Action
Left (any)	Toggle clock in/out (clocks out other in-mode tasks first)
Right (short)	Complete the task
Right (long)	Toggle pin (`next` tag)
Bottom-left (long)	Next page (mode-aware: project / tag / auto / focus / normal)

Clock-in/out and completion also push a full task detail card to the Telegram Tasks topic, and (in Auto mode) append to data/auto-curator/engagement.jsonl so the curator can learn which auto-tasks get touched.

View modes

Module-level state in todo_console.py (_focused_project, _focused_tag, _auto_mode, _focus_mode, plus per-mode page counters) selects which task list _refresh_displays() paginates. Modes: normal console, project focus, tag focus, auto mode, and focus mode (due / scheduled / in_progress / next_actions — the same buckets the stream dock's row-3 task-view buttons toggle).

Todo console API

Router prefix: /todo-console (note: no /api/v1 here). Relative to http://localhost:33800:

Method & path	Purpose
`POST /todo-console/callback`	Device button callback (form-encoded)
`POST /todo-console/refresh`	Re-render current page
`POST /todo-console/refresh-if-changed`	Re-render only if the visible set differs (avoids flicker)
`POST /todo-console/shuffle`	Random reselect (or next-page when focused)
`GET /todo-console/status`	Display state + resolved device URL
`POST /todo-console/tasks`	Body `["text", ...]` — create `console`-tagged tasks
`POST /todo-console/set-order`	Body `["key", ...]` — force display order (≤4)
`POST /todo-console/pin/{task_key}` · `/unpin/{task_key}` · `/add/{task_key}`	Tag mutations
`POST /todo-console/focus/{path}` · `/unfocus` · `/project-next-page` · `/project-page-one`	Project focus
`POST /todo-console/focus-tag/{tag}` · `/unfocus-tag`	Tag focus
`POST /todo-console/auto-mode` · `/auto-next-page` · `GET /auto-status`	Auto view
`POST /todo-console/focus-mode/{mode}` · `/unfocus-mode` · `GET /focus-status`	Task-view focus
`GET /todo-console/task-counts`	Counts the stream dock renders into row-3 icons

curl -X POST http://localhost:33800/todo-console/tasks \
     -H 'content-type: application/json' -d '["draft the spec", "review PR"]'
curl -X POST http://localhost:33800/todo-console/refresh
curl http://localhost:33800/todo-console/status

The device URL resolves through resolve_console_base_url(device_id="todo-console", fallback=...), which uses the registry's last-known IP and falls back to VITA_TODO_CONSOLE_FALLBACK_URL (default http://todo-console.local).

Auto-curator

So the console isn't a manual chore, a scheduler loop scores Precedent tasks and auto-tags the most relevant ones for the Auto view.

Aspect	Detail
Loop	`_auto_curator_loop` (registered as `auto_curator`, `operations`)
Cadence	Every `AUTO_CURATOR_INTERVAL` (60 min), gated to business hours 7am–6pm PT
Work	`run_auto_curator()` → scores and tags tasks; feeds `get_auto_tasks()`
Manual trigger	`POST /api/v1/scheduler/trigger/auto_curator`

The stream dock's full-sync action also re-triggers it so the smart list is fresh after a data refresh.

EPD47 "live fronts" dashboard (superseded)

The original second-EPD47 generator (scripts/eink/epd47_fronts.py) rendered a 960×540 grayscale layout of active "fronts," commitment counts, and daily-closure status to keep priorities physically visible.

It is no longer scheduled. The scheduler service run_epd47_fronts_push() now renders the coach training dashboard (scripts/eink/epd47_coach.py) for that display slot, and explicitly notes the live-fronts generator is "preserved but no longer scheduled." Furthermore, the auto-schedule loop itself was disabled 2026-05-25 in api/src/loop_engine/registry.py: the epd47-fronts device had been offline since 2026-03-06 and was removed from the registry. Manual fire still works via run_epd47_fronts_now() / run_epd47_coach_now() if the device is revived.

# render the legacy live-fronts layout by hand (device must be online)
uv run python scripts/eink/epd47_fronts.py

The active e-paper surfaces (creative canvas + coach dashboard) are documented on E-ink Displays.

File Locations

Path	What
`scripts/streamdock/driver.py`	hidraw driver, HID protocol, JPEG/grid logic
`scripts/streamdock/listener.py`	Config, icon push, poll thread, action dispatch
`scripts/streamdock/manager.py`	Dynamic icons, focus/overlay state, composite actions
`scripts/streamdock/icon_gen.py` · `icons.py`	Dynamic (Pillow) + static (Imagen 4) icon generation
`data/streamdock/config.json` · `state.json`	Button config + manager state
`api/src/routers/streamdock.py`	Stream dock API (`/api/v1/streamdock`)
`api/src/services/streamdock_service.py`	Singleton accessors + health
`api/src/routers/devices.py` · `scripts/devices/registry.py`	Device registry API + library
`data/devices/registry.json`	Persisted device registry
`api/src/routers/peripherals.py` · `services/peripheral_health.py`	Unified health snapshot
`api/src/routers/todo_console.py` (+ `services/todo_console_*.py`)	ESP32 todo console bridge (`/todo-console`)
`data/auto-curator/engagement.jsonl`	Auto-mode engagement log
`scripts/eink/epd47_fronts.py`	Superseded live-fronts generator
`api/src/loop_engine/registry.py`	Scheduler loop registrations (`streamdock`, `auto_curator`)
`.claude/skills/streamdock/` · `.claude/skills/todo-console/`	Operator skills

Where to go next

E-ink Displays — the EPD47 creative canvas and coach training dashboard.
Precedent — the task/habit system the todo console surfaces.
Voice & Location — the silicon-zack voice ESP32 in the same registry.
Scheduling — the loop engine that runs streamdock and auto_curator.
Architecture — ports, routers, and how the API process hosts these loops.
Proactive Outreach — what the dock's "fire agent" button triggers.