Skip to content

Voice and Location APIs

The Voice and Location APIs power the iOS app's voice assistant and location awareness features. The voice system enables hands-free interaction with vita through a server-side turn-processing pipeline with SSE-streamed responses; the location system provides geofencing for context-aware reminders and presence detection.

Two voice surfaces share this domain but use different engines:

  • Turn-based voice sessions (this page) — the iOS app's primary voice assistant. Audio in, SSE out: Groq Whisper STT → shared Claude inference → ElevenLabs TTS.
  • Hardware voice pipeline — a silicon-zack ESP32 device that streams raw PCM over a WebSocket (STT → intent classification → Telegram-topic dispatch → response).

Note: The OpenAI Realtime API voice integration is a separate engine that belongs to the Sync Meetings feature, not the turn-based assistant documented here. The iOS turn pipeline uses Groq + ElevenLabs; Realtime is exclusive to the live meeting-surrogate path.

Overview

+-------------------+      +------------------+
|   iOS App         |      | VITA API         |
|   (Voice UI)      | <--> | (Turn Processing,|
|                   |      |  SSE Streaming,  |
|                   |      |  Tool Execution) |
+-------------------+      +--------+---------+
        |                           |
        |                  +--------v-----------+
        |                  | Shared inference    |
        |                  | factory (task_type  |
        |                  | = "voice")          |
        |                  +---------------------+
        |
+-------v----------+      +-------------------+      +------------------+
| CoreLocation     | ---> | Geofence Triggers | ---> | Presence Monitor |
| (iOS Geofencing) |      | (enter/exit)      |      | (Home Reminders) |
+------------------+      +-------------------+      +------------------+

Security: Both APIs are secured via Tailscale network access (both router module docstrings state "no token auth required"). There is no token authentication when accessed over the Tailscale network. The API listens on port 33800 (./run api).

Note on status: The voice turn pipeline and the hardware WebSocket pipeline are active (the ESP32 device reports live). The GPS-tracking and geofencing paths are currently dormant — the code is fully built, but data/location/transitions.jsonl and the daily history files stopped growing in May 2026, so no live location data is flowing. The location sections below document the engine as built.

Voice API

Endpoints

Method Endpoint Description
POST /api/v1/voice/sessions Create a new voice session
GET /api/v1/voice/sessions List sessions (filterable by status)
GET /api/v1/voice/sessions/{session_id} Get full session detail with turns
PATCH /api/v1/voice/sessions/{session_id} Update session metadata (title, muted, voice_mode)
DELETE /api/v1/voice/sessions Delete all sessions (bulk)
DELETE /api/v1/voice/sessions/{session_id} Delete a single session
POST /api/v1/voice/sessions/{session_id}/clear Clear session context
POST /api/v1/voice/sessions/{session_id}/retry Retry the last turn
POST /api/v1/voice/sessions/{session_id}/turns Submit a turn (audio or text); SSE-streamed response
GET /api/v1/voice/sessions/{session_id}/stream Reconnect to an in-progress turn's SSE stream
GET /api/v1/voice/turns/{turn_id}/audio Get TTS audio for a turn (MP3)
GET /api/v1/voice/turns/{turn_id}/original-audio Get original user recording (M4A)
GET /api/v1/voice/audio List tracked audio files
GET /api/v1/voice/audio/stats Audio storage stats
POST /api/v1/voice/audio/backfill Backfill TTS for existing turns
GET /api/v1/voice/health Health check for connectivity verification
POST /api/v1/voice/debug-log Receive debug logs from iOS
WS /api/v1/voice/hardware/ws Hardware voice pipeline (binary PCM streaming) — see Hardware Voice Pipeline

Session Creation

Create a persistent session before submitting turns:

Request: 

POST /api/v1/voice/sessions
Content-Type: application/json

{
  "title": "Morning check-in"
}

Response: 

{
  "id": "A1B2C3D4-E5F6-7890-ABCD-EF1234567890",
  "title": "Morning check-in",
  "status": "active",
  "muted": false,
  "voice_mode": false,
  "is_processing": false,
  "created_at": "2026-01-30T08:00:00Z",
  "updated_at": "2026-01-30T08:00:00Z",
  "last_message": null
}

voice_mode is a boolean (default false), not a string. All audio processing and AI inference happen server-side. The iOS client submits turns (audio or text) and consumes SSE-streamed responses.

Submitting a Turn

Each turn is submitted as multipart form data with audio or text:

Request (audio): 

POST /api/v1/voice/sessions/A1B2C3D4-.../turns
Content-Type: multipart/form-data

audio=<M4A audio file>

Request (text): 

POST /api/v1/voice/sessions/A1B2C3D4-.../turns
Content-Type: multipart/form-data

text=How did I sleep last night?

For audio input, the server transcribes via Groq Whisper (scripts/integrations/groq_whisper.py, transcribe_audio()) before inference begins; the transcript is emitted as the first SSE event.

The server returns an SSE stream with the following event types:

Event Payload Description
transcript {text} Audio transcription (audio-input turns only)
turn_created {turn_id} Emitted before inference starts; the assistant turn id
text_delta {text} Streaming response chunks
tool_start {id, name} Tool call beginning
tool_end {id, output, is_error} Tool call result
title_updated {title} Auto-generated session title (sent before complete)
keepalive {} Sent every 30s to prevent iOS URLSession timeout
complete {turn_id, text, has_audio} Turn finished. has_audio is hardcoded true — TTS is always pre-generated fire-and-forget after completion
error {message} Processing error

Inference design: shared pipeline + durable background task

The key architectural point is that voice turns are not a bespoke voice agent. They run through the same shared inference factory as Telegram:

from scripts.inference.factory import stream_inference

stream = stream_inference(..., task_type="voice")

Each turn is launched as a background asyncio task (_run_inference_background in voice_turn.py). That task:

  1. Runs stream_inference and pushes events to a StreamBroadcaster.
  2. Always persists the assistant turn (and tool calls) to SQLite regardless of whether the SSE client is still connected.
  3. Pre-generates TTS in the background after completion.

Because persistence is decoupled from the HTTP connection, the iOS client can jump between sessions, background the app, or lose connectivity mid-turn without losing the result.

Stream reconnection (StreamBroadcaster)

A StreamBroadcaster accumulates all SSE events for an active turn (text_delta, tool_start/tool_end, transcript, title_updated, keepalive, complete, error) and broadcasts to multiple subscribers, replaying past events to late joiners. This lets the iOS app reconnect to an in-progress turn:

GET /api/v1/voice/sessions/{session_id}/stream

The server replays accumulated events then continues with live ones. If there is no active stream for the session, it returns 204.

iOS App                          VITA API
   |------ POST /sessions --------->|
   |<----- session id --------------|
   |  [User speaks or types]        |
   |------ POST /sessions/{id}/turns (audio or text) -->|
   |<====== SSE stream ============|
   |     transcript                 |
   |     turn_created               |
   |     text_delta, text_delta ... |
   |     tool_start / tool_end      |
   |     title_updated              |
   |     keepalive (every 30s)      |
   |     complete (turn_id, has_audio=true) |
   |  [Retrieve TTS if desired]     |
   |------ GET /turns/{turn_id}/audio ----------------->|
   |<----- MP3 bytes ---------------|
   |  (sessions persist until deleted)

Voice Tools

The voice prompt builder defines two OpenAI-compatible inline tools (scripts/voice/prompt_builder.py, get_tool_schema()):

Tool Purpose Parameters
ask_vita General information lookup / task execution — calendar queries, data lookups, file operations, any question about bio-zack's life. Routes back into internal Claude inference. query (required)
set_reminder Create a location- or time-based reminder. task (required), trigger ∈ home|work|gym|time (required), time (ISO timestamp, required when trigger="time")

Tool calls are executed server-side during turn processing.

Voice TTS (ElevenLabs)

Assistant turns are synthesized to speech with an ElevenLabs cloned voice (scripts/voice/tts.py):

Setting Value
Model eleven_multilingual_v2
Voice id 4NykfJgp4HqPHbp1OwTB (cloned "Zack" voice)
Output MP3 (also convertible to raw 16-bit LE PCM @ 16kHz for hardware)

Behavior:

  • TTS is pre-generated fire-and-forget after each turn completes, so replay is instant. has_audio on the complete event is therefore always true.
  • TTS speaks the exact stored assistant text — there is no generative speech-rewrite layer that could diverge from the chat transcript.
  • Voice sessions pin ElevenLabs with allow_fallback=False, so replay audio never silently swaps to a different (Edge TTS) voiceprint.
  • Original user recordings are stored as M4A and retrievable; synthesized audio is cached to disk. An audio_files table tracks all stored audio, with stats and a backfill endpoint.
Endpoint Returns
GET /api/v1/voice/turns/{id}/audio TTS MP3 for an assistant turn
GET /api/v1/voice/turns/{id}/original-audio Original user recording (M4A)
GET /api/v1/voice/audio List of tracked audio files
GET /api/v1/voice/audio/stats Counts / bytes / durations
POST /api/v1/voice/audio/backfill Generate TTS for turns missing it

Voice Session Auto-titling

Sessions auto-generate a short (3–5 word) title from the conversation, regenerated on each user message to reflect current focus (scripts/voice/auto_title.py). Titles are produced via google/gemini-3.1-flash-lite-preview over OpenRouter and pushed to the client over SSE as title_updated (before the complete event).

Voice System Prompt Builder

scripts/voice/prompt_builder.py assembles the silicon-zack voice system prompt from identity sources and fills the template at prompts/voice-system.md:

Placeholder Source
{who_i_am_section} The "Who I Am" section extracted from .claude/CLAUDE.md
{voice_profile} Spoken voice profile from profile/communication.json (traits, patterns, vocabulary, things to avoid)
{current_time} Natural-language spoken time (e.g. "Wednesday January 15th, 3:18 in the afternoon")
{current_location} Optional location context (e.g. "at home"), or "location unknown"
{recent_turns} Last few conversation turns

Session Storage

Sessions and turns are stored in a SQLite database at data/voice/sessions.db (scripts/voice/session_store.py). Audio files (original recordings and TTS) are stored on disk with their paths recorded in the database. The schema uses journal_mode=WAL and runs additive migrations for columns added after initial release.

sessions table: id, title, status (active|closed), claude_session_key, muted, voice_mode, created_at, updated_at

turns table: id, session_id, role (user|assistant), text, has_audio, input_mode (text|voice), status (complete|pending|processing|error), error_message, tool_calls (JSON), created_at

audio_files table: id, turn_id, session_id, type (original|tts), format (m4a|mp3|wav), path, size_bytes, duration_secs, sample_rate, created_at

Reviewing Sessions

GET /api/v1/voice/sessions
GET /api/v1/voice/sessions?status=active
GET /api/v1/voice/sessions/{session_id}        # full session with turns
GET /api/v1/voice/turns/{turn_id}/audio        # TTS MP3
GET /api/v1/voice/turns/{turn_id}/original-audio

Hardware Voice Pipeline

A WebSocket endpoint serves the silicon-zack ESP32 device — a distinct path from the HTTP/SSE assistant. It streams raw audio, transcribes it, classifies intent, dispatches to a Telegram topic thread, runs Claude through that topic's session, and returns either spoken audio or OLED-friendly text.

ESP32 ──WS──> /api/v1/voice/hardware/ws
   |  1. server sends {"type": "ready"}
   |  2. client streams binary 16kHz/16-bit mono PCM frames
   |  3. client sends {"type": "end_recording", "response_mode": "voice"|"text"}
   v
process_hardware_voice()  (scripts/voice/hardware_voice.py)
   |  wrap PCM in WAV header
   |  STT via Groq Whisper
   |  classify intent (scripts/voice/intent_router.py, Gemini over OpenRouter)
   |  resolve intent -> Telegram topic thread
   |  run Claude via the topic's session
   v
response:
   - text mode  -> {"type": "text_response", "text": "..."}  (128x64 OLED)
   - voice mode -> binary frame of raw 16kHz PCM audio (speaker)

Intent → topic map (INTENT_TOPIC_MAP): training, habits, tasks, interviews, calendar, meetings, general (unknown falls back to General). Routing the command to the matching topic's ClaudeSession keeps hardware conversations threaded with their Telegram counterparts.

The WebSocket processes buffered audio even on early/abnormal disconnect, defaulting to voice response mode.

Hands-free Control (iOS)

The iOS app adds eyes-free operation on top of the turn API (Swift, in vita-ios/):

  • State machine: off / idle / recording / processing / speaking (HandsFreeController.swift).
  • BLE media remote: play/pause, next, prev, vol+/- buttons map (per-state, configurable) to actions — start/send/discard recording, jump/new/delete session, repeat last, stop playback.
  • Acoustic triggers: an on-device WhistleDetector (1024-pt FFT, Hann window, spectral ratio) and a ClickDetector (2.5 kHz high-pass biquad + ratio/decay gates).

The detection algorithms have offline, WAV-based threshold-tuning scripts that replicate the iOS detectors exactly:

uv run python scripts/voice/tune_whistle_detector.py analyze recording.wav
uv run python scripts/voice/tune_click_detector.py analyze recording.wav

Location API

Status: Dormant. Code is fully built; no live GPS data is flowing (history/transitions stopped in May 2026).

Endpoints

Method Endpoint Description
POST /api/v1/location/update Record a single GPS location (also runs server-side geofence check)
POST /api/v1/location/batch Record a batch of locations (offline sync; geofence check on most recent)
POST /api/v1/location/geofence-trigger Handle an iOS CLRegion geofence enter/exit event
GET /api/v1/location/places Get all places (for display)
GET /api/v1/location/places/geofence Get priority places for geofencing (max 20)
POST /api/v1/location/places Create a new place
PUT /api/v1/location/places/{place_id} Update place coordinates / radius
DELETE /api/v1/location/places/{place_id} Delete a place
GET /api/v1/location/history/{date} Location history for a date
GET /api/v1/location/history Location history for a date range (max 30 days)
GET /api/v1/location/summary/{date} Daily location summary
GET /api/v1/location/dates List dates with location data
GET /api/v1/location/insights Location insights and patterns

Recording Locations

The iOS LocationManager (CoreLocation) does significant-location-change monitoring plus optional continuous 60s tracking, buffers pings offline in UserDefaults, and batch-syncs them. Readings with accuracy worse than 10 km are dropped client-side (both on upload and on pending-load) so one bad reading does not 422 the whole batch.

Single location update: 

POST /api/v1/location/update
Content-Type: application/json

{
  "latitude": 37.7749,        // illustrative SF placeholder
  "longitude": -122.4194,
  "accuracy": 10.5,
  "timestamp": "2026-02-02T15:30:00Z",
  "speed": 1.2
}

Note: All coordinates in this doc are generic San Francisco placeholders. Real home/place coordinates live in profile/places.json, which is gitignored and never reproduced here.

Batch update (offline sync): 

POST /api/v1/location/batch
Content-Type: application/json

{
  "locations": [
    {"latitude": 37.7749, "longitude": -122.4194, "accuracy": 10.5, "timestamp": "2026-02-02T15:30:00Z"},
    {"latitude": 37.7752, "longitude": -122.4190, "accuracy": 12.0, "timestamp": "2026-02-02T15:35:00Z"}
  ]
}

Server stores pings in daily JSONL files (data/location/history/) and computes summaries/insights (distance, top places, home-time %).

Geofencing

Geofencing runs in two layers — iOS CLRegion monitoring with a server-side haversine backup — so background-killed or inaccurate-coordinate misses still fire reminders.

Layer 1 — iOS CLRegion

  1. iOS fetches priority places via GET /api/v1/location/places/geofence.
  2. iOS registers a CLCircularRegion for each (max 20, the iOS limit — get_priority_places(max_count=20)).
  3. CoreLocation monitors regions in the background.
  4. On boundary crossing, iOS sends POST /api/v1/location/geofence-trigger.
  5. vita processes the event (logs the transition, fires reminders if applicable).

Layer 2 — server-side haversine backup

POST /location/update and POST /location/batch call check_and_fire_transitions() (scripts/location/geofence.py) on the incoming ping(s). This haversine-checks each ping against known places and fires the same enter/exit handlers as the iOS CLRegion path, covering CLRegion misses. An ENTRY_COOLDOWN_SECONDS = 3600 (1 hour) guard prevents duplicate entry firings; geofence state is persisted in data/location/geofence_state.json.

Place priority

iOS can only monitor 20 geofence regions, so places are prioritized by type (TYPE_PRIORITY in scripts/location/places.py):

Priority Type Example
0 current_home Primary residence
1 work Office
2 gym Fitness center
99 (default) All other places

A place may also carry an explicit priority field that overrides the type default.

Geofence event

POST /api/v1/location/geofence-trigger
Content-Type: application/json

{
  "place_id": "home",
  "event": "enter",
  "timestamp": "2026-02-02T18:30:00Z"
}

{
  "status": "ok",
  "place_id": "home",
  "event": "enter",
  "reminders_fired": [42, 43]
}

Transition logging

All enter/exit events are logged to data/location/transitions.jsonl:

{"ts": "2026-02-02T08:30:00", "place_id": "home", "event": "exit", "source": "ios"}
{"ts": "2026-02-02T09:15:00", "place_id": "gym", "event": "enter", "source": "ios"}
{"ts": "2026-02-02T10:30:00", "place_id": "gym", "event": "exit", "source": "ios"}
{"ts": "2026-02-02T11:00:00", "place_id": "home", "event": "enter", "source": "ios"}

Places Configuration

File: profile/places.json (gitignored)

{
  "schema_version": 1,
  "places": {
    "home": {
      "name": "Home",
      "latitude": 37.7749,
      "longitude": -122.4194,
      "radius_meters": 150.0
    },
    "gym": {
      "name": "Fitness Center",
      "latitude": 37.7699,
      "longitude": -122.4469,
      "radius_meters": 100.0,
      "type": "gym"
    }
  }
}

Field Reference

Field Type Required Default Description
name string Yes – Display name for the place
latitude float No – GPS latitude (-90 to 90)
longitude float No – GPS longitude (-180 to 180)
radius_meters float No 150.0 Geofence radius (>0, ≤10000m)
type string No – current_home, work, gym, or custom
priority int No (by type) Lower = higher priority for iOS slot allocation

Adding / updating places via API

POST /api/v1/location/places
Content-Type: application/json

{
  "name": "Coffee Shop",
  "latitude": 37.7795,
  "longitude": -122.4195,
  "radius_meters": 50.0
}

{ "status": "ok", "created": true, "place_id": "coffee_shop" }

Place ID is auto-generated from the name (lowercase, underscores for spaces). PUT /api/v1/location/places/{place_id} updates coordinates / radius.

Integration with the Proactive System

Location-triggered reminders

The voice set_reminder tool (executed server-side during a turn) creates a presence-based reminder when trigger is home, work, or gym (or a time-based reminder when trigger="time"):

User: "Remind me to take my supplements when I get home"
         |
         v  (server-side tool execution during SSE turn)
+--------+----------+
| set_reminder      |
| trigger: "home"   |
+--------+----------+
         |
         v
+--------+----------------+
| schedule_when_home()    |  --> enqueue in proactive/queue.db
| (scripts/.../triggers.py:1156) |  with a presence trigger
+--------+----------------+
         |
         v  [Later, on arrival — iOS CLRegion OR server-side haversine]
+--------+----------+
| geofence trigger  |  event: "enter"
+--------+----------+
         |
         v
+--------+------------------+
| handle_geofence_entry()   |  --> fire reminder(s) via the message queue
| (scripts/.../ios_presence.py:122) |
+---------------------------+

On enter, handle_geofence_entry() fires pending presence reminders for the place and its aliases (and supports delayed firing); on exit, handle_geofence_exit() fires exit hooks. Reminders are enqueued via the message queue rather than sent inline — the old inline path ran a sync send inside the async loop and silently dropped messages while marking them sent.

Per-place hooks

Place IDs map to home / work / gym / skatepark triggers. Hooks include capability examples like: arriving at the skatepark sends a goal-driven message (current trick list pulled from a goal file), and leaving enqueues a Claude prompt to ask how the session went and auto-log the workout.

Current location context

Scripts can query current presence (scripts/proactive/ios_presence.py):

from scripts.proactive.ios_presence import get_current_location_context

context = get_current_location_context()
# {
#   "current_place": "home",
#   "last_transition": {"ts": "...", "place_id": "home", "event": "enter"},
#   "is_at_home": True
# }

This context flows into proactive messages (e.g. different prompts for home vs. away) and into the voice system prompt's {current_location} placeholder.

Location History and Insights

GET /api/v1/location/summary/2026-02-02

{
  "date": "2026-02-02",
  "ping_count": 156,
  "first_ping": "2026-02-02T06:30:15Z",
  "last_ping": "2026-02-02T22:45:30Z",
  "distance_km": 12.4,
  "places_visited": ["home", "gym", "home"]
}


GET /api/v1/location/insights?days=7

{
  "period_days": 7,
  "total_pings": 1045,
  "total_distance_km": 78.3,
  "avg_daily_distance_km": 11.2,
  "top_places": [{"place": "home", "visits": 14}, {"place": "gym", "visits": 4}],
  "home_time_percentage": 72.5,
  "days_with_data": 7
}

File Locations

Voice

File Purpose
api/src/routers/voice.py Voice + hardware WebSocket endpoints
api/src/services/voice_turn.py Turn processing, SSE streaming, StreamBroadcaster, background inference
api/src/schemas/voice.py Pydantic request/response schemas
scripts/voice/session_store.py SQLite session/turn/audio storage
scripts/voice/prompt_builder.py System prompt assembly + tool schemas
scripts/voice/tts.py ElevenLabs text-to-speech
scripts/voice/auto_title.py Session auto-titling (Gemini over OpenRouter)
scripts/voice/hardware_voice.py ESP32 pipeline (STT → intent → topic dispatch)
scripts/voice/intent_router.py Intent classification
scripts/integrations/groq_whisper.py Groq Whisper STT (shared by both pipelines)
prompts/voice-system.md Voice system prompt template
profile/communication.json Spoken voice profile source
data/voice/sessions.db SQLite database (sessions, turns, audio metadata)
vita-ios/.../Services/HandsFreeController.swift iOS hands-free state machine

Location

File Purpose
api/src/routers/location.py Location API endpoints
scripts/location/places.py Place management + priority ordering
scripts/location/geofence.py Server-side haversine geofence backup
scripts/location/history.py Location history storage / summaries
scripts/proactive/ios_presence.py Geofence enter/exit handlers + presence context
scripts/proactive/triggers.py schedule_when_home()
profile/places.json Place definitions (gitignored)
data/location/history/ Daily location JSONL files
data/location/transitions.jsonl Geofence enter/exit log
data/location/geofence_state.json Server-side geofence cooldown state

Error Handling

Voice API

Error HTTP Status Response
No audio and no text provided – error SSE event: {"message": "No input provided"}
Session not found 404 {"detail": "Session not found"}
Transcription / inference error SSE error event {"message": "..."} in the stream
No active stream to reconnect 204 (empty)
Audio not found 404 {"detail": "..."}

Location API

Error HTTP Status Response
Invalid date format 400 {"detail": "Invalid date format. Use YYYY-MM-DD"}
Date range too large 400 Range capped at 30 days
Invalid coordinates / radius 422 Pydantic validation error
Place not found 404 {"detail": "..."}

Where to go next

  • iOS App — the client that drives both APIs
  • Proactive Outreach — the message queue that delivers presence reminders
  • Telegram — topic threads the hardware pipeline dispatches into
  • Model Guidance — which models back task_type="voice" and the auxiliary STT/title/intent calls