Skip to content

Photos, Faces & Events

End-to-end photo management: ingest from multiple sources, extract metadata via LLM vision, detect and cluster faces, organize into events, and browse via a multi-zoom timeline UI.

System Architecture

INGEST                    EXTRACT                    DETECT & CLUSTER
+-----------------+       +-----------------+        +------------------+
| Google Photos   |       | gemini-3-flash  |        | InsightFace      |
| Google Takeout  | ----> | (premium, most) | -----> | RetinaFace det.  |
| Local Folders   |       | gpt-4.1-nano    |        | ArcFace 512-dim  |
| iOS Uploads     |       | (budget <30KB)  |        | HDBSCAN cluster  |
+-----------------+       +-----------------+        +------------------+
                                |                           |
                                v                           v
                    +-------------------+        +-------------------+
                    | extractions.db    |        | faces.db          |
                    | - photos table    |        | - persons         |
                    | - events table    |        | - face_detections |
                    | - screenshots     |        | - person_centroids|
                    +-------------------+        | - face_clusters   |
                                                 +-------------------+
                                |                           |
                                +-------------+-------------+
                                              |
                                              v
                                  +---------------------+
                                  |     Web UI          |
                                  | /photos  /faces     |
                                  | /events             |
                                  +---------------------+

The iOS Uploads ingest path is served by a dedicated router (api/src/routers/photo_upload.py); all other sources land via the script pipeline. See Ingest & Pipelines for the two orchestrators.

Note: Photo content and face identities are the most personal data in vita. This page documents the engine — schema, models, control flow — never the people in the library.

LLM Vision Extraction

Each photo is analyzed by an LLM to extract structured metadata:

Field Values
life_domain family, social, work, solo, fitness, travel, food, creative, pets, nature, documents
emotional_valence warm, joyful, tense, neutral, melancholy, excited, proud, playful, serene, intimate
significance low, medium, high (with reason)
media_type photo, screenshot, document, video, video_still

Also extracts: description, people_context, activity, place_name, place_type, objects, text_visible (OCR), temporal_cues.

Two-tier routing (via OpenRouter): premium model (google/gemini-3-flash-preview) for most photos, budget model (openai/gpt-4.1-nano) for tiny photos below BUDGET_SIZE_THRESHOLD (30 KB) where the premium model adds no value. Parallel extraction via ThreadPoolExecutor (15 workers default). Model IDs live in scripts/photos/extract.py (PREMIUM_MODEL / BUDGET_MODEL); for vita-wide model routing policy see Model Guidance.

uv run python scripts/photos/extract.py [--limit N] [--workers N] [--dry-run]
uv run python scripts/photos/extract.py --status    # show progress
uv run python scripts/photos/extract.py --test      # test on 5 samples

Face Detection & Clustering

InsightFace buffalo_l model (RetinaFace detection + ArcFace 512-dimensional embeddings, CPUExecutionProvider) with HDBSCAN cosine clustering and a DBSCAN fallback. Embeddings are stored as float32 blobs; blob_to_embedding also auto-detects legacy dlib float64/128-dim embeddings from the pre-InsightFace backend.

The pipeline stages (run via the detect_faces.py CLI):

  1. detect — Run face detection on all photos, extract embeddings
  2. assign — Nearest-neighbor match to named people (cosine similarity >= NN_SIMILARITY_THRESHOLD, 0.6)
  3. consolidate — Pull unknown clusters into named people (high-confidence NN pass)
  4. cluster — HDBSCAN on remaining unknowns (adaptive density, cosine metric)
  5. name — Interactive naming session for top clusters
uv run python scripts/photos/detect_faces.py detect [--limit N]
uv run python scripts/photos/detect_faces.py assign [--threshold 0.6]
uv run python scripts/photos/detect_faces.py consolidate [--threshold 0.6]
uv run python scripts/photos/detect_faces.py cluster [--min-cluster-size 8] [--cluster-eps 0.24] [--refine-eps 0.24]
uv run python scripts/photos/detect_faces.py name
uv run python scripts/photos/detect_faces.py qa
uv run python scripts/photos/detect_faces.py clear-labels
uv run python scripts/photos/detect_faces.py status
uv run python scripts/photos/detect_faces.py reset

The full CLI command choices are: detect, cluster, assign, consolidate, name, qa, clear-labels, status, reset.

Note: There is no auto-split CLI subcommand and no --eps flag. Auto-split is exposed only as an API endpoint (POST /api/v1/faces/clusters/{id}/auto-split) and a web-UI feature. The relevant clustering flags are --cluster-eps (default 0.24) and --refine-eps (default 0.24), not --eps.

Key hyperparameters (scripts/photos/detect_faces.py):

Constant Default Purpose
EMBEDDING_DIM 512 ArcFace embedding dimension
NN_SIMILARITY_THRESHOLD 0.6 Nearest-neighbor cosine match to named people
QUALITY_MIN_DET_SCORE 0.5 Detection-time quality floor
DEFAULT_CLUSTER_EPS / DEFAULT_REFINE_EPS 0.24 Clustering / refinement eps
DEFAULT_MIN_CLUSTER_SIZE 8 HDBSCAN min cluster size
DEFAULT_MIN_SAMPLES 10 HDBSCAN min samples
DEFAULT_CLUSTER_MIN_DET_SCORE 0.72 Detection-score filter at cluster stage
DEFAULT_CLUSTER_MAX_ABS_YAW 22.0 Max abs(yaw) degrees at cluster stage

Ingest & Pipelines

Two distinct orchestrators wrap the per-stage scripts:

Orchestrator When to use Behavior
scripts/photos/pipeline.py Full / batch reprocessing Runs every stage over the whole library
scripts/photos/sync_pipeline.py Incremental, after a sync Runs each stage only on unprocessed rows (WHERE phash IS NULL, person_id IS NULL, extracted_at IS NULL, etc.); meant to be called right after sync_gphotos.py downloads new photos

pipeline.py run executes a 12-step sequence (LLM steps 10/12 skipped unless --extract):

  1. Import takeout (import_takeout.py)
  2. Redate HEIC files (redate.py)
  3. Recover dates from filename patterns (recover_dates.py)
  4. Classify — phash, screenshot detection, thumbnails (classify.py)
  5. Detect faces (detect_faces.py detect)
  6. Assign known faces (detect_faces.py assign)
  7. Consolidate unknown clusters into named people, pre-cluster (detect_faces.py consolidate)
  8. Cluster remaining unknowns (detect_faces.py cluster)
  9. Consolidate again, post-cluster, to pull split fragments into named people
  10. Extract — LLM vision (extract.py, costs money, --extract only)
  11. Detect events — temporal-spatial clustering (events.py detect)
  12. Synthesize events — LLM titles/summaries (events.py synthesize, --extract only)
uv run python scripts/photos/pipeline.py run [--extract]   # full pipeline
uv run python scripts/photos/pipeline.py status            # per-stage status
uv run python scripts/photos/sync_pipeline.py              # incremental
uv run python scripts/photos/sync_pipeline.py --extract    # incremental + LLM
uv run python scripts/photos/sync_pipeline.py --status     # unprocessed counts

Web UI

Photos Page (/photos)

Six zoom levels (ZOOM_LEVELS in PhotosPage.tsx): Photos > Days > Weeks > Months > Years > All

Level Layout Details
Photos Justified grid per day Virtualized rendering, domain badges, significance stars, video play overlay
Days Hero photo + thumbnail fan 160x120 hero, domain bar, event links, description snippet, thumbnail strip with hover fan-out
Weeks Representative aggregation Diversity-weighted sampling by significance + media_type
Months Representative aggregation Domain breakdown per period
Years Representative aggregation Scrollable/zoomable navigation
All Full collection overview Top-level summary

Filters: search text, life_domain[], significance[], media_type[], emotional_valence[], year, date range, person_id, exclude_screenshots.

Lightbox: Context strip (18 surrounding photos), prev/next respecting filters, detail panel.

Faces Page (/faces)

Cluster management interface:

  • Filter modes: Unnamed (needs naming), Actionable, Named, Ignored, All
  • Sort: Count, Recent, Quality (avg detection score), Oldest
  • Card display: Face crop + sample strip, name, relationship badge, year range, detection count, quality score, co-appearing faces (top 5)
  • Merge mode: Select 2 clusters, confirm merge (source -> target)
  • Detail panel: Detection grid (4 columns, virtualized), manual split (select + name), auto-split presets (Default, Tight, Cleanup with DBSCAN params), unassign/reassign, clear-name

Events Page (/events)

Two views: List (default) and Map.

List view — virtualized with dynamic card sizing:

Significance Layout Height
High Hero card: full-width photo background with gradient, summary 260px
Medium Standard card: 160px thumbnail + 3-line text 164px
Low Compact row, batched in groups 52px

Year headers with event counts and domain bar. Year scrubber on right edge for fast navigation.

Features: Debounced search (300ms), filter dropdowns (domain, significance, year, people), keyboard shortcuts (/ search, f filters, Esc clear), scroll position restoration, infinite pagination (200 per request).

Map view: Lazy-loaded component showing all geotagged events.

API Endpoints

Photos (/api/v1/photos)

Served by api/src/routers/photos.py.

Method Endpoint Purpose
GET /buckets Month-level photo counts for timeline navigation
GET /bucket/{month} Photos for a specific month (grid rendering)
GET /timeline Multi-level timeline (year/month/week/day/moment)
GET /stats Aggregate stats for filter dropdowns
GET /{photo_id} Full photo metadata + faces + event
GET /{photo_id}/adjacent Next/prev photo under current filters
GET /{photo_id}/context Context strip (18 photos around current)
GET /thumbnail/{path} Cached/resized thumbnail (width, height, fit, quality)
GET /original/{path} Original photo (HEIC->JPEG conversion on demand)

Photo Upload (/api/v1/photos)

The iOS-upload data path, served by api/src/routers/photo_upload.py (shares the /api/v1/photos prefix).

Method Endpoint Purpose
POST /upload Upload a photo from the iOS app
POST /check-hashes Client sends content hashes; server replies which are already present (skip re-upload)
POST /process Trigger processing of newly uploaded photos

Faces (/api/v1/faces)

Served by api/src/routers/faces.py.

Method Endpoint Purpose
GET /clusters List all face clusters with co-occurrence
GET /crops/{filename} Serve face crop image
POST /clusters/{person_id}/name Name a cluster
POST /clusters/{person_id}/clear-name Clear a cluster's name (back to unnamed)
POST /clusters/clear-all-names Clear names from all clusters
POST /clusters/merge Merge two clusters
POST /clusters/{person_id}/ignore Mark cluster as ignore/unignore
POST /clusters/{person_id}/split Manual split (select detection IDs)
POST /clusters/{person_id}/auto-split ML-driven split via DBSCAN refinement
GET /clusters/{person_id}/detections All face detections for a person
GET /clusters/{person_id}/events Events where person appears
DELETE /clusters/{person_id} Delete cluster (unassign detections)

Events (/api/v1/photos/events)

Served by a separate router, api/src/routers/events.py (prefix /api/v1/photos/events).

Method Endpoint Purpose
GET /stats Aggregate stats for filter dropdowns
GET /timeline Year-by-year event counts with domain breakdown
GET /geo All geotagged events for map view
GET `` List events with pagination and filters
GET /{event_id} Event detail with photos, adjacent events, faces
PATCH /{event_id} Update event fields
POST /merge Merge source event into target

Database Schema

Two SQLite catalogs, both initialized in scripts/photos/db.py (init_extractions_db, init_faces_db, migrate_faces_db).

extractions.db

photos: id (SHA256 prefix), path, taken_at, camera_make, camera_model, width, height, file_size, mime_type, media_type, quality, phash, burst_group, is_duplicate, description, people_context, activity, place_name, place_type, objects (JSON), text_visible (OCR), life_domain, emotional_valence, significance, significance_reason, temporal_cues, event_id, imported_at, extracted_at, extraction_model, extraction_cost

events: id (evt_YYYYMMDD_slug), date, time_start, time_end, location_name, latitude, longitude, place_type, people (JSON array of names), summary, photo_count, representative_photo_id, life_domain, significance, synthesized_at

Note: The events table has no title column — the human-facing label is derived from summary / location_name. people is a JSON array and synthesized_at records when the LLM titling/summary step ran.

screenshots: photo_id (FK -> photos), screenshot_type (conversation/article/app/map/recipe/other), extracted_text (full OCR), topic, source_app

faces.db

persons: id (person_XXXXXXXX), name, relationship, first_seen, last_seen, photo_count, named_at, is_ignored, ignored_at, ignore_reason

face_detections: id, photo_id (-> extractions.db photos), person_id, confidence (legacy dlib distance), det_score, pose_yaw/pose_pitch/pose_roll, embedding_version (default arcface), bbox_x/bbox_y/bbox_w/bbox_h, embedding (512-dim float32 blob), assignment_method (cluster/nn/manual)

person_centroids: person_id, centroid (512-dim float32 L2-normalized blob), detection_count, updated_at

face_model_info: key/value store (e.g. embedding dim / model identifier)

face_clusters: id, cluster_label (DBSCAN label), person_id, detection_count, representative_detection_id

Scripts

Script Purpose
scripts/photos/extract.py LLM vision extraction (hybrid Gemini/GPT routing)
scripts/photos/detect_faces.py Face detection, clustering, naming, QA
scripts/photos/events.py Event detection and synthesis from photos
scripts/photos/classify.py Media-type classification, phash, screenshot detection
scripts/photos/ingest_folder.py Local folder import with EXIF/mtime recovery
scripts/photos/sync_gphotos.py Google Photos API sync
scripts/photos/download_takeout.py Download Google Takeout files via browser CDP
scripts/photos/import_takeout.py Import downloaded Takeout zips into the catalog
scripts/photos/redate.py Re-date HEIC files that fell through to zip mtime
scripts/photos/recover_dates.py Recover taken_at from filename patterns
scripts/photos/pipeline.py Full 12-step pipeline orchestration
scripts/photos/sync_pipeline.py Incremental pipeline (unprocessed rows only)
scripts/photos/regenerate_thumbnails.py Batch thumbnail generation
scripts/photos/db.py DB initialization and connection helpers

Thumbnail System

  • Base: 1200x1200 (fit="inside"), quality 86
  • On-demand variants: ?width=X&height=Y&fit=cover&quality=Q (fit is inside or cover)
  • Caching: SHA256(path + mtime + dimensions + fit + quality) as the cache key
  • Format conversion: HEIC/HEIF -> JPEG via pillow-heif (register_heif_opener), video thumbnails via ffmpeg

Face Crops

  • Display crops: 200x200 with 30% padding around bounding box
  • Aligned crops: 112x112 normalized from keypoints (for embedding inspection)
  • Storage: data/photos/face_crops/{photo_id}_{face_idx}.jpg

File Locations

Path Contents
data/photos/originals/ Original photo files (NAS-mounted symlink -> /mnt/zacknas/photos/originals)
data/photos/extractions.db Photo metadata + events + screenshots
data/photos/faces.db Face detections + clusters
data/photos/face_crops/ Cropped face images
data/photos/thumbnails/ Cached thumbnails
data/photos/thumbnail_variants/ On-demand resized thumbnail variants
scripts/photos/ All photo processing scripts
api/src/routers/photos.py Photos API
api/src/routers/photo_upload.py iOS upload API
api/src/routers/events.py Events API (/api/v1/photos/events)
api/src/routers/faces.py Faces API
web/src/features/photos/ Photos feature components (zoom levels, lightbox)
web/src/routes/photos/ Photos route
web/src/routes/faces/ Faces route
web/src/routes/events/ Events route

Where to go next

  • Search — how photo metadata feeds long-term memory search
  • iOS App — the client behind the photo upload endpoints
  • Model Guidance — vita-wide model routing policy (extraction tiers defer here)
  • Architecture — where the API/web/scripts layers fit