Hybrid Search

Hybrid search is silicon-zack's long-term memory. It indexes the entire data layer into a single SQLite database and retrieves over it with a hybrid of vector (semantic) and BM25 (keyword) search, fused by rank, decayed by recency, enriched with known-person profiles, and reranked for source diversity. When silicon-zack needs to recall anything about bio-zack's life — a past conversation, a health day, a relationship, a decision — it searches.

The module lives at scripts/search/. The public surface is two functions:

from scripts.search import search, reindex

search() reads; reindex() writes. Everything else is internal.

Why hybrid

Neither retriever alone is sufficient, so the system runs both and merges.

Method	Strengths	Weaknesses
Vector (semantic)	Catches synonyms, related concepts, paraphrases	Misses exact terminology, proper nouns
BM25 (keyword)	Precise matches for names, technical terms	Misses semantically similar content
Hybrid	Semantic understanding plus exact matching	More moving parts to tune

A query like "date night ideas" benefits from the vector side (finds "romantic dinner", "evening out") and the BM25 side (finds the exact words and proper nouns) at once. Rank fusion then combines them without either retriever's score scale dominating.

Data sources

The index covers the whole data layer — roughly two dozen source types, dominated in volume by the messaging archives. Chunk counts are illustrative of shape, not exact (the DB is rebuilt continuously); the messaging types are orders of magnitude larger than the identity types.

Source type	Origin	Relative volume
`slack`	Synced Slack channel history (`data/slack/*.jsonl`)	very large
`telegram`	Telegram DMs + proactive conversation log	large
`meeting`	Processed meeting transcripts (`data/meetings/processed/`)	medium
`email_catalog`	Curated important-email catalogs	medium
`event`	Photo events index	medium
`garmin`	Daily health metrics, one summary per day	small
`signal`	A synced Signal thread	small
`transcript`	Identity interview transcripts	small
`voice`	Voice-memo transcript segments	small
`entity_file`	`memory/entities/{people,projects,topics}/*.md`	small
`codebase_doc`	`memory/codebase/*.md` knowledge-graph docs	small
`quote` / `observation` / `story`	Identity layer (statements, patterns, narratives)	small
`checkin` / `journal` / `experiment`	Health/journaling layers	small
`rwgps_memory`	Work-context memory notes	small
`podcast_digest`	Podcast digest entries	small
`board_summary`	Board-meeting summaries	small
`scratch`	Daily / weekly working notes	small
`interview_meta` / `interview`	Interview session metadata	small
`cgm`	Daily continuous-glucose summary	varies

The full routing table (path pattern → source type) is FILE_ROUTES in scanner.py; that list is the canonical answer to "what is indexed." To add a source: add a route there and register an extractor (see Extractors).

Note: The live database is multi-gigabyte and contains hundreds of thousands of chunks, almost entirely messaging history. Plan storage and maintenance accordingly — see Index maintenance.

The search pipeline

search(query, limit=10, source_types=None) runs a fixed sequence (searcher.py, function search):

query
  │
  ├─ embed query  ──────────────►  embed_single() → 1536-dim vector
  │
  ├─ vector search (KNN) ───────►  chunks_vec MATCH, k = limit*5
  ├─ BM25 search (FTS5) ────────►  chunks_fts MATCH, LIMIT limit*5
  │
  ├─ RRF merge ─────────────────►  score = Σ 1/(k + rank), k=60
  ├─ recency decay ─────────────►  score *= 2^(-age/half_life)  (floored)
  ├─ entity injection ──────────►  synthesize a known person's profile row
  ├─ source_type filter ────────►  (optional caller filter)
  └─ diversity rerank ──────────►  MMR, cap 3 per source type
                                        │
                                   results[:limit]

Each stage is its own helper, so the control flow is easy to trace. The stages in order:

1. Candidate fetch

Both retrievers fetch fetch_limit = limit * 5 candidates, not just limit. The oversized pool gives the fusion and diversity stages enough minority-source candidates to work with.

2. Vector search (`_vector_search`)

The query is embedded once with OpenAI text-embedding-3-small (1536 dimensions, embedder.embed_single). The vector is packed into a binary blob and matched against the chunks_vec sqlite-vec vec0 virtual table doing KNN inside SQLite:

SELECT c.id, v.distance, c.content, c.source_file, c.source_type, c.content_date
FROM (
    SELECT rowid, distance FROM chunks_vec
    WHERE embedding MATCH ? AND k = ?
) v
JOIN chunks c ON c.rowid = v.rowid
ORDER BY v.distance

distance_metric=cosine, so similarity is 1 - distance. Vectors are not loaded into Python for cosine math — the KNN runs in the DB engine, which is what makes the multi-GB index searchable in milliseconds.

3. BM25 search (`_bm25_search`)

The query is sanitized then matched against the chunks_fts FTS5 table with bm25() ranking. FTS5 returns negative scores (more negative = better), which the code negates so higher = better. matched_terms is computed by checking which query terms actually appear in each result's content.

FTS5 sanitization (_sanitize_fts_query) is required because raw natural-language queries break MATCH:

Strips FTS5 operators and special characters ((){}[]"'\^~:;!@#$%&+=<>|/).
Drops stop words from a built-in _FTS5_STOP_WORDS set (articles, pronouns, prepositions, common interrogatives).
Falls back to the first three raw tokens if every token was a stop word.

The whole BM25 path is also wrapped in a try/except returning [], so a malformed query degrades to vector-only rather than raising.

4. Reciprocal Rank Fusion (`_rrf_merge`)

This is the merge algorithm — not a weighted score sum. Each retriever's results are scored purely by rank:

rrf_score(item) = Σ over retrievers  1 / (k + rank + 1)      (k = 60)

Items appearing in both result sets accumulate both contributions. RRF is rank-based, so it is immune to the two retrievers having incompatible score scales (cosine similarity vs. BM25), which is the failure mode of normalized weighted-sum fusion.

Note: The old weights=(0.7, 0.3) weighted-sum scheme is gone. search() still accepts a weights argument for backward compatibility, but it is ignored — the docstring marks it deprecated. Do not tune it; it does nothing.

5. Recency decay (`_apply_recency_decay`)

Each chunk carries a content_date. Its fused score is multiplied by an exponential decay keyed to its source type's half-life:

score *= max(DECAY_FLOOR, 2^(-age_days / half_life))

DECAY_FLOOR = 0.25 — nothing decays below 25% of its score, so old-but-relevant identity content is never erased, only deprioritized. Half-lives (SOURCE_HALF_LIVES) encode "how fast does this kind of content go stale":

Half-life	Source types	Rationale
365 d	`observation`, `story`, `quote`, `interview*`, `entity_file`, `codebase_doc`	Identity-level facts age slowly
180 d	`rwgps_memory`	Work context, slow drift
120 d	`transcript`	Interview material
90 d	`email_catalog`
60 d	`meeting`, `board_summary`, `voice`, `experiment`, `event`, `journal` (and the `DEFAULT_HALF_LIFE` fallback)
35 d	`slack`, `telegram`, `signal`	Conversational, ages fast
30 d	`podcast_digest`, `checkin`
14 d	`garmin`, `cgm`, `scratch`	Transient health/working data

content_date is set per chunk by the extractor where possible; the schema.py migration _ensure_content_date_column backfills it for existing rows (parsing message-window dates, chunk-ID dates, or falling back to indexed_at).

6. Entity injection (`_inject_entity_results`)

If the query names a known person, that person's record from profile/relationships.json is synthesized into a SearchResult and inserted. The matcher checks the longest known name keys first (so "first last" beats a bare first name), formats the person's relationship/context/notes/interests/milestones into searchable text (_person_to_content), and gives the synthetic row a score near the median of existing results so it surfaces without leapfrogging strong lexical matches. One injection per query, skipped if that profile is already present. The entity table is loaded once and cached.

7. Source-type filter

If the caller passed source_types=[...], results are filtered to those types after fusion and decay. This is a post-filter, so a narrow filter still benefits from the full candidate pool.

8. Diversity rerank (`_diversity_rerank`)

A final MMR-style pass prevents one source type (e.g. slack, which dominates by volume) from flooding the head of the results:

Diversifies the top diversity_target = 7 results.
Builds a candidate pool, capping how many of each source type can enter it.
Greedily selects by mmr = λ·relevance - (1-λ)·max_similarity, with λ = 0.7. Similarity between two candidates is 1.0 if same source file, 0.5 if same source type, 0.0 otherwise.
Hard cap: at most 3 results per source type (max_per_source_type = 3) in the diverse head.

The diversified head is returned first, then the remaining pool, then the tail — and the whole thing is truncated to limit.

Result format

@dataclass
class SearchResult:
    id: str                          # chunk ID (or "profile:..." for injected entities)
    content: str                     # the text
    source_file: str                 # path the chunk came from
    source_type: str                 # 'observation', 'garmin', 'slack', ...
    score: float                     # fused + decayed relevance score
    matched_terms: list[str] | None  # BM25 terms found in content
    content_date: str | None         # ISO date used for recency decay

Illustrative result (placeholder content):

SearchResult(
    id='obs_20260115_abc123',
    content='Gets energy from showing projects to an engaged audience...',
    source_file='/home/<user>/work/vita/data/identity/observations.jsonl',
    source_type='observation',
    score=0.0163,
    matched_terms=['energy', 'projects'],
    content_date='2026-01-15',
)

Note: score is an RRF-derived value (on the order of 1/k), then decayed — it is not a 0–1 similarity. Use it for ordering, not as a calibrated confidence.

Usage

from scripts.search import search

# Search everything
for r in search('skateboarding', limit=10):
    print(f'{r.source_type}: {r.content[:100]}...')

# Filter to specific source types
results = search('parenting philosophy', source_types=['observation', 'story'])

# A person query auto-injects their profile from relationships.json
results = search('lunch with <name>', limit=5)

Parameter	Type	Default	Description
`query`	`str`	required	Natural-language query text
`limit`	`int`	`10`	Max results returned
`weights`	`tuple[float, float] \\| None`	`None`	Deprecated, ignored — kept for backward compat
`source_types`	`list[str] \\| None`	`None`	Post-filter to these source types

When silicon-zack searches

Search is the recall layer, used constantly and without announcement:

When bio-zack mentions something and the context is unclear.
When making a recommendation that should align with stated values/preferences.
When he references a past event, person, decision, or experience.
During /checkin, /interview, and /reflect to ground questions in history.
Before answering "what do I think about X" / "how do I feel about Y."

The behavioral rule (in CLAUDE.md) is: don't ask permission, don't announce it, just search.

Indexing

Pipeline

Scanner (scanner.py) discovers files via FILE_ROUTES and detects changes against the index_state table.
Extractors (extractors/) parse each file into Chunks — one extractor per source type.
Embedder (embedder.py) batches chunk texts to OpenAI text-embedding-3-small.
Indexer (indexer.py) writes chunks + embeddings; triggers keep the FTS5 and vec0 tables in sync.

Change detection is two-level

The scanner first filters by mtime: a file is a candidate only if it is new or its mtime differs from index_state. Then, for each candidate file, the indexer diffs at the chunk level by content_hash (_index_file_chunks):

Chunks whose id exists with a matching content_hash → skipped, not re-embedded (unchanged).
Chunks that are new or whose hash changed → re-embedded and upserted.
Chunks present in the DB but absent from the fresh extract → deleted (deleted).

This content-hash diff is what makes reindexing a large append-only log (like a Slack channel) cheap: only the new windows get embedded. It replaced an older "delete-all, re-embed-everything" approach that caused both wasteful OpenAI calls and sqlite-vec dead-slot bloat.

from scripts.search import reindex

stats = reindex()
print(stats.files, stats.chunks, stats.unchanged_chunks, stats.deleted_chunks)

reindex() returns IndexStats(files, chunks, skipped, unchanged_chunks, deleted_chunks), where chunks is the number actually embedded and written (not total seen). Embeds are batched (EMBEDDING_BATCH_SIZE = 50 per OpenAI call), texts are truncated to 20,000 chars before embedding (the model's ~8191-token limit), and large files are sub-batched (CHUNK_BATCH_SIZE = 200) with explicit GC to bound peak memory. Progress is committed per file, so a crash mid-reindex loses at most one file's work.

When reindex runs

Trigger	Detail
Hourly messaging sync	After each messaging-sync tick (Slack/Telegram/Signal), the scheduler runs `reindex` as a subprocess — `api/src/services/scheduler_messaging.py::run_search_reindex`, called from the messaging-sync loop in `api/src/scheduler.py` (hourly, 5am–10pm PT). This is the primary path that keeps the conversational corpus fresh.
Nightly `/sleep`	The nightly self-improvement cycle reindexes. See Improve / Sleep.
After new interviews	Interview processing reindexes the new transcript.
Manual	A direct `reindex()` call or `python -m scripts.search.indexer`.

Index maintenance

The sqlite-vec vec0 table stores fixed-size vector BLOBs with a validity bitmap for soft deletes. Dead slots are never reclaimed in place, so chunk updates/deletes accumulate bloat in the shadow table chunks_vec_vector_chunks00. maintenance.py::rebuild_vec_index() drops and recreates chunks_vec from the authoritative chunks.embedding column, then VACUUMs to release reclaimed pages:

python -m scripts.search.maintenance            # rebuild + VACUUM
python -m scripts.search.maintenance --no-vacuum  # rebuild only

It returns a stats dict (size before/after, reclaimed bytes, row counts, timings). The caller must ensure no other writer is active during the rebuild.

Extractors

Each source type has a dedicated extractor (subclass of BaseExtractor) that converts a file into Chunks. The registry is EXTRACTORS in extractors/__init__.py; get_extractor(source_type) returns an instance.

A Chunk carries id, source_file, source_type, content, metadata, content_hash, and content_date. Chunk.compute_hash(content) is a 16-hex-char SHA-256 prefix used for change detection.

Extractor	Source type(s)	Chunking strategy
`ObservationExtractor` / `InterviewExtractor`	`observation` / `interview`	One chunk per JSONL record
`MarkdownTranscriptExtractor`	`transcript`	Paragraph/section chunks from markdown
`GarminExtractor`	`garmin`	One human-readable daily summary chunk
`CGMExtractor`	`cgm`	One daily glucose summary chunk
`VoiceExtractor`	`voice`	Segments from voice transcripts
`QuoteExtractor` / `StoryExtractor`	`quote` / `story`	One chunk per item with context
`InterviewMetaExtractor`	`interview_meta`	Session metadata
`ExperimentExtractor` / `CheckinExtractor` / `JournalExtractor`	`experiment` / `checkin` / `journal`	One chunk per record
`ScratchExtractor`	`scratch`	Working-note chunks
`BoardSummaryExtractor`	`board_summary`	Meeting summaries
`EventExtractor`	`event`	Photo-event entries
`SlackExtractor`	`slack`	Messages grouped into conversation windows (~10 messages each)
`TelegramExtractor`	`telegram`	Conversation windows
`SignalExtractor`	`signal`	Conversation windows
`MeetingExtractor`	`meeting`	Transcript segments
`PodcastDigestExtractor`	`podcast_digest`	One chunk per digest
`EmailCatalogExtractor`	`email_catalog`	One chunk per catalog entry
`RwgpsMemoryExtractor`	`rwgps_memory`, `codebase_doc`	Markdown doc chunks (`codebase_doc` is a dynamic subclass)
`EntityFileExtractor`	`entity_file`	Memory entity markdown

Note (health/PII): the garmin and cgm extractors fold daily health metrics into searchable summaries. The mechanism is generic — a daily file becomes one human-readable chunk so silicon-zack can recall a given day. The contents are private; this page documents the engine, not the readings.

Database schema

One SQLite file, data/search/vita.db (FTS5 + WAL, synchronous=NORMAL). Three tables plus a vec0 virtual table, kept consistent by triggers.

-- Authoritative chunk store (embedding packed as 1536 float32s)
CREATE TABLE chunks (
    id TEXT PRIMARY KEY,
    source_file TEXT NOT NULL,
    source_type TEXT NOT NULL,
    content TEXT NOT NULL,
    metadata TEXT,
    embedding BLOB,
    indexed_at TEXT NOT NULL,
    content_hash TEXT NOT NULL,
    content_date TEXT          -- added by migration; drives recency decay
);

-- BM25 keyword index (external-content FTS5 over chunks)
CREATE VIRTUAL TABLE chunks_fts USING fts5(
    content, source_file, source_type,
    content='chunks', content_rowid='rowid'
);

-- KNN vector index (sqlite-vec)
CREATE VIRTUAL TABLE chunks_vec USING vec0(
    embedding float[1536] distance_metric=cosine
);

-- Incremental-index bookkeeping (mtime-level change detection)
CREATE TABLE index_state (
    source_file TEXT PRIMARY KEY,
    mtime REAL NOT NULL,
    indexed_at TEXT NOT NULL
);

Triggers keep both derived indexes in lockstep with chunks:

chunks_ai / chunks_ad / chunks_au mirror INSERT/DELETE/UPDATE into chunks_fts.
chunks_vec_ai / chunks_vec_ad / chunks_vec_au / chunks_vec_au2 mirror them into chunks_vec (insert only fires when embedding IS NOT NULL).

schema.py is self-migrating: ensure_vec_index() lazily creates the vec0 table and triggers, backfills chunks_vec from existing embeddings if empty, and adds/backfills the content_date column — all idempotent and a fast no-op after the first call.

CLI

# List everything the router would index
python scripts/search/scanner.py --discover

# Show only files that currently need (re)indexing
python scripts/search/scanner.py

# Reindex changed files
python -m scripts.search.indexer

# Rebuild the vec0 index + VACUUM (reclaim soft-delete bloat)
python -m scripts.search.maintenance [--no-vacuum]

A small adjacent helper, scripts/slack_saved.py, treats Slack "save for later" as a lightweight todo/respond queue. It exposes list_saved(), count_saved(), and clear_saved(key); the queue surfaces into /now and the comms radar and is nudged proactively when it backs up. It is independent of the search index but lives in the same recall-tooling neighborhood.

File locations

Path	Purpose
`scripts/search/searcher.py`	`search()`, RRF, recency decay, entity injection, diversity rerank
`scripts/search/indexer.py`	`reindex()`, content-hash chunk diffing, batched writes
`scripts/search/scanner.py`	`FILE_ROUTES`, change detection, discovery CLI
`scripts/search/embedder.py`	Batched OpenAI embedding calls with retry + truncation
`scripts/search/schema.py`	Schema, triggers, vec0 + `content_date` migrations
`scripts/search/config.py`	DB path, model, dimensions, batch size
`scripts/search/maintenance.py`	`rebuild_vec_index()` + VACUUM
`scripts/search/extractors/`	Per-source-type chunk extractors + registry
`data/search/vita.db`	The index (chunks, FTS5, vec0, index_state)
`secrets/openai.json`	OpenAI API key for embeddings
`docs/references/search-reference.md`	Internal quick reference

Where to go next

Memory & Trends — the broader memory architecture this index serves.
Document Filing — how raw inputs become indexable files.
Improve / Sleep — the nightly cycle that reindexes and consolidates.
Model Guidance — model routing (embedding model choice lives here, in config.py, not the router).