Skip to content

Proactive Research

The research system runs autonomous background research on topics derived from goals, projects, and heartbeat topics. It maintains a prioritized topic queue, executes LLM-powered analysis, stores findings in daily JSONL files, and synthesizes accumulated findings into coherent summaries.

Warning — current state (instrumentation-lying): the research loop runs on schedule (status: success every 3h through 2026-06-21, ~6–8 min/run), the queue keeps growing (now 5,782 topics, 3.3 MB), and last_researched_at keeps updating — yet no new findings JSONL has been written since 2026-04-05 and no synthesis since 2026-04-04. The loop is green while persisting zero findings: either the LLM analysis returns empty findings (which research_topic() treats as a no-op, see Research Execution) or the writes are silently failing. A reimplementer should treat the storage write path described below as the intended behavior, not the live one, and add an alert that ties "loop ran" to "findings written." See Sentinel for the watchdog pattern this gap should be wired into.

Why Proactive Research?

Silicon-Zack needs to stay informed on topics that matter to bio-Zack without waiting for explicit requests. The research system:

  1. Extracts topics automatically from active goals, projects, and heartbeat topics
  2. Prioritizes intelligently using age decay, source weighting, and research fatigue
  3. Avoids repetition by feeding previous findings back into prompts
  4. Synthesizes over time - individual findings accumulate into coherent summaries
  5. Surfaces high-priority findings via Telegram when they warrant interruption

System Architecture

TOPIC EXTRACTION
+------------------+     +------------------+     +------------------+
|   Heartbeat      |     |   Active Goals   |     |   Projects       |
|   Tag Mapping    |     |   LLM Extraction |     |   LLM Extraction |
+--------+---------+     +--------+---------+     +--------+---------+
         |                        |                        |
         +------------------------+------------------------+
                                  |
                                  v
                  +-------------------------------+
                  |        TOPIC QUEUE             |
                  |  data/research/queue.json      |
                  |  - Deduplication by query      |
                  |  - Priority scoring            |
                  |  - Auto-retirement             |
                  +---------------+---------------+
                                  |
                                  v
                  +-------------------------------+
                  |      RESEARCH EXECUTION        |
                  |  - Top 5 due topics per loop   |
                  |  - LLM analysis (deepseek-v4-flash) |
                  |  - Novelty checking            |
                  +---------------+---------------+
                                  |
                     +------------+------------+
                     |                         |
              All findings              2+ findings exist
                     |                         |
                     v                         v
              +-----------+         +-------------------+
              |  STORE    |         |    SYNTHESIZE     |
              |  findings/|         |  syntheses/       |
              |  YYYY.jsonl|        |  {topic_id}.json  |
              +-----------+         +-------------------+
                                           |
                                    (if notify + score >= 80)
                                           |
                                           v
                                  +-------------------+
                                  |  TELEGRAM ALERT   |
                                  +-------------------+

Topic Queue

Queue File

Location: data/research/queue.json

{
  "topics": [
    {
      "id": "f8e57d42-...",
      "query": "VO2 max training protocol improvement strategies",
      "source": "goal",
      "source_id": "maintain-cardiovascular-fitness-001",
      "priority": 0.7,
      "created_at": "2026-02-13T05:20:42Z",
      "last_researched_at": "2026-02-16T03:52:13Z",
      "research_count": 3,
      "status": "active",
      "context": { "goal_title": "Improve VO2 Max" },
      "findings_count": 3,
      "tags": ["fitness", "endurance", "training"]
    }
  ]
}

Topic Sources

Location: scripts/research/topic_sources.py

Topics are extracted from three sources during each queue refresh:

Source Extractor Priority Range Method
Heartbeat extract_from_heartbeat() 0.4 Tag-based mapping via RESEARCHABLE_TAG_MAP
Goals extract_from_goals() 0.3-0.8 LLM generates research queries from goal descriptions
Projects extract_from_projects() 0.3-0.6 LLM generates competitive intelligence queries

Heartbeat tag mapping is deterministic - specific tags map to predefined research queries:

RESEARCHABLE_TAG_MAP = {
    "rwgps-game": ["location-based cycling games", "gamification fitness apps retention"],
    "ceo-mode": ["CEO transition founder playbooks", "engineering manager to CEO frameworks"],
    "health": ["endurance athlete recovery optimization"],
    # ...
}

Goal and project extraction uses LLM inference (research_extract task type) to generate 3-8 research queries from active goal descriptions or project registries. Falls back to simple title-based queries if the LLM call fails.

Biographical topics (type project_interview) are filtered out. Only active status heartbeat topics are considered.

Topic Statuses

Status Meaning
pending New topic, never researched
active Has been researched at least once
completed Manually marked as done
retired Auto-retired or manually retired

Priority Scoring

Effective priority determines research order. Calculated as:

effective = base_priority * age_decay * freshness * research_decay * source_multiplier

Factors:

Factor Effect
Age decay Linear decay to 0.1 over 30 days
Freshness boost 1.2x for topics < 3 days old, 1.0x for 3-7 days, 0.8x after
Research fatigue Each research reduces priority by 20% (floor 0.1)
Source multiplier conversation: 1.3, goal: 1.2, media_followup: 1.0, project: 0.9, heartbeat: 0.7

Auto-Retirement

Topics are automatically retired when: - Researched 5+ times with 0 findings (nothing to find) - Older than 30 days with 3+ research attempts (stale)

Retirement only flips a topic's status to retired; retired topics are filtered out of get_due_topics() but are not deleted from queue.json. There is no topic-count cap, so the queue file grows without bound — in the live system it has reached 5,782 topics / 3.3 MB. Since get_due_topics() loads and scans the entire file every loop (and add_topic() does a full linear dedup scan on each insert), this is an O(n) cost that grows with queue size; a reimplementer should consider periodic pruning of retired topics or an indexed store.

Deduplication

add_topic() deduplicates by case-insensitive query matching. If a matching non-retired topic exists, it updates the priority (if new is higher) and merges tags.

Minimum Interval

A topic cannot be researched more than once per 12 hours.

Research Execution

Location: scripts/research/researcher.py

Note: research_topic() returns early (no finding stored) whenever the parsed LLM response has an empty findings list or fails to JSON-parse. This is the most likely cause of the silent-success state noted at the top of this page — the loop completes and reports success while append_finding() is never reached.

The Research Loop

research_loop() is the main entry point, called by the scheduler:

  1. Refresh queue - Run all topic extractors, deduplicate, update queue
  2. Get due topics - Top 5 highest-priority topics that pass filters
  3. Research each - LLM analysis with novelty checking
  4. Update metadata - Increment research_count, update last_researched_at, transition pending to active
  5. Synthesize - For topics with 2+ findings, generate/update synthesis

Research Prompt

Each topic is analyzed using a structured prompt that includes: - The research query and context - Previous findings (up to 15 items from last 30 days) for novelty avoidance - Instructions for structured JSON output

The LLM returns: 

{
  "findings": ["Key finding 1", "Key finding 2"],
  "actionable_insights": ["Recommendation 1"],
  "relevance_score": 85,
  "confidence": "medium",
  "notify": false,
  "notify_reason": "",
  "source_title": "Analysis: Topic Name"
}

High-Priority Surfacing

Findings with notify: true AND relevance_score >= 80 are sent to Telegram via the proactive outreach queue (researcher.py:_surface_finding):

enqueue_message(
    trigger="research_finding",
    message=formatted_finding,        # query + first finding + notify_reason + score
    send_at=datetime.now(),
    dedup_key=f"research-{finding.id}",
    topic_id="0",                     # routes to the default Telegram topic
)

See Proactive Outreach for how enqueue_message and dedup keys work.

Rate Limiting

RESEARCH_CONFIG = {
    "max_topics_per_loop": 5,
    "max_findings_per_topic": 10,
    "max_tokens_per_topic": 50_000,
    "loop_interval_hours": 3,
    "active_hours": (8, 22),        # 8am-10pm PT
    "max_notifications_daily": 5,
    "max_research_cost_daily_usd": 0.50,
}

Inference Routing

Research uses cheap, high-volume inference rather than a frontier model. All three research task types route to deepseek/deepseek-v4-flash in .aloop/config.json (current as of this writing — for the live routing map and rationale see Model Guidance, which is auto-maintained):

Task Type Purpose
research_extract Extract research queries from goals/projects
research_analyze Analyze a topic and produce findings
research_synthesize Synthesize accumulated findings

Storage

Location: scripts/research/storage.py

Findings

Daily JSONL files at data/research/findings/YYYY-MM-DD.jsonl.

Each line is a ResearchFinding:

{
  "id": "3005c4f6-...",
  "topic_id": "eb2a780f-...",
  "query": "founder to CEO transition delegation frameworks",
  "source_url": "llm-knowledge",
  "source_title": "Founder-to-CEO Transition: Leadership Frameworks",
  "findings": ["Finding 1", "Finding 2", "Finding 3"],
  "actionable_insights": ["Action 1", "Action 2"],
  "relevance_score": 85,
  "confidence": "medium",
  "notify": false,
  "notify_reason": "",
  "model": "deepseek/deepseek-v4-flash",
  "researched_at": "2026-02-24T02:59:36Z",
  "tokens_in": 10844,
  "tokens_out": 386,
  "cost_usd": 0.0037
}

source_url is currently always "llm-knowledge" since the system uses LLM training knowledge. Web search integration is planned for the future.

Syntheses

Per-topic JSON files at data/research/syntheses/{topic_id}.json.

{
  "topic_id": "799fab46-...",
  "query": "founder to CEO transition delegation frameworks time allocation",
  "synthesis": "Multi-paragraph summary of all findings...",
  "key_findings": [
    "Hiring executives in first 30 days is highest-leverage activity",
    "Delegation failure stems from expertise trap, not trust",
    "Quarterly OKRs with 'what I will stop doing' formalize role transition"
  ],
  "next_steps": [
    "Research hiring frameworks for first-30-day executive recruitment",
    "Investigate time audit methodologies for CEO transition"
  ],
  "last_updated": "2026-02-24T17:40:33Z",
  "total_research_count": 2,
  "total_findings_count": 8,
  "total_cost_usd": 0.0052
}

Syntheses are regenerated whenever a topic has 2+ findings and is researched again. The synthesis prompt includes up to 50 finding items from the last 90 days.

Index File

data/research/index.json contains metadata for manually curated research artifacts (papers, ideas, experiments) separate from the automated research pipeline.

Data Layout

data/research/
├── queue.json                          # Topic queue (all topics with metadata)
├── index.json                          # Manual research index (papers, ideas)
├── papers/                             # Manually curated research papers
│   └── 2512.05765-missing-layer-agi.md
├── findings/                           # Daily JSONL finding files
│   ├── 2026-02-13.jsonl                 # first file
│   ├── 2026-02-14.jsonl
│   └── ...                              # 52 files, 2026-02-13 .. 2026-04-05 (none since)
└── syntheses/                          # Per-topic synthesis JSON
    ├── 799fab46-...-8af90ee8ec23.json
    ├── d9a8ae0b-...-a5d3eaf78c66.json
    └── ...                              # 676 files (newest written 2026-04-04)

Note: the findings/ and syntheses/ directories stopped growing in early April 2026 even though the loop still runs — see the warning at the top of this page. The counts above are a snapshot; in a healthy system they would grow daily.

Scheduling

The research loop runs every 3 hours during active hours (8am-10pm PT), managed by VitaScheduler._research_loop().

Setting Value
Interval 3 hours
Active hours 8am-10pm PT
Initial delay 180 seconds
Timeout 600 seconds (10 min)
Execution tracking track_execution("research")

VitaScheduler._run_research() delegates to run_research_cycle() (imported as run_research_cycle_inference) in api/src/services/scheduler_inference_jobs.py. That helper is where the subprocess is actually spawned, with PYTHONPATH set to the scripts/ directory:

# api/src/services/scheduler_inference_jobs.py
async def run_research_cycle(*, project_root, env):
    async with track_execution("research"):
        proc = await asyncio.create_subprocess_exec(
            "python", "-c",
            "from research import research_loop; import asyncio; asyncio.run(research_loop())",
            cwd=str(project_root),
            env=_with_scheduler_pythonpath(env, project_root),
            ...
        )
        stdout, stderr = await asyncio.wait_for(proc.communicate(), timeout=600)

Note: research is one of a cluster of inference jobs that share this subprocess-spawn-plus-track_execution pattern in scheduler_inference_jobs.py — alongside run_speculative_computation(), run_deep_analysis_cycle(), and run_daily_synthesis() (the media deep-analysis and synthesis jobs). If you reimplement the scheduler side, treat these as one family: each spins up a short-lived python -c "from <module> import …" subprocess, wraps it in track_execution(<name>), and returns (ok, output_or_error).

Manual Trigger

curl -X POST http://localhost:33800/api/v1/scheduler/trigger/research

The endpoint is POST /api/v1/scheduler/trigger/{task_name} (router prefix /api/v1/scheduler); task_name=research maps to scheduler.run_research_now(), which awaits _run_research() immediately.

Python API

Add a topic manually

from scripts.research import add_topic

topic = add_topic(
    query="zone 2 training heart rate drift",
    source="conversation",
    source_id="session-2026-02-24",
    priority=0.7,
    context={"reason": "bio-zack mentioned HR drift on rides"},
    tags=["cycling", "training"],
)

Get recent findings

from scripts.research import get_findings

# Last 24 hours (default)
findings = get_findings()

# Last 72 hours
findings = get_findings(hours=72)

Run the full loop

from scripts.research import research_loop
import asyncio

stats = asyncio.run(research_loop())
# stats = {
#   "topics_researched": 3,
#   "findings_count": 3,
#   "syntheses_updated": 2,
#   "cost_usd": 0.012,
#   "queue_refresh": {"heartbeat_extracted": 6, "goals_extracted": 5, ...}
# }

Queue management

from scripts.research.queue import load_queue, get_due_topics, retire_topic

# List all topics
topics = load_queue()
for t in topics:
    print(f"{t.status}: {t.query} (researched {t.research_count}x)")

# Get next batch
due = get_due_topics(max_count=3)

# Retire a topic
retire_topic("topic-uuid-here")

Synthesis

from scripts.research.researcher import synthesize_topic
import asyncio

synthesis = asyncio.run(synthesize_topic("topic-uuid-here"))
if synthesis:
    print(synthesis.synthesis)
    print(synthesis.key_findings)

Data Types

ResearchTopic

Field Type Description
id str UUID
query str The research query
source str heartbeat, goal, project, conversation, media_followup
source_id str ID of the originating item
priority float Base priority 0.0-1.0
created_at str ISO timestamp
last_researched_at str or None Last research timestamp
research_count int Times researched
status str pending, active, completed, retired
context dict Source-specific context for the LLM
findings_count int Total findings produced
tags list[str] Grouping tags

ResearchFinding

Field Type Description
id str UUID
topic_id str Parent topic UUID
query str Research query
source_url str Currently "llm-knowledge"
source_title str LLM-generated title
findings list[str] 2-4 key findings
actionable_insights list[str] 1-2 recommendations
relevance_score int 0-100
confidence str high, medium, low
notify bool Whether to alert via Telegram
notify_reason str Why notification warranted
model str Model used
researched_at str ISO timestamp
tokens_in int Input tokens
tokens_out int Output tokens
cost_usd float Inference cost

ResearchSynthesis

Field Type Description
topic_id str Parent topic UUID
query str Research query
synthesis str 2-3 paragraph summary
key_findings list[str] Top 3-5 findings
next_steps list[str] 1-3 follow-up suggestions
last_updated str ISO timestamp
total_research_count int Total research sessions
total_findings_count int Total findings across all sessions
total_cost_usd float Cumulative inference cost

File Locations

File Purpose
scripts/research/__init__.py Public API: research_loop, add_topic, get_findings
scripts/research/types.py Data classes (ResearchTopic, ResearchFinding, ResearchSynthesis)
scripts/research/queue.py Queue management, priority scoring, auto-retirement
scripts/research/storage.py JSONL/JSON persistence for findings and syntheses
scripts/research/researcher.py Core execution: research_topic, synthesize_topic, research_loop
scripts/research/topic_sources.py Topic extraction from heartbeat, goals, projects
api/src/scheduler.py Scheduler loop: _research_loop (timing/window), _run_research (delegates to the inference-jobs helper), run_research_now
api/src/services/scheduler_inference_jobs.py run_research_cycle() — spawns the actual research subprocess (and sibling speculative/deep-analysis/synthesis jobs)
api/src/routers/scheduler.py Manual trigger endpoint (/api/v1/scheduler/trigger/{task_name})
.aloop/config.json Inference routing for research_extract / research_analyze / research_synthesize task types
data/research/queue.json Topic queue
data/research/findings/*.jsonl Daily findings
data/research/syntheses/*.json Per-topic syntheses
data/research/index.json Manual research index

Known Limitations

  1. Silent success / no findings written - The loop reports status: success every cycle but has written no findings since 2026-04-05 and no synthesis since 2026-04-04 (see the warning at the top of this page). Nothing ties "loop ran" to "findings persisted," so the failure is invisible to the execution log. This is the highest-priority limitation for a reimplementer to fix — wire a watchdog on findings-write recency, not just loop success.
  2. LLM-only sources - Uses LLM training knowledge only (source_url is hardcoded to "llm-knowledge"). No web search or live data fetching. The codebase notes web search integration as a future addition.
  3. No API endpoints for browsing - Research data has no dedicated REST endpoints. Access is through Python imports or file inspection.
  4. Cost tracking is per-finding - No aggregated daily budget enforcement like the media system has. The max_research_cost_daily_usd config exists but is read by nothing in the loop, so it is not enforced.
  5. Unbounded queue growth - Retired topics stay in queue.json; there is no topic-count cap or pruning. The file is scanned in full on every loop and every add_topic().
  6. Findings accumulate indefinitely - No rotation or archival of old findings files.
  7. Synthesis overwrites - Each synthesis run replaces the previous one rather than appending history.

Where to go next

  • Sentinel — the watchdog pattern this loop's silent-success gap should be wired into
  • Model Guidance — live inference routing for the research_* task types
  • Proactive Outreach — how high-priority findings reach Telegram
  • Media — sibling inference jobs (deep-analysis, daily synthesis) that share the same subprocess pattern
  • Heartbeat / Project Registry — sources the topic queue draws from