Model Benchmarks

The benchmark suite measures language models against the work vita actually does — not against public leaderboards — across every harness vita runs. It answers two questions per use-case: which model is best? and what is the cheapest model that is still good enough? Definitions live in git as diffable YAML; results live in an append-only SQLite store; a pinned LLM jury does the grading; a read-only API serves the leaderboard, case matrix, inspector, and calibration views.

This is a separate subsystem from the routing recommendations in Model Guidance (which is auto-maintained by a flow) and from raw token accounting in Token Analytics. The benchmark suite produces the evidence; model-guidance is where the resulting routing decisions get written down.

Note: This is a young subsystem (initial cut June 2026). It runs via a CLI and a cross-repo ingester; there is no UI route or scheduled-run wiring yet. The API and store are live.

Why it exists

vita routes work to many models across many harnesses. Picking models by vibes or by public benchmarks is a guess — public benchmarks don't measure vita's summarization, vita's agentic tool-use, or the specific quality bar a given surface needs. The suite turns "is model X good enough for use-case Y?" into a falsifiable, reproducible measurement:

Offload — find the cheapest/fastest model that still clears the bar for a use-case currently served by a more expensive model.
Unlock — track capabilities that fail today (the idea stubs) so a new model release can be tested against the exact bar that's blocking them.
Calibration — keep the automated jury honest by comparing its scores to bio-zack's own ground-truth ratings.

Architecture at a glance

  benchmarks/*.yaml          (definitions in git — diffable, reviewable)
        │  models / judge panels / use-case roster
        ▼
  CLI / ingester  ──>  Runner ──> Attempt{final, trace, telemetry}
   (run.py)                            │
                                       ▼
                              Grader ──> [Score]   (jury votes + gates)
                                       │
                                       ▼
              append-only SQLite store + content-addressed artifacts
                     (data/benchmarks/benchmarks.db)
                                       │
                                       ▼
                  views.py  (leaderboard / case-matrix / inspector / calibration)
                                       │
                                       ▼
                  api/src/routers/benchmarks.py  (read-only, port 33800)

The system is a generic spine plus one plugin per use-case. The spine — store, registries, eligibility, aggregation views — is built once. A use-case is just a registered triplet implementing three interfaces.

The plugin contract

Defined in scripts/benchmarks/contract.py. A use-case plugs into the spine through exactly three interfaces:

Case ──> Runner(model) ──> Attempt{final + trace} ──> Grader ──> [Score]

Interface	Role
`CaseSource`	Produces the use-case's graded `Case`s (floor, ceiling, etc.).
`Runner`	Drives a specific harness; runs one case against one model, returns an `Attempt`.
`Grader`	Turns `(case, attempt)` into one-or-more normalized `Score`s.

The Attempt is a trace-aware envelope, not a bare string — Attempt.final is the output text and Attempt.trace is a list of Steps (thought | tool_call | tool_result | message). Single-turn use-cases leave the trace empty; agentic/tool-use use-cases populate it, so adding agentic benchmarks needs no new top-level axis.

Domain types

Type	Key fields
`Case`	`id`, `version`, `use_case_id`, `kind` (`floor\\|ceiling\\|frontier\\|stub`), `input` (plugin payload), `reference` (gold output / key facts), `threshold` (ship/unlock bar), `metadata`. `input_hash()` content-hashes the resolved input so "the input changed" is falsifiable.
`Attempt`	`final`, `trace`, `telemetry`, `status` (`completed\\|failed\\|timeout\\|tool_error`), `error`, `runner_provenance`, `raw`.
`Telemetry`	`cost_usd`, `latency_ms`, `ttft_ms`, `input_tokens`, `output_tokens`, `cache_tokens`, `tokens_per_sec`. Normalized across providers/harnesses.
`Score`	`metric`, `score_type` (`absolute\\|boolean\\|ordinal\\|label`), `value`, `value_label`, `jury_votes`, `inter_judge_disagreement`, `confidence`, `critical_failure`, `gate_meta`, `reasoning`, `grader_version`. One run yields one `Score` per metric.

Definitions in git

Three YAML files under benchmarks/ are the human-edited, diffable definitions. They are loaded by scripts/benchmarks/specs.py. Editing them is a reviewable git change; a panel change bumps a version, which makes a new benchmark_version and flags older results stale (they are never deleted).

File	Contents
`benchmarks/models.yaml`	Model registry — display/metadata tags only (`family`, `weights` open/proprietary, `tier` flagship/mid/small). Used for labels, coloring, filtering. Real cost comes from runs, not from here.
`benchmarks/judge_panels.yaml`	Pinned LLM jury panels. The `default` panel carries a `version` (e.g. `jury@v1`) that rides into every `Score`'s `judge_panel_version`.
`benchmarks/use_cases.yaml`	The use-case roster: each entry is `idea` (no harness yet — the wishlist / unlock tracker) or `runnable` (a plugin exists and produces scores).

Tip: For an idea stub, the ceiling field describes the bar that stops the capability today and unlock the decision it would unblock. These become the ceiling cases when the stub goes runnable.

Use-case lifecycle

# benchmarks/use_cases.yaml (illustrative)
use_cases:
  - id: summarization
    project: vita
    status: runnable                  # plugin exists, produces scores
    modality: { input: text, output: text }
    plugin: benchmarks.plugins.summarization
    deployed: { model: "<provider/model>", harness: aloop, source: "<config path>" }

  - id: example_idea
    project: vita
    status: idea                      # wishlist — fails today, no harness yet
    modality: { input: text, output: audio }
    unlock: "What shipping this would unblock."
    ceiling: "The bar that stops us today."

The deployed block names the incumbent each use-case currently runs in production — the thing to beat or offload from. The leaderboard view uses it to compute delta-vs-deployed and to flag whether the deployed model has even been benchmarked.

How a run works

The orchestrator scripts/benchmarks/run.py runs, for a (benchmark_version, use-case): fleet × cases × samples → Runner → Grader → append-only store. Before any results are written it freezes a BenchmarkSpec snapshot (including the judge panel) so results stay reconstructable even if the definitions later change.

Two run depths trade cost for precision:

Depth	Cases	Samples	Use
`limited`	floor cases only (or a subset)	1	Cheap screen. Low-precision scores, rendered with an asterisk. For models you probably won't run.
`thorough`	full case set	N (default 2)	Decision-grade precision.

# Run the summarization benchmark over two models, cheap screen:
python -m benchmarks.run \
    --use-case summarization \
    --models <provider/model-a>,<provider/model-b> \
    --depth limited

# Decision-grade run, override sample count:
python -m benchmarks.run --use-case summarization --depth thorough --samples 3

Note: Run all benchmark commands from the vita repo root (PYTHONPATH includes scripts/, so the import is benchmarks.run, not scripts.benchmarks.run).

For each (model, case, sample) the orchestrator:

Calls Runner.run(case, model_id) → Attempt.
Stores the output text and trace as content-addressed artifacts (returns sha256 refs).
Inserts one runs row with telemetry (cost, latency, tokens) and provenance.
Calls Grader.grade(case, attempt) → list of Scores.
Inserts one scores row per metric.

The CLI prints a leaderboard ranked by the headline metric mean, with pass_rate (share of cases clearing their threshold), total cost, and p50 latency. A trailing * marks limited-depth (low-precision) runs.

The Runner: benchmarking what prod runs

The default AloopRunner (scripts/benchmarks/runners/aloop_runner.py) drives vita's own aloop harness in-process via inference.factory.complete. Benchmarking through aloop is benchmarking what vita runs in production. It consumes case.input keys prompt (required), system_prompt, max_tokens, temperature, and records cost/latency/tokens into Telemetry. Single-turn runs leave the trajectory empty; agentic harnesses get their own Runner that populates Attempt.trace. Cross-codebase harnesses shell out to a foreign repo's harness via a JSON contract.

The pinned jury (grading)

scripts/benchmarks/judge.py implements the grader's panel of LLM judges. The rationale (research-backed): a panel of diverse, disjoint-family judges correlates better with human judgment than a single frontier judge, has less self-preference bias, and is cheaper. The decisive reason for pinning the panel is consistency — one panel judging every model makes scores apples-to-apples and turns residual family bias into a constant that cancels in relative comparisons.

Each juror is sent a blinded grading prompt (jurors never learn which model produced the candidate), asked to return a strict JSON object with numeric fields (0..1), boolean gate fields, and a one-sentence rationale.
The jury aggregates: mean for numeric fields, majority vote for booleans, population stdev as inter_judge_disagreement, and a confidence derived from mean disagreement.
Jurors run at temperature=0 and request a JSON response format; malformed output is salvaged by a JSON-extraction parser.
The panel is loaded from benchmarks/judge_panels.yaml (falling back to a built-in default), and the chosen panel + version is frozen into the spec snapshot. The grader's headline metric is a weighted composite of its rubric dimensions, and a critical-failure gate can cap the composite when a disqualifying condition trips (e.g. an unsupported/fabricated claim in a summary). The gate is recorded as critical_failure plus gate_meta on the score.

Note: The jury is the automated anchor. The ground-truth anchor is bio-zack's own ratings (see Calibration), recorded via POST /rate and compared against the jury per use-case.

Where state lives

The store (scripts/benchmarks/db.py) is a standalone SQLite database with three hard invariants:

Append-only. Nothing is ever UPDATEd. A re-run is a new row; "current" is computed as a view (latest by version/depth). Old-benchmark_version rows are kept and flagged stale by the read layer, never deleted.
Isolated from prod data — its own DB file under data/benchmarks/.
Artifacts by reference. Inputs/outputs/traces/judge-reasoning/stdout are content-addressed files at data/benchmarks/artifacts/<sha[:2]>/<sha>; the DB stores only sha256 refs. Storing a blob is idempotent (write-if-absent).

Path	What
`data/benchmarks/benchmarks.db`	The results store (overridable via `VITA_BENCH_DB`).
`data/benchmarks/artifacts/`	Content-addressed blobs (overridable via `VITA_BENCH_ARTIFACTS`).

Schema (core tables)

Table	Purpose
`spec_snapshots`	Frozen `BenchmarkSpec` per `benchmark_version` (PK). Immutable — first write wins (`INSERT OR IGNORE`). Carries `spec_hash`, full `spec_json`, `home_repo`, human-readable `methodology`.
`runs`	One row per execution (append-only). Model, harness, case, depth, status, telemetry, artifact refs, provenance JSON.
`scores`	≥1 row per run (multi-metric). Metric, score_type, value, jury votes, disagreement, confidence, `critical_failure`, `gate_meta`. FK → `runs`.
`owner_ratings`	bio-zack's ground-truth ratings of a run's output (`0..1`, label `ship\\|edit\\|reject`). Append-only; latest per run wins (a view).
`deployments`	Appended deployment history — "what's running where, over time."
`calibration_results`	Judge-vs-owner agreement report cards.
`volume_snapshots`	`usage.jsonl` rollup per use-case, for offload weighting.
`comparisons`	Pairwise/relative results. Schema-ready; not populated by the MVP.
`artifacts`	Content-addressed blob index (`sha256` PK → path, size, content_type).

Connection conventions mirror the feed store: WAL journaling, foreign keys on, Row factory. Eval timestamps are always written in UTC (unambiguous, never PT). The store is stdlib-only and dependency-light so it has no import cycle with the contract/engine layers.

Read layer & API

scripts/benchmarks/views.py is the aggregation layer — kept separate from the router so it's testable without FastAPI. The store keeps every sample as its own row; "current" is computed here as a view: group by (model, harness, case), average the headline across samples, then across cases for the leaderboard. Stale-benchmark_version rows are flagged (is_current) rather than hidden — comparability is the read layer's job.

The leaderboard view also computes the decision: which model is deployed, the best current model, and the cheapest good-enough model (within a small band of the best headline). It never averages across depths — a decision-grade row prefers thorough.

The router api/src/routers/benchmarks.py is mounted at prefix=/api/v1/benchmarks on the main API (port 33800). It is read-only over the store — writes happen via the orchestrator CLI, not the API — with one exception: the /rate endpoint records bio-zack's ground-truth ratings.

Method	Endpoint	Returns
`GET`	`/api/v1/benchmarks/use-cases`	Use-cases with a spec snapshot, run counts, methodology.
`GET`	`/api/v1/benchmarks/roster`	Full roster incl. `idea` stubs (the "unlock board").
`GET`	`/api/v1/benchmarks/leaderboard?use_case=…`	Per `(model × harness)`: headline, cost, latency, precision, confidence, deployed flag, decision.
`GET`	`/api/v1/benchmarks/case-matrix?use_case=…`	case × model grid of headline values.
`GET`	`/api/v1/benchmarks/inspector?use_case=…&case_id=…`	Per-model candidate output + all metric scores + jury reasoning, for side-by-side comparison.
`GET`	`/api/v1/benchmarks/calibration?use_case=…`	Jury-vs-owner agreement for the current version.
`GET`	`/api/v1/benchmarks/spec/{benchmark_version}`	The frozen spec snapshot + methodology.
`POST`	`/api/v1/benchmarks/rate`	Record an owner rating (`{run_id, rating?, label?, note?}`).

The leaderboard, case-matrix, and inspector accept an optional benchmark_version query parameter to pin to a specific frozen spec; omitting it uses the current version. (/spec/{benchmark_version} takes it as a required path param; the other endpoints don't accept it.) Those same three endpoints accept the use-case as a use_case query parameter.

# Leaderboard for a use-case
curl 'http://localhost:33800/api/v1/benchmarks/leaderboard?use_case=summarization'

# Record a ground-truth rating (label maps: ship=1.0, edit=0.6, reject=0.2)
curl -X POST http://localhost:33800/api/v1/benchmarks/rate \
  -H 'Content-Type: application/json' \
  -d '{"run_id": "<run_id>", "label": "ship", "note": "clean, no fabrication"}'

Calibration (the jury's report card)

The calibration view answers "do I trust the jury?" — over runs bio-zack has rated on the current version, it compares his rating to the jury's headline, returning per-item deltas, a mean absolute delta, and an agreement score (1 − mean_abs_delta). This keeps the automated grader honest against the human anchor.

Use-cases shipped

Use-case	Project	Status	Modality	Notes
`summarization`	vita	runnable	text → text	The reference plugin. Floor + ceiling cases; jury-graded fact-recall, faithfulness, conciseness with a critical-hallucination gate.
`gumball_kids_qa`	external	runnable	text → text	Abstraction-validation plugin #2: same spine, different rubric (accuracy gate + age-appropriateness + curiosity-driving follow-ups).
`cadence_agentic`	external	runnable	text → trajectory	Agentic tasks mirrored from a foreign repo via the ingester (below).
`tts`, `image_generation`, `code`, `telegram_agent`	vita	idea	various	Stubs on the unlock board — capabilities that fail today, each with a `ceiling`/`unlock` describing the bar and the decision it would unblock.

Use-case examples are intentionally generic. The contents of cases (sources, prompts, gold answers) are authored placeholders or harvested from vita's own data; the page documents the engine, not any private content.

A plugin is a package under scripts/benchmarks/plugins/<use_case>/ exporting a CaseSource and a Grader (the Runner is usually shared). Adding a use-case means writing those two classes against the contract and registering the triplet — the spine, store, and views need no changes.

Federated (cross-repo) ingest

Some benchmarks are produced in other codebases. scripts/benchmarks/ingest/cadence.py mirrors a foreign repo's model-benchmark into vita's central store as the cadence_agentic use-case. The pattern is generic:

Read-only against the foreign repo (opens its SQLite store in read-only mode).
Re-runnable: each sync deletes the prior mirror for that source and reinserts (delete_imported_runs(use_case, source)), then refreshes the frozen spec snapshot (home_repo = foreign_repo@commit). Natively-produced rows are never touched — only mirrored rows tagged with the source provenance.
Two fidelities: per-case rows (the agent's real response, tool-call trajectory, per-dimension judge justifications → full case-matrix + inspector) where a completed judge run exists, with an aggregate fallback row so in-progress models still appear on the leaderboard.

python -m benchmarks.ingest.cadence

This is how a benchmark whose spec lives in another repo still shows up on vita's single leaderboard without re-running every model locally.

Relationship to model-guidance and tokens

Subsystem	Role
Model Benchmarks (this page)	Produces measured evidence: per-use-case scores, cost, latency, and the best / cheapest-good-enough decision.
Model Guidance	The auto-maintained routing doc — which model to use where. Benchmark results inform it; it is not restated here.
Token Analytics	Raw aggregate token/cost accounting from CLI session logs — the spend picture, not per-task quality.

This suite is distinct from the older inference /test endpoint; it is the dedicated /benchmarks subsystem.

File Locations

Path	Purpose
`benchmarks/models.yaml`	Model registry (display/metadata tags).
`benchmarks/judge_panels.yaml`	Pinned jury panels (versioned).
`benchmarks/use_cases.yaml`	Use-case roster (`idea` / `runnable`).
`scripts/benchmarks/contract.py`	The 3-interface plugin contract + domain types.
`scripts/benchmarks/run.py`	Orchestrator + CLI (`python -m benchmarks.run`).
`scripts/benchmarks/judge.py`	The pinned LLM jury.
`scripts/benchmarks/db.py`	Append-only SQLite store + content-addressed artifacts.
`scripts/benchmarks/specs.py`	Git-config definitions loader.
`scripts/benchmarks/views.py`	Read/aggregation layer (leaderboard, matrix, inspector, calibration).
`scripts/benchmarks/runners/aloop_runner.py`	In-process aloop Runner.
`scripts/benchmarks/plugins/`	One package per use-case (CaseSource + Grader).
`scripts/benchmarks/ingest/cadence.py`	Cross-repo federated ingester.
`api/src/routers/benchmarks.py`	Read-only API router (mounted on port 33800).
`data/benchmarks/benchmarks.db`	The results store.
`data/benchmarks/artifacts/`	Content-addressed blobs.

Where to go next

Model Guidance — the routing decisions benchmark evidence feeds.
Token Analytics — aggregate token/cost accounting.
Architecture — services and ports (API on 33800).