Token Analytics

Aggregates Claude CLI and OpenAI Codex CLI token usage from session logs into daily/monthly summaries, enabling cost tracking and usage pattern analysis across projects and models.

Note: This subsystem measures subscription CLI session usage (Claude Code + Codex). It is distinct from the inference usage log, which records paid API spend. Both feed the unified cost view via scripts/inference/costs.get_unified_costs.

Overview

Token analytics captures every API call made through the Claude CLI (~/.claude/projects/), optionally folds in OpenAI Codex CLI sessions (~/.codex/), and provides:

Aggregated Usage - Monthly JSON files with daily/hourly/project/model breakdowns
Session Tracking - Individual session extraction with metadata
LLM Summarization - On-demand session summaries via OpenRouter
Cost Tracking - Summarization costs logged and archived

The system runs on a 15-minute schedule via the FastAPI scheduler and supports both full rebuilds and incremental updates.

System Architecture

                              Token Analytics Flow

+-----------------------------------------------------------------------------+
|              Claude CLI Sessions              Codex CLI Sessions             |
|  ~/.claude/projects/-home-****-work-****/    ~/.codex/state_5.sqlite         |
|      <uuid>.jsonl   (no sessions/ subdir)    ~/.codex/sessions/*.jsonl       |
|  ~/.claude/projects/-home-****-work-****/                                    |
|      <uuid>.jsonl                                                            |
+-----------------------------------------------------------------------------+
                                     |
          +--------------------------|---------------------------+
          |                          |                           |
          v                          v                           v
+--------------------+    +----------------------+    +----------------------+
| Aggregation        |    | Session Extraction   |    | Summarization        |
| (Every 15 min)     |    | (On-demand API)      |    | (On-demand API)      |
+--------------------+    +----------------------+    +----------------------+
| aggregate-tokens.py|    | scripts/tokens/      |    | scripts/tokens/      |
| --incremental      |    | extractor.py         |    | summarize.py         |
| (+ codex_extractor)|    |                      |    |                      |
+--------------------+    +----------------------+    +----------------------+
          |                          |                           |
          v                          v                           v
+--------------------+    +----------------------+    +----------------------+
| data/tokens/       |    | data/tokens/         |    | data/tokens/         |
| YYYY-MM.json       |    | sessions/YYYY-MM-DD  |    | summaries/YYYY-MM-DD |
+--------------------+    +----------------------+    +----------------------+
          |                          |                           |
          +--------------------------|---------------------------+
                                     |
                                     v
+-----------------------------------------------------------------------------+
|                            API Endpoints                                     |
|  GET  /api/tokens/month/{month}     - Monthly aggregated data               |
|  GET  /api/tokens/day/{date}        - Day detail with hourly breakdown      |
|  GET  /api/tokens/sessions/{date}   - Paginated session list                |
|  GET  /api/tokens/session/{id}      - Single session detail                 |
|  GET  /api/tokens/session/{id}/messages  - Paginated session messages       |
|  POST /api/tokens/session/{id}/summarize - Generate/cache summary           |
|  POST /api/tokens/extract/{date}    - Trigger session extraction            |
|  GET  /api/tokens/summary-costs     - Summarization cost tracking           |
+-----------------------------------------------------------------------------+

Why This Exists

Understanding token usage matters for several reasons:

Cost Visibility - Claude Code uses multiple models (Opus, Haiku, Sonnet) with different pricing
Project Allocation - See which projects consume the most tokens
Usage Patterns - Hourly breakdowns reveal when you're most productive
Session Context - Summaries help you remember what each session accomplished

Source Data Structure

The Claude CLI stores session data as JSONL files placed directly in each project directory — there is no sessions/ subdirectory. Each file is named by session UUID:

~/.claude/projects/
├── -home-****-work-****/
│   ├── abc123-def456-....jsonl
│   ├── 35b0df76-....jsonl
│   └── ...
├── -home-****-work-****/
│   └── ...

Codex CLI sessions live separately under ~/.codex/ (see Codex Token Ingestion).

Assistant Message with Usage (Source Record)

Each assistant response in the JSONL contains usage metadata:

{
  "type": "assistant",
  "timestamp": "2026-01-01T15:23:50.245Z",
  "message": {
    "model": "claude-opus-4-8",
    "id": "msg_0123456789ABCDEF",
    "role": "assistant",
    "content": [...],
    "usage": {
      "input_tokens": 2,
      "output_tokens": 58,
      "cache_creation_input_tokens": 22551,
      "cache_read_input_tokens": 12832,
      "cache_creation": {
        "ephemeral_5m_input_tokens": 22551,
        "ephemeral_1h_input_tokens": 0
      },
      "service_tier": "standard"
    }
  }
}

Token Types Explained

Field	Description	Cost Implication
`input_tokens`	Fresh tokens sent to the model (not from cache)	Full input price
`output_tokens`	Tokens generated by the model	Full output price
`cache_creation_input_tokens`	Tokens written to prompt cache	1.25x input price
`cache_read_input_tokens`	Tokens read from prompt cache	0.1x input price

Total tokens per request = input_tokens + output_tokens + cache_creation_input_tokens + cache_read_input_tokens

This matches the calculation used by claude-monitor for consistency.

Aggregated Data Structure

Monthly File: `data/tokens/YYYY-MM.json`

{
  "month": "2026-01",
  "days": {
    "2026-01-04": {
      "total": 586464036,
      "by_hour": {
        "12": 3808273,
        "14": 15234567
      },
      "by_model": {
        "claude-opus-4-8": 569280487,
        "claude-fable-5": 16624764,
        "gpt-5.4": 558785,
        "<synthetic>": 0
      },
      "by_model_detail": {
        "claude-opus-4-8": {
          "input_tokens": 120345,
          "output_tokens": 84210,
          "cache_creation": 4221330,
          "cache_read": 564854602
        }
      },
      "by_project": {
        "project-a": 562073893,
        "project-b": 12104723,
        "project-c": 8728550,
        "sys": 3556870
      },
      "timezone": "America/Los_Angeles",
      "day_of_year": 4
    }
  },
  "weekly_totals": [1780994303],
  "monthly_total": 1780994303,
  "_meta": {
    "year": 2026,
    "year_total": 1780994303
  },
  "updated_at": "2026-01-04T20:29:20.272349+00:00"
}

Field Descriptions

Field	Type	Description
`month`	string	Month identifier (YYYY-MM format)
`days`	object	Daily breakdowns keyed by YYYY-MM-DD
`days.*.total`	int	Total tokens processed that day
`days.*.by_hour`	object	Tokens per hour (0-23), local timezone
`days.*.by_model`	object	Total tokens per model ID
`days.*.by_model_detail`	object	Per-model breakdown: `input_tokens`, `output_tokens`, `cache_creation`, `cache_read`
`days.*.by_project`	object	Tokens per project directory
`days.*.timezone`	string	Timezone used for hourly bucketing
`days.*.day_of_year`	int	1-365/366 for trend analysis
`weekly_totals`	array	Tokens per ISO week (Monday-Sunday)
`monthly_total`	int	Sum of all daily totals
`_meta`	object	Metadata for hero metrics
`updated_at`	string	ISO timestamp of last aggregation

Aggregation State: `data/tokens/.aggregation-state.json`

Tracks file positions for incremental processing (file is ~700KB due to many session files):

{
  "files": {
    "/home/****/.claude/projects/-home-****-work-****/abc123.jsonl": {
      "position": 535105
    }
  },
  "last_run": "2026-01-04T05:00:00.000Z"
}

Module Architecture

The token system is organized into a reusable package:

scripts/tokens/
├── __init__.py          # Public exports
├── storage.py           # Data models, paths, atomic writes
├── extractor.py         # Claude session extraction from JSONL
├── codex_extractor.py   # OpenAI Codex session extraction (sqlite + rollouts)
├── session_sync.py      # Smart session sync with change detection
└── summarize.py         # LLM summarization via OpenRouter

storage.py - Data Layer

Provides data classes and file operations:

from scripts.tokens import SessionRecord, SessionSummaryRecord

# Data classes
SessionRecord(
    id="abc123",
    slug="abc12345-Jan04-09",  # Human-readable: {id[:8]}-{MonDD-HH}
    start_ts="2026-01-04T09:15:00Z",
    end_ts="2026-01-04T10:30:00Z",
    duration_seconds=4500,
    token_count=125000,
    model="claude-opus-4-8",
    project="-home-****-work-****",
    source_file="/home/****/.claude/projects/..."
)

# Paths
SESSIONS_DIR = data/tokens/sessions/
SUMMARIES_DIR = data/tokens/summaries/
COSTS_LOG_PATH = data/tokens/summary-costs.jsonl

extractor.py - Session Extraction

Extracts session metadata from Claude CLI JSONL files:

from scripts.tokens.extractor import extract_sessions_for_date, ExtractionLock

# Extract with backpressure control
with ExtractionLock(date) as acquired:
    if acquired:
        result = extract_sessions_for_date(
            date="2026-01-04",
            max_sessions=100,      # Backpressure limit
            max_file_size_mb=50,   # Skip huge files
            timezone="America/Los_Angeles"
        )
        # result.extracted, result.remaining, result.status

Key features: - File locking - Prevents concurrent extractions for same date - Backpressure - Limits sessions per extraction (default 100) - Large file skip - Ignores files >50MB - Progress callbacks - Optional progress reporting

summarize.py - LLM Summarization

Generates session summaries via OpenRouter with rate limiting and cost tracking:

from scripts.tokens.summarize import generate_summary_async, rate_limiter

# Check rate limit (5 requests/minute, persisted to disk)
allowed, wait_seconds = rate_limiter.check_and_record()
if not allowed:
    return f"Retry in {wait_seconds}s"

# Generate summary
result = await generate_summary_async(
    session_id="abc123",
    session_content="...",  # Full session content
    date="2026-01-04",
    api_key=os.environ["OPENROUTER_API_KEY"],
    model="google/gemini-3.1-flash-lite-preview",  # DEFAULT_MODEL, cheap
    force=False  # Use cache if available
)
# result.summary, result.error, result.cached, result.cost_usd

Key features: - Persistent rate limiter - Survives restarts via .rate-limit.json - Caching - Summaries stored in summaries/YYYY-MM-DD.jsonl - Cost tracking - Every API call logged to summary-costs.jsonl - Monthly archiving - Cost logs archived at month boundaries

session_sync.py - Smart Session Sync

Automatically extracts sessions with change detection, running every 15 minutes via the scheduler:

from scripts.tokens.session_sync import sync_today, sync_all, backfill, show_status

# Default: sync today (re-checks for new/modified session files)
result = sync_today(quiet=True)
# result.extracted, result.skipped, result.reason

# Backfill historical days (extract once, never re-extract)
results = backfill(quiet=True, max_days=50)

Key features: - Change detection - Tracks source file mtimes to detect modifications - Today always re-checks - New sessions appear throughout the day - Past days are immutable - Extracted once, then skipped - Monthly file updates - Writes session_count back to monthly JSON for UI display - State tracking - data/tokens/.session-sync-state.json records extraction status per day

CLI:

python scripts/tokens/session_sync.py                    # Sync today
python scripts/tokens/session_sync.py --all              # Sync all days
python scripts/tokens/session_sync.py --backfill         # Backfill historical
python scripts/tokens/session_sync.py --status           # Show sync status

codex_extractor.py - Codex Token Ingestion

aggregate-tokens.py also pulls OpenAI Codex CLI sessions and folds them into the same aggregate format, so the monthly JSON is a unified Claude + Codex view. Codex ingestion runs by default; pass --no-codex to skip it.

Codex stores its state differently from Claude — a sqlite database plus per-session rollout JSONL files:

Source	Path	Used for
Thread index (sqlite)	`~/.codex/state_5.sqlite` (`threads` table, read-only)	session id, `created_at` epoch, `tokens_used`, `rollout_path`, `cwd`, model_provider, title
Rollout files (JSONL)	`~/.codex/sessions/*.jsonl`	per-session token breakdown (final `token_count` event) and `turn_context` model

How it maps into the Claude format:

Only threads with tokens_used > 0 are read; an optional since_date filters by created_at (PT).
model defaults to gpt-5.4 and is overridden by the rollout's turn_context.model when present.
The rollout's total_token_usage populates input_tokens, output_tokens, cached_input_tokens (mapped to cache_read_input_tokens) and reasoning_output_tokens (kept as reasoning_tokens). cache_creation_input_tokens is always 0 for Codex.
The project name is derived from cwd using the same "segment after work" convention as Claude (see Project Name Extraction).
Each emitted entry is tagged "source": "codex".

Note: When merged into the monthly file via aggregate-tokens.py, only the four Claude-shaped token fields propagate into by_model_detail (input_tokens, output_tokens, cache_creation, cache_read); the Codex reasoning_tokens field is dropped at that boundary. It is preserved by the standalone summary helper below.

The module also exposes a daily summary helper and a CLI:

from tokens.codex_extractor import extract_codex_entries, get_codex_daily_summary

# All Codex sessions (optionally since a date)
entries = extract_codex_entries(since_date="2026-03-01")

# Aggregated single-day view (PT) — matches monthly-JSON day shape plus source/reasoning
summary = get_codex_daily_summary("2026-03-06")
# {total, by_model, by_model_detail, by_project, by_hour, session_count, source}

python scripts/tokens/codex_extractor.py              # List recent Codex sessions
python scripts/tokens/codex_extractor.py --summary [YYYY-MM-DD]   # Daily summary (default: today PT)

File Locations

Path	Purpose
`scripts/aggregate-tokens.py`	Main aggregation script (Claude + Codex)
`scripts/tokens/`	Package: storage, extractor, codex_extractor, summarize, session_sync
`scripts/tokens/codex_extractor.py`	OpenAI Codex session extraction (sqlite + rollouts)
`data/tokens/YYYY-MM.json`	Monthly aggregated data
`data/tokens/.aggregation-state.json`	Incremental aggregation state
`data/tokens/.session-sync-state.json`	Session sync state (per-day extraction tracking)
`data/tokens/sessions/YYYY-MM-DD.jsonl`	Extracted session metadata
`data/tokens/summaries/YYYY-MM-DD.jsonl`	Cached session summaries
`data/tokens/summary-costs.jsonl`	Summarization cost log
`data/tokens/summary-costs-archive/`	Archived monthly cost logs
`data/tokens/.extraction-locks/`	File locks for extraction
`data/tokens/.rate-limit.json`	Rate limiter state
`api/src/routers/tokens.py`	API endpoints (`/api/tokens`)
`api/src/schemas/token_sessions.py`	Pydantic schemas
`api/src/scheduler.py`	Background scheduler (15-min interval)
`~/.claude/projects/<project>/<uuid>.jsonl`	Source Claude JSONL session files
`~/.codex/state_5.sqlite` + `~/.codex/sessions/`	Source Codex sessions

Aggregation Logic

Project Name Extraction

Project names are extracted from the JSONL file path:

Path: ~/.claude/projects/-home-****-work-****/<uuid>.jsonl
                                       ^^^^
                                       Extracted: "****"

Path: ~/.claude/projects/-home-****-work-****-****/<uuid>.jsonl
                                       ^^^^
                                       Extracted: "****" (first segment after "work")

Logic (from extract_project_name()): 1. Split directory name on - 2. Find work in the parts 3. Take the first segment after work 4. Fallback to last part if work not found

Deduplication

Streaming responses send the same usage data multiple times with the same message.id. The aggregator tracks seen message IDs per file to avoid double-counting:

if message_id in seen_message_ids:
    continue  # Skip duplicate
seen_message_ids.add(message_id)

Incremental Mode

When --incremental is passed: 1. Load previous file positions from .aggregation-state.json 2. Seek to last position for each file 3. Only parse new bytes 4. Merge new data into existing monthly files 5. Save updated positions

This makes the 15-minute scheduler efficient - only processing new session data.

Hourly Bucketing

The aggregator converts all timestamps to local timezone (America/Los_Angeles) before bucketing by hour. This ensures: - Consistent hourly patterns across DST changes - Hours 0-23 represent local time, not UTC

local_ts = ts.astimezone(LOCAL_TZ)
hour_key = str(local_ts.hour)  # "0" to "23"

Weekly Totals

Uses ISO week numbers (Monday-Sunday):

for day_str, day_data in days.items():
    week_num = datetime.strptime(day_str, "%Y-%m-%d").isocalendar()[1]
    weeks[week_num] += day_data["total"]

Scheduling

Two scheduled loops run via api/src/scheduler.py:

Token Aggregation (every 15 min)

async def _token_aggregation_loop(self):
    # Run immediately on startup
    await self._run_token_aggregation()

    while self._running:
        await asyncio.sleep(15 * 60)  # Every 15 minutes
        await self._run_token_aggregation()

Execution:

uv run python scripts/aggregate-tokens.py --incremental --quiet

Setting	Value
Interval	15 minutes
Startup	Runs immediately
Timeout	120 seconds
Mode	Incremental (only new entries)

Session Sync (every 15 min)

async def _session_sync_loop(self):
    # Run immediately on startup
    await self._run_session_sync()

    while self._running:
        await asyncio.sleep(15 * 60)
        await self._run_session_sync()

Execution:

python scripts/tokens/session_sync.py --quiet

Setting	Value
Interval	15 minutes
Startup	Runs immediately
Scope	Today only (past days are immutable)
Change detection	File mtime tracking

CLI Usage

# Full aggregation (reprocess everything)
python scripts/aggregate-tokens.py

# Incremental (only new entries, used by scheduler)
python scripts/aggregate-tokens.py --incremental

# Show stats without writing files
python scripts/aggregate-tokens.py --stats

# Quiet mode for automated runs
python scripts/aggregate-tokens.py --incremental --quiet

# Skip Codex ingestion (Claude sessions only)
python scripts/aggregate-tokens.py --incremental --no-codex

API Endpoints

Legacy Endpoints (Backward Compatible)

GET /api/tokens/data

List available month files.

Response:

{
  "files": ["2025-12.json", "2026-01.json"]
}

GET /api/tokens/data/{filename}

Get raw monthly JSON file.

New Session-Based Endpoints

GET /api/tokens/month/{month}

Get aggregated data for a month with full breakdown.

GET /api/tokens/day/{date}

Get detailed data for a specific day including session count.

Response:

{
  "date": "2026-01-04",
  "total": 586464036,
  "by_hour": {"12": 3808273, "14": 15234567},
  "by_model": {"claude-opus-4-8": 569280487},
  "by_project": {"project-a": 562073893},
  "session_count": 15,
  "sessions_extracted": 15,
  "timezone": "America/Los_Angeles",
  "day_of_year": 4
}

GET /api/tokens/sessions/{date}?offset=0&limit=20&sort=newest

Get paginated list of sessions for a day.

Query parameters: - offset - Pagination offset (default 0) - limit - Page size (1-100, default 20) - sort - newest, oldest, or most_tokens

Response includes X-Extraction-Status header: complete, pending, partial, in_progress

GET /api/tokens/session/{session_id}?date=YYYY-MM-DD

Get single session detail. Does NOT auto-summarize.

GET /api/tokens/session/{session_id}/messages?date=YYYY-MM-DD&offset=0&limit=50

Get paginated session messages with full content blocks (text, tool_use, tool_result, thinking). Returns conversation flow for session detail view.

POST /api/tokens/session/{session_id}/summarize?date=YYYY-MM-DD

Generate or retrieve cached summary.

Response:

{
  "session_id": "abc123",
  "summary": "Built token analytics dashboard...",
  "error": null,
  "cached": true,
  "cost_usd": 0.000123
}

Rate limited: 5 requests/minute. Returns 429 with Retry-After header when limited.

POST /api/tokens/extract/{date}

Trigger session extraction for a date. Uses file locking to prevent concurrent extractions.

GET /api/tokens/summary-costs?since=YYYY-MM-DD&until=YYYY-MM-DD

Get summarization cost tracking with optional date filter.

Cost Calculation

Token Aggregation Costs

Not currently implemented for token aggregation. The system tracks raw counts but does not calculate costs.

Reasons: 1. Pricing varies by model tier (frontier vs lightweight) by an order of magnitude per MTok 2. Cached tokens have different pricing (cache read is 0.1x, cache write is 1.25x) 3. Pricing changes over time 4. The subscription plan has different economics than per-token API billing

Tip: For paid spend rollups (vendor-billed API calls, not subscription sessions), see the inference cost API. For the live model roster and which task uses which model, see Model Guidance.

Summarization Costs

Implemented for the summarization feature. Each summary generation is logged:

{
  "date": "2026-01-04",
  "session_id": "abc123",
  "model": "google/gemini-3.1-flash-lite-preview",
  "input_tokens": 45000,
  "output_tokens": 150,
  "cost_usd": 0.000123
}

Pricing table in summarize.py (USD per million tokens):

DEFAULT_MODEL = "google/gemini-3.1-flash-lite-preview"

PRICING = {
    "google/gemini-3.1-flash-lite-preview": {"input": 0.25, "output": 1.50},
}

Known Limitations

1. Project Name Extraction is Brittle

The path parsing assumes a specific directory structure:

-home-****-work-{project}/...

Nested paths like -home-****-work-****-today extract ****, losing subdirectory info.

Impact: Projects in subdirectories get grouped with parent.

Fix: Add configurable project name mappings or parse more intelligently.

2. No Retroactive Rebuilds of Monthly Files

If aggregation logic changes, old monthly files are not automatically rebuilt.

Workaround: Delete .aggregation-state.json and monthly files, then run full aggregation.

3. `<synthetic>` Model Entries

Some entries have model <synthetic> with 0 tokens - these appear to be CLI-generated records, not actual API calls. They're included in output but could be filtered.

4. Large State File

.aggregation-state.json grows with every session file ever seen (currently ~700KB). Consider pruning entries for files that no longer exist.

5. UI Dashboard

The web UI at /dashboard/tokens provides a full visualization: - Hero Section - Monthly total, daily average, year-to-date stats - Day Grid - Calendar view with token counts and hourly sparklines - Day Deep Dive - Expanded view with session list, extraction status - Session Cards - Individual sessions with model, duration, token breakdown - Session Detail - Full content view with LLM-generated summaries - Mobile Deep Dive - Responsive slide-up panel for mobile

Components in web/src/components/tokens/: - HeroSection.tsx - Animated monthly stats - DayGrid.tsx + DayCard.tsx - Calendar-style day view - DayDeepDivePanel.tsx - Expanded day details - SessionList.tsx + SessionCard.tsx - Session browsing - HourlySparkline.tsx - Mini hourly usage chart - TokenHeatmap.tsx - Usage heatmap visualization

6. Timezone Handling

All timestamps are converted to Pacific Time for bucketing. If multi-timezone analysis is needed, consider storing both UTC and local timestamps.

7. Session Extraction is Automatic

Session extraction runs automatically every 15 minutes via session_sync.py. Today's sessions are re-checked on each run (mtime-based change detection); past days are extracted once and then skipped. Manual extraction is also available via the API (POST /api/tokens/extract/{date}).

8. Summarization Requires API Key

OpenRouter API key must be set in environment. If missing, summarization returns an error rather than failing silently.

Improvement Opportunities

Token Cost Calculation - Add pricing table and compute estimated costs per day/project/model
Usage Alerts - Notify when daily usage exceeds threshold
Retention Policy - Archive or compress old monthly files
Project Mappings - Allow explicit project name overrides for cleaner grouping
Rate Tracking - Calculate tokens per minute/hour for rate limit awareness
Session Correlation - Link usage back to conversation content for analysis
~~Automatic Extraction~~ - Implemented via session_sync.py (every 15 min)
Multi-Model Cost Tracking - Calculate accurate costs per model with cache multipliers
Keyboard Navigation - Add j/k navigation and shortcuts in the dashboard

Token analytics covers subscription CLI sessions (Claude Code + Codex), where per-token cash cost is effectively zero. Paid, vendor-billed API calls are tracked by a separate, append-only log so cost dashboards reflect real cash spend only:

Concern	This subsystem	Inference usage log
What it measures	Subscription CLI session tokens	Every paid inference call (model, task_type, tokens, cost, duration)
Store	`data/tokens/YYYY-MM.json`	`data/inference/usage.jsonl` (append-only)
Cash cost	Not computed (subscription)	Real per-call cost; Claude OAuth calls forced to `$0`
Entry point	`scripts/aggregate-tokens.py`, `/api/tokens/*`	`scripts/inference/usage.py`, `GET /api/v1/inference/usage`

The /api/v1/inference router doubles as an observability dashboard: it lists LLM consumers (scheduler tasks, sentinel poll-agents, telegram) with live status and resolved model, exposes unified Claude + paid cost rollups (/costs), task-level activity (/activity), model-registry CRUD (/models), and a /test endpoint to ping a model. Both data paths converge in scripts/inference/costs.get_unified_costs.

Endpoint	Purpose
`GET /api/v1/inference/usage`	Paid usage rollup
`GET /api/v1/inference/consumers`	LLM consumers + live status
`GET /api/v1/inference/costs`	Unified subscription + paid cost view
`GET /api/v1/inference/activity`	Task-level activity from `usage.jsonl`
`GET/POST /api/v1/inference/models`	Model registry CRUD
`POST /api/v1/inference/test`	Ping a model

For the model roster and routing policy that the registry resolves against, see Model Guidance.

Where to go next

Model Guidance — live model roster and per-task routing
Architecture — how the API, scheduler, and scripts fit together
Data Browser — inspecting data/ files like data/tokens/

Token Analytics

Overview

System Architecture

Why This Exists

Source Data Structure

Assistant Message with Usage (Source Record)

Token Types Explained

Aggregated Data Structure

Monthly File: data/tokens/YYYY-MM.json

Field Descriptions

Aggregation State: data/tokens/.aggregation-state.json

Module Architecture

storage.py - Data Layer

extractor.py - Session Extraction

summarize.py - LLM Summarization

session_sync.py - Smart Session Sync

codex_extractor.py - Codex Token Ingestion

File Locations

Aggregation Logic

Project Name Extraction

Deduplication

Incremental Mode

Hourly Bucketing

Weekly Totals

Scheduling

Token Aggregation (every 15 min)

Session Sync (every 15 min)

CLI Usage

API Endpoints

Legacy Endpoints (Backward Compatible)

New Session-Based Endpoints

Cost Calculation

Token Aggregation Costs

Summarization Costs

Known Limitations

1. Project Name Extraction is Brittle

2. No Retroactive Rebuilds of Monthly Files

3. <synthetic> Model Entries

4. Large State File

5. UI Dashboard

6. Timezone Handling

7. Session Extraction is Automatic

8. Summarization Requires API Key

Improvement Opportunities

Related: Paid Inference Usage

Where to go next

Monthly File: `data/tokens/YYYY-MM.json`

Aggregation State: `data/tokens/.aggregation-state.json`

3. `<synthetic>` Model Entries