Skip to content

Scratch Pads

Working memory for daily and weekly notes. The daily pad captures in-progress thoughts, actions, and observations throughout the day. The weekly pad accumulates via a nightly rollup that uses an LLM to summarize the day into the week. Both are exposed as a read-only API for the web UI and other tools, and both are indexed by search.

How It Works

Throughout the day:
  silicon-zack appends to data/scratch/daily.md

00:05 PT nightly rollup (rolls up YESTERDAY's notes):
  ensure weekly.md is not older than yesterday's ISO week
  yesterday's daily.md --> LLM-summarized into a brief weekly entry --> weekly.md
  daily.md --> archived verbatim even if the LLM fold fails
  ensure weekly.md is not older than today's ISO week
  fresh daily.md created for today
  retry at most one earlier failed fold from its daily archive
  search index reindexed

Whenever weekly.md's header week is older than the required anchor week:
  stale weekly.md --> archived under its own YYYY-Wxx header
  fresh weekly.md created for the anchor week

Sunday maintenance (independent of pad rotation):
  full sqlite-vec index rebuild + VACUUM

The rollup is a small state machine, not a raw concatenation. The interesting parts are (1) the weekly entry is generated by the Claude daemon, with a guardrail to protect weekly.md, (2) file rotation no longer depends on one particular Monday run succeeding, and (3) raw daily notes rotate independently from the fallible LLM fold. The archive checks make successful rotations idempotent and safe to re-run.

The Nightly Rollup

Source: scripts/memory/daily_rollup.py. Entry point: python scripts/memory/daily_rollup.py, which prints a JSON status to stdout and exits non-zero only on error.

State machine

The rollup always targets yesterday's notes (today - 1 day, computed in America/Los_Angeles). Before doing anything it inspects the current daily.md date header and the archive, then branches:

Condition Action Status
archive/daily/{yesterday}.md already exists No-op (already rolled) already_done
daily.md missing Create a fresh daily for today skipped_empty
daily.md header shows today's date Already rotated by a prior run; no-op already_done
daily.md header is unparseable or older than yesterday Archive the stale file as-is, skip the LLM, create fresh daily skipped_stale
daily.md header is yesterday Normal rollup (below) rolled_up
LLM weekly update raises Append a loud failure pointer to weekly.md, archive the daily verbatim, and continue rotation rolled_up_unfolded

The implementation can return rolled_up | rolled_up_unfolded | skipped_empty | already_done | skipped_stale | error; the API schema intentionally exposes status as a string. Idempotency comes from two checks: the archive-exists short-circuit and the today-date header check. Re-running the script after a completed rotation is a no-op.

Generating the weekly entry (LLM)

When yesterday's daily has real content (_has_real_content ignores headers, the <!-- ... --> comment, and (No entries...) placeholders), the rollup calls the Claude daemon via scripts.claude.run_simple(prompt, timeout=60) to produce the entire updated weekly.md. The prompt includes yesterday's daily notes plus the current weekly content and asks the model to add a brief, concise entry for the day while preserving file references exactly.

Warning: The daemon sometimes returns a chat-style summary of its edit ("I added an entry for...") instead of the file body. Writing that reply would destroy weekly.md — which actually happened on 2026-06-08 and 2026-06-09. The guardrail (_is_weekly_body) checks that the output starts with a <!-- scratch:weekly week=... --> header. If not, it retries once with an explicit correction. If that also fails, the helper raises; the rollup keeps the existing weekly body, writes a visible failure pointer, and archives the raw daily notes for a bounded later retry.

Optionally, a Micro-Grit stats block (scripts.identity.microgrit.get_stats → 7-day completion, current/longest streak) is injected into the prompt when there is data, with instructions to mention it in the weekly only if progress or drift is notable.

Archiving and rotation

Weekly rotation is self-healing and runs against two anchors:

  1. Before the fold, _ensure_weekly_for(yesterday) makes sure an older weekly cannot absorb yesterday's notes under the wrong week. A missing weekly is created. An unparseable weekly is preserved as archive/weekly/unparseable-<timestamp>.md. A weekly whose header is older than yesterday's week is archived under its own header week, then replaced with a fresh weekly for yesterday.
  2. The LLM fold runs when the daily has real content. Failure does not block the remaining steps: the error pointer names archive/daily/{yesterday}.md, where the source notes will be preserved.
  3. The daily is always archived verbatim to data/scratch/archive/daily/{yesterday}.md after the fold attempt.
  4. After the archive, _ensure_weekly_for(today) rotates whenever the weekly header is older than today's week. On a normal Monday run this ordering folds Sunday's notes into the completed week first, archives that week, and only then creates the new weekly. If Monday's run was missed entirely, Tuesday's run still catches up rather than waiting another week.
  5. A fresh daily.md is created for today.

Weekly archive collisions never overwrite an existing file: recovery writes YYYY-Wxx-recovered-<timestamp>.md instead. The comparison only repairs weekly headers older than the anchor; it does not rewrite a future-dated header.

This behavior was added after weekly.md remained frozen at W24 for roughly three weeks. The recovery split the accumulated material into W24, W25, and W26 archives and reconstructed W27 from the preserved daily archives. That reconstruction was a one-time data repair; the durable contract is the pre-fold/post-archive self-healing check above.

Failed-fold retry

After rotation, _retry_failed_folds() looks for weekly fold FAILED markers in the current weekly.md and retries one per nightly run, oldest failed date first. It reads the raw daily archive, removes the marker only in the candidate weekly content, and writes the updated weekly only after a valid fold succeeds.

  • A failed retry leaves the marker in place with an incremented attempt annotation.
  • After three failed retries the marker becomes an honest ABANDONED terminal record pointing readers to the daily archive.
  • A missing daily archive becomes UNRESOLVABLE; it is not retried forever.
  • Archives over 15,000 characters are condensed in bounded chunks before folding; more than six chunks is rejected rather than sending an unbounded prompt.
  • Retry folding gets a 180-second timeout, longer than the ordinary nightly fold.

The retry scan only examines the current weekly.md. If a week-boundary rotation has already moved a failure marker into an archived weekly, that marker is not automatically retried.

Writes use an atomic temp-file-plus-rename pattern (atomic_write_text) so a crash mid-write never corrupts a pad.

Search index maintenance

After rotation the rollup keeps the search index current:

Step When What
scripts.search.reindex() Every completed normal rotation, including rolled_up_unfolded Incremental reindex of changed files
scripts.search.maintenance.rebuild_vec_index(vacuum=True) When the rollup runs on Sunday (today.weekday() == 6) Full sqlite-vec index rebuild + VACUUM to reclaim soft-deleted vector slots (sqlite-vec leaves deleted vectors as dead weight until rebuilt)

Both steps are non-fatal: a failure here is logged but does not fail the rollup.

The Header-Comment Contract

The first line of each pad is an HTML comment that drives both parsing and the rollup state machine. Get this format wrong and the API reports period: "unknown" and the rollup treats the file as stale.

<!-- scratch:daily date=2026-01-02 -->
# Friday, January 2

(No entries yet)

<!-- scratch:weekly week=2026-W01 start=2025-12-29 end=2026-01-04 -->
# Week 1: Dec 29 - Jan 4

(No entries yet)
Pattern Used by
<!--\s*scratch:daily\s+date=(\d{4}-\d{2}-\d{2})\s*--> API period extraction; rollup branch selection (today vs. yesterday vs. stale)
<!--\s*scratch:weekly\s+week=(\d{4}-W\d{2}) API period extraction; weekly-body guardrail (_is_weekly_body)

What Gets Logged

  • Significant actions with file refs: processed interview -> `path/to/file`
  • Things in progress: "waiting on a result"
  • Quick observations: "energy low today, possibly from poor sleep"
  • Timestamps for time-sensitive items: "3pm: made the call"

Not logged here: Check-in data (goes in check-ins), full content (link to source files).

API Endpoints (/api/v1/scratch-pads)

Source: api/src/routers/scratch_pads.py. Served by the API on port 33800 (./run api). Read-only except for the manual rollup trigger.

Method Endpoint Purpose
GET /daily Current daily scratch pad
GET /weekly Current weekly scratch pad
GET /archive/{pad_type}/{date_str} Archived pad (pad_type = daily|weekly; date_str = YYYY-MM-DD for daily, YYYY-Wxx for weekly)
POST /rollup Manually trigger the nightly rollup

GET /archive/... validates pad_type against {daily, weekly} and date_str against a strict YYYY-MM-DD / YYYY-Wxx regex before touching the filesystem — this is the path-traversal guard. A request for a missing or empty pad returns a normal response with is_empty: true rather than a 404.

POST /rollup delegates to scheduler.run_rollup_now(), which calls the same _run_rollup() used by the nightly loop under an asyncio.Lock (self._rollup_lock) so a manual trigger can never race the scheduled one. The scheduler runs scripts/memory/daily_rollup.py as a subprocess; the nightly _rollup_loop fires at 00:05 PT (api/src/scheduler.py).

Response fields (ScratchPadResponse):

Field Type Meaning
content_md string Raw markdown of the pad
period string Date/week parsed from the header comment, or "unknown"
updated_at string | null File mtime as ISO timestamp; null if the file is missing
is_empty bool True when only the header/markdown title/placeholder remain

RollupResponse returns status, rolled_date, and message (mirrors the rollup script's JSON output).

File Locations

Path Contents
data/scratch/daily.md Current day's working notes
data/scratch/weekly.md This week's accumulating summary (LLM-rolled)
data/scratch/archive/daily/YYYY-MM-DD.md Archived daily pads
data/scratch/archive/weekly/YYYY-Wxx.md Archived weekly pads (self-healing ISO-week rotation)
scripts/memory/daily_rollup.py Rollup state machine, LLM weekly update, rotation recovery, and failed-fold retry
api/src/routers/scratch_pads.py Read-only API + manual rollup trigger
api/src/schemas/scratch_pads.py ScratchPadResponse, RollupResponse
api/src/scheduler.py _rollup_loop (00:05 PT), run_rollup_now, mutex

Where to Go Next

  • Search — both pads are indexed (source_type: scratch); the rollup keeps the index fresh.
  • Document Filing — how facts get extracted and persisted across the system.
  • Memory & Trends — the wider memory layering that the scratch pads feed into.