Data Browser

The Data Browser is a web-based file explorer for viewing all Vita health data files. It provides a resizable split-pane interface with collapsible folder navigation, JSON syntax highlighting, and JSONL pagination for large log files.

Overview

The Data Browser exposes structured health data through a secure API that enforces strict path validation (10-step defense-in-depth) to prevent directory traversal attacks. Only whitelisted directories are accessible: data, profile, goals, conditions, trends, insights, and meta.

Why it exists: Debugging health data and understanding system state requires browsing raw JSON files. Rather than using terminal commands, the Data Browser provides a visual interface with syntax highlighting, search, and pagination for large JSONL files.

How to Access

1. Navigate to /data in the web UI
2. The sidebar displays a file tree of all allowed directories
3. Click any file to view its contents in the main panel

User Interface

+-------------------------+--------------------------------------------------+
|                         |                                                  |
|  SIDEBAR (resizable)    |              CONTENT AREA                        |
|  200px - 480px          |                                                  |
| +---------------------+ | +----------------------------------------------+ |
| | [Search files...]   | | |  Home > data > checkins > 2026-01-03.json   | |
| +---------------------+ | +----------------------------------------------+ |
| [+ Expand] [- Collapse] |                                                  |
|                         | +----------------------------------------------+ |
| v data/          (12)   | | [File] 2026-01-03.json  [JSON]               | |
|   v checkins/    (31)   | |                           [Copy] [Download]  | |
|     2026-01-01.json     | +----------------------------------------------+ |
|     2026-01-02.json     |                                                  |
|     2026-01-03.json  <- | |  1  {                                        | |
|   > garmin/      (31)   | |  2    "mood": 8,                             | |
|   > identity/    (4)    | |  3    "energy": 7,                           | |
| v profile/       (3)    | |  4    "sleep_quality": "good"                | |
|   basics.json           | |  5  }                                        | |
|   preferences.json      |                                                  |
|   communication.json    | +----------------------------------------------+ |
|                         | |  [<<] [<] [1] [2] ... [10] [>] [>>]          | |
| ----------------------- | |  (JSONL pagination only)                     | |
| 7 root folders          | +----------------------------------------------+ |
+-------------------------+--------------------------------------------------+
           ^                                    ^
           |                                    |
     Drag handle                     Floating back button (mobile)
     (desktop only)                  + bottom action bar (mobile)

Data Flow

User clicks file in tree
        |
        v
+------------------+        +------------------+        +------------------+
| TanStack Router  | -----> | useDataFile hook | -----> | API: GET /file/  |
| updates URL      |        | (React Query)    |        | {path}?offset=0  |
+------------------+        +------------------+        +------------------+
                                    |
                                    v
                           +------------------+
                           | FileViewer       |
                           | - Formats JSON   |
                           | - Highlights     |
                           | - Paginates      |
                           +------------------+

Layout Components

Component	Description	Behavior
DataBrowserLayout	Split-pane container	Manages sidebar width state, drag-resize handling
FileTree	Collapsible file tree	Default expands first level, memoized for performance
FileViewer	Code viewer with actions	Breadcrumbs, syntax highlighting, copy/download
Resize Handle	Drag divider (desktop)	Min 200px, max 480px, cursor changes during drag

Key UI Features

Feature	Implementation	Why
Real-time search	matchesSearch() recursive filter	Find files without scrolling
Auto-expand on search	effectiveExpanded set from all dirs	Show matches in context
Memoized tree nodes	React.memo(TreeNode)	Prevent re-renders on sibling clicks
Lazy child rendering	Only render children when expanded	Large trees stay fast
JSON syntax highlighting	String replacement with HTML spans	No external dependencies
Copy with feedback	2-second checkmark + "Copied!" text	Confirm action succeeded

Visual Elements

• Folder icons: Closed folder (indigo-400), Open folder (indigo-500)
• File badges: JSON (gold gradient), JSONL (blue gradient)
• Active file: Indigo left border + gradient background
• Child count: Hover reveals item count per folder (visible on group-hover)
• Skeleton loading: Shimmer animation while fetching tree or file

Mobile Behavior

On screens below the md breakpoint (768px):

• Sidebar: Full width, collapses to show file list only
• Floating back button: Fixed position top-left to return to tree
• Bottom action bar: Copy/Download buttons in fixed footer
• Pagination: Shows X / Y compact format instead of page numbers
• Tree items: Taller (48px) for touch targets

Data Sources Exposed

Directory	Contents	File Format
data/checkins/	Daily health check-ins	JSON per day
data/garmin/	Synced Garmin Connect data (sleep, HRV, steps)	JSON per day
data/rwgps/	RideWithGPS cycling data	JSON per sync
data/identity/	Observations, interviews, quotes	JSONL append-only
data/proactive/	Telegram message queue	SQLite (not browsable) + JSONL
data/scheduling/	Frequency goal occurrences	JSONL per month
data/tokens/	Token usage aggregates, sessions, summaries	JSON/JSONL
data/board/	Board meeting data and resolutions	JSON per meeting
data/objectives/	Daily and weekly objectives	JSON per day/week
data/experiments/	Experiment protocols and results	JSON per experiment
data/media/	Media consumption (HN, RSS, podcasts, X)	JSONL per day
data/location/	Location history	JSONL per day
data/activities/	Unified activity data (cycling, runs, etc.)	JSONL per year
data/coaching/	Coaching effectiveness and cooldowns	JSON/JSONL
profile/	User settings (basics, preferences, communication)	JSON
goals/active/	Current health goals	JSON per goal
goals/achieved/	Completed goals archive	JSON per goal
insights/	AI-generated health insights	JSON
meta/	System metadata, feature intents	JSON/JSONL
trends/	Computed trend data	JSON
conditions/	Health conditions tracking	JSON

Architecture

Component Hierarchy

/data (index.tsx)                 /data/{path} ($.tsx)
        |                                 |
        v                                 v
+------------------+              +------------------+
| DataBrowserLayout|              | DataBrowserLayout|
+------------------+              +------------------+
        |                                 |
   +---------+                    +-----------+-----------+
   |         |                    |           |           |
FileTree  WelcomeContent       FileTree  FileViewer  (skeletons)

State Management

The Data Browser uses TanStack Query for server state with these caching strategies:

Query	Stale Time	Why
Tree (/api/v1/data/tree)	5 minutes	Directory structure rarely changes
File content	30 seconds	Files may be updated by check-ins/syncs
File metadata	5 minutes	Line counts stable unless file appended

Local state is minimal: - sidebarWidth in DataBrowserLayout (drag resize) - expandedPaths in FileTree (folder collapse state) - searchQuery in FileTree (filter text) - offset in route pages (pagination)

File Locations

Component	Path	Responsibility
Route: Index	web/src/routes/data/index.tsx	Welcome screen, tree only
Route: File View	web/src/routes/data/$.tsx	Tree + file viewer
Layout	web/src/components/data/DataBrowserLayout.tsx	Split pane, resize logic
File Tree	web/src/components/data/FileTree.tsx	Navigation, search, expand/collapse
File Viewer	web/src/components/data/FileViewer.tsx	Display, highlight, paginate
Index	web/src/components/data/index.ts	Barrel exports
Hooks	web/src/hooks/useDataBrowser.ts	API integration (3 hooks)
API Router	api/src/routers/data.py	Path validation, file reading
Service	api/src/services/data_browser.py	Tree building, content extraction
Schemas	api/src/schemas/data_browser.py	Pydantic models

Data Hooks

// Fetch entire tree structure (cached 5 min)
useDataTree(): { data: DataTreeResponse, isLoading, error }

// Fetch file content with pagination (cached 30s)
useDataFile(path, offset, limit): { data: DataFileResponse, isLoading, error }

// Fetch metadata only (for line counts without content)
useDataFileMeta(path): { data: DataFileMeta, isLoading, error }

API Endpoints

Method	Endpoint	Description	Response
GET	/api/v1/data/tree	Directory tree of allowed roots	DataTreeResponse
GET	/api/v1/data/file/{path}	File contents with optional pagination	DataFileResponse
GET	/api/v1/data/file/{path}/meta	Metadata without content	DataFileMeta

Tree Response Schema

{
  "roots": [
    {
      "name": "data",
      "type": "dir",
      "children": [
        {
          "name": "checkins",
          "type": "dir",
          "children": [
            {
              "name": "2026-01-03.json",
              "type": "file",
              "size": 1234,
              "modified": 1704326400.0
            }
          ]
        }
      ]
    }
  ]
}

File Response Schema

{
  "path": "data/checkins/2026-01-03.json",
  "content": { "mood": 8, "energy": 7 },
  "is_jsonl": false
}

For JSONL files:

{
  "path": "data/identity/observations.jsonl",
  "content": [
    { "id": "obs-001", "text": "..." },
    { "id": "obs-002", "text": "..." }
  ],
  "is_jsonl": true,
  "total_lines": 847,
  "offset": 0,
  "limit": 100
}

Query Parameters

Param	Type	Default	Range	Description
offset	int	0	>= 0	Starting line for JSONL pagination
limit	int	100	1-1000	Lines per page

Security Model

The Data Browser implements defense-in-depth with a 10-step path validation:

1. Reject empty input
2. Decode URL encoding (with warning log)
3. Reject backslashes
4. Use PurePosixPath for consistent parsing
5. Reject absolute paths
6. Check for .. traversal BEFORE resolution
7. Verify first component in ALLOWED_ROOTS
8. Join with base and resolve
9. Verify resolved path still under base
10. Block symlinks in any component

Constants

Constant	Value	Purpose
ALLOWED_ROOTS	data, profile, goals, conditions, trends, insights, meta	Whitelisted directories
MAX_FILE_SIZE	10 MB	Prevent memory exhaustion
MAX_JSONL_LINES	1000	Max lines per request
MAX_TREE_DEPTH	10	Prevent deep recursion

Code Patterns

JSON Syntax Highlighting

The highlightJSONToHTML() function uses regex replacement instead of a parsing library:

// Pattern: escape HTML, then apply spans for keys, strings, numbers, booleans, null
const escaped = json.replace(/&/g, '&amp;').replace(/</g, '&lt;').replace(/>/g, '&gt;');
return escaped
  .replace(/"([^"\\]*(?:\\.[^"\\]*)*)"\s*:/g, '<span class=\'json-key\'>"$1"</span>:')
  .replace(/"([^"\\]*(?:\\.[^"\\]*)*)"/g, '<span class=\'json-string\'>"$1"</span>')
  // ... numbers, booleans, null

Why regex instead of a parser? Zero dependencies, smaller bundle, fast enough for typical file sizes. The trade-off is no semantic understanding (can't distinguish nested arrays from top-level).

Memoized Tree Rendering

TreeNode is wrapped in React.memo() to prevent re-renders when siblings change:

const TreeNode = memo(function TreeNode({ node, path, ... }) {
  // Only renders children when expanded (lazy rendering)
  {isExpanded && (
    <div className="tree-children">
      {visibleChildren.map((child) => <TreeNode key={child.name} ... />)}
    </div>
  )}
});

Drag-Resize Pattern

DataBrowserLayout handles resize with mouse events:

const handleMouseDown = useCallback((e) => {
  e.preventDefault();
  setIsDragging(true);
}, []);

useEffect(() => {
  if (!isDragging) return;
  const handleMouseMove = (e) => {
    const newWidth = e.clientX - containerRect.left;
    setSidebarWidth(Math.max(200, Math.min(480, newWidth)));
  };
  document.addEventListener('mousemove', handleMouseMove);
  document.addEventListener('mouseup', handleMouseUp);
  return () => { /* cleanup */ };
}, [isDragging]);

Why document listeners? Mouse can move outside the handle during drag. The overlay <div className="fixed inset-0"> prevents text selection.

Known Limitations

Limitation	Impact	Workaround
Read-only	Cannot edit files in browser	Use CLI or API for writes
No binary files	Images, SQLite not viewable	Open in native apps
O(n) line counting	Slow metadata for huge JSONL	Pre-compute line counts
No content search	Can only filter by filename	Use grep in terminal
No diff view	Cannot see file history	Use git log/diff
No keyboard nav	Mouse-only tree navigation	Planned improvement
URL-encoded paths	Special chars need encoding	Handled in encodePath()

Improvement Opportunities

These are prioritized by estimated impact and implementation complexity:

High Value, Low Effort

1. Keyboard navigation - Arrow keys for tree, Cmd+C for copy, Cmd+F to focus search
2. Add onKeyDown handler to FileTree
3. Track focusedPath in state

4. Effort: ~2 hours

5. Persist sidebar width - Store in localStorage

6. Read on mount, write on mouseup

7. Effort: ~30 minutes

8. Dark mode syntax theme - Already have CSS classes

9. Add .dark .json-key { color: #... } rules
10. Effort: ~1 hour

Medium Value, Medium Effort

1. Content search - Full-text search across files
2. Add /api/v1/data/search?q= endpoint using grep
3. Display results with file path + matching lines

4. Effort: ~4 hours

5. Lazy tree loading - Fetch children on expand

6. Add /api/v1/data/tree/{path} for subdirectory
7. Initial render just top-level

8. Effort: ~3 hours

9. Schema validation warnings - Highlight malformed JSON

10. Parse with try/catch, show error banner
11. Effort: ~2 hours

High Value, High Effort

1. Edit mode - Inline JSON editing with validation
2. ContentEditable or Monaco editor
3. POST to new /api/v1/data/file/{path} endpoint
4. Use atomic_write() for safety

5. Effort: ~8 hours

6. Git diff viewer - Compare with previous versions

7. Run git show HEAD~1:{path} via API
8. Side-by-side diff view

9. Effort: ~6 hours

10. Real-time updates - WebSocket for file changes

11. Watch filesystem with inotify
12. Push updates to connected clients

13. Effort: ~8 hours

14. Bulk export - Download multiple files as ZIP

    • Select files with checkboxes
    • Stream ZIP from API
    • Effort: ~4 hours