Performance harness and baselines

Measures the three surfaces CodeVetter can control so every optimization is proven against numbers.

CodeVetter is a local-first desktop tool. Its performance is won by not doing wasteful work, not by changing languages — the native side is already Rust and the dominant wall-clock cost (LLM calls) is network-bound and unfixable by us.

This harness measures the three surfaces we can control, so every optimization is proven against numbers instead of vibes. Measure → change → measure.

Running it

From apps/desktop/:

pnpm bench          # build + bundle budget + Rust benches (everything)
pnpm bench:bundle   # JS chunk sizes vs budget (needs a prior `pnpm build`)
pnpm bench:rust     # serialized index, graph, history, and FTS benches
pnpm qualify:graph     # enforced canonical-graph backend + UI data-path budgets
pnpm qualify:graph:browser # history-slider browser interaction qualification

The Rust benches are #[ignore]d (src-tauri/src/commands/perf_bench.rs) so they never gate normal cargo test. Comparison benches print tables without timing assertions. qualify:graph sets CV_ENFORCE_GRAPH_BUDGETS=1; on the calibrated Apple M5 Pro profile, the real-repository structural benchmark enforces the release envelope below. Shared release runners set CV_GRAPH_BUDGET_MODE=report-only, retaining correctness and resource measurement without treating variable hosted-runner timing as comparable. qualify:graph:browser likewise enables its absolute frame-time ceilings only outside report-only mode; normal browser CI still exercises every scrub input, final revision, accessibility label, and concurrent indexing state. The script forces one test thread so independent CPU, SQLite, and filesystem benches do not contaminate each other’s baselines. Bigger inputs:

cd src-tauri
CV_BENCH_MAX_MB=256 cargo test --release perf_bench::bench_index_parse -- --ignored --nocapture

Numbers below are machine-relative (captured 2026-06-19, Apple Silicon, release build). They are a baseline to diff against, not an absolute spec. Re-run on your machine before/after a change and compare deltas.

1. Session indexing — the headline cost

history.rs already skips files whose mtime is unchanged. The waste is in the append case: when a live session file grows, the whole file is re-read via std::fs::read_to_string and re-parsed. Parsing runs at ~400 MB/s:

It grows linearly with file size. On this machine the largest real transcript is 211 MB, so one ~4 KB append currently triggers a ~525 ms full re-parse.

bench_incremental_waste quantifies what an incremental byte-offset reader saves:

base file:        64 MB
full re-parse:    162.5 ms   (current cost per append)
incremental tail: 0.0104 ms  (4 KB only — target cost)
waste factor:     15,619x

At 211 MB the waste factor is ~50,000x.

✅ Fixed — incremental byte-offset indexer (v1.1.90)

cc_sessions now carries last_indexed_byte_offset + last_indexed_line_count. When an indexed file only grows, the indexer seeks to the saved offset, parses just the appended tail (up to the last newline, so half-flushed events are never indexed), and merges deltas into the session — appending archive rows with continued message_index/source_line, bumping day buckets, summing token totals, and recomputing cost from the new totals. A shrunk/rotated file falls back to a clean full reparse. (history.rs::index_adapter_session.)

Two guarantees, both tested:

• Correctness — incremental_index_matches_full_reindex_byte_for_byte proves an incremental index is byte-identical to a one-shot full re-index (totals, cursor, cost, every archive row, day buckets). file_shrink_falls_back_to_full_reparse covers rotation.

• Speed — bench_incremental_reindex_vs_full on a 23.5 MB indexed file:

  full reparse:       1275.9 ms   (old behavior, every append)
  incremental append:    2.114 ms (new behavior, 4 KB tail)
  speedup:             604x

  The gap widens with file size — the old path also rewrote all ~80k archive + FTS rows on every append; the new path writes only the handful that arrived.

cargo test --release bench_incremental_reindex_vs_full -- --ignored --nocapture

bench_index_parse above is unchanged — a cold first index is still linear (you must read the file once). The win is on every subsequent append.

2. FTS query latency

bench_query seeds 20,000 archived messages across 50 sessions and times the archive search users hit from the Roadmap page:

seeded:     20,000 rows across 50 sessions in ~343 ms
search avg: ~14.3 ms/query (limit 25, 200 iters)

But that 14 ms is the worst case: a term present in every one of the 20k rows, so bm25 ranks all of them. The number users actually feel is the selective case — a term matching a handful of rows:

worst case:   14.5 ms/query  (term in every row)
realistic:     0.05 ms/query  (selective term, ~25 matches) — ~300x faster

So real-world archive search is ~50 microseconds. There is no query problem to fix. datetime(a.timestamp) is only a tiebreaker (the primary sort is rank ASC), not the bottleneck, so changing it would buy nothing and risks reordering results. Left as-is, by measurement rather than assumption.

3. Frontend — desktop reality + render

This is a Tauri desktop app: the frontend loads from local disk, so network transfer and cache reuse are not the startup constraint. Route splitting still matters because JavaScript outside the entry route is not parsed, compiled, or rendered at startup. The useful startup metric is therefore the entry module’s static import closure plus the default Home route—not the sum of every lazy route.

• Bundle: 1,601 KB total / 445 KB gzip across all lazy routes, while the initial + Home closure is 452.8 KB raw. QuickReview, Repo, Settings, and AgentPanel remain lazy and do not block Home startup. The suspected “heavy” deps (react-markdown, @xterm/*, rehype-highlight, remark-gfm) are imported nowhere — dead dependencies, tree-shaken out of the bundle entirely. Removing them from package.json is install/supply-chain hygiene, not a runtime win.
• Render: QuickReview is already heavily memoized (79 memo hooks / 87 states). The one real inefficiency was the diff renderer: the parser joined hunk lines into a string and the render re-split them on every re-render. Fixed — hunks now carry pre-split lines (computed once in the memoized parse). Remaining opportunity (only if large diffs ever feel janky): virtualize the diff line list / wrap the per-file diff in React.memo. Deferred — speculative without a profile, and risky in a 6k-line file.

Bundle budget guard (bench:bundle)

Reads Vite’s manifest to compute the actual entry + Home static closure and fails if that exceeds 550 KB raw, if any individual chunk exceeds 500 KB raw, or if the complete lazy distribution exceeds 1,800 KB raw. This catches startup regressions without treating intentionally deferred code as startup work:

4. Release-history graph — backfill, time travel, and scrubbing

The temporal graph reads immutable Git objects without checkout, builds exact release/HEAD checkpoints, and stores commit-level materialization deltas. The history path is incremental in four places:

• changed revisions read only changed/deleted Git paths and reuse the previous structural snapshot;
• compatible cached deltas resume without rebuilding either side;
• historical source excerpts are omitted while path/line/column anchors remain;
• checkpoints and deltas are zlib-compressed in SQLite instead of duplicating a fully normalized graph for every commit.

flate2 is a deliberate native dependency here. The payloads are immutable, highly repetitive local JSON, and compression reduced the measured 24-commit database from about 1.59 GiB to 23.88 MiB without adding a service or network boundary.

Backend baseline

Captured 2026-07-13 on Apple Silicon against this repository in a release build, using a bounded 24-commit window (310 files, 19,055 nodes, 30,356 edges, two releases, four checkpoints):

The original full-snapshot implementation took about 95.2 seconds for the same 24-commit shape, produced about 1.59 GiB of SQLite data, and peaked near 1.89 GiB RSS. Storage and latency are now practical; peak memory during a cold long-lived backfill remains the main measured pressure point. History stays usable because backfill runs off the UI thread, publishes progress/coverage, supports cancellation, and makes HEAD/release checkpoints useful first.

The 2026-07-14 release qualification covers 445 files, 35,775 nodes, and 58,344 edges. Serialized full construction is 369.54 ms and a one-file refresh is 235.79 ms. Delete and rename repair on a deterministic repository fixture are 0.02 ms and 0.05 ms and assert that deleted/old paths leave no stale nodes. Snapshot transfer costs 25.51 ms, warm status is 1.5589 ms, persistence is 854.91 ms, cold SQLite hydration is 157.08 ms, and in-memory search is 0.1338/0.1481 ms p50/p95. The normalized SQLite graph consumes 82.97 MiB and the maximum sampled process RSS is 436.5 MiB. Candidate ordering, ambiguity, repair, and evidence semantics remain deterministic.

The 2026-07-18 candidate indexes 854 files into 81,307 nodes and 143,860 edges. It measured 1,189.64 ms full construction, 842.13 ms one-file refresh, 0.06 ms delete repair, 0.08 ms rename repair, 6.654 ms warm status, 3,479.25 ms persistence, 665.33 ms cold hydration, and 2.0978/2.4725 ms search p50/p95. The normalized database was 242.21 MiB and sampled peak RSS was 1,037.4 MiB.

The 2026-07-25 workbench rebaseline compared the same current graph engine against exact origin/main and release-candidate trees. The 844-file baseline produced 85,282 nodes, 150,686 edges, a 253.48 MiB database, and 1,109.9 MiB sampled peak RSS. The 876-file candidate produced 87,687 nodes and 154,932 edges; its repeated database measurement was 260.24 MiB and sampled RSS ranged from 1,136.1 to 1,194.6 MiB. Full build, incremental refresh, persistence, hydrate, and search remained inside their existing ceilings. The resource ceilings were therefore rebaselined to 272 MiB and 1,280 MiB for this measured source growth; this is not a larger-repository or asymptotic scaling claim.

The signed-release workflow runs this gate before the Tauri build. These are fixed ceilings for the current named-machine repository profile, with measured headroom over the candidate. They are a regression/resource envelope, not a claim about asymptotic scaling. Material corpus growth requires a separately recorded multi-size scaling run before any rebaseline:

The benchmark runner forces one test thread; its previous parallel execution introduced CPU/SQLite contention and produced incomparable numbers. The cold build ceiling includes headroom for the observed 1.19–1.91 second named-machine range while still catching a material regression above 2.2 seconds.

Query relevance uses the checked repository-owned structural-coverage-v1 fixture. It covers a cross-package Rust symbol-isolation case and a cross-file Swift extension case. Across three expected-answer queries, CodeVetter and the in-memory raw-text baseline both covered 3/3; CodeVetter ran at 0.0026 ms p50 / 0.0036 ms p95 versus raw search at 0.0004 / 0.0004 ms. On the current 81,324-node CodeVetter candidate, both covered 3/3 expected files; graph retrieval ran at 1.2433 ms p50 / 1.5140 ms p95 versus the preloaded raw-text scan at 0.8853 / 1.9763 ms. This does not claim universal ranking: graph retrieval was slower at the median and faster at p95 for this corpus and query set. Each latency result covers 200 iterations of three deterministic queries; the raw baseline excludes filesystem I/O so it does not make graph retrieval look artificially favorable.

cargo test --release perf_bench::bench_structural_graph_query_relevance -- --ignored --nocapture --test-threads=1

The causal query benchmark seeds 10,000 evidence events: 4.78 ms p50 / 5.12 ms p95, with a 7.24 MiB database. Re-run both backend benches from apps/desktop/src-tauri/:

CV_HISTORY_BENCH_COMMITS=24 cargo test --release bench_history_backfill_incremental_and_as_of_real_repo -- --ignored --nocapture
cargo test --release perf_bench::bench_history_causal_query -- --ignored --nocapture

UI budget

The deterministic data-path benchmark uses 1,500 nodes, 2,200 edges, 500 graph transitions, and 2,000 revisions:

• topology transition: 1.053 ms p50 / 1.174 ms p95 (8 ms p95 gate);
• bounded revision search: 0.186 ms p50 / 0.203 ms p95 (4 ms p95 gate);
• heap used: 26.5 MiB (64 MiB gate).

The Playwright scrub test delays mocked background indexing for 1.2 seconds and measures at least 40 animation frames while the slider changes. The latest calibrated local qualification measured 8.3 ms p50 / 10.2 ms p95 / 10.3 ms max, against enforced 50 ms p95 / 120 ms maximum bounds. Shared hosted runners report these timings while keeping deterministic interaction assertions enforced. This is a browser-level responsiveness proxy; the Rust benchmark above separately measures native backfill CPU, memory, and I/O.

Production Chrome audit

Captured 2026-07-14 from the optimized Vite output on an unthrottled local Chrome session. The browser preview loads the same route chunks as the packaged application; Tauri serves them from local application assets instead of the temporary loopback preview server.

All requests stayed local, with no image, font, or third-party-origin startup requests. Chrome attributed 0 ms estimated FCP/LCP savings to the stylesheet and found no useful preconnect opportunity.

The interaction trace started a 1.2-second history backfill and scrubbed 60 revision inputs concurrently. It observed 27 ms INP (1 ms input delay, 6 ms processing, 19 ms presentation), 0 CLS, and no estimated interaction savings. The scrub itself produced 58 measured frames at 8.3 ms p50 / 9.3 ms p95 / 9.3 ms max, and the 96-node structural projection remained rendered. This audit also exposed and fixed a foreground/prefetch request-serial race that could leave the graph stuck on loading revision; the Playwright contention test now asserts that all 96 nodes render before measuring the slider.

pnpm bench:history-ui
pnpm exec playwright test tests/e2e/repo-unpacked.spec.ts

Coverage is intentionally explicit: history is bounded to the requested recent commit limit; shallow repositories, unsupported languages, missing Git objects, and parser failures remain visible as gaps. Mandatory reachable-release and HEAD checkpoints are exact for their recorded coverage. Intermediate states reconstruct from ordered materialization deltas and fall back to exact Git-object extraction when a compatible chain is unavailable.

5. Local MCP sidecar

pnpm bench:mcp builds the release binary, creates an isolated WAL-mode fixture database, launches the real stdio process, verifies zero TCP listeners and zero target-repository mutation, then measures initialization and three progressive query shapes. Captured 2026-07-14 on Apple Silicon after caching HEAD and the release-tag fingerprint together for one second while retaining per-request enablement checks:

The long-lived fixture contains 10,000 evidence events in a 0.92 MiB database; the release binary was 7.04 MiB and idle RSS was 12.31 MiB. The earlier small fixture measured 5.30 ms / 6.18 ms cold initialize p50/p95, 1.37–1.40 ms warm p50, and 12.22 MiB idle RSS. The first scoped read after one idle second refreshes Git HEAD and release tags; subsequent queries reuse both while every request still rechecks repository enablement. A regression that recomputed the tag fingerprint through Git on every result raised warm p50 to about 9 ms; sharing the bounded freshness cache restored 2.16–2.34 ms p50 without weakening live disable or tag-aware staleness. One of 25 launches was a 442.81 ms cold outlier; p95 remained 7.90 ms.

bench:mcp always fails on listeners, repository mutation, protocol framing errors, or query failures. Its hardware-specific latency and memory ceilings apply only on the named Apple M5 Pro profile and are listed with the current qualification below; the sidecar binary ceiling remains 10 MiB.

Rust remains the implementation choice: a Go sidecar would duplicate the canonical Rust query contracts or pay an IPC hop, while the measured native path is already roughly 2.2 ms warm with a small standalone footprint. See MCP-SDK-EVALUATION.md for the full dependency and Rust-versus-Go decision.

6. Local history MCP

The MCP benchmark uses a separate temporary Git repository and SQLite database; it never writes to the repository being protected. The qualification fixture has 65 commits, 64 tagged releases, 10,000 history events, 512 structural nodes, and 1,024 edges. Before timing, the harness verifies strict read-only schemas, non-empty graph/history/evidence results, complete resource pagination, redaction, the 256 KiB response ceiling, zero TCP listeners, and unchanged protected-repo HEAD/status.

Run from apps/desktop/:

pnpm bench:mcp:smoke            # quick correctness check; never enforces budgets
pnpm bench:mcp                  # full named-machine qualification
pnpm bench:mcp --skip-build     # reuse an already-built release sidecar

Qualification refreshed 2026-07-18 on an Apple M5 Pro with a release sidecar, 3 process warmups, 50 recorded starts, 10 workload warmups, and 200 recorded rounds. The sidecar now exposes 22 schema-validated tools; each round includes five individual read workloads plus a true four-request concurrent batch.

The 8.90 MiB sidecar finished at 33.92 MiB RSS and grew 3.16 MiB across the second half of the recorded rounds. The fixture database shape is unchanged and the process opened no TCP listeners. Compared with the earlier 13-tool profile, the broader 22-tool schema and result surfaces cost latency and binary/RSS headroom; the table records that regression rather than carrying forward the older measurements.

Absolute gates apply only to the named Apple M5 Pro qualification profile: initialize 25 ms p95; every individual query 8 ms p50; graph query 12 ms p95; release list and broad history 15 ms p95; evidence hydration and resource list 10 ms p95; mixed concurrency 22/30 ms p50/p95; final RSS 36 MiB; second-half growth 8 MiB; binary 10 MiB. Other machines still run every correctness and safety check but report timings without claiming that these hardware-specific gates passed.

7. Warm local browser verification

The warm-verification qualification uses the checked-in 20-scenario manifest, one persistent loopback target, and one persistent Playwright Chromium process. Each recorded invocation includes exact Git worktree collection, deterministic capability selection, 20 fresh browser contexts, automatic observation, reporting, and context teardown. Intentional observer-negative fixtures remain in correctness tests and are excluded from timing samples.

Run from apps/desktop/:

pnpm bench:verify

The 2026-07-15 qualification on the Apple M5 Pro used Chromium revision 1217, two excluded warm-up batches, and 20 recorded batches. Cold harness startup was 1054.265 ms (148.949 ms browser launch; 845.355 ms Vite server readiness). The qualification target runs React through Vite and installs client-scoped named state through the real MSW state bridge. Vite’s HMR client and target modules were ready in 787.844 ms before a recorded 250 ms settle window completed.

Parallelism 4 is therefore the fastest stable default on the recorded machine. The independent 20-sample gate at that setting passed with 3605.560 ms p50, 4792.196 ms p95, and 5320.379 ms max, against the required p95 below 30 seconds.

The machine-readable report at tests/fixtures/warm-verification/qualification-2026-07-17.json preserves all 20 invocation durations, target/config/manifest identities, exact benchmark and app source hashes, machine and browser details, cold startup, HMR conditions, parallelism profiles, and per-stage summaries. Per-scenario stage values are summed work time and can overlap under parallel execution; whole_invocation is the wall-clock release gate.

The normal small changed-capability path is measured separately with one exact mapped scenario; it does not replace or relax the 20-scenario release gate. Run:

pnpm bench:verify:stability

After two warm-ups, 20 whole focused invocations recorded 506.426 ms p50, 512.035 ms p95, and 515.900 ms max. The focused regression budget is 2000 ms, leaving operating headroom while remaining materially tighter than the independent 30-second full-corpus gate.

The same command executed 100 additional warm batches: 80 passes, 10 intentional deterministic regressions, and 10 cancellations triggered only after scenario execution started. Every batch closed all contexts and retained the same Vite and Chromium identities. Peak Node RSS grew 13,582,336 bytes against a 134,217,728-byte budget; second-half median RSS did not grow. Retention finished at its 20-run cap using 4470 bytes, below its 104,857,600-byte cap. The measured path recorded only its 110 required Git subprocess calls and zero Cargo, Tauri, or production-build invocations. Its raw samples, exact source hashes, resource gates, command audit, and temporary-root cleanup proof are in tests/fixtures/warm-verification/stability-2026-07-17.json.

8. Warm-verification implementation growth

The third cleanup gate measured the complete warm-verification surface against 75f1deb1, the parent of the first runtime implementation commit. These are source-line changes, not bundle size:

The number is intentionally reported rather than described as small. It includes 5688 lines of unit tests plus checked-in browser fixtures and raw qualification evidence. The production core is still substantial and should not grow by copying another runtime or control surface.

The cleanup removed the unused review-specific warm-run column/filter/index, the backend run-ID fallback, a duplicate current-identity type, an unused CLI error field, and a 20-row T-Rex read where only the newest row was rendered. It also found that the existing projection adapter was test-only; deleting it would have hidden an incomplete spec. The adapter is now used by a bounded read-only Review history, timeline, same-flow comparison, and historical execution-finding surface without duplicating legacy QA rows, preferences, controls, or persisted review-finding indices. That correction made the cleanup slice net +122 lines across 14 feature files (+228/-106) while closing the missing production path.

9. Agent-facing runtime diagnosis (experimental)

The repository CLI can profile one exact Node test, standalone Node benchmark script, Vitest test, or Go benchmark and turn the bounded capsule into a deterministic agent-facing diagnosis. This is separate from the desktop performance harness above: it is for understanding a target application’s runtime path.

For Node and Vitest workloads, the local flow lane packages this engine as a machine-queryable product capability rather than a profiling playbook:

pnpm --silent runtime:capture-flow -- \
  --repo /path/to/project \
  --adapter node-test \
  --target test/http.test.mjs \
  --name "exact request flow" \
  --samples 3 \
  --warmups 1 \
  --json

pnpm --silent runtime:mcp -- --repo /path/to/project

--silent is required when pnpm is the launcher because stdout is the CLI JSON or MCP protocol stream. MCP clients may instead execute node scripts/runtime-failure-capsule/mcp.mjs --repo /path/to/project directly.

The repository-scoped local MCP exposes four closed operations: capture_local_flow, inspect_local_flow, explain_local_flow, and verify_local_optimization. Captures are held in memory for the current MCP session, capped at eight, and use opaque IDs. The MCP does not accept arbitrary commands or repository paths after startup.

One unprofiled pass supplies the root timing distribution, one separate diagnostic pass observes loopback fetch, Node HTTP server activity, and request-scoped built-in node:sqlite operations, and two independent V8 profile passes test source-candidate repeatability. A separate bounded V8 coverage pass records repository-owned named function call counts without arguments, return values, or duration. Node tests use raw V8 coverage; Vitest uses its repository-local V8 provider so transformed modules map back to original TypeScript. An AsyncLocalStorage request context nests SQLite operations beneath the server flow; startup SQL outside a request is ignored. The capsule records operation, normalized SQL shape, outcome, and duration, but never bind arguments or returned rows.

HTTP query strings are discarded, variable-looking path segments and SQL literals are normalized, and all owned raw artifacts are deleted after normalization. Child interval union provides accounted and unaccounted time inside one diagnostic request without double-counting concurrent children. Because the root timing comes from different executions, child durations are still not subtracted from the root median.

A V8 source candidate is actionable only when both profile passes select the same repository-owned file/function and each pass contributes at least five samples, 10 ms self time, and 10% sample share. Weaker evidence returns no_confidence with a scale-up experiment. Optimization comparison separately reports whether movement is mechanically confirmed, materially useful, and strong enough to recommend shipping. Function frequency becomes a repeated-work candidate only when its source range intersects CPU evidence. Frequency alone never assigns duration. The MCP verification response returns a compact comparison and the two capture IDs; it does not duplicate the complete stored capsules.

Vitest leaf names work inside nested describe blocks. CodeVetter accepts the scope only when exactly one assertion executes; a missing or duplicate leaf name fails closed. If the repository does not provide Vitest’s V8 coverage provider, function frequency is reported as unavailable while independently complete timing and CPU evidence remain usable.

pnpm runtime:diagnose-performance -- \
  --repo /path/to/project \
  --adapter vitest \
  --target src/recommendations.scale.test.ts \
  --name "recommendation catalogue scale profiles representative catalogue sizes" \
  --samples 3 \
  --warmups 1 \
  --json

Supported adapters are node-test, node-script, vitest, and go-bench. The target must be a repository-contained file and the optional workload name is applied as an exact selector. node-script accepts only .js, .mjs, or .cjs, supplies no caller arguments, and does not describe exit success as a correctness check. Profiling uses local runtime primitives and removes its owned temporary profiles after normalization.

pnpm --silent runtime:diagnose-performance -- \
  --repo /path/to/project \
  --adapter node-script \
  --target benchmarks/parser.mjs \
  --samples 3 \
  --warmups 1 \
  --json

Failed performance passes retain only bounded redacted operational error, stdout, and stderr on the exact failed execution. Every execution also reports whether bounded output confirmed the requested workload. This distinguishes a workload failure from a runner/profiler incompatibility without retaining raw profiles or successful-run logs.

The runtime-performance-diagnosis/v1 result separates:

• observed: ranked measurements, scale curves, and repository source samples;
• inferred: deterministic interpretations that reference observation IDs;
• unverified: hypotheses with an explicit falsification step;
• next_action: one bounded experiment instead of a list of speculative fixes;
• verification: the identical adapter, target, name, and sample policy for a before/after comparison;
• performance_capsule: the complete bounded source evidence.

For explicit ms/op scale metrics, the diagnosis also applies an absolute-cost guardrail. If the largest recorded supported input is at or below 0.1 ms/op, CodeVetter keeps the workload as a regression baseline but does not recommend source optimization merely because the profiler can identify a hot function.

For runtime-selected repository locations, the diagnosis also includes a redacted bounded source window and exact matched pattern lines. The first pattern family recognizes full sorting before a bounded slice, eager mapping, repeated traversal of the same source collection, and nested collection lookups. Function anchors are corrected against the original TypeScript source when a transformed V8 location points above the declaration. These observations influence a hypothesis only when they intersect runtime evidence; CodeVetter does not run a repository-wide performance linter.

Node and Vitest console benchmark metrics are captured in samples independent, unprofiled executions and normalized by median. The profiled execution remains separate for CPU attribution. This prevents profiler overhead or one noisy host interval from confirming an optimization.

V8 source attribution requires the same repository-owned candidate in two independent profiles plus minimum sample and self-time evidence. A candidate can be material relative to the full profile, or—when test-runner and dependency frames dominate—its file can own a majority of the captured application CPU as long as application work itself clears a bounded total-profile floor. The report records which materiality mode qualified the candidate.

Use /performance_capsule from the diagnosis as the baseline JSON for the next run. A diagnosis never edits source or invokes a model. actionable means the measurements identify a candidate worth testing; it does not mean the candidate has been proven causal. A startup-dominated or incomplete run asks for better evidence instead of naming application code.

After changing one candidate, verify the identical scope against either the saved performance capsule or the full saved diagnosis:

pnpm runtime:verify-optimization -- \
  --repo /path/to/project \
  --adapter vitest \
  --target src/recommendations.scale.test.ts \
  --name "recommendation catalogue scale profiles representative catalogue sizes" \
  --baseline artifacts/baseline-diagnosis.json \
  --samples 3 \
  --warmups 1 \
  --json

runtime-optimization-verification/v1 compares matching scale inputs and units, or matching Go benchmark timing and allocation metrics. It returns confirmed, rejected, inconclusive, or no_confidence; a failed or incompatible workload can never confirm an optimization. Explicit in-workload benchmark metrics are compared independently from process startup timing, so noisy runner wall time cannot override a stable exact-scope ms/op series.

When both revisions are independently runnable, prefer the paired lane:

pnpm runtime:verify-paired-optimization -- \
  --baseline-repo /path/to/baseline-checkout \
  --repo /path/to/candidate-checkout \
  --adapter node-test \
  --target test/performance.test.mjs \
  --name "exact performance workload" \
  --samples 3 \
  --warmups 1 \
  --json

The two repositories must be distinct, contained runnable checkouts with the same relative target and exact workload name. CodeVetter alternates baseline and candidate order for each sample, records paired_schedule, and labels the report evidence_mode: paired_interleaved. Any failed run or missing comparable metric yields no_confidence. This operation captures timing/domain metrics only; use diagnose-performance separately for source attribution.

Explicit Playwright flow evidence

An existing repository-owned Playwright test can be selected explicitly as a local performance scope. Qualification lists browser candidates but never declares one representative automatically:

pnpm runtime:profile -- \
  --repo /path/to/react-app \
  --adapter playwright \
  --target tests/checkout.spec.ts \
  --name "completes checkout" \
  --samples 3 \
  --warmups 1 \
  --json

The capsule records the exact test duration reported by Playwright separately from owned-process wall time. Missing, failed, retried, ambiguous, or truncated reporter evidence yields no_confidence. Paired Playwright verification uses the same scope with runtime:verify-paired-optimization; direct comparisons require at least three samples, while campaign promotion retains the ten-sample floor. Confirmation requires at least 10% and 10 ms median test-duration improvement with no process-wall regression above 20%.

An existing Vite output can be inspected without running a build:

pnpm runtime:profile -- \
  --repo /path/to/react-app \
  --adapter playwright \
  --target tests/checkout.spec.ts \
  --name "completes checkout" \
  --vite-build-dir dist \
  --vite-entry index.html \
  --json

Only a bounded initial HTML/static-JavaScript closure is read. Raw and gzip bytes are always labelled existing_unverified_vite_artifact; they cannot confirm or keep an optimization because CodeVetter did not run or attest the build. MCP exposes the same closed inputs as profile_local_performance and verify_paired_performance; neither accepts commands or installs packages.

This compact lane deliberately does not restore the retired general browser runtime. It does not capture Chrome traces, Core Web Vitals, React commits, browser memory, request waterfalls, production traffic, or automatic patches. Those gaps remain explicit in every Playwright performance capsule.

10. Official 1BRC Node artifact

The repository-owned Node adaptation of the official One Billion Row Challenge provides enough application work for the runtime tools without a production database or giant fixture:

pnpm bench:1brc

It deterministically generates 20,000, 200,000, and 800,000 station;temperature rows, checks exact count/min/max/sum aggregates, validates the official sorted min/mean/max output contract with UTF-8 station names, and emits the existing size<N>=<duration>ms/op contract. Dataset construction is outside the timed region. The benchmark’s README contains the exact diagnosis command, while its artifact record records attribution and the differences from the official Java challenge.

The initial qualification used CodeVetter to select the parser at 77.64% CPU share before source inspection. Replacing per-row string splitting with a single cursor pass reduced the 800,000-row median from 118.928 to 35.744 ms/op, and the identical-scope verifier confirmed a 69.945% improvement. A second micro-optimization improved the largest case by only 8.863% and regressed the smallest by 4.079%; it was reverted after the verifier returned inconclusive. See the active OpenSpec qualification for the complete evidence and limitations.

The challenge is not an official submission and does not claim one- or nine-billion-row performance. Its current boundary is one in-memory generated string at 800,000 rows. The next useful product lane is chunked input plus observed peak RSS and bytes processed, not an extrapolated completion time.

11. CodeVetter-on-CodeVetter qualification

The runtime tools also have a permanent self-profiling workload around V8 function-coverage normalization:

node --test scripts/runtime-failure-capsule/function-coverage-performance.test.mjs

The synthetic ten-sample stress fixture grew from 1.138 ms/op at 80 source anchors to 405.185 ms/op at 3,200. CodeVetter localized 56.54% of repository-owned CPU samples to collectV8FunctionCoverage and captured its repeated source-offset scan. A line-start index plus binary-search lookup reduced the 3,200-anchor measurement to 4.830 ms/op, a 98.808% improvement for that deliberately adversarial one-file fixture while exact source-line output remained unchanged. This is not an end-to-end or representative CodeVetter speedup.

A follow-up replay used 91 real V8 coverage documents emitted by the full runtime suite. After filtering the collector’s own coverage document so source offsets remained comparable, 11-run medians improved from 39.873 to 32.652 ms, or 18.11%, with identical normalized function-output SHA-256. The representative result is therefore 18.11% for the coverage-normalization stage; 98.808% remains only the worst-case regression fixture.

The self-trial is also an accuracy fixture. CodeVetter’s first inference blamed a bounded coverage-filename sort even though the workload held file count at one. That hypothesis was rejected from the captured workload identity before the observed offset helper was optimized. The active OpenSpec qualification records the complete evidence, the conservative shipping limitation, and the decision-explanation bug found and fixed during the loop.

A later four-project robustness pass found one additional material candidate: qs flat query parsing improved 45.086% at 40,000 parameters in a ten-pair interleaved run and passed all 1,045 upstream tests. Picomatch and Pixelmatch candidates stayed below the 10% materiality policy or regressed, while GJSON’s selected benchmark already reported zero allocations and no material bottleneck. These negative results are part of the product evidence, not failed qualification runs.

12. Autonomous local optimization campaigns

An optimization campaign turns the single-run profiling primitives above into a resumable local laboratory. The coding agent still proposes and applies one bounded source edit. CodeVetter owns the immutable evaluation scope, correctness-first execution, screening verdict, paired promotion, incumbent, budgets, and tamper-evident history.

Create .codevetter/optimization-campaigns/<id>/manifest.json in the target repository before initialization. The manifest is closed and versioned:

{
  "schema_version": "optimization-campaign-manifest/v1",
  "campaign_id": "parser-loop",
  "repository_revision": "0123456789abcdef0123456789abcdef01234567",
  "artifact_directory": ".codevetter/optimization-campaigns/parser-loop",
  "allowed_files": ["src/parser.js"],
  "correctness": [
    {
      "adapter": "node-test",
      "target": "test/parser.test.js",
      "name": "preserves parser behavior",
      "timeout_ms": 10000
    }
  ],
  "performance": {
    "adapter": "node-test",
    "target": "test/parser.performance.test.js",
    "name": "profiles representative parser input",
    "timeout_ms": 30000,
    "screening": { "samples": 3, "warmups": 1 },
    "promotion": { "samples": 10, "warmups": 1 }
  },
  "budgets": {
    "max_experiments": 12,
    "max_elapsed_minutes": 120,
    "max_consecutive_non_improvements": 4,
    "max_consecutive_crashes": 2
  }
}

Evaluator targets cannot overlap writable allowed_files. Unsupported adapters, ambiguous test selection, changed manifests, escaping paths, out-of-scope diffs, incomplete evidence, or altered ledger/evidence digests all fail closed. Screening can only return promising; only correctness-preserving, limitation-free, ten-sample paired evidence can return keep and advance the incumbent.

pnpm --silent runtime:campaign -- init \
  --repo /path/to/candidate \
  --campaign .codevetter/optimization-campaigns/parser-loop --json

pnpm --silent runtime:campaign -- baseline \
  --repo /path/to/candidate \
  --campaign .codevetter/optimization-campaigns/parser-loop --json

pnpm --silent runtime:campaign -- screen \
  --repo /path/to/candidate \
  --campaign .codevetter/optimization-campaigns/parser-loop \
  --hypothesis "Remove one redundant allocation without changing output." --json

pnpm --silent runtime:campaign -- promote \
  --repo /path/to/candidate \
  --incumbent-repo /path/to/independent-incumbent \
  --campaign .codevetter/optimization-campaigns/parser-loop \
  --hypothesis "Remove one redundant allocation without changing output." --json

The same six campaign operations are available from runtime:mcp. Start that server with --repo /path/to/candidate --incumbent-repo /path/to/incumbent so the promotion checkout is fixed outside tool arguments. The checked-in agent program defines the loop and its authority boundary.

Campaign artifacts are local JSON under the declared directory: one manifest, an atomic append-only ledger.ndjson, and bounded evidence blobs. Status is derived from those records, so restarts cannot reset experiment, elapsed-time, plateau, or crash budgets. Recovery means fixing the local checkout or reproducing the incumbent in a separate worktree and calling status again; do not edit the ledger. The initialization record also pins a digest of the CodeVetter evaluator implementation; an evaluator upgrade can inspect old history but must start a new campaign before making new decisions. Removing a finished campaign is a manual local cleanup choice. The engine installs nothing, invokes no production or cloud endpoint, and never commits, pushes, resets source, or manufactures a checkout.

Candidate challenge and contribution closeout

A campaign keep proves only the declared local scopes. Commit the candidate before paired promotion so the kept record names the future PR head; uncommitted source at challenge time fails closed. Before publishing, challenge that exact retained SHA:

pnpm --silent runtime:campaign -- challenge \
  --repo /path/to/candidate \
  --campaign .codevetter/optimization-campaigns/parser-loop \
  --selected-sequence 4 \
  --justification "The direct lookup adds no cache or fallback state." --json

The challenge binds the kept record, current commit, diff digest, complexity, and deterministic risk observations. It requires either a directly comparable qualified candidate or a bounded reason that a simpler comparison is not applicable. Candidate comparison retains the largest scale or latency value as the target and treats smaller scale points, bytes/op, and allocations/op as controls when they exist. The simpler candidate must stay within the same 5% tolerance on every retained metric. Risk tokens request evidence; they are not a model score and cannot offset correctness or performance.

Inspect one published pull request with the returned challenge path:

pnpm --silent runtime:campaign -- inspect-contribution \
  --repo /path/to/candidate \
  --campaign .codevetter/optimization-campaigns/parser-loop \
  --challenge .codevetter/optimization-campaigns/parser-loop/closeout/challenge-<sha>-<digest>.json \
  --pr https://github.com/owner/repository/pull/123 \
  --trex-policy optional --json

trex-policy is optional, required, or not_applicable. A supplied T-Rex preview receipt is read from a contained local path and must identify the same candidate SHA. CodeVetter does not launch T-Rex, a browser, preview, or hosted load from this operation.

The contribution receipt keeps correctness, performance, patch quality, T-Rex, head freshness, checks, review threads, and merge authority as independent gates. It distinguishes failed checks from fork-workflow approval, retains outdated and resolved feedback without treating either as current work, and reports waiting_for_maintainer when upstream owns merge. Use refresh-contribution for one explicit reread; there is no polling.

The append-only receipt also carries bounded maintainer-feedback learning when a reviewed candidate is superseded: exact before/after SHAs and complexity, the actionable thread, prior deterministic risk signals, the revised campaign hypothesis, repeated local gates, and observed upstream disposition. This is evidence from one contribution, not a universal style rule.

closeout/publication.json is a concise projection of the newest current receipt. A different PR head marks an existing projection stale while keeping its original receipt digest; only a newly challenged, current receipt can regenerate it. The receipt ledger remains the authority.

The GitHub adapter uses one fixed read-only GraphQL query. It cannot comment, resolve a thread, request review, approve, merge, deploy, install an app, create a required check, or change pull-request metadata. Raw receipts remain local; an author can copy a concise summary separately.

The challenge, inspection, and refresh operations are also available through runtime:mcp; their repository is fixed when the server starts.

13. Qualifying a workload before profiling

Runtime detection proves that a repository contains a supported toolchain; it does not prove that an arbitrary test is representative enough to profile. Use the read-only qualification planner before selecting a target:

pnpm --silent runtime:qualify -- --repo /path/to/repository --json

The planner reads bounded manifests and test source without running project code. It returns package-scoped candidates, exact literal workload identities, score evidence, possible external-operation flags, and one of ready, needs_selection, no_representative_workload, unsupported, or inaccessible. Only ready includes a recipe for the existing profiler; even then, the result is a candidate and not proof of production representativeness.

A portfolio scan uses an operator-owned manifest so CodeVetter does not assume a sibling-directory layout or compile private project paths into the product:

{
  "schema_version": "runtime-qualification-portfolio-manifest/v1",
  "repositories": [
    { "id": "catalog-api", "path": "../catalog-api" },
    { "id": "web-client", "path": "../web-client" }
  ]
}

pnpm --silent runtime:qualify-portfolio -- --manifest ./portfolio.json --json

Entries are inspected sequentially, output preserves manifest order, and absolute paths are omitted. The operation does not install dependencies, start services, use a browser, contact databases or networks, or execute the selected workloads. Agents can also call the read-only MCP operation qualify_runtime_repository before capture_local_flow or an explicit performance profile.

14. Preserving a profiling attempt under failure

Use the outer supervisor when a workload may crash, receive a signal, exceed its deadline, or produce unusable output. The supervisor writes an initialized receipt before it launches the existing diagnosis pipeline, updates a bounded heartbeat, and atomically records one terminal state:

pnpm --silent runtime:supervise-performance -- \
  --repo /path/to/repository \
  --run-id parser-baseline-01 \
  --adapter node-script \
  --target benchmark/parser.mjs \
  --samples 3 \
  --warmups 1 \
  --timeout-ms 30000 \
  --json

pnpm --silent runtime:inspect-performance-run -- \
  --repo /path/to/repository \
  --run-id parser-baseline-01 \
  --json

Artifacts live under .codevetter/performance-runs/<run-id>/. receipt.json is atomically replaced as lifecycle evidence changes; a successful run also owns one validated, digest-addressed result.json. Starting a duplicate run ID fails closed. The start command exits 0 only for succeeded and 2 for every complete but non-success state. Inspection is read-only and is also available through the MCP tool inspect_performance_run, whose repository is fixed when the MCP server starts.

The supervisor accepts only existing closed adapters and their exact target and test-name scope. It does not accept arbitrary executable arguments, inherit application secrets, install dependencies, contact cloud services on its own, or alter source control. Child output is byte-bounded and redacted before it can become failure evidence; successful JSON is redacted, schema-validated, and hashed before preservation.

Recovery is deliberately conservative. A killed profiling child, timeout, ordinary exit, spawn failure, or malformed result gets a terminal receipt and authorizes no performance conclusion. If the supervisor or entire machine dies, the last initialized/running heartbeat remains, but this version does not reconcile it automatically or resume partial profiling. Local CPU and memory limits still come from the workload and per-execution timeout; OS-level resource budgets are future work.

15. Choosing the next local flow

Qualification finds runnable workloads; the flow-campaign planner screens a bounded safe subset and decides which one deserves an optimization campaign:

pnpm --silent runtime:plan-flow-campaign -- \
  --repo /path/to/repository \
  --max-flows 3 \
  --samples 3 \
  --warmups 1 \
  --timeout-ms 30000 \
  --json

Automatic screening requires direct benchmark timing. Remote-network, database, browser, integration, and required-argument signals remain excluded; loopback-only HTTP may run through the closed Node or Vitest adapters. Workloads that merely contain URL fixture data remain eligible unless they invoke a network client. Workloads execute sequentially so their CPU profiles do not contaminate one another. Raw profiles are removed after normalization.

The planner ranks only comparable application metrics: the largest explicit ms/op input or Go ns/op converted to milliseconds. Runner wall time is not treated as product latency. A project can add reviewable product context:

{
  "schema_version": "runtime-flow-priority-manifest/v1",
  "flows": [
    {
      "candidate_id": "0123456789abcdef",
      "frequency_weight": 8,
      "user_impact_weight": 5,
      "rationale": "Runs for every interactive request."
    }
  ]
}

Pass it with --priority-manifest path/to/priorities.json. The deterministic ranking is supported_scale_ms × frequency_weight × user_impact_weight. Missing entries use neutral weights and remain explicitly unverified; the planner never fabricates production frequency. An already-fast flow stays as a regression guardrail, while the highest-ranked actionable flow becomes the one handoff to runtime:campaign. MCP clients can invoke the equivalent closed plan_flow_optimization_campaign tool.

Principle

A feature is on-budget when it doesn’t make the app re-do work proportional to data it has already seen, and doesn’t grow the initial payload without cause. The benches encode that: re-reading 211 MB for a 4 KB append is the canonical thing we refuse to keep doing.