← CodeVetter

Verification field guide

AI code review vs. coding-agent verification

Validate AI-generated code with a runnable failure-and-fix check. Compare review with executable verification and inspect the evidence.

Try an AI code validation check

Requirement: preserve an explicit false setting, the default, and the existing label. A truthiness fallback (input.enabled || true) breaks that requirement; a nullish fallback (input.enabled ?? true) preserves it.

The published synthetic task repeats the intended baseline failure twice and the known-good pass twice. Inspect the executed qualification receipt, then run the same failure-and-fix check locally.

The receipt qualifies this task and its checks; it does not measure an agent's success rate.

AI code review and coding-agent verification solve related but different problems. Review searches a change for risks and explains suspicious code. Verification determines whether the requested behavior works and preserves evidence for the verdict. Teams using coding agents usually need both.

Question AI code review Agent verification
Primary input Diff and repository context Task, exact change, environment, and checks
Main output Findings and explanations Pass, fail, or unverified with evidence
Strong at Breadth, suspicious patterns, maintainability clues Reproducing behavior and proving closure
Main limitation A plausible finding may not reproduce A check can miss risks outside its behavioral boundary
Best use Risk discovery and reviewer focus Acceptance, regression protection, and auditability

Why review alone is not enough

A reviewer can notice an authorization branch that looks unsafe, but only an authoritative check can show whether an unauthorized request is accepted. A model can praise a state update while a browser journey still loses user input. Review remains valuable because it can propose where to test; it should not be promoted into runtime proof.

Why tests alone are not enough

A green suite can be irrelevant to the requested task, stale, or incomplete. Verification strengthens test output by binding it to the task, revision, environment, and expected behavior. It also preserves explicit unknowns instead of treating an absent check as a pass.

A practical combined loop

  1. Review the diff and repository context to identify risk.
  2. Translate material risks and acceptance criteria into focused checks.
  3. Run repository-owned tests plus the smallest missing behavioral check.
  4. Package findings and execution results separately.
  5. Re-run after fixes and retain the before-and-after evidence.

CodeVetter's current direction is this combined evidence loop, with execution-backed verification as the authority. The public recognition benchmark remains useful for one review dimension and openly states that it does not prove production PR performance.

A public-repository case

Inspect a React hydration visibility regression: one CodeVetter review finding, two failing visibility cases before repair, and four passing checks afterward. The article links pinned public source, a redacted evidence summary and a focused replay. This is one observed case, not a catch-rate benchmark.

Worked verification recipes

These worked recipes run public synthetic task fixtures. Each retained qualification repeats the intended baseline failure twice and the supplied known-good pass twice, with complete cleanup. The linked receipts were recorded on 14 November 2026 (UTC) with Node.js 24.22.2. They qualify the task and checks; they do not measure an agent, the installed app or production reliability. Qualification runs trusted corpus code without an operating-system sandbox.

Use Node.js 24 or newer. These qualification drivers use Node built-ins; no package install, coding agent or provider key is required. In a fresh clone:

git clone https://github.com/Codevetter/codevetter.git codevetter-recipes
cd codevetter-recipes
git switch --detach 28ee304927cd431966588136d099cf7659b03087

Then run the task-specific command below from that checkout. Read the corpus guide for the full contract.

Turn a suspicious HTTP status into a reproducible finding

Requirement: A response normalizer must retain a non-success upstream status without changing its body or headers.

Exercise the boundary: Pass normalizeResponse an upstream result with status 422, a retained headers object and a retained body object containing error: invalid. Require output.status === 422, output.body === body and output.headers === headers. The identity checks protect forwarding behavior as well as the status requirement.

Starting behavior:

return { status: 200, headers: response.headers, body: response.body };

Supplied known-good replacement (relevant fragment):

return { status: response.status, headers: response.headers, body: response.body };

Review can flag the hardcoded 200 and explain why it looks risky. Execution narrows that concern to an observed contract failure: this 422 becomes 200 before repair, then stays 422 after repair. Keep the review finding and the check result as different evidence types. Neither needs to stand in for the other.

node scripts/agent-task-corpus/qualify-cli.mjs --task preserve-upstream-http-status --json

Inspect the pinned task, driver and acceptance checks · Read the observed qualification receipt. Expected qualified: true, baseline.status: intended_failure, known_good.status: pass and cleanup.status: complete.

Boundary of this evidence: The check invokes a normalizer, not a live HTTP server. It does not establish that a framework emits the returned status on the wire. Verify the real endpoint and its client-visible response when the changed boundary includes HTTP serialization or middleware.

CTA: Start with the verification workflow, then inspect the public benchmark and its limitations.