Verification field guide
How to verify AI-generated code
Verify AI-generated code with a runnable Node.js failure-and-fix example, executable acceptance checks, regression checks and reproducible evidence.
To verify AI-generated code, start from the requested behavior and finish with reproducible evidence. Do not begin by asking a second model whether the patch looks good. That can help triage risk, but it cannot establish that the software works.
1. Freeze the task and revision
Write down the requested outcome, the repository revision before the run, and the exact agent change. If the task is ambiguous, record the ambiguity instead of allowing the verifier to silently choose a friendlier interpretation.
2. Identify the behavioral boundary
Name what users or callers can observe: a browser state transition, an API response, an authorization denial, a persisted record, or a regression that must remain fixed. Prefer checks against this boundary over checks that merely mirror the implementation.
3. Run the smallest authoritative checks
Use repository-owned tests and commands first. Add a focused browser or API check when the existing suite does not exercise the changed behavior. Capture the command, environment, exit state, and bounded output. A passing command without identity or context is weak evidence.
4. Separate failures
Distinguish an agent regression from an existing repository failure, an environment problem, a timeout, and an unverified requirement. This prevents infrastructure noise from becoming a false product verdict.
5. Package the evidence
Bind the task, change, checks, outputs, artifacts, and verdict into a machine-readable bundle. A reviewer should be able to answer: what ran, against which revision, what changed, what failed, and what remains unknown?
6. Re-check after a fix
A fix is not complete because the diff changed. Re-run the failing check and the smallest relevant regression set. Preserve both the failure and the later pass so the evidence shows closure rather than only the final state.
CodeVetter follows this task-to-evidence shape and is designed to fail closed when required evidence is absent. Its current public benchmark demonstrates a narrower recognition task and publishes its own limitations; it should not be read as a production-wide guarantee.
Try it: catch a real class of AI-code bug
Suppose an agent is asked to preserve an explicit false setting while keeping the default and label behavior. A truthiness fallback can look reasonable in a review and still violate that requirement. This small, synthetic example is adapted from CodeVetter's published explicit-false task; it illustrates the check, rather than a measured agent success rate.
With Node.js installed, save this as verify.mjs and run node verify.mjs:
import assert from 'node:assert/strict';
function transform(input) {
return {
enabled: input.enabled || true,
label: input.label ?? 'Default',
};
}
assert.equal(transform({ enabled: false }).enabled, false);
assert.equal(transform({ label: 'Keep this' }).label, 'Keep this');
assert.equal(transform({}).enabled, true);
The first assertion fails: false || true becomes true. Change only input.enabled || true to input.enabled ?? true and run the same file again. All three assertions should pass. The default and label checks protect behavior that the fix must preserve.
For your own patch, replace these assertions with checks at the changed API, browser or storage boundary. A unit example cannot establish that a deployed route or permission-dependent workflow works. Record the failing revision, the fixed revision and the same check's output for both.
Reproduce the CodeVetter corpus check
To inspect the product's evidence mechanism, clone the repository and follow its corpus setup and qualification guide. From the repository root, run:
pnpm corpus:qualify --task preserve-explicit-false --json
Qualification executes the committed baseline and known-good change in fresh temporary workspaces and checks that the intended failure and later pass repeat. It does not call a model provider or demonstrate that an agent found the fix. Inspect the task identity, required check, preserved-label regression and cleanup state in the receipt before treating it as evidence.
A public-repository case
Inspect a React hydration visibility regression: one CodeVetter review finding, two failing visibility cases before repair, and four passing checks afterward. The article links pinned public source, a redacted evidence summary and a focused replay. This is one observed case, not a catch-rate benchmark.
Worked verification recipes
These worked recipes run public synthetic task fixtures. Each retained qualification repeats the intended baseline failure twice and the supplied known-good pass twice, with complete cleanup. The linked receipts were recorded on 14 November 2026 (UTC) with Node.js 24.22.2. They qualify the task and checks; they do not measure an agent, the installed app or production reliability. Qualification runs trusted corpus code without an operating-system sandbox.
Use Node.js 24 or newer. These qualification drivers use Node built-ins; no package install, coding agent or provider key is required. In a fresh clone:
git clone https://github.com/Codevetter/codevetter.git codevetter-recipes
cd codevetter-recipes
git switch --detach 28ee304927cd431966588136d099cf7659b03087Then run the task-specific command below from that checkout. Read the corpus guide for the full contract.
- Verify that stale search results cannot overwrite a newer request
- Verify that a success result waits for transaction commit
Verify that stale search results cannot overwrite a newer request
Requirement: An older response must leave the active search state intact, while the active request can update its results and preserve the query.
Exercise the boundary: Start with requestId: 2, query: current and items: [current]. Apply a response for request 1 containing [stale], then independently apply a response for request 2 containing [fresh]. The stale branch must return the original state object and keep current; the active branch must return fresh and preserve the query.
Starting behavior:
return { ...state, items: response.items };Supplied known-good replacement (relevant fragment):
if (response.requestId !== state.requestId) return state;
return { ...state, items: response.items };Both inputs are successful responses, so a happy-path-only test misses the defect. The supplied repair rejects the outdated identity before writing results. The driver chooses response order deterministically instead of relying on sleeps or a fast network to reproduce a race.
node scripts/agent-task-corpus/qualify-cli.mjs --task ignore-stale-search-response --jsonInspect the pinned task, driver and acceptance checks · Read the observed qualification receipt. Expected qualified: true, baseline.status: intended_failure, known_good.status: pass and cleanup.status: complete.
Boundary of this evidence: The boundary is a state-transition helper. It does not launch a browser or issue concurrent HTTP requests. Add the real interaction that creates and propagates request identities when verifying a search UI; cancellation and component unmounting remain separate requirements.
Verify that a success result waits for transaction commit
Requirement: Persistence may report committed: true only after the commit promise settles. Preserve the written value and write-before-commit order.
Exercise the boundary: Use a transaction whose write resolves but whose commit promise is held open by the driver. Start persistValue(transaction, value-1), flush the driver’s two microtask checkpoints, and inspect whether the outer promise has settled. It must remain pending. Release commit, then check the success result and the exact call order: write(value-1), commit.
Starting behavior:
await transaction.write(value);
transaction.commit();
return { committed: true };Supplied known-good replacement (relevant fragment):
await transaction.write(value);
await transaction.commit();
return { committed: true };A check that awaits only the final result can see committed: true in both versions. Holding commit unresolved exposes the premature-success window. The known-good patch changes the waiting behavior, not the success shape or write value.
node scripts/agent-task-corpus/qualify-cli.mjs --task await-transaction-commit --jsonInspect the pinned task, driver and acceptance checks · Read the observed qualification receipt. Expected qualified: true, baseline.status: intended_failure, known_good.status: pass and cleanup.status: complete.
Boundary of this evidence: This is an in-memory promise boundary, not a measurement of durable storage. Real database acceptance also needs a committed record read through the appropriate connection and a failure/recovery check. The fixture does not establish crash durability or isolation level.
Decide what is ready to ship
A passing example covers only its declared requirement. Before shipping your change, also exercise relevant integration paths, permission denials and failure recovery. If a required environment or check is unavailable, record it as unverified and state what remains to run. Keep the evidence bundle alongside the patch so another developer can assess the result.
CTA: Use the checklist with your next agent patch, then inspect how CodeVetter structures verification evidence.