← CodeVetter

Verification field guide

Coding-agent verification

Verify code inside the agent loop with a runnable pagination example: define checks, reproduce failure and repair, preserve regressions, and decide when to stop.

Coding-agent verification asks whether an agent completed the requested software task and which executable checks support that answer. Put the check inside the agent loop: define the behavior, preserve the patch, run the check, then use the result to decide whether to repair, stop or gather missing evidence.

The five-part loop

  1. Task: write the requested outcome and the behavior that must stay intact.
  2. Change: record the base revision and the exact agent-produced patch.
  3. Execution: run repository-owned checks against the changed boundary.
  4. Evidence: retain the command, environment, output and source identity.
  5. Verdict: state what passed, failed or remains unverified.

Review can help find a suspicious change. Execution establishes whether the checked behavior fails. A green command is useful only when it exercises the requirement at the right boundary.

Worked example: forward a pagination cursor

An agent is asked to update a list wrapper so the caller's cursor reaches an integration client, while preserving the requested limit and returned response. The published synthetic task supplies the starting source, acceptance contract, check driver and known-good change.

The starting wrapper calls client.list({ limit }), dropping its cursor argument. The known-good change calls client.list({ limit, cursor }). Before changing code, turn the request into these three checks:

  • Requested behavior: call listNext(client, 25, 'cursor-2') and check that the integration receives cursor: 'cursor-2'. The check is named pagination-cursor-forwarded.
  • Preserved input: the same call must pass limit: 25. The regression check is pagination-limit-preserved.
  • Preserved output: the wrapper must return the exact response object from the integration. The regression check is list-response-preserved.

The authoritative boundary here is the wrapper's call to client.list and its returned value. The check captures that call with a small test client. It does not contact a live integration or prove that an external API accepts the cursor. In your repository, add the real integration check when that is part of the requested task.

Run the failure-and-fix check

Use Node.js 24 or newer and the repository's pinned pnpm 10.33.2. In a fresh clone, run:

git clone https://github.com/Codevetter/codevetter.git codevetter-cursor-check
cd codevetter-cursor-check
git switch --detach 83b09906d44eab9ddd15bbc8264d10781e3d2474
corepack pnpm install --frozen-lockfile
corepack pnpm corpus:qualify --task forward-pagination-cursor --json

The corpus guide explains setup and the full receipt contract. Qualification runs the committed baseline twice and the known-good replacement twice in fresh temporary workspaces. It does not launch a coding agent or call a model provider. It executes trusted corpus check code without an operating-system sandbox.

For this pinned task, the expected receipt has qualified: true, baseline.status: "intended_failure", known_good.status: "pass" and cleanup.status: "complete". The command exits 0 when all four attempts and cleanup satisfy the contract. The baseline is supposed to fail the cursor check while passing both regression checks; an arbitrary failure does not qualify it. The known-good version must pass all three checks on both attempts.

Keep the complete JSON receipt. Its task ID and SHA-256 identities bind the manifest, fixture, acceptance contract and known-good change. The attempt outcomes and result hashes distinguish repeated expected behavior from a single green run. Record your local Node version and command alongside it.

Decide whether to repair, recheck or stop

  • The changed behavior fails: give the agent the failing requirement and bounded output. After a repair, rerun the same check and the relevant regressions against the new patch. Preserve the earlier failure.
  • The requested behavior passes but a regression fails: continue the repair. Forwarding the cursor while changing the limit or response is incomplete work.
  • All declared checks pass: stop the repair loop for those requirements. Review any remaining integration, permission or recovery requirements before release; the fixture's pass does not cover them.
  • Setup fails, a check times out, attempts disagree or evidence is missing: keep the outcome inconclusive. Fix or obtain the missing environment or evidence, then rerun. Do not ask the agent to alter working product code just to hide a test-environment failure.

This synthetic replay qualifies a task and its checks. It does not measure an agent's task-completion rate, time saved or return on investment. Verification can guide the next action for an exercised requirement; showing that it improves agent-loop effectiveness requires a separate comparison on representative tasks with the same scoring rules.

Worked verification recipes

These worked recipes run public synthetic task fixtures. Each retained qualification repeats the intended baseline failure twice and the supplied known-good pass twice, with complete cleanup. The linked receipts were recorded on 14 November 2026 (UTC) with Node.js 24.22.2. They qualify the task and checks; they do not measure an agent, the installed app or production reliability. Qualification runs trusted corpus code without an operating-system sandbox.

Use Node.js 24 or newer. These qualification drivers use Node built-ins; no package install, coding agent or provider key is required. In a fresh clone:

git clone https://github.com/Codevetter/codevetter.git codevetter-recipes
cd codevetter-recipes
git switch --detach 28ee304927cd431966588136d099cf7659b03087

Then run the task-specific command below from that checkout. Read the corpus guide for the full contract.

Connect an agent’s timeout change to the integration boundary

Requirement: The report wrapper must forward the caller’s timeout to the integration client while preserving the report path and response.

Exercise the boundary: Call fetchReport(client, report-7, 1500) with a client that captures its get arguments and returns a retained response object. Require options.timeoutMs === 1500, path === /reports/report-7 and the exact returned response object. This checks the receiving boundary rather than whether the wrapper merely accepts a timeout parameter.

Starting behavior:

return client.get(`/reports/${reportId}`);

Supplied known-good replacement (relevant fragment):

return client.get(`/reports/${reportId}`, { timeoutMs });

The baseline exposes the parameter but drops it at the call. In an agent loop, return request-timeout-forwarded as the failing requirement; after a repair, rerun it with the path and response regressions. Changing a type declaration alone cannot make the captured option appear.

node scripts/agent-task-corpus/qualify-cli.mjs --task forward-request-timeout --json

Inspect the pinned task, driver and acceptance checks · Read the observed qualification receipt. Expected qualified: true, baseline.status: intended_failure, known_good.status: pass and cleanup.status: complete.

Boundary of this evidence: Forwarding 1500 is not proof of a 1500 ms deadline, cancellation or socket cleanup. This driver never waits on a network. If the request must actually time out, add a controlled delayed integration using the real client’s timeout semantics and record its cancellation behavior.

Verify shared in-flight work without caching results forever

Requirement: Concurrent calls for the same profile must share one operation. Once it settles, a later call must start a fresh load and preserve returned profile values.

Exercise the boundary: Start two loadProfile(profile-1, fetchProfile) calls together with Promise.all and count fetchProfile invocations. Require one invocation for that concurrent pair and profile-1 in both results. After settlement and the driver’s microtask checkpoint, make a third call and require exactly one additional invocation with the same returned profile ID.

Starting behavior:

inFlight.set(id, request);
inFlight.delete(id);
return request;

Supplied known-good replacement (relevant fragment):

inFlight.set(id, request);
request.finally(() => inFlight.delete(id));
return request;

Deleting the entry immediately makes both concurrent callers fetch independently. Keeping the entry forever would fix that symptom but fail settled-load-restarts. Feed both requirements back into the agent loop so the repair preserves the intended lifetime, rather than optimizing one assertion at the expense of freshness.

node scripts/agent-task-corpus/qualify-cli.mjs --task share-inflight-profile-load --json

Inspect the pinned task, driver and acceptance checks · Read the observed qualification receipt. Expected qualified: true, baseline.status: intended_failure, known_good.status: pass and cleanup.status: complete.

Boundary of this evidence: This fixture uses successful in-memory promises for one profile. It does not exercise rejection cleanup, cross-key isolation, cancellation or a synchronous throw from the loader. In particular, do not treat the finally fragment as a complete error-handling recipe; use the separate rejection-and-retry example for that boundary.

Use the loop on your next change

Run the published verification recipe, then replace the sample requirement with your own task, source revision and authoritative checks. Keep an evidence bundle with the patch.

For an observed public-repository failure, inspect the React hydration case. It links the reviewed source, before-and-after visibility checks and a focused replay. For local product availability, install CodeVetter.