Verification field guide
Read a coding-agent verification receipt
Inspect a real JSON qualification receipt: match its source, read baseline and known-good results, and spot missing evidence.
A green verdict is useful only when you can tell which task, source and checks it belongs to. This guide reads a real CodeVetter JSON receipt and shows where its evidence stops. Use it when reviewing a verification result in CI or deciding what to retain for a later re-check.
First identify the kind of receipt
CodeVetter has several versioned receipt contracts. This example uses codevetter.agent-task-qualification.v2: it checks whether a synthetic task fixture fails for the intended reason and passes after its supplied known-good patch. It is not an agent-run receipt, a production verification bundle or a measurement of how often an agent solves tasks.
The general receipt-design article discusses what other evidence a system may need. The concrete contract below is narrower; do not assume a field exists because another receipt format has it.
Inspect the complete retained example
The public keep-zero-pagination-limit task asks the implementation to preserve an explicit zero, while keeping positive limits and the null fallback unchanged. The JSON below is the complete retained receipt at revision 0052cc9f. It contains public fixture identities and results; no private repository, user or credential data has been substituted.
Open the raw receipt and its pinned v2 schema to inspect the full contract independently.
{
"schema_version": "codevetter.agent-task-qualification.v2",
"task_id": "keep-zero-pagination-limit",
"manifest_sha256": "20266168d91738834547858dd0d95cc9e36375192a2e8f173affb3f63eaeabae",
"fixture_sha256": "0c87583841de15a41e275028894edf3aaccac6643a1759e7acfc1ca9ee5f697a",
"acceptance_contract_sha256": "bf72420e58ac1470b8ea7b71e07a14b4186116dd8b1e3d9516837075f695d9d3",
"known_good_sha256": "83de72637c2c9defe60db1a1a5e65a74409fc1adcde89520492b0b4642178420",
"workspace_policy": "public_fixture_and_task_packet_v1",
"qualified": true,
"baseline": {
"status": "intended_failure",
"attempts": [
{
"attempt": 1,
"outcome": "intended_failure",
"result_sha256": "9e1f290057de20e00c728a0273ddc6b6bfabb5ebad8cb087cca8a4a205d484a5"
},
{
"attempt": 2,
"outcome": "intended_failure",
"result_sha256": "9e1f290057de20e00c728a0273ddc6b6bfabb5ebad8cb087cca8a4a205d484a5"
}
]
},
"known_good": {
"status": "pass",
"attempts": [
{
"attempt": 1,
"outcome": "pass",
"result_sha256": "0d15f42fce15868ccbc1089f603975202739ec6ff17369f02b982f9f0b74608a"
},
{
"attempt": 2,
"outcome": "pass",
"result_sha256": "0d15f42fce15868ccbc1089f603975202739ec6ff17369f02b982f9f0b74608a"
}
]
},
"cleanup": {
"status": "complete"
},
"limitations": [
"Qualification executes trusted corpus check code without an operating-system sandbox."
]
}
Read these fields before trusting qualified
schema_versionandtask_idselect the contract and task. A qualification receipt answers whether the fixture is usable; it does not say that an agent produced a correct change.- The four input hashes bind the manifest, starting fixture, acceptance contract and supplied known-good patch. Compare them with those exact inputs. The task name alone cannot identify a changed fixture.
baseline.statusisintended_failure, with two attempts. This is a useful negative control: the starting fixture failed the required acceptance check, rather than merely crashing during setup.known_good.statusispass, also with two attempts.result_sha256identifies each check result; the receipt itself does not embed the underlying test output. Inspect that output when the decision needs more detail.cleanup.statusiscomplete. The v2 validator derivesqualifiedfrom the intended baseline failure, known-good pass and complete cleanup;qualifiedis not a model confidence score.
A passing flag cannot replace missing identity
Delete manifest_sha256 from this example and the receipt is invalid, even if qualified remains true. A renamed or missing identity is not repaired by copying a hash from an unrelated run. Recover the original input evidence or repeat qualification against the intended inputs.
Shape validation and source matching are separate checks. A correctly formed 64-character hash can still refer to the wrong input. Hashes help identify content; they do not authenticate the producer or prove which code it executed.
What this receipt does not establish
The limitations field states that qualification runs trusted corpus checks without an operating-system sandbox. workspace_policy describes which task material is exposed; it is not proof of OS isolation. This format does not establish production behavior, agent effectiveness, test coverage, memory use, network isolation or business outcomes. Two repeat attempts are not a general reliability estimate.
Worked verification recipes
These worked recipes run public synthetic task fixtures. Each retained qualification repeats the intended baseline failure twice and the supplied known-good pass twice, with complete cleanup. The linked receipts were recorded on 14 November 2026 (UTC) with Node.js 24.22.2. They qualify the task and checks; they do not measure an agent, the installed app or production reliability. Qualification runs trusted corpus code without an operating-system sandbox.
Use Node.js 24 or newer. These qualification drivers use Node built-ins; no package install, coding agent or provider key is required. In a fresh clone:
git clone https://github.com/Codevetter/codevetter.git codevetter-recipes
cd codevetter-recipes
git switch --detach 28ee304927cd431966588136d099cf7659b03087Then run the task-specific command below from that checkout. Read the corpus guide for the full contract.
- Keep rollback evidence separate from the original failure
- Prove recovery after a rejected cached request
Keep rollback evidence separate from the original failure
Requirement: A failed settings commit must trigger rollback and keep the original commit failure as the rejection. A successful update must not roll back.
Exercise the boundary: Make commit throw a retained Error object and record write, commit and rollback calls. Require exactly that call order and compare the caught rejection with the original object. In a second transaction, allow commit to succeed and require saved: true with zero rollback calls.
Starting behavior:
await transaction.write(value);
await transaction.commit();
return { saved: true };Supplied known-good replacement (relevant fragment):
try {
await transaction.write(value);
await transaction.commit();
return { saved: true };
} catch (error) {
await transaction.rollback();
throw error;
}The baseline already rejects on commit failure, so a test that merely expects a rejection is insufficient. The required check distinguishes a rejected transaction with no cleanup from one that ran rollback. Retain the call-order evidence alongside the failure rather than replacing the earlier record with a later green verdict.
node scripts/agent-task-corpus/qualify-cli.mjs --task rollback-failed-settings-update --jsonInspect the pinned task, driver and acceptance checks · Read the observed qualification receipt. Expected qualified: true, baseline.status: intended_failure, known_good.status: pass and cleanup.status: complete.
Boundary of this evidence: The driver makes rollback succeed. It does not test a second failure during rollback, storage state after partial writes, or crash recovery. The supplied patch therefore supports this bounded contract only; it is not a universal transaction implementation.
Prove recovery after a rejected cached request
Requirement: Remove a rejected in-flight load so a later call can retry; continue sharing an operation while it is pending and preserve the original failure.
Exercise the boundary: Start two loadOnce calls for profile before settling the first loader and require the same pending promise. Reject it with temporary, observe both rejections, then make a fresh call whose loader returns recovered. Require two total loader invocations, the recovered result and the retained failure message.
Starting behavior:
request.then(
() => inFlight.delete(key),
() => {}
);
return request;Supplied known-good replacement (relevant fragment):
const tracked = request.finally(() => inFlight.delete(key));
inFlight.set(key, tracked);
return tracked;The baseline deletes only after success, so it reuses an already rejected promise on the next attempt. The repair stores and returns the cleanup-tracked promise. A complete evidence record preserves the failed first attempt and the successful retry; publishing only the final recovered value would hide whether failure recovery was exercised.
node scripts/agent-task-corpus/qualify-cli.mjs --task clear-inflight-request-after-rejection --jsonInspect the pinned task, driver and acceptance checks · Read the observed qualification receipt. Expected qualified: true, baseline.status: intended_failure, known_good.status: pass and cleanup.status: complete.
Boundary of this evidence: The driver checks a retained message, not exact error-object identity or real network recovery. It does not cover synchronous loader throws, eviction or simultaneous different keys. Use a controlled real integration failure when those are release requirements, and retain the missing checks as unverified.
Reproduce the check, then keep the right evidence
Use the pinned source and prerequisites in the corpus guide to run qualification. Keep the emitted receipt with that source revision and the input artifacts. New output hashes can differ; compare the intended identities and outcomes rather than treating every byte difference as a regression.
corepack pnpm corpus:qualify --task keep-zero-pagination-limit --json
For a real agent change, retain its own revision-bound execution evidence and missing requirements. Keep an earlier failure alongside its later re-check; a new passing receipt does not erase the failed attempt.
Run the published corpus recipe, or follow the worked agent-loop guide to turn a task into executable checks.