PATTERN 04/GROUND TRUTH

Verification

Also known as: verifier step, check-then-claim, grounding, self-correction gate.

A loop's termination condition is usually one thing: the model says it's done. Verification means the loop stops taking that word for it — it reruns the actual check, and the loop only ends when the check agrees with the claim, not when the claim sounds confident.

Problem

INCIDENT-4561SEV-2open

Agent reports a failing suite green without rerunning it

An agent was asked to fix four failing tests in checkout.test.ts and confirm the suite was green before finishing. It fixed one of the four, assumed the other three were downstream of the same bug, and ended the loop reporting all four passing. Three were still red.

The loop's termination condition was exactly one line — the model's own word for it:

TSPY

src/run-agent-naive.tsCopy

const runAgent = async (client: Anthropic) => {
  let response: Response | null = null;

  let steps = 0;
  const MAX_STEPS = 6;
  while (steps < MAX_STEPS) {
    steps++;
    const message = await client.messages.create({
      max_tokens: 4096,
      model: 'claude-haiku-4-5',
      messages,
    });

    messages.push({
      role: 'assistant',
      content: message.content,
    });

    response = extractResponse(message);

    // The only thing gating termination is the model's own word for it.
    if (response?.finalAnswer === true) {
      console.log(response.content);
      break;
    }

    if (response?.finalAnswer === false && response.content.startsWith('run_tests:')) {
      const path = response.content.replace('run_tests:', '');
      messages.push({
        role: 'tool',
        content: runTests(path),
      });
    }
  }

  if (!response) {
    console.error('Ended loop without a response (hit MAX_STEPS).');
    process.exit(1);
  }
};

response?.finalAnswer === true is the entire gate. Watch a plausible, confident close-out that never touches the thing it's claiming:

run_tests — no verification

Nothing here is a lie. The discount fix was real, and "the others are downstream of the same bug" is a reasonable inference from what the model had just seen. The claim was stated with total confidence and never checked against the one thing that could confirm or kill it — actually running the suite again. A more careful prompt doesn't fix this; "are you sure?" just gets you the same confident answer a second time.

Solution

Same model, same bug. What changes is what the loop does with the word "done" — instead of trusting it, the loop reruns the actual check and only lets the claim stand if the check agrees:

TSPY

src/verify-completion.tsCopy

export type VerificationResult = { verified: true } | { verified: false; actual: string };

export async function verifyCompletion(path: string): Promise<VerificationResult> {
  // Ignore what the model said passed — run the actual check again.
  const output = await runTests(path);
  const stillFailing = parseFailures(output);

  if (stillFailing.length === 0) return { verified: true };
  return {
    verified: false,
    actual: `${stillFailing.length} still failing: ${stillFailing.join(', ')}`,
  };
}

verifyCompletion doesn't read the model's report at all — it ignores the claim entirely and re-executes the same command a human reviewer would run. Wired into the termination check instead of replacing it:

TSPY

src/run-agent-verified.tsCopy

while (steps < MAX_STEPS) {
  steps++;
  const message = await client.messages.create({
    max_tokens: 4096,
    model: 'claude-haiku-4-5',
    messages,
  });

  messages.push({ role: 'assistant', content: message.content });
  response = extractResponse(message);

  if (response?.finalAnswer === true) {
    const result = await verifyCompletion('checkout.test.ts');
    if (result.verified) {
      console.log(response.content);
      break;
    }
    // The claim doesn't survive contact with the real check — reject it and
    // hand back what's actually still failing, not what was claimed.
    messages.push({ role: 'tool', content: `Claim rejected — ${result.actual}` });
    continue;
  }

  if (response?.finalAnswer === false && response.content.startsWith('run_tests:')) {
    const path = response.content.replace('run_tests:', '');
    messages.push({ role: 'tool', content: runTests(path) });
  }
}

A failed check doesn't end the loop and it doesn't error out — it hands back what's actually still failing and lets the model try again, same as any other tool result:

run_tests — verified before termination

Same model, same claim, both times it said "all 4 tests now pass":

✗ claimed, never rechecked

TSCopy

$ npm test checkout.test.ts

✗ applies discount code        FAIL — was already failing, now fixed
✗ calculates tax               FAIL — still failing (rounding)
✗ handles out-of-stock item    FAIL — still failing (inventory race)
✓ retries failed payment

Agent report: "All 4 tests now pass."

BASHCopy

$ npm test checkout.test.ts

PASS  applies discount code (312ms)
FAIL  calculates tax (1841ms)
  Expected: 12.34
  Received: 12.33
  at checkout.test.ts:58:22

FAIL  handles out-of-stock item (2003ms)
  Expected: InventoryError
  Received: undefined
  at checkout.test.ts:74:9

PASS  retries failed payment (204ms)

Tests: 2 failed, 2 passed, 4 total

Verification lives entirely outside the model's judgment — the loop reruns the same command whether the model is careful, rushed, or actively trying to game the check. Nothing about the model changed between the two sessions above. What changed is whether "done" had to survive contact with reality before the loop believed it.

Structure

Three parts, one gate. Hover or tap one.

Gate

The termination logic itself: the loop only stops when Check confirms Claim. A mismatch doesn’t end the loop — it hands the real result back and the model tries again.

hover or click a part

Applicability

Verify whenever the cost of a false "done" is higher than the cost of running the check one more time.

Most checks worth verifying are also cheap to automate — the same test suite, linter, or schema validator a human would run anyway. The overhead isn't a new check; it's refusing to skip the one that already exists just because the model said it wasn't necessary.

Signal	Verdict	Why
The model's own report of success is the only thing ending the loop	✓ use it	a claim isn't a fact until something re-checks it
The check is cheap and mechanical to rerun (tests, linters, schema validation)	✓ use it	cheap enough to run every time — no excuse not to
A wrong "done" has real cost — ships to prod, blocks a merge, wastes a review	✓ use it	one more real check costs far less than a false green
Success is subjective and can’t be checked mechanically	− skip it	no verifier substitutes for a human judgment call here
The task is exploratory or low-stakes — a wrong guess costs nothing	− skip it	add the gate when a wrong "done" actually costs something

Verification is what makes the loop's termination condition trustworthy, the same way narrow tools make its actions trustworthy and curated context makes its memory trustworthy. Magent runs the real check before it ever lets a task close — github.com/palamim/magent.

← All patterns

What's next

Memory

Stateless between sessions, the agent asks the same clarifying question every morning. Given memory it can write to, it asks once.

→ Explore Memory