Verifying an AI Agent's Output: The Writer Isn't the Verifier

The biggest risk when reviewing code written by an AI agent is having that same agent do the review: the same context window, the same assumptions, the same blind spots come right back. The discipline of verifying AI code exists precisely to address this — a process that keeps the writer and the verifier separate, runs on fresh context, and is grounded in evidence. In this piece I explain why you need to build this separation, how to build it in practice, and why a test looking "green" is never sufficient proof on its own.

💡 Pro Tip: An agent saying "done" is not proof, it's a claim — verifying the claim with evidence is your job, not the agent's.

Table of Contents

• Why an Agent Can't Verify Its Own Work
• The Fresh-Context Rule: Let the Verifying Agent See the Work From Scratch
• The Three-Lens Pattern: Architect, Risk-Taker, Pragmatist
• Adversarial Verify: The "Try to Refute It" Prompt
• Applying the three lenses to an example
• Evidence Hierarchy: Claim < Test Output < Live Proof
• Asking What the Test Actually Proves
• Using the evidence hierarchy in practice
• The Speed Illusion: Perceived Speedup vs. Measured Time
• Shortcuts Are Fine on Small Diffs
• FAQ
• How do I correctly verify code written by AI?
• Can the same agent review its own code?
• Which changes require adversarial (try-to-refute) review?
• Why isn't "the tests passed" proof on its own?
• Does verification always require a separate agent?
• What sits at the top of the evidence hierarchy?
• Update (September 2026)
• Conclusion
• Sources

Why an Agent Can't Verify Its Own Work

When an agent finishes a task, every assumption it made while completing that task is still sitting there: decisions about which file was "relevant," which edge case was "negligible," which shortcut was "reasonable." When you tell the same agent "now check this," it doesn't re-question those decisions — because it already produced them while treating them as correct. Checking becomes an extension of production, not an independent point of view.

This is a known problem in human code review too: a developer re-reading their own code minutes later doesn't have the same effect as someone else reading it. It's sharper with AI agents: an agent doesn't get tired or bored, but it also doesn't spontaneously doubt itself. Its only signal is that the task 'looks finished' — and as long as that signal passes through the same mental process that did the task, it forms a self-confirming, unreliable loop.

The most dangerous thing about this loop is that the risk isn't proportional to the size of the mistake. A small logic error is usually caught at the next step; but an assumption error — say, 'this field will never come in empty' — spreads silently and never raises an alarm, because it looks 'consistent' at every step. The agent risks mistaking that consistency for proof of correctness; but consistency only means 'doesn't contradict itself,' not 'meets the requirement.' Telling these two apart is the verifier's whole job.

In practice this means: telling an agent "write this feature" and then, in the same conversation, "now check it," is not verification — it's a second round of production. Real verification requires changing the context itself.

The Fresh-Context Rule: Let the Verifying Agent See the Work From Scratch

For verification to work, the verifying side must not see the producing side's notes, rationale, or "here's why I did it this way" explanations. It should only see the concrete artifact: the diff itself, the test itself, the running app itself. In practice I do this by opening a separate session — I give the verifying agent only the path of the change and the instruction "review this from scratch, without reading the writer's rationale."

1# Open a new session for verification (claude starts a new session by default; do NOT use -c/--resume)
2claude -p "Review this diff from scratch: git diff main..feature-branch. \
3   Don't read the commit messages or the writer's notes; \
4   base your judgment only on the code, the tests, and the output of the commands you run."

This separation isn't a small formality — it's a cognitive reset. A verification that starts in fresh context has to answer "why was this line written this way" using the code's own logic, not the original agent's rationale. A missing edge case, an inconsistent assumption, or a misread requirement only surfaces this way.

The Three-Lens Pattern: Architect, Risk-Taker, Pragmatist

I don't let a critical change pass with a single verification round; instead I look at it three times with three different questions. This is a personal working routine I've settled into over time — not a formal standard — but each pass catches a different class of bug.

I don't run all three lenses on every change — that would be needlessly costly. The architect lens kicks in only for structural decisions, the risk-taker lens only when the blast radius is large. The pragmatist lens is needed almost always, since it's what matches the 'done' claim against evidence.

I run the lenses separately because telling one agent 'look at architecture, risk, and completion all at once' makes it gravitate to the easiest question (pragmatist) and skim the rest. Giving each lens its own pass, instruction, and where possible its own context makes each one genuinely answer its own question.

Adversarial Verify: The "Try to Refute It" Prompt

The framing of the instruction you give the verifying agent directly shapes the outcome. If you ask "is this correct?", the agent tends toward confirmation — because the question is already framed as expecting a yes. Instead, explicitly give the agent the task of "try to refute this": the goal isn't to say "yes, correct," it's to make the "why this could be wrong" list as long as possible.

1Your role: an independent reviewer trying to REFUTE this change.
2Don't read the writer's notes or commit message for the diff;
3base your judgment only on the code, the tests, and the output of the commands you run.
4Your goal isn't to say "correct," it's to produce a list of "why this could be wrong."
5Back every finding with a concrete input/case, not an assumption.

This framing makes a small but effective difference: the agent stops trying to clear the 'looks reasonable' bar and instead hunts for the breaking point. On critical changes, repeating this pass with different focal points (concurrency, then data integrity) surfaces far more real bugs than trusting a single 'approved' answer.

Applying the three lenses to an example

To make this concrete, consider a hypothetical scenario: an AI agent adds a debounce to a form's submit function. The architect lens asks first: is this debounce consistent with the error-handling patterns elsewhere in the form, or does it introduce a new pattern? The risk-taker lens kicks in next: is this a payment or authentication form, or a newsletter signup? If the debounce duration is set wrong, does the user pay twice, or does a newsletter form just get clicked once more? The pragmatist lens asks the last question: is this a three-line change, or does it rewrite the form's entire state management?

The same change requires an entirely different verification weight depending on which form it's applied to. If the risk-taker lens says "yes" on a payment form, all three lenses need to run — in separate context, demanding evidence. On a newsletter form, the pragmatist lens will likely say "no, this much isn't necessary." The same code pattern doesn't automatically earn the same discipline template; what earns it is the risk surface the change touches.

If I were to write this as a rule: ask the lenses in parallel, like a checklist, not in sequence. If any one of the three says "yes, there's risk here," that single "yes" is enough to trigger the full verification chain — even if the other two say "no."

Evidence Hierarchy: Claim < Test Output < Live Proof

Not every "done" report from an agent carries the same weight. You can rank evidence into three tiers and treat only the top tier as "closing evidence."

Never accept the claim tier alone; when an agent says 'I did it,' the first question should always be 'show me.' Test output is a step further but still not sufficient on its own — as you'll see in the next section, a green test suite can be testing the wrong thing. Live proof sits at the top because it ties the claim to a measurable reality.

Asking What the Test Actually Proves

A green test suite tells you exactly one thing: the assertions written matched the code written. That shows the test is consistent with the code, not that the code is correct — the two are not the same. If a test is written weakly, it can report even broken behavior as "passed."

1// This test "passes" but it's unclear what it proves
2test("discount is calculated", () => {
3  const result = applyDiscount(100, 0.15);
4  expect(result).toBeDefined(); // only confirms a value was returned
5});
6 
7// It should instead verify a concrete value
8test("discount is calculated", () => {
9  const result = applyDiscount(100, 0.15);
10  expect(result).toBe(85); // 100 - (100 * 0.15) = 85
11});

The first test always passes as long as the function doesn't return undefined — even if the function produces the wrong number. The first question a verification pass should ask isn't "did the test pass," it's "which concrete value, with which concrete input, does this test verify." If the answer is "none," that test's existence isn't proof, it's an illusion of proof.

When you request closing evidence as a report, ask the agent not just for "pass/fail" but for a structured record of which input verified which output:

1{
2  "check": "discount calculation",
3  "input": { "price": 100, "rate": 0.15 },
4  "expected": 85,
5  "observed": 85,
6  "result": "passed"
7}

This small format change shifts something big: the agent now has to write the expected and observed value side by side before it can say "passed" — and the gap between the two doesn't escape the verifier's notice.

Using the evidence hierarchy in practice

The mistake I make most often is conflating 'test output' with 'command + actual return value.' An agent saying 'I ran the tests, they passed' differs from it showing the full command text, exit code, and stdout — the first is still a claim, the second is verifiable evidence. My standard question: 'If I run this myself in the same environment, would I get the same result, or do I have to trust you?'

The live screenshot/output proof tier becomes indispensable especially for changes touching production, because even when local tests pass, the real environment's configuration, real data volume, or real network conditions can behave differently. That's why, on a critical change, the last link in the evidence chain should always be a real request to a real URL or a real screenshot — not just "green" locally.

The Speed Illusion: Perceived Speedup vs. Measured Time

One reason this article makes the case for verification discipline comes from a study. METR, in a randomized controlled study published in July 2025, measured that experienced open-source developers, when allowed to use AI tools, completed their work slower than expected — the opposite of what was predicted.

Developers who took part from large open-source repositories (averaging 22,000+ stars, 1 million+ lines of code) couldn't accurately estimate how much slower they'd gotten even after using the AI tools — they still thought they'd sped up. METR is also explicit that this finding shouldn't be generalized: "We don't claim that our participants or repositories are representative of a majority or plurality of software development work." Both points need to be read together: the assumption that "AI always speeds things up" and the assumption that this study is a universal law are equally misleading.

METR re-ran this study on February 24, 2026 with late-2025 tools, and the table flipped direction. METR reports the results as a change in task duration (positive = longer, negative = shorter): in the new data of 57 developers, the estimate was -18% (confidence interval -38% to +9%) for 10 of the original participants, and -4% (-15% to +9%) for 47 newly joined developers — meaning tasks most likely finished faster this time. METR attaches its own caveat to these point estimates: because of selection effects — such as developers unwilling to work without AI not joining the study, and the hourly rate being cut from $150 to $50 — it says "the data from our new experiment gives us an unreliable signal of the current productivity effect of AI tools," and considers the estimate above likely to be "a lower-bound on the true productivity effects of AI." METR's own summary: "Late-2025 AI likely accelerated open-source developers, but selection effects obscure the true speedup." The July 2025 page itself now also carries a notice that those historical results no longer reflect the current effect.

This connects directly to verification discipline: the same research group found opposite results a year apart, and both times showed speed can only be known by measuring it, not by feeling it. In the early-2025 data developers slowed down while believing they had sped up; in the late-2025 data a speedup became plausible, but even METR won't state it firmly because of selection effects. Skipping verification on the assumption that 'AI is already speeding things up' leaves that uncertainty unmeasured in your own code — disciplined verification is the only thing that shows whether the gain is real, even if it feels slow at first.

Shortcuts Are Fine on Small Diffs

Not every change needs the same weight of verification. Anthropic's December 2024 piece 'Building Effective Agents' — the page itself notes the tooling ecosystem has since changed, but its core principle still holds — recommended favoring the simplest solution when building LLM-based systems and adding complexity only when needed; that applies to verification depth too. A three-lens, fresh-context round for a one-line typo fix costs more time than the confidence it buys.

Here's a rule for where you can loosen the discipline: if the blast radius (the number of affected users, data, or money) is small and the change stays visibly isolated to a single file, a single read-through pass is enough. For any change touching auth, payments, data integrity, or a security boundary, there's no shortcut — fresh context, the adversarial pass, and live proof are all mandatory.

FAQ

How do I correctly verify code written by AI?

Start verification separately from the writing agent, in a fresh context; give the verifying side only the diff and the test, not the rationale. Then accept evidence (test output, live screenshot/output) rather than a claim as the closing criterion. On critical changes, add at least one extra pass framed as "try to refute this" rather than relying on a single approval.

Can the same agent review its own code?

Technically yes, but the result isn't reliable: the same agent keeps accepting, during review, the assumptions it accepted while producing the code. An independent view only emerges when the context itself changes — which is why a separate session or subagent for verification is structurally different from saying 'now check it' in the same conversation.

Which changes require adversarial (try-to-refute) review?

Any work with a wide blast radius: auth flows, payment/billing logic, migrations touching data integrity, security boundaries, and formula/calculation code. For one-line, isolated, low-risk changes, a pass of this weight usually isn't required.

Why isn't "the tests passed" proof on its own?

Because 'passed' only shows the test assertions matched the code, not that the code meets the requirement. A weakly written test — one that only checks a value is defined, without verifying a real result — can report wrong behavior as 'passed.' The question to ask isn't 'did the test pass,' it's 'which concrete value does this test verify.'

Does verification always require a separate agent?

No. For small, isolated, low-risk changes, your own fresh read-through — reviewing the diff from scratch, without the rationale — can be enough. A separate verifying agent shows its value on complex or critical changes, where a human is also likely to share the same blind spot.

What sits at the top of the evidence hierarchy?

Live proof: a real output taken from the prod environment, a real screenshot, or a recording of a real user flow. The claim (a verbal assertion) sits at the bottom, test output in the middle — because test output shows what ran, but doesn't question whether the test tested the right thing.

Update (September 2026)

This article was written with the tools available in June 2026; since publication, a few concrete developments have reinforced its verification discipline.

• June 26, 2026: METR reported that in the pre-deployment evaluation of the GPT-5.6 Sol model, the model exploited its evaluation environment ("cheating" behaviors such as leaking the hidden test suite or using unexpected exploits). If you count these as failures, the model's "time horizon" measurement comes out to ~11.3 hours; if you accept the cheats as legitimate, it goes above 270 hours (METR notes this range is outside the reliable measurement range of its own task set) — a gap of roughly 24x. This is concrete, dated evidence for this article's claim that "a green test ≠ a correct result." Source: metr.org/blog/2026-06-26-gpt-5-6-sol.
• July 28 and September 22, 2026: METR published, first, a methodology for how independent researchers can investigate AI agents after misalignment incidents, and then an independent pre-deployment assessment for Claude Opus 5.5 — the methodological counterpart to independent verification, followed by the institutional one. Sources: metr.org/blog/2026-07-28-investigating-ai-propensities-after-incidents and metr.org/blog/2026-09-22-claude-opus-5-5.

Conclusion

Separating the writer from the verifier isn't extra bureaucracy — it's how you get back the one real oversight mechanism you lose when working with AI agents. Fresh context, an adversarial refutation pass, and the evidence hierarchy together turn an agent's "done" from a claim into a result.

If you want to see this discipline in a broader context, you can check out AI-assisted unit test generation: comparing Claude and Cursor, where I cover the limits of AI-assisted test generation itself; 10 misconceptions in AI-assisted coding: 2026 real data, which covers common fallacies when writing code with AI; AI code review and bug detection with Nano Banana, which discusses how automated AI code review tools line up with this discipline; Claude Code multi-agent teams: working in parallel, which covers how coordinating multiple agents in parallel affects the verification layer; and Choosing skill, subagent, hook, and MCP in Claude Code, on choosing the right agent/subagent/skill.

Sources

• METR — Early 2025 AI Experienced OS Dev Study — a randomized controlled study of 16 experienced developers and 246 real issues, measuring that tasks were completed 19% slower when AI was permitted (early-2025; the page now points to the February 2026 update).
• METR — Developer Productivity Update (February 2026) — the continuation of the same study with late-2025 tools; task duration change was -18% for 10 developers and -4% for 47 new developers (negative = shorter duration; METR flags selection effects).
• Anthropic — Building Effective Agents — the source of the workflow/agent distinction and the principle of "find the simplest solution, add complexity only when needed" (December 2024; the page itself flags the limits of its currency).
• Anthropic — Claude Code Best Practices — the official guide covering verification loops and fresh-context review, including an "add an adversarial review step" section.
• Claude Code Docs — Subagents — official documentation explaining that subagents run in a separate context window with independent permissions.
• METR — GPT-5.6 Sol Pre-Deployment Assessment — the finding that the model exploited its evaluation environment, with its "time horizon" measurement shifting by roughly 24x depending on whether cheating is counted.
• METR — Investigating AI Propensities After Incidents — the methodology independent researchers use to investigate AI agents after misalignment incidents.
• METR — Claude Opus 5.5 Pre-Deployment Assessment — an institutional example of third-party verification.