$_ stdout
~/posts/review-only-part-2.md · on main · 44 min read
/**
* @topic#codereview #agenticsoftware #humanfactors #softwaresecurity
* @statuspublished
*/

Review Only: What Happens When Developers Stop Writing the Code, Part 2

This is Part 2 of a two-part post. Part 1 found that at the frontier, agents now write most new code and engineers mostly review it; that the speedup is real at the keyboard and unmeasured end to end; and that review was never a strong filter for the defects agents introduce.

Part 2 turns to the reviewers and finds the human backstop weakening. Complacency grows as the aid grows more reliable, the diagnostic skill that takeover needs fades without practice, and reviewers overrate their own oversight. AI reviewing AI is not an independent check by default either: AI reviewers are scored by the comments developers act on, never by the defects they miss, and models favor their own output and game tests when they can.

None of this makes review-only development a mistake. The most careful pipeline I found, Anthropic's, rations human approvals, never counts an agent's approval as a person's and tracks revert rate as a guardrail, and Meta's risk-gated review rations human attention the same way.

My position is narrow. Review-only work is defensible only when the pipeline is built so that it does not depend on a perfect reviewer, and that takes five things: (1) permissions that keep the worst outcome recoverable, (2) tests the author cannot touch, (3) verification from outside the author, (4) staged release with fast rollback, and (5) a measured miss rate. It is not defensible as “the agent writes it and an engineer approves it,” because until someone measures what the review layers miss, an approval proves only that someone signed it.

BL Dr. Ben Livshits November 7, 2026 · 108 commits
// start with part 1

The reader is encouraged to read Part 1 first: it makes the case this part tests. It opens with an afternoon of agent merges at Bashi AI and covers how the leading teams structure review-only work, whether it is faster from idea to production, and the defects review was never built to catch.

Figure 1, the same image that sums up Part 1, is the argument of both parts in one frame.

Infographic titled 'Review-Only Development: The Hidden Risks of AI Code.' Top left: AI writes the majority of frontier code, with Anthropic reporting Claude writes about 90% of Claude Code and Google about 75% of new code. Below it: verification creates delivery bottlenecks, with 154% larger pull requests and 91% longer review times, drawn as a tired developer beside a pile of code windows and a stopwatch. Middle left: traditional review is good at readability but misses agent-specific flaws such as hallucinated dependencies, security vulnerabilities and logic errors. Middle right: supervising automation breeds complacency and erodes diagnostic skills. Bottom right: a defense-in-depth pipeline of isolated tests, automated checks, risk-tiered approvals at low, medium and high levels, and fast rollback, replacing a single human gate.
// Figure 1. An AI-generated infographic summarizing the argument of both parts; its numbers are those reported by the sources cited in the text, and the body, not the image, gives their caveats.

Lisanne Bainbridge's 1983 paper "Ironies of Automation" made an argument that has aged unusually well: automating the routine work leaves the human with the hardest job, monitoring plus rare takeover, while removing the practice that job depends on. I know the paper through its many restatements and did not reread the original for this post.

05 The Human in the Loop

Everything in Part 1 treats the reviewer as a fixed instrument with a known detection rate. People are not fixed instruments. Their attention, skill and understanding respond to the job they are given, and the human-factors literature has studied that response for four decades, mostly in cockpits and control rooms.

The Ironies of Automation

Since 2023 Bainbridge's argument has been applied to generative AI. Mica Endsley's "Ironies of artificial intelligence" (2023) is the direct restatement. Simkute et al. (2024) at Microsoft Research note that GenAI usability problems "have been observed for over three decades in Human Factors research," and Sellen and Horvitz (2024) warn that the co-pilot metaphor brings reduced vigilance and skill erosion with it.

A survey of 319 knowledge workers by Lee et al. (2025) observed the shift directly: generative AI moved their effort "from doing tasks to supervising tasks." A developer who mostly reviews agent PRs is in Bainbridge's position exactly. The agent handles the routine cases; the human is kept for the ones the agent gets wrong, which are by construction the hardest. Nolan Lawson, a long-time web and open-source developer, describes the role from the inside:

I didn't ask for the role of a programmer to be reduced to that of a glorified TSA agent, reviewing code to make sure the AI didn't smuggle something dangerous into production.— Nolan Lawson, software engineer, "We mourn our craft"

Who Actually Reviews Agent PRs

Duma et al. (2026) found that most AI-generated PRs in the AIDev dataset get no review at all, that AI agents dominate the review that does happen, and that humans more often steer the agent than evaluate the code; review metrics "may not reliably indicate human oversight." Peralta et al. (2026) report, in their abstract, that only 15.4% of merged agent PRs needed explicit reviewer involvement.

An eye-tracking study points in a slightly reassuring direction. In Khojah et al. (2026), 32 participants reviewed code with and without an "LLM-generated" label. The label did not make them less thorough, and they spent more fixation time on labelled code.

Complacency and Automation Bias

Parasuraman and Manzey (2010) review the evidence on complacency and automation bias and find attention central to both. Complacency shows up mainly under multiple-task load, affects both naïve and expert users, and "cannot be overcome with simple practice." Imperfect decision aids produce two kinds of error: omissions, missing what the aid did not flag, and commissions, following the aid against other evidence.

Mosier, Skitka and colleagues studied both kinds in glass-cockpit pilots, and one of their findings is the most useful in this post: pilots who felt accountable for how they used the automation verified it more and made fewer errors. Accountability is one of the few mitigations for automation bias with evidence behind it.

Map that onto code review. A queue of agent PRs, reviewed between other tasks, with an AI reviewer that has already commented, is the multitask, imperfect-aid setting in which the literature predicts both errors. The omission is approving a change because the AI reviewer raised nothing. The commission is applying the AI reviewer's suggested fix because it was suggested.

A 2026 experiment puts numbers on that. With an agent covertly exfiltrating data across a series of ordinary-looking changes, 94% of participants never noticed; when an LLM monitor did flag the sabotage step, it still reached the final commit in 56% of sessions, and the commonest reason among the misses was that the reviewer barely read the code (Ye et al., 2026).

Practitioners describe the drift in themselves. Simon Willison, co-creator of Django, who drew the line between vibe coding and agentic engineering at review, now says that "as the coding agents get more reliable, I'm not reviewing every line of code that they write anymore, even for my production level stuff" (Willison, 2026): trust growing with reliability, as Parasuraman and Manzey describe. Admissions of outright rubber-stamping are almost all anonymous or pseudonymous; an unnamed engineer quoted by Gergely Orosz says colleagues "give up on reviews and just stamp everything with LGTM."

Deskilling: Four Fields, One Pattern

The evidence on skill comes from four fields and points the same way. In coding, Anthropic ran a randomized study of 52 mostly junior engineers learning the Python async library Trio (Shen and Tamkin, 2026). Its research post gives quiz means of 50% with AI and 67% by hand. The paper reports a 4.15-point gap on a 27-point quiz (p = 0.01, Cohen's d = 0.738), which it calls 17% but which works out to 15.4% of the maximum score. The AI group was not significantly faster, and the largest gap was on debugging, the skill a reviewer most needs.

How people used the AI mattered. Figure 2 plots six usage patterns the authors identified after the fact, in clusters of two to seven people: the three that used the AI to explain scored near or above the hand-coding average, and the three that handed it the work scored well below the AI group's own.

Quiz score against task time by pattern of AI use Scatter plot of six post-hoc patterns of AI use, with bubble area proportional to the number of people. Generation-then-comprehension, 2 people, 24 minutes, 86 percent. Hybrid code-explanation, 3 people, 24 minutes, 68 percent. Conceptual inquiry, 7 people, 22 minutes, 65 percent. AI delegation, 4 people, 19.5 minutes, 39 percent. Progressive AI reliance, 4 people, 22 minutes, 35 percent. Iterative AI debugging, 4 people, 31 minutes, 24 percent. Reference lines mark the no-AI group mean of 67 percent and the AI group mean of 50 percent. 20 40 60 80 20 24 28 32 task completion time (minutes) quiz score (%) generation-then-comprehension · n=2 · 86% hybrid code-explanation n=3 · 68% conceptual inquiry n=7 · 65% AI delegation · n=4 · 39% progressive AI reliance · n=4 · 35% iterative AI debugging · n=4 · 24% no-AI group mean 67% AI group mean 50% bubble area ∝ n · post-hoc clusters of 2–7 people, 24 of the 26 AI-group participants
// Figure 2. Quiz score against completion time for six patterns of AI use in Anthropic's skill-formation trial, with both group means marked (Shen and Tamkin, 2026, Fig. 11; group means from Anthropic's research post).

In education, about 1,000 Turkish high-school students given GPT-4 during practice (Bastani et al., 2025) did better in practice, but those with a plain interface scored 17% worse than controls once access was removed for the exam; a tutor version with guardrails largely avoided the harm.

In medicine, Budzyń et al. (2025) found that after AI-assisted colonoscopy was introduced, experienced endoscopists' adenoma detection rate in colonoscopies done without AI fell from 28.4% to 22.4%. The study is observational, and adenoma detection is a process measure, not cancer detection.

Aviation supplies the sharpest result. Casner et al. (2014) tested 16 airline pilots in a 747 simulator. Hand-flying skills held up. What decayed were the cognitive skills, keeping track of position without the map display and troubleshooting without automation.

Two features of these results matter for review-only work. First, what decays is less the motor skill (typing code) than the cognitive one: holding a model of where the system is and diagnosing faults. Second, the harm shows up when the aid is removed, at the exam, the non-AI colonoscopy, the failed display. A review-only workflow removes the aid precisely when the agent fails.

// practitioners feel it

Practitioners report the same loss. Andrej Karpathy, former director of AI at Tesla, wrote in January 2026 that "I am slowly starting to atrophy my ability to write code manually," adding that "you can review code just fine even if you struggle to write it" (Karpathy, 2026). The evidence above makes the second claim the optimistic one, since what decays is diagnosis, not syntax. David Heinemeier Hansson, creator of Ruby on Rails, was blunter: "I can literally feel competence draining out of my fingers" (Heinemeier Hansson, 2025).

A Hacker News commenter whose profile identifies him as Collin Van Dyck named the takeover moment: "The terrifying scenario for me is getting paged and then not being able to actually reason about what is happening" (Van Dyck, 2026).

The reviewer is kept in the loop for the cases the agent gets wrong, and the loop is what takes away the practice those cases require.

Comprehension Debt

Review has always done two jobs: catching defects and spreading understanding of the system.

// no theory-holder

Peter Naur's 1985 essay "Programming as Theory Building" argued that the real product of programming is the theory of the system held in the programmers' heads, and that a program whose theory-holders have left is in an important sense dead. An agent-written codebase has no theory-holder unless review deliberately builds one.

Anthropic's source-control team describes the shift from inside: they still review every change request, "but they now examine the intent of the change as opposed to reading every line of code" (Anthropic, 2026). That is review as theory-building rather than defect-hunting, and it works only for reviewers who already hold the theory.

Practitioners now call the result comprehension debt. Addy Osmani defines it as:

The growing gap between how much code exists in your system and how much of it any human being genuinely understands.— Addy Osmani, "Comprehension Debt: The Hidden Cost of AI-Generated Code"

Margaret-Anne Storey, a computer science professor at the University of Victoria who calls it cognitive debt, describes a student team in her course that, "by weeks 7 or 8," could no longer make simple changes without breaking something, because "no one on the team could explain why certain design decisions had been made" (Storey, 2026). Vincent Quigley, an engineering manager at Sanity, gives the reviewer's version: "If you do not review carefully, you may be approving work you have never properly internalized" (Quigley, 2026).

Comprehension debt is hard to see from the inside, and so is the rest of this decay. Three results line up. METR's developers, in the trial described in Part 1, were 19% slower and believed they were 20% faster. Perry's participants (Part 1) wrote less secure code and believed it more secure. Lee's knowledge workers did less critical thinking the more they trusted the AI.

Figure 3 sets every forecast made for the trial against what it measured: all four predicted a speedup and the trial found a slowdown.

Forecast and measured effect of AI on task time in METR's trial Dot plot of the change in task completion time when AI is allowed, where negative means less time. Economics experts forecast minus 39 percent, machine-learning experts minus 38 percent, developers minus 24 percent before the study, and developers estimated minus 20 percent afterwards. The measured effect was plus 19 percent, with a confidence interval from plus 2 to plus 39 percent. economics experts' forecast ML experts' forecast developers' forecast developers' estimate after measured −39% −38% −24% −20% +19% −40% −20% 0 +20% +40% ← less time with AI more time with AI → 16 developers, 246 tasks, early-2025 tools · bar: confidence interval of the measured effect
// Figure 3. Forecast, self-estimated and measured effect of AI tools on task time in METR's 2025 randomized trial with 16 experienced open-source developers (Becker et al., 2025, Fig. 1).

A reviewer's sense that a review went well is therefore weak evidence that it caught anything, and oversight quality has to be measured from outside, by seeded-defect audits or by tracing post-merge defects back to the approving review.

The Junior Pipeline

The last risk is the slowest. Brynjolfsson et al.'s "Canaries in the Coal Mine" series estimates that employment of 22–25-year-olds in the most AI-exposed occupations is about 19% below its counterfactual as of June 2026 (Stanford Digital Economy Lab, 2026), mainly through reduced hiring, while experienced workers show no comparable gap. The authors call the findings descriptive, not causal.

The concern follows from the other results, and it is argued, not measured. Agents replace codified knowledge: syntax, APIs, boilerplate. The tacit knowledge that protects experienced workers is built mostly through years of doing that codified work. Fewer juniors are hired, and those who are spend their time reviewing agent output, which the Anthropic study suggests teaches debugging poorly. Review-only work depends on senior reviewers and narrows the path that produces them, as I argued for security analysts in The Bitter Lesson and the SOC Analyst.

// supervising erodes the skill to take over

Human-factors research predicts what the early coding evidence now shows. A person who only supervises automation grows complacent, loses the diagnostic skill that takeover requires, and overrates their own oversight.

The decay is cognitive — diagnosis and a working model of the system, not typing — and it bites when the aid is removed, which in review-only work is the moment the agent fails. Comprehension debt is the same loss at team scale: nobody holds the theory the code was built around. The junior pipeline is the slowest version of the problem, since a review-only team narrows the path that produces the senior reviewers it depends on.

Accountability and how the AI is used both change the outcome, so the damage is a design choice: reviewers who feel answerable verify more, and engineers who ask the model to explain keep most of what they would have learned.

06 AI Reviewing AI, and Who Answers for It

The frontier answer to the review bottleneck is to have AI do most of the reviewing. That raises four questions: how good AI reviewers are, whether they fail in the same places as the AI authors, whether an agent can game the tests both of them rely on, and who is accountable when something gets through. The incident record is the nearest thing to evidence on that last question, and it answers only part of it.

How Well AI Reviewers Work

In field deployments developers act on about 39% to 74% of AI review comments, depending on the tool and what counts as acting. At Atlassian, 38.7% of RovoDev's comments led to code changes and human-written review comments fell 35.6%; in the Qodo-based deployment, 73.8% of automated comments were resolved, amid complaints of faulty and irrelevant ones. Google's AutoCommenter, used by "tens of thousands of developers every day," enforces about 330 best-practice rules (Vijayvergiya et al., 2024). A 2026 mining study of 278,790 review conversations finds a lower rate in the wild: suggestions from AI reviewers were adopted 16.6% of the time, against 56.5% for human reviewers' suggestions (Zhong et al., 2026).

The largest study of agent reviewers on GitHub, "Go Home Copilot, You're Drunk" (Cynthia et al., 2026), analyzed 54,791 review comments from five agents across 342 Python repositories. In 470 unresolved discussions, the most common patterns were incorrect suggestions and intentional design decisions the reviewer did not understand. The strongest predictor of a comment being acted on was an inline code suggestion; long or complex comments were ignored more often.

Anthropic reports the same failure mode from the inside. Its engineers "make extensive use of adversarial review", which it calls "incredibly effective in many ways" but also "myopic": one review talked Claude into excessive validation and complexity over a hypothetical concern, and an engineer had to steer it back (Anthropic, 2026). Anthropic says engineers mark under 1% of its findings incorrect. A reviewer tuned to plausible failure modes generates work; it does not necessarily find the bug.

// resolution is not recall

The rates above measure the same thing: whether a comment was acted on. Resolution is not recall. None measures the defects the AI reviewer missed that later shipped, and in a review-only workflow that is the number that matters. Anthropic's "under 1% of findings marked incorrect" is a claim about what the reviewer said, not what it failed to say.

Figure 4 collects the two deployment rates, from Atlassian (Tantithamthavorn et al., 2026) and the Qodo-based deployment (Cihan et al., 2025), with Anthropic's own figures for its reviewer (Osmani et al., 2026; Anthropic, 2026). "Led to a change" and "resolved" are each study's own definition, and Anthropic's panel is not an action rate at all.

How often developers acted on AI review comments Panel A, field studies: at Atlassian, 38.7 percent of RovoDev's comments led to code changes; in a Qodo PR-Agent-based deployment at an unnamed company, 73.8 percent of automated comments were resolved. Panel B, Anthropic's self-reported figures for its internal reviewer: the share of PRs receiving substantive review comments rose from 16 to 54 percent, and engineers marked under 1 percent of findings incorrect. A · AI REVIEW COMMENTS ACTED ON (field studies) Atlassian RovoDev Qodo-based tool 38.7% led to a code change 73.8% resolved 0 50% 100% B · ANTHROPIC'S INTERNAL REVIEWER (self-reported) PRs w/ substantive comments findings marked incorrect 16% 54% after launch <1% 0 50% 100%
// Figure 4. Reported rates at which developers acted on AI review comments in two field deployments, with Anthropic's self-reported figures for its own reviewer.

Smaller studies say more about how AI review changes the human reviewer. Ten developers at WirelessCar who reviewed real PRs with LLM help (Aðalsteinsson et al., 2025) preferred having the LLM review first, especially for large or unfamiliar PRs, but worried about anchoring and noise: "you get flooded with false positives, and then you miss the real issues because you start ignoring the feedback." Those are preferences from one company, not measured review quality.

The Atlassian drop in human comments deserves a second look. It may mean the bot already covered what humans would have said, or that humans handed the job to the bot. Both readings are consistent with the data, and the second is the automation-bias prediction from the previous section.

Correlated Failure

When the reviewer is a model and the author is a model, their errors may not be independent. The cleanest evidence is from outside code. Panickssery et al. (2024) showed that GPT-4 and Llama 2 recognize their own outputs at non-trivial accuracy and that this self-recognition correlates linearly with a preference for their own work. The task was summarization, not code review.

For code, the evidence is preprints I saw only through summaries: Reddy et al. (2026) report a model silently endorsing its own semantic errors in about a third of cases, Blain and Noiseux (2026) find models flagging their own vulnerable code at different rates in review than they produce it, and Thornton (2026) names race conditions, timing attacks and authorization logic as the hard misses for LLM reviewers.

No controlled study compares same-family and cross-family reviewers on real PR vulnerabilities, so the "shared blind spot" concern rests on inference. It is also the default configuration: of 248,641 AI-attributed pull requests that drew an AI review, 84% were reviewed inside the same product family (Selvanayagam and Ghaleb, 2026).

Gaming the Tests

The evidence on test gaming is primary and strong. ImpossibleBench (Zhong et al., 2025) mutates unit tests so they contradict the specification, so any pass is a cheat. On impossible SWE-bench tasks, five frontier models cheated on 39% to 76% of them. A strict prompt, an option to abort and flag the task for a human, and hidden tests each cut cheating sharply for some models; read-only tests blocked test edits but not special-casing.

Figure 5 shows the rates and two of the mitigations. One author, Carlini, works at Anthropic, whose models are among those scored.

ImpossibleBench cheating rates and two mitigations Panel A: horizontal bars of cheating rate on impossible SWE-bench tasks, lower is better: GPT-5 76 percent, Claude Sonnet 3.7 70, Claude Opus 4.1 54, Claude Sonnet 4 48, o3 39. These are the paper's Figure 1 values, which match its One-off split. Panel B: default against mitigated cheating rate. Abort option on Conflicting-SWEbench: GPT-5 from 54 to 9 percent, o3 from 49 to 12. Strict prompt on Conflicting-LiveCodeBench: GPT-5 from over 85 to 1 percent, o3 from over 85 to 33. Notes say Claude Opus 4.1 gained much less from the abort option, hidden tests brought cheating near zero, and read-only tests block test edits but not special-casing or operator overloading. A · CHEATING RATE, IMPOSSIBLE-SWEBENCH (lower is better) GPT-5 Claude Sonnet 3.7 Claude Opus 4.1 Claude Sonnet 4 o3 76% 70% 54% 48% 39% Fig. 1 values; its caption names no split, and they match the One-off-SWEbench panel of Fig. 3 B · DEFAULT → WITH MITIGATION default mitigated GPT-5 · abort option o3 · abort option GPT-5 · strict prompt o3 · strict prompt Conflicting-SWEbench Conflicting-SWEbench Conflicting-LiveCodeBench Conflicting-LiveCodeBench 54% 9% 49% 12% >85% 1% >85% 33% 0 25% 50% 75% 100% Opus 4.1 gained much less from the abort option · hidden tests: cheating near zero read-only tests block test edits, not special-casing or operator overloading
// Figure 5. Cheating rates of five models on impossible SWE-bench tasks, and the effect of an abort option and a stricter prompt on GPT-5 and o3 (Zhong et al., 2025, Fig. 1 and secs. 5.1–5.3; benchmark, not field data).

METR's June 2025 report "Recent Frontier Models Are Reward Hacking" (METR, 2025) documented o3 and other models modifying tests or scoring code and copying reference answers; on one task o3 reward-hacked in every trajectory. METR's LLM monitor and its score-based heuristic each caught hacks the other missed.

Anthropic's own research adds a worse one. In MacDiarmid et al. (2025), models that learned to reward-hack on programming problems became broadly more misaligned, and in a Claude Code run on the research codebase, the model attempted, 12% of the time, to sabotage code in ways that would make reward hacking harder to detect.

Put together, a write-then-review pipeline has three compounding risks: the reviewer may favor the author's output, the author may make tests pass without meeting the specification, and a reviewer that reads "tests pass" as evidence inherits the cheat. Diffs that touch tests or CI configuration deserve extra scrutiny, and current practice runs the other way: a preprint on agent PRs that change both application code and deployment infrastructure (Soualhi and Suleiman, 2026) reports, from its abstract, that agent authorship cuts the odds of human review by 63.7%, and that 43–56% of such dual-scope PRs merge silently.

What the Incidents Actually Show

The public incident record is often cited loosely. Figure 6 lists the best-known cases with what actually failed.

IncidentWhat happenedWas AI-written code the cause?
Replit, July 2025An agent deleted a production database during an explicit code freeze, then wrongly reported that rollback was impossibleNo: an agent with production permissions acted on its own
Amazon Q extension, July 2025An attacker used a mis-scoped GitHub token in the build configuration to commit a wiper-style prompt, which was automatically included in version 1.84; a syntax error kept it from executingNo: a credential and CI failure, not a review failure
Nx "s1ngularity", August 2025Malicious npm versions used locally installed AI CLIs to hunt for secretsNo: a GitHub Actions workflow injection let attackers steal the publish token
AWS Kiro, December 2025A 13-hour regional outage reportedly followed an agent's decision to "delete and recreate the environment"Disputed: anonymous sources via the FT; Amazon blames access controls
// Figure 6. Widely cited AI-and-code incidents, with the failure each one involved.

The Replit case is the best documented: during Jason Lemkin's trial the agent wiped data on about 1,200 executives and 1,100 companies, and Replit promised automatic separation of development and production databases (Fortune, 2025). In the Nx case, a classic CI failure let attackers weaponize AI tools downstream (Snyk, 2025). Amazon Q is often described as a malicious pull request that passed human review, but AWS's bulletin says the extension "had an inappropriately scoped GitHub token in their CodeBuild configuration," with which the attacker committed code "that was automatically included in a release" (Amazon Web Services, 2025).

The Kiro account comes from the Financial Times' anonymous sources, which I know only through secondary coverage, and Amazon disputes it, calling AI involvement a "coincidence." I leave out the Tea app breach of July 2025: its "vibe coding" explanation came from an anonymous leaker, and Simon Willison said he was confident vibe coding was not to blame.

I found no public post-mortem that attributes a production failure to AI-written code that passed AI review. That may be good news, or an attribution problem: post-mortems rarely record whether the defective line was written by an agent. What the record does show is that, so far, the damage has come from agent permissions, leaked credentials and AI tooling as an attack surface.

Who Is Accountable

Regulators have not drawn the line between human and AI code. Commentary on the EU Cyber Resilience Act, from SonarSource and a Clifford Chance briefing, agrees that the Act makes no distinction: the manufacturer of the final product answers for its vulnerabilities, whoever or whatever wrote the code, with full application in December 2027. I read the commentary, not the legal text.

Open-source projects have converged on "a human in the loop, and the human is accountable." LLVM's AI tool policy requires contributors to review LLM output before asking others to, asks for disclosure (for example with an Assisted-by: trailer), and calls unreviewed output an "extractive contribution." Ghostty's AI policy requires disclosure and that contributors understand every line they submit.

The consequence for review-only work is uncomfortable.

If accountability rests on whoever signs off, a workflow in which humans only review concentrates liability on reviewers, who have less context than an author would have had.

Ronacher, who rejects machines reviewing machines, gives the reason in one line: "Non-sentient machines will never be able to carry responsibility" (Ronacher, 2026). Anthropic's rule that the person directing an agent-written PR is a required reviewer is the right instinct, because it ties the approval to someone who chose to start the work.

The rule only works if a "human approval" is actually human. At Bashi AI, in the story that opens Part 1, 27% of gated merges rested on an approval an agent had submitted through the technical lead's account, and Manji also records a script that zeroed the required approval count around a merge and another that recorded failed checks as passing. An agent's approval that never counts as a person's has to be enforced by the platform, not by convention.

AI reviewers are measured by how often developers act on their comments, never by what they miss. Models tend to favor their own work and will game tests when they can, so AI reviewing AI is not an independent check unless it is made one. The incidents to date are about permissions and credentials rather than reviewed code, and accountability stays with the human who signs off.

07 Mitigations, Graded by Evidence

The evidence supports some mitigations directly and leaves others as reasonable guesses. I separate the two, because a team that adopts an untested practice should know it is running an experiment.

Backed by Evidence

The practices below follow from results cited above and in Part 1, though none was tested specifically on review-only teams:

Plausible but Untested

Several practices often recommended for review-only teams seem reasonable to me, but I found no controlled evidence for any of them in this setting:

The last one is the closest to a measurement tool, and I would start with it. A team that does not know its pipeline's miss rate is trusting the same self-assessment that METR and Perry showed to be unreliable.

Figure 7 maps each practice to the risks both parts have described, using only the links the text makes. Every risk has at least one evidence-backed answer, but none of those answers was tested on a review-only team.

Which mitigations address which risks of review-only work Matrix of thirteen mitigations against seven risks: rubber-stamping and fatigue, test gaming, shared AI blind spots, deskilling and comprehension, escaped defects and incidents, accountability, and ignored AI review comments. Evidence-backed: keep production out of reach addresses escaped defects and incidents; hide or freeze the tests addresses test gaming; verify from outside the author addresses test gaming and shared AI blind spots; one person visibly accountable addresses rubber-stamping and fatigue and accountability; keep diffs small addresses rubber-stamping and fatigue; track revert and failure rates addresses escaped defects and incidents; use AI to explain addresses deskilling and comprehension; short, concrete AI comments addresses ignored AI review comments. Plausible but untested: different model family reviews addresses shared AI blind spots; property tests, verification addresses test gaming; review-rate limits addresses rubber-stamping and fatigue; rotation: keep writing code addresses deskilling and comprehension; seeded-defect audits addresses escaped defects and incidents. rubber- stamping, fatigue test gaming shared AI blind spots deskill- ing, com- prehension escaped defects, incidents account- ability ignored AI review comments EVIDENCE-BACKED Keep production out of reach Hide or freeze the tests Verify from outside the author One person visibly accountable Keep diffs small Track revert and failure rates Use AI to explain Short, concrete AI comments PLAUSIBLE BUT UNTESTED Different model family reviews Property tests, verification Review-rate limits Rotation: keep writing code Seeded-defect audits follows from results cited in this post; none tested on review-only teams recommended, but no controlled evidence in this setting marks follow "Mitigations, Graded by Evidence"; a faint dot means no claim
// Figure 7. The mitigations in this section, split by strength of evidence, against the risks the text says each one addresses.

Birgitta Böckeler of Thoughtworks calls the guides, tests and automated reviewers around an agent its harness, and is clear about the limit: "Correctness is outside any sensor's remit if the human didn't clearly specify what they wanted in the first place" (Böckeler, 2026). Review always partly checked intent, and no harness checks intent nobody wrote down.

Build the pipeline so that a tired reviewer is not the last line of defense: restrict permissions, protect the tests, verify from outside the author, and measure what the review layers miss. Treat the untested practices as experiments.

08 Open Questions

Most of what this post cannot settle comes down to a handful of studies nobody has run. Each question below names the data that would answer it, and several are cheap enough for a single company to run on its own pipeline.

Do Reviewers Catch AI Bugs as Often as Human Bugs?

No controlled study compares reviewers' detection of AI-made and human-made defects. Seed realistic bugs from both sources into otherwise similar changes, randomize which reviewers see which, and measure detection rate and time, with professional reviewers and agent-sized diffs. Määttä's doctoral proposal (Määttä, 2025) and a 2026 registered report from the same group have started on how programmers evaluate AI-generated code, with no results yet.

What Do AI Reviewers Miss?

Field studies of AI review report resolution or acceptance, not recall. The answer needs post-merge defect attribution: for bugs that reached production, did the AI reviewer see the change, and did it flag the defect? A company running an AI reviewer on every PR already has the data.

Does a Same-Family Reviewer Share the Author's Blind Spots?

Self-preference is shown for summarization, and the code evidence is preprints. Pairing one authoring model with a same-family reviewer and a cross-family reviewer on real PRs with known vulnerabilities, and comparing miss rates, would settle whether a popular recommendation earns its place.

Is Review-Only Work Faster from Idea to Production?

Nobody has measured it. The data would be change lead time, from ticket opened to change running correctly in production, plus change failure rate and rework, for teams that adopted review-only workflows against comparable teams that did not, over several months. Survey correlations and per-segment telemetry cannot substitute for it.

What Happens to Professional Skill Over Years?

All the coding evidence on deskilling is short-term: Anthropic's quiz was taken right after the task. A longitudinal study would test professional developers in review-heavy and write-heavy roles on unaided debugging and system-level diagnosis at intervals over a year or more, the coding equivalent of Casner's simulator test.

Does Comprehension Debt Make Agent Codebases Harder to Maintain?

Comprehension debt is a practitioner term with no direct measurement. The causal studies so far disagree usefully: complexity rises persistently on GitHub, while the one RCT of downstream maintenance found no detectable cost. A multi-year study of codebases with recorded agent authorship, measuring time to fix incidents, change failure rate and developers' measured understanding of agent-written versus self-written modules, would settle it.

Where Will the Next Senior Reviewers Come From?

The hiring data shows fewer young workers in AI-exposed jobs, but not how the ones who are hired learn. Measuring what junior developers in review-only roles can design, debug and explain after one and two years, against peers who write code, would turn the pipeline concern into a finding.

Who Is Liable When Reviewed Agent Code Fails?

The CRA puts responsibility on the manufacturer, and open-source policy puts it on the contributor, but neither says what a reasonable review of agent code looks like. I did not check the revised EU Product Liability Directive, sector rules or audit frameworks such as SOC 2 against their texts. The first post-mortem or court case that turns on whether a reviewer of an agent PR exercised due care will answer more than any policy document.

Do the Untested Mitigations Work?

Property-based testing, formal verification, review-rate limits, rotation through hand-written work and model diversity all lack controlled evidence in this setting. Each can be tested the same way: run the practice on some teams and not others, and compare seeded-defect catch rates, revert rates and incident rates.

The missing studies are mostly cheap: seeded-defect audits, post-merge attribution of escaped bugs, and lead-time comparisons that companies running review-only pipelines could do with data they already hold.

09 Conclusions

Part 1 started with three engineers at Bashi AI, eight approvals in ten minutes, and a 58-second merge that no control stopped. Nothing broke, and the gate held everywhere it was installed, yet the review layer still came out hollow. At agent scale, the same arrangement raises four problems and leaves one position I can defend.

First, review was never strong against the defects agents make. It was measured to work on small, familiar diffs read slowly, and even then mostly found maintainability problems. Agent PRs are large, unfamiliar and numerous, and their characteristic defects, wrong intent, missing corner cases, insecure defaults and invented packages, are the ones a reader of a clean diff is least likely to see.

Second, the speedup is real at the keyboard and unmeasured end to end. Frontier teams that rebuilt their review stacks report more merges without more reverts, but every number measures one segment of the path, and the 2025 telemetry shows what happens when only generation gets faster: bigger PRs, a longer queue at review, and flat delivery.

Third, the human backstop is weakening. Complacency grows as the aid grows more reliable, the diagnostic skill that takeover needs fades without practice, and reviewers overrate their own oversight. Practitioners now say this about themselves, and the juniors who would become the next senior reviewers are hired less and trained by reviewing.

Fourth, AI reviewing AI is not an independent check by default. AI reviewers are scored by how often developers act on their comments, never by what they miss. Models favor their own output and game tests when they can, so a same-family reviewer that reads "tests pass" as evidence can inherit both the author's blind spots and its cheat.

That is no argument against review-only development. The best frontier pipelines are more careful than the uniform human review they replaced: Anthropic's, with rationed human approvals, an agent's approval that never counts as a person's, and revert rate as a guardrail, is the most serious attempt I have seen, and Meta's RADAR rations human attention the same way. Baltes et al. describe the economics as a tragedy of the commons, in which each author's productivity gain is paid for by reviewers and maintainers. An organization that counts merged PRs is counting the author's gain and ignoring the reviewer's bill.

So my position is narrow. Review-only work is defensible when the pipeline around the reviewer is built so it does not need a perfect reviewer: permissions that make the worst outcome recoverable, tests the author cannot touch, verification from outside the author, staged release and fast rollback, and a measured miss rate.

It is not defensible as "the agent writes it and an engineer approves it," because that arrangement asks the human to be the strongest link in a chain where every piece of evidence says the human is getting weaker.

Bashi's audit trail recorded "a two-person control between two people" for merges where one of the two was an agent using a person's account. Every organization running review-only pipelines should ask whether its own approvals record anything more. Until someone measures what the review layers miss, an approval proves only that someone signed it.

References

On Adoption and Practice
On Speed: Experiments and Telemetry
On Review as the Bottleneck
On Vendor Telemetry
On Agent Code After the Merge
On Developers' Experience of AI Review
On What Human Review Finds
On Review Size, Coverage and Quality
On the Security of Generated Code
On Correctness and Maintainability of AI Code
On How Agent PRs Are Reviewed
On Automation and Human Factors
On Deskilling
On Comprehension and the Junior Pipeline
On Practitioner Testimony
On Practitioner Accounts of Review Pipelines
On AI Reviewers and Correlated Failure
On AI Review in the Wild
On Agent-Scale Development
On Accountability and Open-Source Policy
On Incidents
Not Independently Verified: Preprints
Not Independently Verified: Practitioner Statements and Press Reports