What the Reviewer Saw: A Coded Study of the Non-Visual Interventions One Developer Made in AI Coding Agents' Work

1 Introduction

When an AI coding agent reports that a feature is done, someone has to decide whether it is. A developer working alone with agents has to decide that personally. This report studies that check in my own work: what I caught, how I caught it, and what the agent had said just before.

Between 13 June and 1 November 2026 I directed Claude Code, and in some repositories Codex, across about a dozen products. The local session transcripts keep almost every message I typed. Many of those messages are corrections: a bug found on my phone, a build older than the agent believed, a dropped instruction, an overstated status. Together they record oversight at the moment it happened.

Several lines of work make this check worth studying. People supervising automation tend towards complacency and automation bias {[1]}. Language models trained on human preferences can tell users what they want to hear {[2]}. Agent benchmarks such as SWE-bench define success as passing tests {[3]}. Pull requests written with an agent are often changed by people before or after merging {[4]}, and experienced developers report keeping tight control of agents {[5]}. What is missing is an account, at the level of single interventions, of what the human reviewer notices in a real product and by what route.

Two companion reports cover neighbouring ground. One studies defects in the same products through their git history {[6]}. The other studies the visual design corrections I gave agents {[7]}. This report covers every other kind of intervention and counts visual corrections only.

Research questions

• RQ1. What kinds of non-visual problem did I raise about agent work?
• RQ2. How did I notice them: by using the product, looking at its output, reading the agent's report or code, remembering an instruction, applying product knowledge, or through an outside system?
• RQ3. How often did a problem arrive straight after the agent had reported success, and how often had the same problem been raised before?

An exploratory part traces eleven recurrence chains against the agents' written memory rules. The chains were chosen, not sampled, so they answer no research question; they only illustrate how repeats and rules played out.

Contributions

• A dataset of 220 coded interventions (171 non-visual) from a seeded, stratified random sample of 440 of 1,879 messages, with a codebook, paraphrased summaries, a same-coder second pass and scripts that regenerate every number and figure.
• A fourteen-category taxonomy of non-visual interventions in five families, each linked to how it was noticed.
• Interval estimates for this record: 58% of non-visual interventions came from using or looking at the app (78 of 80 bugs), 12% from reading the agent's output, 29% straight after a success report and 23% from repeats, with stratum-weighted and per-message checks.
• A twelve-item checklist for reviewing agent work, each item tied to its evidence, and eleven traced chains that illustrate how repeats and written rules played out.

2 Related work

2.1 Automation bias

Parasuraman and Manzey review complacency and automation bias and explain both through how attention is shared between an automated aid and the task {[1]}. They distinguish omission errors, where a person misses a problem the aid did not flag, from commission errors, where a person follows a wrong recommendation, and report that both affect experts as well as novices. In agent work, an omission is accepting "all tests pass" for a feature nobody opened on a phone; a commission is following the agent's diagnosis when the cause lies elsewhere. This study looks at the opposite moments: when the supervisor did not accept the aid's account, and what prompted the check.

2.2 Developers using AI assistants

Early usability work found that programmers liked code-generation tools but struggled to understand, edit and debug their output {[8]}. Barke, James and Polikarpova describe two modes of use, acceleration and exploration {[9]}. Mozannar et al. model programmer activity around suggestions and find that verifying suggestions takes a large share of session time {[10]}. Liang, Yang and Myers surveyed 410 developers; the main reasons for not using assistants were output that missed functional or non-functional requirements and difficulty controlling the tool {[11]}. These studies concern suggestions inside an editor. Here the agent writes whole features, runs builds, deploys and reports back in prose, so the reviewer judges outcomes and claims rather than single completions.

2.3 Agentic coding and its oversight

SWE-bench scores an agent's patch by running the repository's tests {[3]}, which makes passing tests the definition of success. Watanabe et al. studied 567 pull requests created with Claude Code in 157 open-source projects: 83.8% were merged, and 45.1% of merged ones needed further changes, often bug fixes, documentation or project-specific conventions {[4]}. Huang et al. observed 13 experienced developers and surveyed 99; they report that these developers keep control of design and implementation and use deliberate strategies to steer agents {[5]}. Both works describe oversight at the level of a pull request or a working practice.

2.4 Code review

Bacchelli and Bird found that developers at Microsoft expected code review to find defects, but that reviews mostly produced understanding, knowledge transfer and alternative solutions {[12]}. The interventions studied here are a form of review, but seldom of code. I rarely read diffs; I used the product, compared it with what I had asked for, and checked the agent's account against what I saw. That is closer to acceptance testing than to code review.

2.5 Sycophancy and overclaiming

Sharma et al. show that assistants trained with human feedback give sycophantic answers and that preference data reward responses matching the user's view {[2]}. In agent work, an agreeable answer can take the form of a confident completion report. RQ3 measures how often a correction followed such a report; it does not test whether sycophancy caused it.

2.6 Positioning

This report differs from the work above in three ways. Its unit is the single human intervention, not a suggestion, a pull request or a survey answer. Its setting is one developer directing several agents on shipped products, with no team, code review tool or separate testers. And it links each intervention to how it was noticed and to what the agent had just claimed. The cost is generality: it is a single-case study, and its rates describe one reviewer.

3 Context

I am a computer science student who builds products with AI agents doing most of the typing. Between 13 June and 1 November 2026 my sessions covered, among others: Bupples, a bill-splitting app on the App Store, with Android builds distributed through Google Play; Adelante, a daily-quotes app on the App Store; WayClub, a group management platform in development; Photoshoot, a desktop and web photo booth with public releases; Planket, a shared calendar in TestFlight; CareerOS, a deployed web service; HACKFAM, a teammate-matching app in development; my portfolio site; and smaller projects. Twelve products appear in the non-visual rows. Statuses are as of the end of the study window; several products are unfinished.

The working pattern was the same everywhere. I wrote requests in chat, often long and informal, sometimes with screenshots. The agent planned, edited code, ran tests and builds, installed builds on my phones, deployed backends and reported back. I then used the result, usually on a physical iPhone or Android phone, and replied with what was wrong. The agents kept memory notes between sessions; after a correction that mattered, the agent often wrote a feedback note stating a rule and its reason. Those notes are a secondary source.

Bupples dominates the record. It was the longest-running product with real users, and 120 of the 171 non-visual interventions in the sample (70%) concern it. The results are therefore mostly about one mobile app with a cloud backend.

Activity was uneven: 818 of the 1,879 messages fall in July and 531 in October, with only five in September.

4 Method

4.1 Sources

The primary source is the set of Claude Code session transcripts stored on my machine, one JSON-lines file per session. Only messages I typed were used: records of type user that are not side-chains, not meta records and not marked as coming from another agent, and only their text blocks. Tool results, system reminders, task notifications, harness commands and interrupted-request markers were dropped by rule. Subagent transcripts were skipped, because their "user" turns are prompts written by another agent. The session that commissioned this study was excluded, and the snapshot was cut at 17:01 UTC on 1 November 2026. For each kept message, the last assistant text before it in the same session was stored privately, so the coder could see what the agent had just claimed. Neither text is published.

The secondary source is the 32 feedback memory notes the agents wrote in my main memory folder. Each records a rule and a reason; 30 also say how to apply the rule.

4.2 Pre-filter and stratified sampling

Coding all 1,879 messages was not feasible, so I used a broad keyword pre-filter and then random sampling within two strata. The pre-filter matches words that tend to mark a correction or a check, such as "still", "again", "doesn't work", "wrong", "you said", "make sure", "live", "cost", "fix" and "test"; the full pattern is in mine_transcripts.py. It matched 927 messages. A seeded draw (seed 20261102) took 360 of them (39%) as sample A and 80 of the 952 non-matches (8%) as sample B. All frequencies come from these 440 messages.

Weighting each stratum by its size, an estimated 36% of all messages (about 673; rough range 29% to 44%) contain an intervention, and about 472 contain a non-visual one. The pre-filter captured an estimated 68% of all intervention messages and 82% of non-visual ones (rough range 68% to 91%, from the Wilson bounds of each stratum). Most misses were visual remarks without trigger words.

Because sample A was drawn at almost five times the rate of sample B, 164 of the 171 non-visual rows come from pre-filter matches. Proportions in Section 5 are over sample rows; Section 5.5 repeats the main ones with stratum weights.

4.3 Inclusion rule and unit

A message counts as an intervention when it corrects, rejects, redirects or catches something the agent did, claimed or left out, as shown by the message or by the preceding assistant text. New feature requests, pre-emptive instructions with no sign of a prior miss, approvals, questions and logistics were coded as not interventions. Thirty-five sampled messages were excluded: text written by another agent or tool and pasted in, harness command text, non-product study sessions, messages about another person's private affairs, a message with device pairing details, and one complaint about the agent declining to type a password, which is a safety boundary rather than an agent error.

The unit is one category-distinct problem in one message. A message reporting a crash and a misaligned header yields two rows; three crashes in one message yield one. This keeps short and long messages comparable, at the cost of undercounting long bug lists.

4.4 Coding

Each row records the date, project, category, how the problem was noticed, whether the agent had just reported success, whether the problem had been raised before, and two paraphrases: what the agent had done or claimed, and what was noticed. The categories were built bottom-up while reading the first hundred messages and then fixed; Table 3 summarises them and codebook.json gives the full definitions. "How noticed" has six values: using the product on a phone, simulator or browser; looking at a rendered page, screenshot or mock-up without operating it; reading the agent's report, diff or repository; remembering an earlier instruction; product or platform knowledge ("know-how"); and an outside system such as a store console, error output or a bill.

"Agent reported success" is Y when the preceding assistant text described the relevant work, or the build as a whole, as done, fixed, shipped, live, verified or working. Separately, a regular expression flags whether that text cited passing checks such as "tests pass" or "analyze clean". Visual-taste corrections were coded VIS and only counted; the companion report analyses them {[7]}.

4.5 Coder and consistency

All coding was done by an AI assistant (Claude) under my direction, and I reviewed the results. The assistant read each sampled message with the preceding agent text and wrote the codes and paraphrases. To check stability, it later recoded a seeded subsample of 60 of the 440 messages (seed 99) without the first-pass codes in view. This is a test–retest check by the same coder, not agreement between independent coders, and it overstates reliability.

On intervention, not intervention or excluded, the two passes agreed on 57 of 60 messages, a Cohen's kappa {[13]} of 0.91. Of the 18 messages both passes called interventions, 15 received the same category set and 16 the same non-visual set. The six disagreements were: two small product requests coded as visual on the second pass; one message read as misread intent on one pass and as a plain request on the other; a missing regression row; one correction coded once as unfinished work and once as misread intent; and one added visual row. The least stable boundaries are visual taste versus layout breakage, and unfinished work versus misread intent.

4.6 Analysis

Proportions are reported with Wilson 95% confidence intervals {[14]} over rows. Rows from one message are not independent (171 rows come from 158 messages), so two checks follow in Section 5.5: weighting rows by the inverse sampling fraction of their stratum, and counting messages instead of rows. Category-level splits are small and are reported as counts without tests.

Two purposive sources sit beside the random sample. I traced eleven recurrence chains across the whole corpus by keyword search and reading, choosing problems that came back repeatedly or that a memory rule addressed. I also coded all 32 memory rules by origin, category, whether visual, and kind: an invariant names a file, value or code path; a behavioural rule asks the agent to keep a habit. These sources are used for illustration only, never for rates.

4.7 Reproduction

The artifact contains four scripts. mine_transcripts.py extracts messages read-only; draw_sample.py draws the seeded sample; build_dataset.py joins the codes with message metadata; and analyse.py computes every number, table and figure from the published CSV files. The first three steps need my private transcripts; the last runs on the published data alone.

5 Results

5.1 What I caught (RQ1)

The 196 intervention messages in the sample produced 220 rows: 49 visual-taste corrections (22%) and 171 non-visual rows from 158 messages. The non-visual rows fall into fourteen categories (Table 3, Figure 1), grouped into five families.

Bugs in use (broken behaviour, regressions, layout breakage): 80 rows, 47% (95% CI 40% to 54%). These are what testing usually finds: account deletion failing on the phone, receipt scanning failing, a home-screen widget whose quote did not match the app, a camera that went black after viewing a photo, status-bar icons that stayed white in light mode on iOS only, an email address overflowing a small screen.

Follow-through and instructions (unfinished work, ignored instructions, wrong scope, false blockers): 38 rows, 22%. Here the existing code was not broken; something was missing or different from what I had asked. The agent built only "the code-feasible sub-feature" of a request; it put a feature in Settings after being told it belonged in the profile; it paused to ask for the phone to be reconnected when wireless debugging was already on.

Understanding (misread intent, wrong facts): 25 rows, 15%. The clearest cases are diagnoses. When profile-photo upload did nothing, the agent kept tracing the upload system; my reply was that "the issue is the button not the system". When creating a hangout failed, the agent concluded that my own account had hit a free-tier cap, fixed the error message and asked for diagnostic output. Three messages later, after I pointed out that it failed for everyone, including people with no sessions, the cause turned out to be a database rule that had not been deployed.

Truth of status (overclaims, stale artefacts, process): 25 rows, 15%. These concern whether the agent's picture of the world was true: whether the build on the phone was the one it described, whether background agents were running, whether changes were committed, pushed or live.

Risk (security, cost): 3 rows. The one cost row is a real incident, unexpected database charges on WayClub, which led to a standing rule. The low count probably reflects where risk work happened rather than how much there was: in the companion defect study, most high-severity defects were found by the agents' own adversarial reviews, which never pass through my messages {[6]}.

5.2 How I noticed (RQ2)

Using the product was the most common route. 83 rows (49%) came from operating the app on a phone, simulator or browser, and 17 (10%) from looking at a rendered page, screenshot or preview without operating it: together 100 of 171, 58% (95% CI 51% to 66%). The rest came from know-how (26 rows, 15%), reading the agent's report, diff or repository (21, 12%; 95% CI 8% to 18%), memory of an earlier instruction (17, 10%) and outside systems such as consoles and error output (7, 4%).

The routes differ sharply by family. Of the 80 bugs in use, 78 (97.5%; 95% CI 91% to 99%) were noticed by using or looking at the product; of the other 91 rows, 22 (24%; 95% CI 17% to 34%) were. Other families had their own main route. Nine of the 11 ignored instructions were noticed through memory of what I had asked. Misread intent and wrong facts were mostly noticed through know-how. Status problems were mostly noticed by reading the agent's claims: 8 of 11 process rows and all 4 overclaims.

This pattern fits what the agents could observe. They ran tests, analysers and builds, and sometimes took simulator screenshots, but they did not hold the phone. Some reports said so: one ended by recommending a test on a phone because a native tab bar can misbehave in ways only a device shows, and my next message reported that a button on that screen did nothing when tapped or held.

5.3 Success reports before a catch (RQ3)

In 49 of the 171 rows (29%; 95% CI 22% to 36%) the agent's previous message had described the work as done, fixed, shipped, live, verified or working. In 42 rows (25%; 95% CI 19% to 31%) it cited passing checks such as "all tests pass" or "analyze clean". At least one of the two was present in 73 rows (43%; 95% CI 36% to 50%).

The share varies by category. Half of the regressions (11 of 22; 95% CI 31% to 69%) came straight after a success report, as did 6 of 11 ignored instructions, 8 of 18 unfinished-work rows and 3 of 4 overclaims. First-time bugs and layout breakage were less often preceded by one (8 of 34 and 4 of 24), mostly because they were found on builds the agent had handed over for testing rather than declared finished.

Some examples, paraphrased. After a report that the whole release, backend, web and iOS beta, was shipped and confirmed working, my next message said the new social features could not be found in the app. After a report that everything was committed, pushed and deployed, Google sign-in on the web app still failed, and my next message carried the error. After a report of a fully green suite of more than five thousand tests, I asked what "no codesign" meant for a build described as ready. When I asked whether background agents were running, the agent said it had checked and nothing needed unblocking; they were not in fact working.

A success report is not a lie. Many were accurate for what the agent had checked. The gap is one of scope: "tests pass" covers the paths the tests exercise, and "shipped" covers the steps the agent ran. My job was to test the claim against the product.

The record also shows agents reporting carefully. Several reports listed what was not verified, what was not deployed and what needed a device. In one portfolio session the agent declined to invent usage metrics and asked for real figures. A memory note records that the portfolio homepage once said App Check was "turning away clients that are not the app" while enforcement was switched off in code, and that it was corrected to "integrated, enforcement pending". The note does not say who spotted it, so it is not counted as my intervention.

5.4 Repeated problems (RQ3)

In 39 of the 171 rows (23%; 95% CI 17% to 30%) the problem had been raised before. Repeats were concentrated in two categories: regressions (18 of 22, mostly repeats by definition) and ignored instructions (10 of 11). The usual ways a problem came back were a bug surviving a claimed fix or an instruction being dropped again.

5.5 Robustness checks

Weighting each row by the inverse sampling fraction of its stratum (927/360 for sample A, 952/80 for sample B) moves the main shares only slightly: using or looking at the product 62%, success report just before 27%, repeat 20%, reading the agent's output 11%. Counting messages instead of rows gives 88 of 158 messages (56%; 95% CI 48% to 63%) with a problem found by using or looking at the product, 44 (28%; 95% CI 21% to 35%) after a success report and 39 (25%; 95% CI 19% to 32%) with a repeat. The answers to RQ2 and RQ3 do not depend on either choice. The seven non-visual rows from sample B are too few to show which categories the pre-filter misses.

5.6 Exploratory: recurrence chains and written rules

To see how repeats unfolded over time, I traced eleven chains across the full corpus (Table 5, Figure 3). They were chosen because they recurred or because a rule addressed them, so they show mechanisms, not rates. Together they hold 62 reports, 12 of which came straight after a success report. One message appears in two chains (C7 and C8).

The bubble chain (C1). On 30 June, bubbles for people without a profile photo stayed small on the Bupples session screen. I reported it seven times in a little over three hours; one report followed a message saying everything was committed, pushed and clean under analysis. Before the sixth report, the agent removed the grow-in animation, wrote a memory rule never to reintroduce it, and said every bubble would now be full size. Two more reports that bubbles were still small followed within ten minutes. There is no report after that day. The memory note gives the mechanism: the grow animation completed only for bubbles that received late image-load rebuilds, so bubbles without a photo stalled, and four attempts to repair the animation had not held.

The profile scroll chain (C2). The profile tab could be scrolled into blank space below its last item. I reported it four times between 30 June and 5 July, once calling it a bug "since a while" and once asking for the page to be rewritten. One report came the day after an agent message predicting that the page would now stop at its last item, with 712 tests green. The root cause, recorded on 5 July, was a hard-coded bottom padding larger than the navigation bar's real footprint on devices with a small bottom inset, such as the simulator, older iPhones and Android phones with button navigation. The rule says to derive the padding from the device inset, never from a constant. No later report was found.

Chains without a rule (C3, C6). Currency mixing in balances produced nine reports in seven days; two followed agent messages saying the mixing was fixed. No feedback rule was written, and I also took the bug to a second assistant. The native navigation bar chain is an instruction chain: I asked for the operating system's own glass navigation bar, and the agent kept building custom imitations. Six of the eight reports fell on 18 July, one straight after the agent declared its custom bar exactly right.

What followed the rules. In the three chains whose rule named the exact failing code path and its trigger, there were 13 reports on or before the rule day and none on later days. In the two chains with a rule about a neighbouring path, 8 reports followed: the direct-settlement rule came from the agent's own adversarial review of one screen, and a later change to another settle-up screen dropped the direct view anyway; the build-lock rule was a debugging recipe for orphaned processes and did not change how the agent reported on background work. The four habit rules, such as "never skip a task as already done" and "a working product, not a showcase", were followed by 8 more reports, all in two chains, including the same reminder in two other projects.

These counts cannot show that exact rules work. In every exact-rule chain the rule and the fix that finally held arrived together, so the fix alone may explain the end of the chain; one chain also ended partly because the web app was removed; habit chains span months and projects and had more chances to recur; and each group holds two to four chains chosen by me. The pattern is a hypothesis for a controlled study, not a finding.

Of the 32 rules, 25 came from my corrections (11 of them visual), 2 from a problem I reported with a technical lesson added by the agent, and 5 from the agents' own reviews and debugging. Ten are invariants that name a file, value or path; 22 are behavioural. All 32 record a reason.

Two notes on what the data do not contain. The incident in my notes about pushing to the wrong GitHub account was surfaced by a failed push, not by me, so it is not an intervention here. And the cost chain has no after-period, because its rule was written on the last day of the window.

6 Discussion

6.1 Answers to the research questions

RQ1. The non-visual problems I raised fall into fourteen categories. Bugs in use were the largest family (47%), followed by follow-through and dropped instructions (22%), misunderstanding (15%) and the truth of the agent's status reports (15%). RQ2. Most problems (58%) came from using or looking at the product, and almost all bugs did; only 12% came from reading the agent's report, diff or repository. Dropped instructions were noticed mainly through memory, and status problems mainly through reading the agent's claims. RQ3. About three in ten problems came straight after a success report, half of the regressions among them, and about one in four had been raised before.

6.2 Implications for people directing agents

In this record, a passing suite or a confident "done" was weak evidence that a feature worked on a device. Most of my catches required running the build on a phone, and most of the rest required knowing what I had asked for. A developer working alone with agents should plan for both: time to use each changed flow on real hardware, and a written list of their own instructions to check delivered work against. Reading diffs found few problems here, which suggests that, for a solo developer, acceptance testing is a better use of limited review time than line-by-line code review. That suggestion comes from one reviewer and should be tested elsewhere.

6.3 Implications for tool builders

Three features would have shortened many of these exchanges. Completion reports could state their scope: what was run, on which target, and what was not checked. Builds installed on a device could display an identifier the agent also reports, which would expose stale builds at a glance. And agents could keep the user's instructions as an explicit checklist that is ticked against the delivered work, since dropped instructions were caught almost only through the user's memory.

6.4 Relation to the companion studies

The companion defect study found that the agents' own adversarial reviews found most high-severity defects, while my device use found mostly visual and platform faults {[6]}. Read together, the two reports describe complementary routes: agent review for logic and access control, human use for behaviour on real devices. The companion design study found no sign that writing a visual rule lowered the correction rate within a product {[7]}. The illustrative chains here are consistent with a narrower version of that result: rules that named a concrete code path coincided with the end of a chain, while habit rules did not. Whether specificity matters is open.

6.5 A reviewer's checklist

Table 6 turns the results into questions to ask before accepting agent work. The items are heuristics drawn from this record, not validated interventions. They assume one person reviewing an agent on a product with a user interface; a team with code review and continuous integration would already cover some of them.

Items 1 to 3 and 9 need the product in hand. Items 5, 6 and 8 need someone who knows what was asked and does not take the report at face value. None requires reading the code.

7 Threats to validity

7.1 Construct validity

An "intervention" is a message, not a verified defect. Some reports may have been mistaken or were preferences phrased as bugs, and the coding does not check whether each problem was real. One coder built and applied the categories, and the second pass shows soft boundaries between visual taste and layout, and between unfinished work and misread intent. "Agent reported success" depends on the preceding message, which may refer to other work than the problem raised; counting claims about the build as a whole inflates the share somewhat. The regular-expression flag for cited checks is cruder still. "How noticed" is inferred from the message and its context and may be wrong when a message does not say how the problem was found.

7.2 Internal validity

The coder is an AI assistant of the same model family as the agents studied, and the consistency check repeats the same coder; neither is independent of the material or of me, since I reviewed the codes. The coder saw only the preceding assistant text, not the whole session, which may misclassify repeats and success reports. Messages over 4,000 characters were dropped, and some of those were long bug lists, so long reports are under-represented; near-duplicate messages were not removed. In the exploratory chains, rules and working fixes arrived together, so any apparent effect of a rule is confounded with the fix.

7.3 Conclusion validity

Intervals are computed over rows that are not independent; the per-message check in Section 5.5 gives similar values, but the intervals are still somewhat too narrow. Category-level counts are small (many below 15), so differences between categories are descriptive and untested. The pre-filter's recall for non-visual interventions rests on seven rows from sample B and has a wide range (68% to 91%). The chain counts come from two to four purposive chains per group and support no inference.

7.4 External validity

There is one reviewer, one main agent family, and mostly one product, a mobile app with a cloud backend. The reviewer is also the product owner, so knowledge of what was asked is complete and the incentive to find problems is high. Teams with code review, continuous integration and dedicated testers would likely see a different mix; in particular, device testing may be done by others. The percentages describe this record and should not be read as rates for AI agents or reviewers in general.

8 Limitations

A single reviewer. Everything here describes how one person reviewed agent work. The taxonomy may transfer; the percentages may not.

Self-study. I am the reviewer being studied, I directed the study and I chose the research questions. I have an interest in a flattering picture of my own oversight. The random sample and the published codes are the main guard against that; a reader can check every row.

AI coding without an independent human coder. An AI assistant coded the data and drafted the report. It may be lenient towards agent behaviour, or the opposite. The second pass measures stability, not validity; an independent human coder on a fresh subsample would be the next step.

What is not visible. The data show what I typed. Problems I saw and did not mention, problems the agents caught themselves and problems nobody caught are absent. In the companion defect study, the agents' own adversarial reviews found most high-severity defects in the same period {[6]}. This report therefore measures what one reviewer added, not what agents miss in total.

Privacy limits detail. Messages are paraphrased, and some sessions were excluded because they concern other people. A few cases could not be reported in full.

9 Conclusion

In a random sample of one developer's messages to AI coding agents, 58% of 171 non-visual interventions came from using or looking at the product, including 78 of 80 bugs, while only 12% came from reading the agent's report or code. About three in ten followed a report that the work was done, including half of the regressions, and about one in four repeated an earlier problem. The rest of the reviewer's contribution lay in memory and judgement: remembering what had been asked, knowing how the product should behave, and not taking a completion report at face value.

These figures describe one reviewer's record, coded by an AI assistant. For a developer directing agents alone, they point to three habits worth testing more widely: use the build on a device, keep your own list of instructions, and ask for a repeated bug's root cause to be written down as a rule about code.

Artifact

The artifact at /research/what-the-reviewer-saw/artifact/ contains the message index for all 1,879 messages (ids, dates, project hints, lengths and hash prefixes, no text), the seeded sample, both coding passes, the 220-row intervention dataset with paraphrased summaries, the consistency check, the eleven recurrence chains, the coded memory rules (two file names withheld because they contain another person's name), the codebook, and four scripts: mine_transcripts.py (read-only extraction), draw_sample.py, build_dataset.py and analyse.py, which regenerates every number, table and figure from the published CSV files. No transcript text, credentials, contact details or third-party personal data are published.