This post is for engineers and tech leads who merge pull requests that an agent wrote, and who suspect their review is mostly reading for style. Reviewing AI-written code fails in a predictable way: the diff compiles, the tests in the same diff pass, and the bug sits in a line nobody had a reason to doubt. Instead of offering another checklist built on opinion, this post measures one. It takes a small order library, seeds fourteen defects of the kinds that published studies of LLM output describe, and then runs six review checks against every one. The result is a map of which check catches which bug, how long each check takes, and the one defect that only a human with the spec could find.
The numbers below came from this environment:
Measured 2026-11-03
Windows 11 Pro 25H2 (build 26200), AMD Ryzen 5 8600G (6C/12T), 16 GB RAM
Dart SDK 3.9.0 (stable), package:lints 6.1.0, package:test 1.31.0
One clean package, fourteen single-defect variants, six checks per variant
Dart was chosen because its analyzer, lint sets and test runner ship with the SDK, so every check here runs from one install. The defect classes, however, are not Dart-specific. Each one has a direct equivalent in TypeScript, Python, Java and Go, and the closing sections map the checks across.
What Goes Wrong in AI-Written Code?
AI-written code mostly fails in ways that compile. Studies of LLM output find misread requirements, missing corner cases, calls to methods or packages that do not exist, insecure choices such as string-built SQL, and weak error handling. Because the code looks idiomatic and arrives with passing tests, review has to target those classes on purpose rather than read for style.
That summary rests on four sources worth reading in full. The first is Tambon et al.’s empirical study of bugs in LLM-generated code, which sorted 333 buggy samples into ten patterns. Misinterpretation, missing corner case and hallucinated object are the ones this bench reuses. Next, Spracklen et al., presented at USENIX Security 2025, generated 576,000 code samples across 16 models and found an average hallucinated-package rate of at least 5.2 percent for commercial models and 21.7 percent for open-source ones.
On the security side, Veracode’s 2025 GenAI Code Security Report tested more than 100 models on 80 tasks and found that 45 percent of the generated code samples introduced a security flaw. Finally, CodeRabbit’s December 2025 report compared 470 open-source pull requests and counted about 1.7 times as many review findings in AI-co-authored ones, with error-handling gaps nearly twice as common.
None of these studies says AI code is worse than a hurried human’s code in every way. However, they agree on where the defects cluster, and that is exactly the information a review process needs.
The Bench: Fourteen Seeded Defects, Six Review Checks
The library is a slice of an order system: cart totals with a coupon and tax, top-SKU reporting, 1-based pagination, email normalization, an order repository over a SQL client interface, and a checkout service that charges a payment gateway with retries. Each function carries a short spec comment, and the clean version passes every check. That mirrors reviewing AI-written code in practice, where the reviewer holds a spec and the code may or may not match it.
Each variant then changes exactly one thing. A small injector applies one exact string substitution to a copy of the clean package, so every variant differs from the baseline by one defect and nothing else. Here is one entry and the part that applies it:
// tool/inject.dart (excerpt): one defect is one exact substitution.
import 'dart:io';
typedef Defect = ({String id, String kind, String file, String from, String to});
const defects = <Defect>[
(
id: 'D08', kind: 'broken access control: ownership check dropped',
file: 'lib/src/orders.dart',
from: r'return order != null && order.ownerId == requesterId ? order : null;',
to: r'return order;',
),
// ... thirteen more
];
void main(List<String> args) {
final defect = defects.singleWhere((d) => d.id == args.single);
final file = File(defect.file);
final source = file.readAsStringSync().replaceAll('\r\n', '\n');
if (!source.contains(defect.from)) {
stderr.writeln('${defect.id}: pattern not found in ${defect.file}');
exit(2); // a silent no-op would report a "missed" defect that was never injected
}
file.writeAsStringSync(source.replaceFirst(defect.from, defect.to));
}
The exit on a missing pattern matters more than it looks. Without it, a typo in a pattern would leave a variant identical to the clean code, and every check would “miss” a defect that does not exist.
The Fourteen Defects and Where They Come From
| ID | Defect | Study category it reproduces |
|---|---|---|
| D01 | Calls firstWhereOrNull from package:collection with no import | Hallucinated object (Tambon) |
| D02 | reduce() on an empty cart instead of fold | Missing corner case (Tambon) |
| D03 | Coupon taken off after tax instead of before | Misinterpretation (Tambon) |
| D04 | 1-based page number used as 0-based offset | Missing corner case, off by one |
| D05 | Money rounding via double division and toInt() truncation | Numeric correctness |
| D06 | Email lowercased but not trimmed | Missing corner case |
| D07 | User input interpolated into a SQL string | CWE-89 injection (Veracode) |
| D08 | Ownership check dropped from an order lookup | Broken access control (Veracode, OWASP A01) |
| D09 | save() future not awaited | Error handling (CodeRabbit) |
| D10 | Failed charge caught and saved as pending | Error handling, swallowed exception |
| D11 | Retry loop with no limit and no backoff | Error handling, resilience |
| D12 | Full card token written to the log | Sensitive data exposure (CWE-532) |
| D13 | sort() on the caller’s list instead of a copy | Non-prompted side effect |
| D14 | Date range excludes its last day | Boundary error |
These defects were seeded by hand, not sampled from a model’s output. Consequently, the bench answers “can this check catch this class of bug”, not “how often does a model produce it”. The frequency question belongs to the studies above.
The Six Checks
Each check is something a team can add to a pipeline in an afternoon:
- analyze-recommended:
dart analyze --fatal-infoswithpackage:lints/recommended.yaml, which is whatdart creategenerates. - analyze-strict: the same analyzer with strict casts, strict inference, strict raw types, and the async and error-handling lints that are off by default.
- test-happy: one typical input per function, asserting the spec’d output. This is the shape of the tests that usually arrive in the same diff as generated code.
- test-edge: one test per sentence of each spec comment, written from the spec rather than from the implementation.
- test-property: 500 seeded random cases per function, checked against invariants or a slow reference model.
- greps: four project-specific regex rules over
lib/.
The strict analyzer configuration is short:
# analysis_options.yaml: recommended set plus strict types and off-by-default rules
include: package:lints/recommended.yaml
analyzer:
language:
strict-casts: true
strict-inference: true
strict-raw-types: true
linter:
rules:
- unawaited_futures
- discarded_futures
- avoid_catches_without_on_clauses
- use_rethrow_when_possible
- only_throw_errors
- avoid_dynamic_calls
- cancel_subscriptions
- close_sinks
- avoid_print
- prefer_final_locals
The grep rules are cruder, on purpose. Each one encodes a mistake this codebase must never contain, and none of them needs a parser:
#!/usr/bin/env bash
# Project-specific review greps over lib/. Prints one line per hit, exits 1 if any rule hits.
cd "$(dirname "$0")/.." || exit 2
hits=0
rule() { # $1 = rule name, $2 = extended regex
local out
out=$(grep -rnE "$2" lib --include='*.dart') && { echo "$out" | sed "s/^/[$1] /"; hits=1; }
}
rule sql-interpolation '(SELECT|INSERT|UPDATE|DELETE)[^;]*\$\{?[a-zA-Z_]'
rule secret-in-log '(log|print)\(.*\$\{?[a-zA-Z_.]*([Tt]oken|[Pp]assword|[Ss]ecret)\b'
rule catch-without-on '^\s*\}? ?catch \('
rule unbounded-loop 'while \(true\)'
exit $hits
What a Spec Test and a Property Test Look Like
The edge tests read like the spec. For example, the order lookup’s spec says another user gets null, the same answer as for a missing order, so the test asserts exactly that:
import 'package:review_bench/review_bench.dart';
import 'package:test/test.dart';
import 'support/fakes.dart';
void main() {
test('another user cannot read the order', () async {
final h = Harness([null]); // payment gateway that succeeds once
await h.place(); // order o-1, owned by u-1
expect(await h.repo.findForUser('o-1', 'u-2'), isNull);
});
}
Property tests, in contrast, do not name a case at all. Instead, they state something that must hold for every input and then throw a few hundred seeded inputs at it. Pagination has a clean invariant: joining every page in order gives back the original list.
import 'dart:math';
import 'package:review_bench/review_bench.dart';
import 'package:test/test.dart';
void main() {
final random = Random(42); // seeded, so a failure reproduces exactly
test('concatenating every page gives back the list', () {
for (var i = 0; i < 500; i++) {
final items = List.generate(random.nextInt(60), (i) => i);
final size = 1 + random.nextInt(12);
final pages = (items.length / size).ceil();
final joined = [for (var p = 1; p <= pages; p++) ...pageOf(items, p, size)];
expect(joined, items, reason: 'case $i: n=${items.length} size=$size');
}
});
}
That one test catches an off-by-one anywhere in the offset math, whether or not anyone thought about the last page. As a result, property tests are the cheapest way to cover the “missing corner case” category, because nobody has to list the corners in advance.
How the Matrix Was Run
A driver script copies the clean package, injects one defect, and runs every check, printing one line per variant and check. A check “catches” a defect when it passes on the clean package and fails on the variant:
#!/usr/bin/env bash
# tool/matrix.sh (excerpt). Usage: tool/matrix.sh matrix | time <runs>
ROOT=$(cd "$(dirname "$0")/.." && pwd)
WORK="$ROOT/../review-work"
run_check() { # $1 = check name; returns 1 when the check flags something
case "$1" in
analyze-recommended|analyze-strict)
cp "analysis/${1#analyze-}.yaml" analysis_options.yaml
dart analyze --fatal-infos ;;
test-*)
timeout 120 dart test "test/${1#test-}_test.dart" ;;
greps)
bash tool/review_greps.sh ;;
esac
}
variant() { # $1 = clean | D01..D14
rm -rf "$WORK/$1" && mkdir -p "$WORK/$1"
cp -r "$ROOT"/{lib,test,tool,analysis,pubspec.yaml,pubspec.lock,.dart_tool} "$WORK/$1/"
cd "$WORK/$1" || exit 2
[ "$1" = clean ] || dart run tool/inject.dart "$1" || exit 2
for c in analyze-recommended analyze-strict test-happy test-edge test-property greps; do
if run_check "$c" > /dev/null 2>&1; then v=pass; else v=CAUGHT; fi
printf '%s\t%s\t%s\n' "$1" "$c" "$v"
done
}
The full script also prints a one-line detail per result, such as the lint name or the test count, which appears in the raw output below. The matrix ran once with tool/matrix.sh matrix. Pass or fail is deterministic here, since the property tests use a fixed seed, so a single run is the result. Timings ran separately with tool/matrix.sh time 5, and the reported value is the median of five runs.
Which Checks Caught Which Bugs, Measured
Here is the raw output of the matrix run, unedited:
clean analyze-recommended pass
clean analyze-strict pass
clean test-happy pass +8: All tests passed
clean test-edge pass +17: All tests passed
clean test-property pass +5: All tests passed
clean greps pass
D01 analyze-recommended CAUGHT error lib\src\users.dart:22:16 undefined_method
D01 analyze-strict CAUGHT error lib\src\users.dart:22:16 undefined_method;error lib\src\users.dart:22:58 argument_type_not_assignable;warning lib\src\users.dart:22:34 inference_failure_on_untyped_parameter;info lib\src\users.dart:22:58 avoid_dynamic_calls
D01 test-happy CAUGHT +0 -1: Some tests failed
D01 test-edge CAUGHT +0 -1: Some tests failed
D01 test-property CAUGHT +0 -1: Some tests failed
D01 greps pass
D02 analyze-recommended pass
D02 analyze-strict pass
D02 test-happy pass +8: All tests passed
D02 test-edge CAUGHT +16 -1: Some tests failed
D02 test-property CAUGHT +4 -1: Some tests failed
D02 greps pass
D03 analyze-recommended pass
D03 analyze-strict pass
D03 test-happy CAUGHT +7 -1: Some tests failed
D03 test-edge pass +17: All tests passed
D03 test-property CAUGHT +4 -1: Some tests failed
D03 greps pass
D04 analyze-recommended pass
D04 analyze-strict pass
D04 test-happy CAUGHT +7 -1: Some tests failed
D04 test-edge CAUGHT +16 -1: Some tests failed
D04 test-property CAUGHT +4 -1: Some tests failed
D04 greps pass
D05 analyze-recommended pass
D05 analyze-strict pass
D05 test-happy pass +8: All tests passed
D05 test-edge CAUGHT +16 -1: Some tests failed
D05 test-property CAUGHT +4 -1: Some tests failed
D05 greps pass
D06 analyze-recommended pass
D06 analyze-strict pass
D06 test-happy pass +8: All tests passed
D06 test-edge CAUGHT +15 -2: Some tests failed
D06 test-property CAUGHT +4 -1: Some tests failed
D06 greps pass
D07 analyze-recommended pass
D07 analyze-strict pass
D07 test-happy pass +8: All tests passed
D07 test-edge CAUGHT +16 -1: Some tests failed
D07 test-property pass +5: All tests passed
D07 greps CAUGHT [sql-interpolation] lib/src/orders.dart:55
D08 analyze-recommended pass
D08 analyze-strict pass
D08 test-happy pass +8: All tests passed
D08 test-edge CAUGHT +16 -1: Some tests failed
D08 test-property pass +5: All tests passed
D08 greps pass
D09 analyze-recommended pass
D09 analyze-strict CAUGHT info lib\src\checkout.dart:70:11 unawaited_futures
D09 test-happy CAUGHT +6 -2: Some tests failed
D09 test-edge CAUGHT +16 -1: Some tests failed
D09 test-property pass +5: All tests passed
D09 greps pass
D10 analyze-recommended pass
D10 analyze-strict CAUGHT info lib\src\checkout.dart:64:7 avoid_catches_without_on_clauses
D10 test-happy pass +8: All tests passed
D10 test-edge CAUGHT +15 -2: Some tests failed
D10 test-property pass +5: All tests passed
D10 greps CAUGHT [catch-without-on] lib/src/checkout.dart:64
D11 analyze-recommended CAUGHT warning lib\src\checkout.dart:47:15 unused_field
D11 analyze-strict CAUGHT warning lib\src\checkout.dart:47:15 unused_field
D11 test-happy pass +8: All tests passed
D11 test-edge CAUGHT +15 -2: Some tests failed
D11 test-property pass +5: All tests passed
D11 greps CAUGHT [unbounded-loop] lib/src/checkout.dart:75
D12 analyze-recommended CAUGHT warning lib\src\checkout.dart:86:17 unused_element
D12 analyze-strict CAUGHT warning lib\src\checkout.dart:86:17 unused_element
D12 test-happy pass +8: All tests passed
D12 test-edge CAUGHT +16 -1: Some tests failed
D12 test-property pass +5: All tests passed
D12 greps CAUGHT [secret-in-log] lib/src/checkout.dart:60
D13 analyze-recommended pass
D13 analyze-strict pass
D13 test-happy CAUGHT +7 -1: Some tests failed
D13 test-edge CAUGHT +16 -1: Some tests failed
D13 test-property CAUGHT +4 -1: Some tests failed
D13 greps pass
D14 analyze-recommended pass
D14 analyze-strict pass
D14 test-happy pass +8: All tests passed
D14 test-edge CAUGHT +16 -1: Some tests failed
D14 test-property CAUGHT +4 -1: Some tests failed
D14 greps pass
The clean rows matter as much as the defect rows. Every check passed on the clean package, so every CAUGHT is a real catch and none is noise from a check that was already failing.
The Catch Matrix at a Glance
| Defect | Recommended lints | Strict analyzer | Happy tests | Spec tests | Property tests | Greps |
|---|---|---|---|---|---|---|
| D01 hallucinated API | yes | yes | yes | yes | yes | no |
| D02 empty cart | no | no | no | yes | yes | no |
| D03 coupon after tax | no | no | yes | no | yes | no |
| D04 off by one | no | no | yes | yes | yes | no |
| D05 truncated rounding | no | no | no | yes | yes | no |
| D06 untrimmed email | no | no | no | yes | yes | no |
| D07 SQL injection | no | no | no | yes | no | yes |
| D08 missing ownership check | no | no | no | yes | no | no |
| D09 unawaited save | no | yes | yes | yes | no | no |
| D10 swallowed charge error | no | yes | no | yes | no | yes |
| D11 unbounded retry | incidental | incidental | no | yes | no | yes |
| D12 token in log | incidental | incidental | no | yes | no | yes |
| D13 in-place sort | no | no | incidental | yes | yes | no |
| D14 last day excluded | no | no | no | yes | yes | no |
| Caught | 3 of 14 | 5 of 14 | 5 of 14 | 13 of 14 | 8 of 14 | 4 of 14 |
“Incidental” marks catches that happened for a reason unrelated to the defect class, and the next section explains each one. Counting only deliberate catches, the recommended lint set caught one defect, and it did so through the compiler’s type check, not through a lint.
What Each Check Costs
Every check is cheap enough to run on every push. Here is the timing output on the clean package:
analyze-recommended median=1053ms runs=[1049 1053 1053 1068 1125]
analyze-strict median=1035ms runs=[1023 1033 1035 1052 1087]
test-happy median=1436ms runs=[1425 1435 1436 1437 1883]
test-edge median=1448ms runs=[1437 1448 1448 1452 1452]
test-property median=1462ms runs=[1444 1451 1462 1468 1468]
greps median=143ms runs=[135 138 143 151 152]
Notably, the strict analyzer ran no slower than the recommended set, so the extra rules cost nothing at check time. Similarly, 2,500 property cases added only 26 ms over the eight happy tests, because nearly all of the 1.4 seconds is the test runner compiling and starting up. In other words, the cost of these checks is writing them, not running them.
Results That Cut Against Expectations
Three results in the matrix do not match the usual advice about reviewing AI-written code, and each one changes how the checks should be used.
Spec Tests Missed the Misread Spec
The spec-derived edge tests caught 13 of 14 defects, which looks like a clear winner. However, the one they missed is the most AI-shaped bug in the set. D03 applies the coupon after tax, a classic misinterpretation, and the edge suite passed it with +17: All tests passed.
The reason is mundane. The edge suite tests the coupon floor and the tax rounding separately, and the only coupon test uses a cart smaller than the coupon, where both orders of operations give 0. No test combined a coupon with tax on a cart large enough to show the difference. Meanwhile, the single happy-path test did exactly that, by accident of picking a realistic input:
00:00 +0 -1: cart total applies the coupon, then tax [E]
Expected: <3000>
Actual: <3100>
The lesson is not that happy tests beat edge tests. Rather, tests written one spec sentence at a time cover each rule in isolation and miss how rules interact. The property test caught D03 because its reference model computed the whole formula at once. Therefore, any function with two or more rules that combine needs at least one test, or one reference model, that exercises them together.
The Analyzer Caught Two Bugs It Has No Rule For
The recommended lint set has no rule about retry limits or logging secrets. Still, it flagged both D11 and D12, with these outputs:
warning - lib\src\checkout.dart:47:15 - The value of the field '_sleep' isn't used. Try removing the field, or using it. - unused_field
The unbounded retry loop no longer calls _sleep, so the backoff field became dead. Likewise, logging the raw token removed the only call to _last4, the helper that masked it. In both cases, the bug was the removal of a safeguard, and the safeguard left a dead field or function behind.
This is incidental, since a version of the same bug that kept the helper in use somewhere else would pass the analyzer. Even so, it suggests a review habit worth keeping: when an AI-written diff makes a helper unused, ask what that helper was protecting. Agents regenerate whole functions more often than they edit lines, and dropping a guard while keeping everything else plausible is exactly what that produces.
The Happy Test Caught the Side Effect by Luck
D13 sorts the caller’s list in place. The happy test caught it, but only because its input was a const list:
00:00 +1 -1: top SKUs come back by quantity [E]
Unsupported operation: Cannot modify an unmodifiable list
With a regular list literal, the same test would have passed, since the returned SKUs are correct and nothing checks the input afterwards. In contrast, the spec test and the property test both assert that the input is unchanged, so they caught it on purpose. Accidental catches like this one make a suite look stronger than it is, which is why the table marks them separately.
The Bug Only a Spec Could Catch
D08 drops the ownership check from findForUser, so any logged-in user can read any order by ID. This is broken access control, number one on the OWASP Top 10, and it is the only defect in the set that exactly one check caught:
00:00 +10 -1: orders another user cannot read the order [E]
Expected: null
Actual: <Instance of 'Order'>
No analyzer rule can catch it, because the broken code is simpler and cleaner than the correct code. A grep cannot catch it either, because there is no bad pattern to match, only a missing condition. A property test would need an invariant like “a user never sees another user’s order”, which is exactly the spec, written down. Happy-path tests, finally, always run as the owner.
In other words, authorization bugs are invisible to every check that does not know who is allowed to do what. That knowledge lives in the spec and in the reviewer’s head, not in the code. Consequently, reviewing AI-written code that touches data access has to include reading every query for its ownership or tenant condition, and at least one test per endpoint that calls it as the wrong user. The API security checklist for production applications covers the broader set of access checks worth testing this way.
How to Review AI-Written Code: A Checking Order
For reviewing AI-written code, the matrix suggests an order that puts the cheapest, most mechanical checks first, so human attention goes where no tool reaches:
- Turn on the strict analyzer and fail the build on infos. It caught the hallucinated API, the unawaited future and the catch-all, in about one second.
- Add project greps for the mistakes that must never ship. String-built SQL, secrets in logs, bare catches and
while (true)each take one line. - Check every new dependency before reading the code. Confirm the package exists, who publishes it and how old it is, since hallucinated names are what slopsquatting attacks register.
- Read the tests before the implementation. List what the spec says and mark which sentences have a test. Untested sentences are where to look first.
- Add one property test or reference model for pure logic. It caught eight of fourteen defects with five test functions.
- Look for safeguards that became dead code. An unused helper, field or parameter in a regenerated function often marks a removed check.
- Trace authorization and data flow by hand. Every query needs an owner or tenant condition, and every endpoint needs a wrong-user test.
Steps one and two belong in CI, not in a reviewer’s memory. Running them as agent hooks also works well, so the agent sees the failure before a human does, as described in automating linting and tests with Claude Code hooks. Teams that are not on an agent workflow can get the same effect from git hooks for code quality checks.
Mapping These Checks to Other Stacks
Nothing here depends on Dart. Every stack used for reviewing AI-written code has an equivalent for each check, although the defaults differ:
| Check | TypeScript | Python | Java / Kotlin |
|---|---|---|---|
| Hallucinated API | tsc --noEmit | mypy --strict or pyright | The compiler |
| Unawaited future | @typescript-eslint/no-floating-promises | RUF006 in Ruff, asyncio debug mode | Error Prone FutureReturnValueIgnored |
| Catch-all handler | ESLint no-empty, a Semgrep rule | Ruff BLE001, S110 | Error Prone, SpotBugs REC_CATCH_EXCEPTION |
| SQL interpolation | Semgrep rules | Bandit B608, Ruff S608 | SpotBugs SQL_NONCONSTANT_STRING_PASSED_TO_EXECUTE, Semgrep |
| Property tests | fast-check | Hypothesis | jqwik |
The catch is that most of these rules are not on by default. In particular, no-floating-promises needs typed linting, and Ruff’s security rules sit in an opt-in group. Consequently, a project that never changed its lint configuration is probably running the equivalent of the “recommended” column, which caught one defect out of fourteen here.
Reading the Diff: What Still Needs a Human
Tools narrow the search, but four questions only a reviewer can answer, and they line up with the defects that needed a spec:
- Does the code do what was asked, or what was plausible? D03 is plausible pricing logic. It is also wrong for this spec, and only someone who knows the spec can tell.
- What did the diff remove? Agents rewrite whole functions, so a dropped condition hides inside a block that looks new. Reading the removed lines in the diff, not only the added ones, puts D08 and D11 in plain view.
- Who is allowed to call this? Access rules are business rules, and the code cannot state them by itself.
- What happens on the second call, the empty input and the failure path? These are the paths that generated tests skip, because the prompt rarely mentions them.
For the human side of review, including how to word feedback on a pull request from a teammate who used an agent, see code review best practices for giving and receiving feedback.
Should an AI Review the AI’s Code?
It can help, as long as it is not the only reviewer. AI review tools are good at spotting known patterns like D07, D10 and D12, which overlap heavily with what the greps and the strict analyzer already catch. On the other hand, they share the author’s blind spot on D03 and D08, because neither the author model nor the reviewer model has the spec unless someone writes it down. The comparison of AI-powered code review tools covers what the current tools check. Treat them as another layer in the sieve, not as the person who signs off.
A Real Scenario: The Feature That Passed Review
Consider a small product team that started using a coding agent for backend work over the past quarter. Pull requests now arrive faster than before, each with green tests, and reviews have shrunk to a skim of the diff and a check that CI passed. A refund endpoint ships that looks up orders by ID, and its tests cover the happy path, a missing order and a refund that exceeds the total.
A few weeks later, a support ticket shows that one customer could see another customer’s refund status by changing the ID in the URL. The ownership check that every other endpoint has was never in the prompt, so the agent never wrote it, and the tests the agent wrote alongside the code ran as the owner every time. Nothing in CI could have failed, because nothing in CI knew the rule.
The fix takes the team an afternoon: a shared wrong user gets 404 test helper, a grep that flags any repository query without an owner_id or tenant_id condition, and one line in the pull request template asking which access rule the change touches. The trade-off is real, since the grep has false positives on admin queries that need an allow comment. Still, that is cheaper than a second incident.
When to Use the Full Review Stack
- The change touches money, authorization, personal data or anything that writes to a database
- An agent generated the implementation and its tests in the same session, from the same prompt
- The function combines two or more business rules, such as discounts, tax and rounding
- The diff rewrites an existing function rather than adding a new one, so a guard could have been dropped
- A new dependency appears that nobody on the team has used before
When NOT to Use the Full Review Stack
- Throwaway scripts and prototypes that will never see production data
- Generated code with its own verification, such as a client produced from an OpenAPI schema
- Pure UI layout changes, where a visual check catches more than any test would
- Mechanical refactors that a compiler and the existing suite already pin down, such as renames
- Spikes whose only purpose is to answer a question and then be deleted
Common Mistakes When Reviewing AI-Written Code
- Accepting the tests in the same diff as evidence, when they were written from the implementation they test
- Leaving the lint configuration at the defaults, which in this bench caught one defect out of fourteen
- Writing one test per rule and never one that combines rules, which is how D03 passed thirteen other checks
- Reading only the added lines of a diff and missing the condition that disappeared
- Counting accidental catches, such as a const input, as coverage
- Installing a suggested package without checking that it exists and who publishes it
- Asking a second model to review the first and treating its silence as approval
What This Measurement Does Not Cover
The fourteen defects were seeded by hand to represent published categories. They were not sampled from model output, so the bench says nothing about how often each class appears. Moreover, the same author wrote the defects and the checks, which favors the checks. In particular, the greps were written knowing the codebase, and an independent reviewer’s rules would catch less on unfamiliar code. The edge suite’s one miss happened despite that advantage, which is part of why it is worth reporting.
Each variant also contains exactly one defect. Real diffs can carry several, and one defect can mask another, for example when a compile error stops every test from running, as D01 did. The bench is 182 non-blank lines of library code and 265 of tests, so it cannot show how review attention degrades on a 2,000-line diff. Finally, nothing here measures reviewer time, false-positive rates on a large codebase, or the speed gains of using an agent at all. Those depend on a team’s own history, not on a bench.
Conclusion
Reviewing AI-written code works when each check targets a defect class instead of reading for style. On this bench, the strict analyzer and four greps caught six of fourteen seeded defects in about 1.2 seconds, while property tests added seven more. Spec tests caught thirteen but missed the misread pricing rule, and the missing ownership check fell to the spec tests alone.
Start this week with the two cheapest changes: turn on the strict analyzer rules for your stack and fail the build on them, then write one wrong-user test for every endpoint that reads by ID. Next, look at the tests an agent writes alongside its code, which is the subject of generating unit tests with large language models on this site.
