This post is for engineers and tech leads whose coverage number went up the month a coding agent started writing tests. An agent can take AI test coverage from 60 percent to 100 percent in an afternoon, which raises a fair question: did the suite get better at catching bugs, or only at running lines? Instead of answering from opinion, this post measures it. It takes one billing module and writes six test suites against it, in the six shapes that agents and busy humans tend to produce. Each suite is then scored three ways: line coverage, the share of 108 automatically generated bugs it catches, and how often it breaks on changes that should not break anything. The result is a ranking of which tests are still worth writing, and which ones an agent should be told to stop producing.
The numbers below came from this environment:
Measured 2026-11-04
Windows 11 Pro 25H2 (build 26200), AMD Ryzen 5 8600G (6C/12T), 16 GB RAM
Dart SDK 3.9.0 (stable), package:test 1.31.0, package:coverage 1.16.0, package:mocktail 1.0.5
One library (195 lines of code), six suites, 114 mutants, four safe changes
Dart was chosen because the test runner, the coverage collector and the mocking library all install with one dart pub get, so every number here comes from one toolchain. However, nothing in the results depends on Dart. The six suite shapes exist in every language, and a later section maps the mutation tools across stacks.
Why AI Test Coverage Stopped Being a Useful Signal
Line coverage measures which lines ran during the tests, not whether any test would fail if those lines were wrong. Before agents, reaching high coverage took effort, so the number loosely tracked care. Now an agent reaches 100 percent in minutes, which means AI test coverage mostly measures how much test code was generated.
That gap was known long before agents. Google’s code coverage best practices already warned that coverage does not guarantee the covered lines were tested correctly, and against treating a high percentage as the goal in itself. Similarly, the State of Mutation Testing at Google paper describes mutation testing as the better signal and shows how to run it on changed code at scale. What changed is the cost. When every test took an engineer real time to write, few people wrote tests that only called functions and checked the result type. An agent writes them in seconds, and a coverage gate rewards every one.
The cost of reading and maintaining those tests, on the other hand, did not drop at all. Every generated test is code that has to be updated when the implementation moves. Consequently, the useful question is no longer “how much coverage do we have” but “which tests earn their maintenance”, and that question has a measurable answer.
The Bench: One Billing Module, Six Test Suites
The library is a small slice of a subscription billing system in three files. pricing.dart holds pure rules: proration for a mid-period plan change, the next billing date with month-end clamping, percent and fixed discounts, tax in basis points, and coupon code normalization. Next, invoice.dart builds an invoice where the discount comes off before tax. Finally, renewal.dart holds a RenewalService that loads an account and its plans, charges a payment gateway, retries timeouts up to three attempts, and saves the invoice as paid, free or failed.
Every function carries a short spec comment, and the spec is what the tests are supposed to check. Here is one of the rules, because it comes back later:
/// The billing date [months] after [anchor]. Keeps the anchor's day of month,
/// clamped to the last day of shorter months: Jan 31 plus one month is Feb 28
/// (Feb 29 in leap years), and plus two months is Mar 31 again.
DateTime nextBillingDate(DateTime anchor, int months) {
if (months < 0) throw ArgumentError.value(months, 'months');
final monthIndex = anchor.month - 1 + months;
final year = anchor.year + monthIndex ~/ 12;
final month = monthIndex % 12 + 1;
final lastDay = DateTime.utc(year, month + 1, 0).day;
final day = anchor.day < lastDay ? anchor.day : lastDay;
return DateTime.utc(year, month, day);
}
The public entry point, lib/billing.dart, hides the two internal helpers, divideHalfUp and daysBetween. That detail matters, because two of the six suites import them from src/ anyway, which is a common result of asking an agent to test a file rather than a feature.
Six Test Shapes and the Instruction Each One Models
Each suite was written for this bench to represent one shape, and every suite except the interaction one covers the whole library in that shape. None of them is a sample of any particular model’s output, so the bench says nothing about how often a given agent produces each shape. All six pass against the clean code.
| Suite | Shape | Instruction it models | Tests | Test lines |
|---|---|---|---|---|
| mirror | Expected values read off the implementation, helpers tested directly, error messages asserted word for word | “Write unit tests for this file” | 14 | 102 |
| coverage | Every branch runs, assertions check type and range only | “Raise line coverage to 100%” | 13 | 130 |
| spec | One test per sentence of each spec comment, public API only, in-memory fakes | “Test this against the spec” | 32 | 185 |
| property | 500 seeded random cases per rule, checked against invariants or a reference | “Find inputs that break this” | 7 | 135 |
| interaction | Mocks that verify calls, arguments, counts and order | “Test the service in isolation” | 5 | 71 |
| snapshot | Whole outputs serialized and compared with recorded golden files | “Lock in the current behavior” | 3 | 112 + 152 golden |
Test lines count non-blank, non-comment lines. In addition, the spec, property and snapshot suites share a 45-line file of in-memory fakes. The interaction suite only tests RenewalService, because mocking has nothing to replace in pure functions, so its whole-library numbers are unfair to it and a per-file breakdown follows below.
What the Shapes Look Like Side by Side
The difference between the coverage-chaser and the spec suite is easiest to see on the same function. Both of these run every line of discountCents:
// coverage_test.dart: every branch runs, nothing checks the arithmetic.
test('discountCents covers all branches', () {
expect(discountCents(1000, null), 0);
expect(discountCents(1000, const PercentOff(10)), isA<int>());
expect(() => discountCents(1000, const PercentOff(101)), throwsArgumentError);
expect(discountCents(1000, const AmountOff(100)), lessThanOrEqualTo(1000));
expect(discountCents(1000, const AmountOff(5000)), lessThanOrEqualTo(1000));
expect(() => discountCents(1000, const AmountOff(-1)), throwsArgumentError);
});
// spec_test.dart: one test per spec sentence, boundaries named explicitly.
test('percent off rounds half up', () {
expect(discountCents(1250, const PercentOff(10)), 125);
expect(discountCents(5, const PercentOff(10)), 1); // 0.5 rounds up
});
test('accepts 0 and 100 percent, rejects outside', () {
expect(discountCents(1999, const PercentOff(0)), 0);
expect(discountCents(1999, const PercentOff(100)), 1999);
expect(() => discountCents(1999, const PercentOff(-1)), throwsArgumentError);
expect(() => discountCents(1999, const PercentOff(101)), throwsArgumentError);
});
The first block reports the same coverage as the second. However, isA<int>() passes for any integer at all, so a discount that rounds the wrong way, or takes 10 percent of the wrong number, sails through.
The service tests split the same way. The interaction suite asserts what the service did to its collaborators, while the spec suite asserts what reached the outside world:
// interaction_test.dart: the call sequence is the assertion.
verifyInOrder([
() => accounts.find('acc-1'),
() => plans.find('basic'),
() => plans.find('storage'),
() => gateway.charge('acc-1', 1870),
() => invoices.save(any(), InvoiceStatus.paid),
]);
// spec_test.dart: the charge and the saved record are the assertion.
expect(await service(gateway).renew('acc-1'), InvoiceStatus.paid);
expect(gateway.charges, [('acc-1', 1870)]);
final (invoice, status) = invoices.saved.single;
expect(invoice.lines.map((l) => l.sku), ['basic', 'storage']);
expect(status, InvoiceStatus.paid);
Both check the charge amount. The difference is that the first one also pins down which repository method produced the plans, which is an implementation detail the spec never mentions. That difference decides one of the safe-change results below.
How the Mutants Were Generated
A mutant is a copy of the code with one small, deliberate bug. If at least one test fails, the suite killed it; if every test passes, the mutant survived, and the suite would not have noticed that bug in a real change. A small runner generates mutants mechanically from the library source, using the same families of operators as PIT and Stryker:
// tool/mutate.dart (excerpt): binary operators only match with spaces around
// them, so generic type arguments like List<int> are never mutated.
const swaps = {
' < ': ' <= ', ' <= ': ' < ', ' > ': ' >= ', ' >= ': ' > ',
' == ': ' != ', ' != ': ' == ', ' && ': ' || ', ' || ': ' && ',
' + ': ' - ', ' - ': ' + ', ' * ': ' + ', ' ~/ ': ' * ', ' % ': ' ~/ ',
'++': '--',
'InvoiceStatus.paid': 'InvoiceStatus.failed',
'InvoiceStatus.failed': 'InvoiceStatus.paid',
'InvoiceStatus.free': 'InvoiceStatus.paid',
'.trim()': '', '.toUpperCase()': '', ".replaceAll('-', '')": '',
'await ': '',
};
// Plus: remove a unary !, integer literal n -> n + 1, every if condition
// replaced by true and by false, and every call-only statement deleted.
String literals and comments are masked before matching, so a mutant never edits text inside a message. Each mutant replaces one line, runs all six suites in one dart test -r json process, records which suites failed, and restores the file. The run produced 114 mutants, and 6 of them did not compile, which leaves 108 scored ones.
One mutant needed special handling. M104 turns attempt++ into attempt-- in the retry loop, and with a gateway that keeps timing out, the loop never ends. Because every await in it completes immediately, the test runner’s own 15-second timeout never gets a turn, so the runner kills the process after 90 seconds instead. That kill cannot tell which suites were still running, so M104 was re-run against each suite alone. The coverage and interaction suites spin forever, which a CI timeout would report as a failure, while the mirror suite passes, and its cell was corrected to survived.
The Two Other Measurements
Line and function coverage come from package:coverage, one suite at a time:
#!/usr/bin/env bash
# tool/coverage.sh: line and function coverage of lib/ for each suite on its own.
set -euo pipefail
for suite in mirror coverage spec property interaction snapshot; do
rm -rf "coverage/$suite"
dart run coverage:test_with_coverage --branch-coverage --function-coverage \
--out "coverage/$suite" --test "test/${suite}_test.dart" > /dev/null
awk -v s="$suite" -F: '
/^LF/ {lf+=$2} /^LH/ {lh+=$2} /^FNF/ {ff+=$2} /^FNH/ {fh+=$2}
END {printf "%-12s lines %3d/%3d %5.1f%% functions %2d/%2d\n",
s, lh, lf, 100*lh/lf, fh, ff}' "coverage/$suite/lcov.info"
done
Branch coverage was requested too, but the collector returned no branch records on this setup, so only line and function coverage are reported.
The third measurement asks the opposite question. A suite that fails on every change is not safe, it is noisy, and a noisy suite trains a team to re-record and move on. Four changes that keep the spec intact were applied one at a time, with every suite run after each:
| Change | What it does | Should any test fail? |
|---|---|---|
| S1 | Renames the two internal helpers | No, both are hidden from the public API |
| S2 | Adds a currency key to the invoice JSON | No, it is additive |
| S3 | Rewords one ArgumentError message | No, the error type is unchanged |
| S4 | Loads plans with one findMany call instead of a find per plan | No, same invoice and same charge |
AI Test Coverage vs Mutation Score, Measured
Here is the coverage output, followed by the mutation summary:
bash tool/coverage.sh
mirror lines 72/ 92 78.3% functions 20/22
coverage lines 92/ 92 100.0% functions 21/22
spec lines 79/ 92 85.9% functions 20/22
property lines 75/ 92 81.5% functions 20/22
interaction lines 44/ 92 47.8% functions 13/22
snapshot lines 86/ 92 93.5% functions 21/22
dart run tool/mutate.dart run mutants.jsonl
dart run tool/report.dart
mutants generated 114, did not compile 6, scored 108
hung (killed by the 90 s limit): M104
suite killed score only this suite
mirror 56/108 51.9%
coverage 48/108 44.4% M1 M4 M8
spec 91/108 84.3%
property 82/108 75.9% M2 M83
interaction 41/108 38.0%
snapshot 66/108 61.1%
mirror + coverage 70/108 64.8%
spec + property 93/108 86.1%
spec + property + snapshot 93/108 86.1%
all six 96/108 88.9%
survived every suite:
M15 pricing.dart:28 [<] if (monthlyCents <= 0) throw ArgumentError.value(monthlyCents, 'monthlyCents');
M16 pricing.dart:28 [literal] if (monthlyCents < 1) throw ArgumentError.value(monthlyCents, 'monthlyCents');
M18 pricing.dart:28 [if->false] if (false) throw ArgumentError.value(monthlyCents, 'monthlyCents');
M43 pricing.dart:46 [<] final day = anchor.day <= lastDay ? anchor.day : lastDay;
M54 pricing.dart:78 [<] if (cents <= 0) throw ArgumentError.value(cents, 'cents');
M55 pricing.dart:78 [literal] if (cents < 1) throw ArgumentError.value(cents, 'cents');
M58 pricing.dart:79 [>] return cents >= subtotalCents ? subtotalCents : cents;
M62 pricing.dart:85 [if->false] if (false) throw ArgumentError.value(basisPoints, 'basisPoints');
M97 renewal.dart:75 [await] invoices.save(invoice, InvoiceStatus.free);
M101 renewal.dart:80 [await] invoices.save(invoice, status);
M112 renewal.dart:90 [if->false] if (false) return InvoiceStatus.failed;
M114 renewal.dart:95 [InvoiceStatus.failed] return InvoiceStatus.paid;
did not compile: M85 M86 M87 M88 M89 M100
The mutation run is deterministic: the property suites use fixed seeds, so the same mutant always produces the same outcome. The full pass over 114 mutants took 268 seconds in a single run.
The Ranking at a Glance
| Suite | Line coverage | Mutants killed | Safe changes that broke it | Median run |
|---|---|---|---|---|
| coverage | 100.0% | 44.4% | 2 of 4 | 591 ms |
| snapshot | 93.5% | 61.1% | 2 of 4 | 348 ms |
| spec | 85.9% | 84.3% | 0 of 4 | 508 ms |
| property | 81.5% | 75.9% | 0 of 4 | 394 ms |
| mirror | 78.3% | 51.9% | 3 of 4 | 401 ms |
| interaction | 47.8% | 38.0% | 1 of 4 | 353 ms |
Sorted by coverage, the suite that ran every line comes first. Sorted by bugs caught, it comes fifth of six, ahead only of the interaction suite, which does not test most of the library at all. In other words, the AI test coverage number ranked these suites almost backwards.
The run times are the median of five runs of each suite, read from the JSON reporter’s final timestamp in milliseconds:
for s in mirror coverage spec property interaction snapshot; do
printf "%-12s" $s
for i in 1 2 3 4 5; do
dart test -r json test/${s}_test.dart | tail -1 | grep -o '"time":[0-9]*' | cut -d: -f2 | tr '\n' ' '
done; echo
done
mirror 463 376 374 401 405
coverage 437 509 797 736 591
spec 717 767 508 456 444
property 406 394 402 392 384
interaction 376 364 353 351 353
snapshot 348 343 350 344 352
Most of each figure is test runner startup, so the gaps between them mean little. Speed is not what separates these suites.
Scores per File
Because the interaction suite only targets the renewal service, the fair comparison is per file:
dart run tool/by_file.dart
pricing.dart 70 mutants mirror 41 coverage 34 spec 58 property 52 interaction 15 snapshot 41
invoice.dart 12 mutants mirror 9 coverage 9 spec 12 property 9 interaction 7 snapshot 9
renewal.dart 26 mutants mirror 6 coverage 5 spec 21 property 21 interaction 19 snapshot 16
On its own ground, the interaction suite killed 19 of 26 renewal mutants, close to the spec and property suites at 21 each. So mocks are not useless at catching bugs in orchestration code. Their problem shows up in the next measurement. Meanwhile, the coverage-chaser killed 5 of 26 renewal mutants even though it ran every line of that file, including both failure paths.
What Combining Suites Adds
The two suites an agent produces for free, mirror and coverage, together reach 100 percent of lines and 64.8 percent of mutants. That is the ceiling of AI test coverage bought without a spec: every line, and about two thirds of the bugs. In contrast, spec and property tests together kill 86.1 percent. Adding the snapshot suite to those two adds nothing: 93 mutants before, 93 after. Every bug the golden files caught, the spec or property tests had already caught.
The “only this suite” column is worth a look as well. The coverage suite’s three unique kills, M1, M4 and M8, are all in the guards of the internal rounding helper, which it calls directly from src/. No public-API test exercises those guards, because the callers validate their inputs first. That direct import is a real gain on this bench, and it is also the reason the same suite stopped compiling after the S1 rename.
Which Suites Broke on Changes That Should Not Break Them
dart run tool/safe_changes.dart
S1 rename the two internal helpers
coverage_test.dart does not compile
interaction_test.dart 0 of 5 tests fail
mirror_test.dart does not compile
property_test.dart 0 of 7 tests fail
snapshot_test.dart 0 of 3 tests fail
spec_test.dart 0 of 32 tests fail
S2 add a 'currency' key to the invoice JSON
coverage_test.dart 0 of 13 tests fail
interaction_test.dart 0 of 5 tests fail
mirror_test.dart 0 of 14 tests fail
property_test.dart 0 of 7 tests fail
snapshot_test.dart 2 of 3 tests fail
spec_test.dart 0 of 32 tests fail
S3 reword one ArgumentError message
coverage_test.dart 0 of 13 tests fail
interaction_test.dart 0 of 5 tests fail
mirror_test.dart 1 of 14 tests fail
property_test.dart 0 of 7 tests fail
snapshot_test.dart 0 of 3 tests fail
spec_test.dart 0 of 32 tests fail
S4 load plans with one findMany call instead of a find per plan
coverage_test.dart 5 of 13 tests fail
interaction_test.dart 4 of 5 tests fail
mirror_test.dart 1 of 14 tests fail
property_test.dart 0 of 7 tests fail
snapshot_test.dart 1 of 3 tests fail
spec_test.dart 0 of 32 tests fail
The spec and property suites survived all four changes without a single false failure. The other four suites each broke at least once, and every break traces to the same cause: a test that knows something the spec does not say.
The mirror and coverage suites import helpers from src/, so renaming them is a compile error rather than a test failure, and nothing in the suite runs at all. Next, the mirror suite asserts an error message word for word, which turns a wording change into a red build. The snapshot suite fails on an additive JSON key, because a golden file cannot tell a new field from a broken one. Finally, S4 is the canonical over-mocking failure: the stubs answer find, the code now calls findMany, and mocktail answers an unstubbed call with null, which fails as a type error. As a result, four of five interaction tests fail on a change that produces the same invoice and the same charge. The guide to mocking external dependencies covers where that line sits in general.
This matters more after AI than before. A team where an agent refactors code daily runs into these false failures daily, and the cheapest fix is to have the same agent update the test until it passes. At that point the test checks nothing, because it was rewritten to match whatever the code now does.
Results That Cut Against Expectations
The 100% Coverage Suite Ran the Worst Bug and Passed
M114 changes the fallback at the end of the retry loop from failed to paid. That line only runs when maxAttempts is zero, which is a configuration mistake, not an everyday input. With the mutant in place, a service configured with zero attempts marks every invoice paid without ever calling the gateway.
The coverage-chaser is the only suite that executes that line, because it set out to reach 100 percent and found the branch. Here is its test:
test('zero attempts', () async {
when(() => gateway.charge(any(), any())).thenAnswer((_) async => 'ch');
final none = RenewalService(accounts, plans, invoices, gateway,
maxAttempts: 0);
expect(await none.renew('a'), isA<InvoiceStatus>());
});
It ran the exact line containing the bug and asserted that the result was some status. As a result, the coverage report shows that line green, and the mutant survived every suite. A coverage report can only show that a test reached a line, never that anything checked what the line did.
Snapshots Added Nothing, Then Certified a Bug
The snapshot suite killed 66 mutants, more than the mirror or coverage suites. However, every one of them was also killed by the spec or property suite, so on top of those two it contributed zero kills and two false failures.
The second problem shows up when the golden files get re-recorded, which is the routine response to a snapshot failure. To test that, the classic month-end bug was injected into nextBillingDate, returning DateTime.utc(year, month, anchor.day) so that Jan 31 plus one month rolls over into March. The snapshot suite failed, as it should. Then the goldens were re-recorded the way a busy reviewer would:
UPDATE_GOLDENS=1 dart test test/snapshot_test.dart 2>&1 | tail -1
diff ../goldens.bak/pricing.json test/goldens/pricing.json
dart test -r expanded 2>&1 | grep -E "\[E\]|Some tests|All tests" | sed 's/^[0-9:+ -]*//'
00:00 +3: All tests passed!
12c12
< "2026-02-28T00:00:00.000Z",
---
> "2026-03-01T00:00:00.000Z",
14,15c14,15
< "2027-02-28T00:00:00.000Z",
< "2026-02-28T00:00:00.000Z",
---
> "2027-03-01T00:00:00.000Z",
> "2026-03-02T00:00:00.000Z",
17,18c17,18
< "2027-02-28T00:00:00.000Z",
< "2026-02-28T00:00:00.000Z",
---
> "2027-03-02T00:00:00.000Z",
> "2026-03-03T00:00:00.000Z",
20c20
< "2027-02-28T00:00:00.000Z"
---
> "2027-03-03T00:00:00.000Z"
test\mirror_test.dart: nextBillingDate clamps Jan 31 to Feb 28 [E]
test\property_test.dart: nextBillingDate matches a month-by-month reference [E]
test\spec_test.dart: nextBillingDate clamps to the end of a short month [E]
test\spec_test.dart: nextBillingDate uses Feb 29 in a leap year [E]
test\spec_test.dart: nextBillingDate crosses a year boundary [E]
Some tests failed.
After re-recording, the snapshot suite passes with the bug written into its golden file, and the diff shows exactly what got certified: customers anchored on the 29th, 30th or 31st now bill in March instead of February. The spec, property and mirror suites still fail. The coverage suite, notably, passes with the bug in place, because its date test only checks that Jan 10 plus one month lands in month 2. The snapshot testing guide covers when golden files are the right tool, and this run shows the failure mode it warns about.
The Mirror Suite Caught the Month-End Bug
That same run produced a surprise in the other direction. The mirror suite, which copied its expected values from the implementation, still caught the month-end bug. The reason is that the spec comment names Jan 31 plus one month as an example, so a suite written by reading the file picked it up.
The mirror suite caught it only because the example was written into the doc comment. Where the comment named no boundary, the mirror suite tested one typical input and moved on. Its 51.9 percent score is the result. Consequently, a mirror-style suite is only as good as the examples the code author left lying around, and it can never catch a bug the author also wrote into the examples.
Random Inputs Missed Zero
The property suite ran 3,500 random cases and still missed four mutants that a single literal would have caught. M15 and M16 make prorateCents reject a monthly price of zero, so prorating a free plan throws. Similarly, M54 and M55 make a fixed discount of zero cents throw.
The generators draw prices from rng.nextInt(100000) and fixed discounts from similarly wide ranges, so any single case has roughly a one in 100,000 chance of producing zero. Across 500 cases per rule, a zero never came up. Meanwhile, the spec suite missed them because the spec says “throws when negative” and never says “zero is fine”, so no test names zero. Property tests are strongest when their generators deliberately include the boundaries, such as 0, 1 and the maximum, rather than relying on luck to find them.
Nothing Caught the Unawaited Save
M97 and M101 remove the await in front of invoices.save. In production, that means a failed write is never observed, and renew can report paid before the invoice exists. Every suite missed both, including the interaction suite, because a mock records the call whether or not anyone awaited it. Likewise, the in-memory fakes complete instantly, so the missing await changes nothing in a test.
No test shape on this bench catches that class of bug. The unawaited_futures lint catches it before any test runs. That is the same conclusion the earlier measurement on reviewing AI-written code reached: some defects belong to static analysis, and no amount of test writing replaces it.
The Twelve Survivors, Read by Hand
A mutation score is only honest after the survivors are read, because some mutants cannot be told apart from the original by any test. Those are called equivalent mutants. Here is the classification of all twelve:
| Mutants | Change | Verdict |
|---|---|---|
| M43, M58 | < to <= in a min-style comparison, where both branches return the same value when equal | Equivalent |
| M112 | Removes the early return failed after the last timeout; the loop ends and the fallback returns failed anyway | Equivalent |
| M18, M62 | Removes the negative-price and negative-tax guards; the rounding helper still throws ArgumentError for a negative input unless the other factor is zero | Nearly equivalent |
| M15, M16, M54, M55 | Rejects a price of zero or a fixed discount of zero | Real gap: no test names zero |
| M97, M101 | Unawaited invoice save | Real gap: no test can observe it, the lint can |
| M114 | Zero attempts returns paid without charging | Real gap: the line ran, nothing checked it |
Excluding the three equivalent mutants leaves 105. On that basis, spec and property tests together kill 93, or 88.6 percent, and all six suites kill 96, or 91.4 percent. The remaining real gaps are fixable with two more spec tests (zero price, zero discount), one more spec test for zero attempts, and one lint rule.
Which Tests Are Worth Writing When an Agent Writes the Code
Ranked by bugs caught and false failures together, the order on this bench is:
- Spec tests, one per sentence of the spec. Highest mutation score (84.3%), zero false failures, written against the public API only.
- Property tests for anything with arithmetic, dates or retries. Second-highest score (75.9%), zero false failures, and two unique kills, but only with boundary values added to the generators.
- Outcome tests with in-memory fakes for collaborators. They are part of the spec suite here: assert the charge, the saved record and the returned status, not the call sequence.
- A static analysis rule for each defect class tests cannot observe. Unawaited futures, swallowed exceptions and unused results belong to the analyzer.
- Snapshots only for formats an outside consumer parses. A public API response or an exported file, where any change really is worth a human look.
- Mirror, coverage-chasing and call-verifying tests, only as temporary scaffolding. They are useful as a safety net during a refactor, as long as they get deleted afterwards rather than maintained.
Put differently, the tests worth writing by hand, or worth reviewing hard when an agent writes them, are the ones that encode intent. Everything that can be derived from the implementation, an agent can regenerate in seconds. Those tests still raise AI test coverage, but they carry no information the code does not already have.
How to Ask an Agent for Tests That Kill Mutants
Agents follow the instruction they get. Asking for AI test coverage gets coverage, while asking for a rule gets a test of the rule. Most of the difference between the coverage suite and the spec suite is therefore in the prompt. A request shaped like this one produces the spec shape instead of the mirror shape:
Write tests for nextBillingDate and prorateCents from the spec below, not
from the implementation. Import only package:billing_bench/billing.dart.
One test per sentence of the spec. Every numeric rule needs a test at 0, at
1, at the boundary named in the spec, and one step past it. Assert exact
values, never isA, isNotNull or greaterThan. Do not assert error message text.
<paste the doc comments, or the ticket, here>
Two details carry most of the weight. Giving the spec rather than the file keeps the agent from copying current behavior into the expected values. In addition, banning type-only matchers removes the assertion style that let M114 survive. After the tests pass, a mutation run on the changed file tells both the agent and the reviewer which rules still have no test. Pasting the survivors back into the conversation is an effective second prompt, because each one is a concrete bug the suite would have missed.
The same idea fits a hook. A Claude Code hook that runs tests on every change can run a mutation pass on changed files instead of a coverage report, which feeds the agent survivors rather than a percentage. For the broader trade-offs of letting a model write tests at all, see generating unit tests with large language models.
Running Mutation Testing on Other Stacks
The runner used here is about 190 lines because the bench needed per-suite attribution. For a real codebase, a maintained tool is the better choice:
| Stack | Mutation tool | Notes |
|---|---|---|
| JavaScript, TypeScript | StrykerJS | Incremental mode re-tests only changed code, which keeps CI times reasonable |
| C#, Scala | Stryker.NET, Stryker4s | Same mutant states and scoring as StrykerJS |
| Java, Kotlin on the JVM | PIT | The documented mutator list maps directly to the operators above |
| Python | mutmut | Works with pytest |
| Rust | cargo-mutants | Needs no changes to the test code |
| Dart, Flutter | mutation_test | Configurable rule-based mutations |
Whichever tool fits, start with changed files on pull requests rather than the whole codebase. Mutation runs scale with mutants times test time, and a full pass on a large service can take hours. Google’s paper describes exactly this diff-based approach, and it is what makes the signal affordable.
A Real Scenario: The Coverage Gate That Rewarded the Wrong Tests
Consider a backend team of six to ten engineers that adopted a coding agent and, at the same time, raised its CI coverage gate from around 70 percent to 90 percent. Over a few weeks the number climbs past the gate on almost every service, mostly through tests the agent writes alongside each change. Review gets faster, because a green check on AI test coverage reads as diligence.
The first sign of trouble is that refactors get slower, not faster. Renaming an internal function breaks dozens of tests that import it directly, and moving a repository call breaks mock setups across several files. The agent fixes those failures by rewriting the tests to match the new code, which keeps the build green. Meanwhile, escaped defects do not fall, because the new tests assert types and call sequences rather than the business rules that change most often.
The fix that tends to work is to change what the gate measures rather than to raise or lower the number. The team keeps AI test coverage as a report but stops failing builds on it. Instead, it runs incremental mutation testing on changed files and asks for no new survivors in the lines a pull request touches. The trade-off is real: CI gets slower by minutes on large diffs, and someone has to triage equivalent mutants. On the other hand, the gate now fails for the M114 kind of gap, and a suite that only runs lines can no longer pass it.
When to Use a Coverage Target
- As a report that shows code no test reaches at all, since 0 percent on a file is still a reliable warning
- On new modules where untested files are the more likely problem than weak assertions
- Alongside a mutation score, where coverage explains why a mutant survived (never reached versus reached but unchecked)
- For finding dead code, since a line no realistic test can reach is often a line nobody needs
When NOT to Use a Coverage Target
- As a build gate on code where an agent writes the tests, because the agent meets it with tests like the coverage suite here
- As evidence that a change was tested, since the 100 percent suite here caught 44.4 percent of the mutants
- As a team or individual performance metric, which rewards exactly the shapes that scored lowest
- On orchestration code verified only through mocks, where high coverage coexists with false failures on every refactor
Common Mistakes with AI Test Coverage
- Asking an agent to “increase coverage” instead of “test this rule”, which produces execution without assertion
- Letting generated tests import from
src/and test private helpers, which turns every rename into a compile error - Accepting
isA,isNotNullandgreaterThan(0)as assertions on values the spec defines exactly - Re-recording snapshots on failure without reading the diff, which certifies whatever bug caused the failure
- Having the agent “fix” a failing test by matching the new output, which deletes the test’s meaning while keeping its line count
- Reading a mutation score without reading the survivors, since some are equivalent and some, like M114, are the most important result in the report
- Treating random generators as boundary tests, when a zero price never came up in 3,500 random cases here
For more on why the number misleads even without agents in the picture, the earlier guide to code coverage metrics and when they lie covers the general case.
What This Measurement Does Not Cover
The six suites were written for this bench to represent common shapes. They were not sampled from any model’s output, so the bench shows what each shape catches, not how often an agent produces it. Moreover, the same author wrote the library, the suites and the mutation operators, which may favor whichever shape the author understood best.
The library is 195 lines of code with no I/O, no concurrency and no UI, so the numbers say nothing about widget, integration or end-to-end tests. The mutants are mechanical, one operator at a time, and real bugs are often larger: a misread requirement or a missing feature will not show up as a flipped <. In addition, branch coverage could not be collected, and the four safe changes are a small sample of the refactors a real codebase goes through. Finally, nothing here measures the time it takes to write each shape by hand or by agent, which depends on the team and the tool.
Conclusion
AI test coverage measures how much code ran, and on this bench that ranked six suites almost backwards. The suite with 100 percent line coverage killed 44.4 percent of 108 mutants and broke on two of four harmless changes. Meanwhile, spec tests reached 85.9 percent coverage, killed 84.3 percent of mutants and broke on none. Spec and property tests together caught 86.1 percent, and snapshots added nothing on top of them.
Start this week by changing one instruction: ask the agent to test a written rule through the public API, with exact values and named boundaries, instead of asking it to raise coverage. Then run a mutation tool on the files the next pull request touches and read the survivors, which is the review the coverage number was supposed to replace.
