The Verdict — Vladimir Mikhalev
Sep 27, 2026 · 19 min · 10 segments
Mutation testing for infrastructure code, done by hand across fifteen repositories. 153 planted bugs, and four green tests that could never have failed. Narrated by an AI clone of the author's voice.
The log cleaner for IIS and Exchange makes three promises.
Old logs of the named kinds go, everything else stays, and a what-if run touches nothing.
The first run caught its what-if test in this shape, checking the files and nothing else.
The plant rewrote the script's own should process check so that it always said yes and the script stopped asking before it deleted.
The test stayed green.
Remove item honors.
What if by itself? So the files survived either way and the test was proving a promise PowerShell keeps rather than one the script keeps.
With the guard bypassed, the script counts every file it believes it deleted and the last line goes red.
In banking, telecom and hyperscaler support, I have watched the same thing happen with every generation of tooling.
The instrument is trusted because it was installed, not because it was ever seen to fire.
The difference was one assertion.
Why this keeps happening? Coverage is the number teams report, and it measures the wrong thing.
In what was then the largest study of the question, Innozemtseva and Holmes generated 31,000 test suites for five Java systems of up to 724,000 lines and used mutation testing to measure how well each suite caught faults.
Once the size of the suite was held constant, coverage correlated with that ability only weakly to moderately, and stronger forms of coverage added nothing.
Their conclusion was that coverage should not be used as a quality target.
A test that runs a line and asserts nothing about it still covers the line.
Infrastructure code gets even less.
Mutation testing is old and well understood in application code.
Pit does it for Java, Striker for JavaScript and TypeScript, Mutmut for Python.
A Terraform module, a PowerShell maintenance script, and a CI example meant to be copied have no tool that understands what matters in them, and a random mutation is the wrong instrument anyway.
Reversing a comparison somewhere in a module tells you little.
Making a database reachable from the internet tells you at once whether the test guarding that promise works.
The badge hides the rest, a workflow that only lints goes green for the same reasons as one that tests and the reader cannot tell which it was.
The log cleaner for IIS and Exchange makes three promises.
Old logs of the named kinds go, everything else stays, and a what-if run touches nothing.
The first run caught its what-if test in this shape, checking the files and nothing else.
The plant rewrote the script's own should process check so that it always said yes and the script stopped asking before it deleted.
The test stayed green.
Remove item honors.
What if by itself? So the files survived either way and the test was proving a promise PowerShell keeps rather than one the script keeps.
With the guard bypassed, the script counts every file it believes it deleted and the last line goes red.
In banking, telecom and hyperscaler support, I have watched the same thing happen with every generation of tooling.
The instrument is trusted because it was installed, not because it was ever seen to fire.
The difference was one assertion.
Why this keeps happening? Coverage is the number teams report, and it measures the wrong thing.
In what was then the largest study of the question, Innozemtseva and Holmes generated 31,000 test suites for five Java systems of up to 724,000 lines and used mutation testing to measure how well each suite caught faults.
Once the size of the suite was held constant, coverage correlated with that ability only weakly to moderately, and stronger forms of coverage added nothing.
Their conclusion was that coverage should not be used as a quality target.
A test that runs a line and asserts nothing about it still covers the line.
Infrastructure code gets even less.
Mutation testing is old and well understood in application code.
Pit does it for Java, Striker for JavaScript and TypeScript, Mutmut for Python.
A Terraform module, a PowerShell maintenance script, and a CI example meant to be copied have no tool that understands what matters in them, and a random mutation is the wrong instrument anyway.
Reversing a comparison somewhere in a module tells you little.
Making a database reachable from the internet tells you at once whether the test guarding that promise works.
The badge hides the rest, a workflow that only lints goes green for the same reasons as one that tests and the reader cannot tell which it was.
The rest of this transcript — segmented and speaker-labeled, so you land on the exact moment something was said
Search every transcript — by keyword, by phrase, or by meaning, across every show Radar indexes
Trends — what is surging across podcasts, measured against its own baseline
Alerts — when a name you follow appears in a newly indexed episode
No account is needed to search Radar.
