Another interesting software quality technique I’ve run across lately is Mutation Testing. Long story short, it makes slightly altered copies of your code, called “mutants” (because it’s mutated them). Then it runs your tests against each mutant. The idea is that each mutant should make at least one test fail. If that doesn’t happen, then there are three main possibilities.
-
You have a gap in your test suite, and should add a test to distinguish between what happens with the “original” and “mutated” versions of your code.
-
You have code that doesn’t really do anything. Maybe it’s unreachable, maybe it’s redundant, maybe it just doesn’t do what you think it does.
-
It’s a false alarm, which happens annoyingly often.
Changes it can make include things like:
- changing a mathematical, logical, or comparison operator to another one, like + to -, * to /, < to >, and to or, and so on.
- changing strings to empty or some constant.
- changing numbers up or down by one, or to some constant (usually something significant such as zero, +/- 1, +/- infinity, etc.), or negating them.
- removing an entire statement.
- removing the control of a condition, including the one that controls a loop.
- removing an argument from a function, or a function call.
- and many more, though not all tools will implement all mutations.
This is available for lots of languages, and was actually invented in 1971! So why wasn’t it popular until very recently? Mainly because it’s rather CPU intensive, and only in the past couple decades or so have consumer-grade machines been powerful enough to get the job done fast enough. To do a decent job of mutation testing on a 1980’s-era PC, let alone a 1970’s micro/mini, would have been much too slow, and doing it on a larger machine would likely have made the machine too slow for the rest of the timesharing users.
UPDATED TO ADD: I now have given several conference talks about this (and a few about ACRUMEN), so I’ve put up a Speaking page. You can see there, where and when I’ve spoken about what, with links to recordings when available. To just go straight to my mutation testing talks, see my Mutation Testing YouTube playlist.