How can we evolve you today?

Blog: Mutation Test to Find Untested Code

Another Quora question:

What are some ways to know if your unit tests are covering all of your application’s code, or if they’re missing some pieces?  What tools can be used to determine this?

The usual answer is a coverage tool.  (But this is wrong.  Hear me out.)  This will tell you what parts of your code (usually lines, sometimes branches or expressions) were executed by at least one test.

Inigo Montoya telling Vizzini 'You keep using test coverage.  I do not think it means what you think it means.  Try mutation testing.'

Test coverage is nearly useless.  It is much better as a negative indicator.  The code that wasn’t excuted, you know definitely isn’t tested.  But what about the code that was executed?  Was it really tested, or only executed?  What’s the difference, you may ask?  The key question is, did the correctness of that code make any difference to whether any test passed or failed?  (That’s what we really mean when we ask if code is “tested”, right?)  Simple coverage doesn’t tell us!

So how can we know?  Maybe you make a change to that code, and see if any test fails (in your otherwise perfectly passing test suite).  If it does, great!  If not, you have either a hole in your test suite (at least regarding the new code), or ineffective meaningless code (such as if it’s unreachable or redundant).

But that’s a royal PITA to do manually.  We’re certainly not going to do that for every constant, expression, function call, etc. in our code.  So what do we do when we have something relatively simple, but tedious?  Automate it!

Enter “mutation testing” tools.  These make lots of copies of your code, with one tiny little change (or “mutation”) each, and see if the mutated copies still let your tests all pass.  They are available for all major programming languages, and are generally easy to set up — but they can be slow to run, and difficult to figure out what to do about the results.  For more information, I have a conference talk on the matter!  My latest recording is: