The Challenge of Coverage

The idea of test coverage is a bit of a holy grail in the software testing world.  When unit testing you’ll often hear about a certain percentage of the code being covered, and with higher level testing you will often hear questions about how well we have covered a feature. As with most things the idea of coverage is a fuzzy idea, but one of the most important lessons I’ve learned about it (from Dorothy Graham in this talk) is to ask the question ‘of what?’ Whenever we realize we are talking about coverage we should be thinking about what we are trying to cover.

It is a very helpful to realize that there are many many forms of coverage. We can never cover all the coverages which is why complete testing of any non-trivial software is impossible, but in some cases we can theoretically cover all (or most) of one type of coverage.  We could for example theoretically get complete line coverage of a code base.

I realized recently though that knowing how to get complete coverage of a particular area can be a bit of double edged sword.  Just because we know how to cover something completely, doesn’t mean we ought to.  In fact, sometimes even using sampling mechanisms like combinatorics testing doesn’t make sense.

I was recently trying to test something that involved the ability to create ‘expressions’ according to certain known rules and inputs.  It was seductive.  I started to make a table of the different ways things could combine together to create different combinations.  I quickly realized that in an n x m matrix like I was dealing with there were many millions of possible combinations and so I started putting in some sampling heuristics to try and reduce the problem space.  As I kept going down this path of creating possible expressions, I eventually realized that I might not be effectively using my time.  Sure I was using a script to help me power through these combinations, but there were little tweaks and changes that needed to be made and then for each combination I would have to run the software for a few seconds.

It was going to take days to get through this all.  Was it worth it?  When I stopped to think about the idea of ‘coverage of what?’ I realized that perhaps I was focusing in on an area where the value of my coverage was low.  There were many other aspects of coverage of this feature that I was not considering because I was so focused on this one area of coverage.  The reality was that the ability to get a high level of coverage in a certain area had seduced me into spending too much time in that area.  Just because I can do something and I know exactly how to do it, doesn’t mean it is the most valuable thing to spend my time on.  I had to leave that area with lower coverage and focus in on other areas instead because the reality was there was a much higher risk of finding problems in those areas.

This is one of the challenges of having measurable coverage.  Some types of coverage are much harder to measure but that doesn’t mean they are less important.  When we have a particular area we can measure it can give us goals to work towards. This can be helpful, but if we let it drive our thinking too much we can easily be doing low value work to meet a coverage goal in the place of some much higher value work on other aspects of coverage.  I think we all tend to bias towards what we know and understand, but don’t forget that it is often in the less explored areas that the nugets of gold are to be found.