I wasn’t feeling very well.
It had seemed like a good idea at the time, but now I was regretting it. That morning I had woken up early while everyone else in the family was still asleep and quietly crept down the stairs and headed to the kitchen. I opened the refrigerator door and there it was: a full carton of eggnog. I pulled it out of the fridge, pushed a chair over to the cupboard, climbed up and found a cup, and poured myself a glass of delicious, creamy eggnog. After the glass was finished I had another, and then another until the entire carton was gone. It had been so delicious, but now my stomach was telling me I shouldn’t have done it. I will spare you the details, but that day I quickly came to realize that you can have too much of a good thing.
Kind of like automated tests.
They are a good thing. They are very helpful, but have you even been in a situation where you just have too much of a good thing? What happens when you have hundreds or even thousands of end-to-end or integration tests?
You start to realize that automated tests aren’t free.
Too Many Tests
How much time are we spending on these automated tests? Well lets add it up. We spend time fixing tests that fail for expected reasons. We spend time debugging intermittent failures. We spend time patching and updating VMs. We spend time creating and maintaining test builds. We spend time figuring out why things have broken. We spend time dealing with requisitions to get new machines in place. We spend money on buying those machines and operating them.
There are a lot of costs that go into tests and the above list in certainly not exhaustive. A lot of those costs can be hidden, but when you have too much of a good thing you can’t ignore them anymore. Your team is overloaded with work and it forces you to realize that automated tests are not free.
So what can you do about it?
You drank the cool aid eggnog, and now you have too many tests. How do you go about fixing the problem? Those tests were initially added for a reason – somebody at some point thought it would be a good idea to add each of them – so how do we reduce the number of tests? We don’t want to just arbitrarily delete tests, but we are already overloaded so we don’t really have time to go through and carefully evaluate them to figure out which ones to get rid of.
What we are face to face with here is something known as ‘the real world.’ There may be some perfect level of automation coverage and there may be some way to get there, but the reality is you can’t get to perfection. You just need to get better. Tackle the problem one bite at a time.
Find the most expensive tests
We outlined above some of the expenses that go into tests. All you need to do is find out which tests are the most expensive. Easy right? Well maybe not, but there are some strategies that can help you focus your attention when cleaning up tests:
Long running tests. That test that runs for 20 minutes – It’s expensive. The time it takes to run and debug that test is high. One really quick way to pick which tests to clean up or remove is to sort your tests by run time
Flaky tests. That tests that fails off and on for no reason – also expensive. You need to look into the failure again and again and you start to learn to ignore it. Do you really need that test? Can you isolate some of what it is doing so that you only check the thing you are interested in? This is another example of tests that are easy to identify (just look through the test reports for how often tests pass or fail) and that would be high value tests to cleanup or remove.
High maintenance tests. The test that always needs to be updated is another class of expensive tests. These can also be reasonably easy to detect if you version control your tests. Just look through the version control logs and see which tests are changed the most. Could you remove some of these, or change what they are checking so they don’t fail for things you don’t care about? Or could you perhaps check the things these tests are looking at more easily with a quick manual check of the product (gasp!). Not everything needs to be automated.
There are more strategies that you can use to help you prioritize your test cleanup activities, but we’ll stop here for now. What about you? Do you have too many tests? Have you started to realize that you need to do something about it? What strategies do you use to approach test cleanup?
offbeattesting 