ROI of Less Automation

I want to reduce the size of my automated regression testing suite.  It takes too long to run and it takes too much work to maintain.  It’s a big clunky beast and I want to reduce the size of it.  The problem is I don’t want to just arbitrarily delete tests.  I want to reduce the size in an intelligent way since there is still value to be had in some of the checks we have in there.  To do this requires work – a fair bit of work at that, and since we are paying people to do that work, it requires money to do this.

So how do I know if it is worth it?  What is the return on investment the business will get out of the work it will take to reduce the size of my automated test suite?

It seems like we don’t often ask the question from this angle.  Go google search queries related to the ROI of test automation and almost everything that comes up will be about how you can justify adding tests and ways to calculate the return on investment your business will get from adding test automation.  However, the reality is that not all tests are created equal.  Some tests do give you a positive return on your investment and some tests give a negative return.  In addition, the return vector can change over time. A test that was very valuable when it was made, might now be costing you more than it gives you.  To satisfy yourself on the fact that not all tests are equal, simply turn back to our good friend Google and ask him (her? it? –  I dunno) about test automation problems. The millions of hits you get speak to the fact that there are many ways for test automation to not add value.

We have a disconnect here.  Many people talk about the ROI of using test automation and many also talk about the how to deal with problematic tests, but are we connecting these two together?  How can we figure out (even in just a rough way) when there is a positive return on removing test automation?

Well let’s talk about the return side of things.  What benefits could your company see from having less automation?

Reduced Test Maintenance

Anyone who has worked in test automation will be able to pretty quickly tell you one of the main benefits – less test maintenance time.  Having less test to run, means less time spent on figuring out failures and updating or changing tests. This seems pretty obvious, but let’s not skip over this point too quickly.  There is more to test maintenance than just the time spent reviewing tests.  The more tests you have, the more machines you need to run those tests on.  Those machines also need to be maintained.  You need to apply patches and reboot them after power failures, or you need to buy licenses and time in the cloud. You also need some way to run those tests, which probably means some kind of build system like TeamCity or Jenkins.  More tests and more machines means more complexity in your build system as well. Those setups and script need maintenance too.

Most of us already know that test maintenance can be pretty expensive, but have you ever stopped think about how much it really cost you?  You might be surprised!

Reduced Machine costs

Machine cost have been driving downwards since computers made their debut and so we might think this one doesn’t matter as much in today’s world, but once again let’s not skip too quickly over this one.  There are a number of factors that play into machine costs.  In addition to the hardware purchases, we have the ongoing electricity costs of running that machine and the licensing costs to have an operating system on that machine.  We also often have other more hidden costs like insurance and the time spent setting up the machines in the first place.  Things like installing the OS and virus scanners, and setting up the machine on the proper domain controllers, etc. etc.  All of these costs add up and mean that there are significant savings to the company for each machine we don’t have to have for running automation.

Increased Coding Productivity

This one requires a little more nuance and thought, but I think there is an argument to be made that in many cases reducing your test automation will make developers become more effective.  This is the opposite of most of the sales pitches you’ll hear about automation, but hey you should probably use your critical thinking skills when it comes to sales pitches anyways.

Snarky comments about sales pitches aside, this probably isn’t as obvious as some of the other things I’ve talked about so I want to dig into it a bit more. Let’s say you have a test suite that takes a long time (definition:  more than 1 hour) to run. Let’s also say it needs to run before code gets merged into the release branch.

At this point we can think though a simple workflow: a coder submits code for merging and the test suite starts running.  An hour later, the build fails for some reason. The coder takes a look at it and it ends up being due to some missing edge case that a test caught.  She makes the fix and re-submits the build and things get in.

This is exactly why we run automated tests right?  To catch those sneaky issues. However, if we extend this thought experiment a bit more we can see where things start to go wrong.  What if you add some more tests that add a couple more minutes to that test run?  And now some more.  And some more.  Keep going with that thought experiment until we have it taking 5 hours to run the test suite.  Now when the developer runs the tests they find the same issue, but she doesn’t find out about it until the next day because the run took so long.  Does waiting this long help or hinder the coder’s productivity?

Of course there are many ways that we try to mitigate this kind of problem (parallel test runs, smoke tests vs. daily tests etc.), but at the root of it all is trying to solve this problem: increasing the run time of a test suite slows down developer productivity.  The converse of this is that by reducing the run time of your test suite, you might just be increasing developer productivity.

So how is your test suite?  Is it worth investing in making it smaller?  Could you help the business by running less test automation?  Count the costs of your test automation – it just might surprise you how expensive it really it!