Don’t be an Automation Miser

I don’t know if you are the kind of person who flips to the end of the book to see what happens, but I’m going to do that in this post.  I’ll start with the end of story.  Here it is: Embrace automation, not because it’s automation, but because of what it can do for you. Now for the story.

I was working on a script to do some fuzz style testing of some of our UI input fields and I got to thinking (because writing code has a way of making my brain wander off into the metaphysical realm) about why I was writing this.  Why bother automating?  This particular script is going to be very different from what we usually think of when we think of automated tests (i.e. regression tests), and so does it have a different goal or is the purpose of it the same as any other automated test?

Once I started off down the rabbit trail of the purpose of automation – oh my!  Things got out of hand quickly.  I had to start scribbling things down and making notes and before long I had a tangled mesh of ideas bouncing around in my head.  Sigh.  This happens a lot.  It’s why I have a blog.  Somewhere to force me to slow down and put some more structure into my thinking. So below are some of my (semi) structured thoughts on why we bother with automation.

Let’s start with something obvious.  We use automation to achieve some kind of goal.  That goal varies from team to team and company to company.  Perhaps it is so that we can ship code more quickly.  Perhaps it is so that we can hire less testers.  Perhaps it is so that I’m not bored. Or perhaps it is so that we can be sure of the quality of the code.  Whatever the reason(s) we have for it and however good or bad they may be, we can be sure of one thing, there is a reason for doing it.  Somebody, somewhere expects to see benefit from it or we would never bother with it in the first place.  If we start with an understanding of the goals we have for the automation we can think about it in two ways.  First of all we can think about if that goal makes sense and is achievable.  Sometime the goals that we have for our automation just aren’t reasonable.  For example if the goal of your automation is to be able to talk about how you have X% of your tests automated, you may need to rethink the goal itself.  That goal isn’t really tied to anything that has value to the company.

Once we have a reasonable goal for our automation we can start to figure out ways to evaluate how well it is meeting that goal.  This becomes a metrics problem and is actually much harder that you might think on the face of it, but for this post I want to focus in on the goals side of thing.  What are some of the goals I have for different automation projects I’m working on?

Automated Regression Tests

The current goal of these tests is to give us confidence that most of the stuff our customers depend on continues to work and to give us quick feedback when existing workflows are broken.  I have some additional goals for these as well that I have been thinking about recently, including using them to increase our agility and using them to more clearly define and establish the interfaces in our product.

Automated ‘Convergence’ Tests

The purpose of these tests is to allow us to experiment with various tweaks to the code in order to evaluate how well they enhance one part of the problem (the convergence) across a huge variety of cases.  The primary goal of these tests is to help us evaluate if a given hypothesis about how a change will affect convergence is true or not.

Automated ‘Fuzzing’ Tests

These tests are a way to help when testing new UI elements.  They can help check that the field limits are set in a way that match what the underlying engine can support.  The goal of these is to help make sure we don’t miss checking for certain things (zero, one, many, none, invalid, on the bounds, near the bounds, outside the bounds, etc. etc.)

So with just this simple survey of a couple of different automation projects, we can see that there are vastly different goals and purposes for each of them.  Automation is an incredibly powerful testing tool and we need to use it to help us achieve goals.  Sometimes we can start thinking like misers when it comes to automation.  A miser wants to have money so that he can have more money.  Of course a more reasonable approach to money is to use it to help you achieve certain goals.  Similarly an automation miser is someone who wants to have automation so that they can have automation.  Don’t be that person. Don’t just use automation because ‘we need too’ or so that you can have a certain number of automated tests. Use it to help you achieve your goals.