When should you automate?
Well let’s think about it for a minute. What is automation good at? Repeating the same thing over and over again. So when should you automate? When you have something you want to repeat over and over right?
But let’s think about it a bit more. What does automation do? Repeats the same thing over and over… What if you want to change something? Your automation might just lock you into something you don’t want any more.
Automation is a powerful way to add leverage. If it is done well it can allow you to get way more done in the same amount of time. However, it can also reduce flexibility. Let’s look at an example. You’ve created a high level test that checks for the existence of a certain value in a table. Now you want to change something in your app so that the value will be in a div instead of a table. At this point you either have to get rid of the test, re-write it completely, or not make the change all. Your automation has made you less flexible.
So we still haven’t answered the question: When should you automate? Let’s take a stab at a couple of heuristics that can be used to help answer that question
When you know what you want
If you have confidence that things will be a certain way for a significant amount of time, it probably makes sense to do some automation. The designs are settled on and we are fairly certain this is the way we will go for the foreseeable future. In other words what we are looking at here is an estimate of the shelf life of the automation. How confident are you that this is something we want to lock ourselves into?
When the automation is simple
Sometimes though, automation still makes sense even when we don’t know what things will look like. For example, when the automation is simple to make. If it only takes me two minutes to automate something it doesn’t really matter to much if the design changes next week and I have to throw that test out.
In a sneaky way, what I’ve been talking about here is your return on investment. How do you know when to automate? Well it comes down to understanding how much leverage your automation will give you and for how long, as compared to the cost of creating that automation. If it is easy to automate something, the ROI comes more quickly and so we can create it in more uncertain environments. If it is hard to automate something, we need to have a lot more confidence that we will be able to use that automation for the long term.
I want to take a step back from this and think about how it applies to test automation in general. I think there is a principle here that bears some reflection. Software creation is an inherently unstable process. We are constantly changing things in response to customer and business needs. In many cases the shelf life of test automation is going to be quite short. It won’t be long before a test fails. If we think about this in terms of ROI doesn’t that imply that we should bias towards smaller tests? Smaller, easier to create automated tests give a return on investment more quickly. The more difficult a test is to create and maintain the more hesitant we should be about automating it.
So when should you automate? When it gives you a good return on your investment.
The table value example you give is a structural change to the object definition/locator. If you have isolated the objects location definition properly you don’t have to throw away the test (or other tests that reference it), you fix the locator information. Once done the test should be able to run again without a problem. Abstraction of functionality, information and data need to be baked into your automation. This will lead to more robust and maintainable code. And I believe you know that.
Regarding when to automate, I can agree with your points. Even if the ROI is low and there is some benefit, and it makes sense, you could automate the process. Notice how I said process, automation can be used for other things in addition to the test itself. Either directly related or not.
Thanks for your comment. I totally agree that there are strategies to help make automation less expensive to create and also less expensive to maintain. Knowing and investing in those kinds of things is certainly a key part to getting good ROI on your automation.
I love what you said about automating the process. There are so many ways to automate besides just creating test cases. You can automate test reporting for example, or data setup, etc. etc.