In my previous post, I promised to expand on how to cleanup your automation so that it could help you be both fast and consistent. In this post I want to share some principles that might help you turn your automation into something that leverages the speed of a hare with the consistent pace of a tortoise.
We’ll start with the speed. What can you do so that your automation helps you move more quickly?
Speed Principle 1 – Don’t Fail
Test failures slow you down. They require debugging and understanding where the failures came from. Now, of course you want your tests to fail when you have introduced defects but realize that every test failure is going to slow you down.
So what do you do if you have a test suite that fails too often?
Could I suggest that step one is figuring how to gather data? What data can you get from your test reporting system? Can you mine anything from git log that will tell you about how often you need to change certain tests? Once you’ve figured out the data part it should be pretty straight forward to figure out which tests are problematic and target those ones for refactoring. Also if you’ve figured out an easy way to gather the data, you should be able to keep on top of this as the project progresses. Keeping your tests useful should be an ongoing job.
Speed Principle 2 – Fail sooner
If you really must let your tests fail, the sooner they do so after the offending changeset the better. This applies whether the failure is for a bug or not. If it is for a bug that was just introduced, the sooner we know about it the easier it is to fix, and if the failure just requires a test update, the sooner we know the easier it is to debug the test and figure out why it is failing and what to update on it.
So how to go about improving this? For tests to fail sooner they need to be run sooner. This means you need to figure out why they are not running right away. Perhaps it is a deliberate thing where you gradually expose the code to wider audiences and run more tests at each stage. In that case you should also have tests that are less and less likely to fail as you progress through each stage.
However, in many cases I suspect the reason we don’t fail sooner is that our tests take too long to run. We don’t have the resources to run them close to the code changes and so they don’t fail early. There are a couple of solutions to this. One is to reduce the run time of your tests, another to increase the machine resources so that you can run more tests.
Speed Principle 3 – Don’t have annoying tests
Tests that are difficult to run or understand are not going to help you move more quickly. If it takes a complex build setup to get your tests to run and only 3 testers on the team know how to run them, do you think they are going to be able to help you move quickly? No, of course not. We need to have tests that are easy for the whole team to run and use. The usability and accessibility of your test framework is very important
To run quickly you need to have fast feedback and this means that developers need to run the tests. If you have a separate team writing your automated tests, you will need to pay special attention to making sure these tests are not annoying for the developers to use and run. Let’s face it, human nature being what it is, they will be unlikely or reluctant to use something they don’t see value in. Having tests that are difficult to run or that they can’t even understand the output of or that are very difficult to debug will make it much less likely that they get run early. Without early feedback on the impact of their changes, it will be hard to move quickly
Consistency Principle 1 – Encourage good coding behavior
Integration tests can be somewhat dangerous. They can encourage making tests pass at the expense of good development practices. Let me explain. Imagine you have two component that need to talk to each other through an API. If all your tests are at the integration level checking that these two components work together, you could encourage sloppy coding at the interface between the two components. For example, if a defect comes along that needs to be fixed quickly, the developers might tempted to put in a fix where one component directly calls a class from the other, rather then changing the APIs. If you have tests that help define the interfaces between these different parts of the products, you will make it harder for people to ‘cheat’ in their coding as the tests themselves will help define the interfaces.
This is just one example of how tests can be used to encourage good coding behaviour. There are other things you could do as well, but this article is already getting too long, so we’ll leave it there.
Consistency Principle 2 – Keep your tests clean
If you want to be consistent you can’t work in spurts. You can’t go from flying along adding new features one month to being bogged down in test maintenance the next. If you do this you’ll be like the hare in Aesop’s fable and you’ll end up at the finish line much later that if your moved at a consistent pace. This means you can’t let test maintenance debt pile up. You can’t leave those failing tests alone for a week or two because you ‘don’t have time’ to update them. The longer you wait for this the further behind you get.
There are also more subtle ways you need to be careful of this. Tests age and start to loose their value over time as the code base changes. You need to regularly spend time looking at how to keep you tests valuable and checking the important things. If you don’t do this, you will eventually get to the point where you realize your tests really aren’t that helpful and then you’ll have a major job on your hands to clean them up. Scheduling time to review and making small changes to keep your tests clean will pay off huge dividends in your team’s ability to be able to consistently produce.
Consistency Principle 3 – Keep your tests short
Here’s the thing with long running tests. When they fail, it takes a long time to figure out what went wrong. If your test takes 15 minutes to run, it will take at least 15 minutes to debug when it fails, and probably much longer. How often do you only run a test once to figure out what went wrong? This might not seem like too big of a deal, but the more long running tests you have, the more time you will end up spending on debugging tests when things go wrong. This leads to a lot of inconsistency in your delivery times, since things will move along smoothly for quite a while, until suddenly you have to spend a few hours debugging some long running tests.
So those are a few principles that could help you clean up your automation if it is in a bad state. I have been working on this kind of cleanup over the last couple months and so these are principles that come from my real life experiences. I will probably be sharing more on this in the coming weeks.
offbeattesting 
2 Comments