As I mentioned before, I have moved to a new team and we are working on a consolidation effort for automated regression tests that we inherited from several different teams. I have spent the last 2 weeks setting up builds to run some of the tests in TeamCity. 2 weeks seems like a long time to get some testing builds up and running and so I stopped to think about why it was taking me so long. I know TeamCity quite well as I have been using it to do similar test runs for years now. The automation framework tool that I’m working with now is new to me, but it is pretty easy and straightforward to use. I would say I only lost about a day figuring out the tool and getting it set up on my VM’s. So what is making this take so long?
The most important factor at play here is how long it takes to run the tests. I broke the tests up between several builds, but each build still takes between 3 and 6 hours to run. What that means is that if I have a wrong configuration on the machine or I’ve setup wrong permission or anything like that, I have a 3 to 6 hour turnaround on getting feedback. I did manage to help that out a bit by creating some dummy test runs that were much faster which allowed me to do some of the debugging more quickly, but at the end of the day, these long run times slowed me down a lot in my ability to get the tests up and running.
But let’s stop and think about that for a minute. What is point of running automated regression tests? To give feedback on any regressions we might have introduced while making changes to the code right? Well, if that is the case, how effective are these test runs at doing that? The length of the feedback loop made it a slow process for me to find and fix issues with my builds and led to it taking much longer to get them setup than it would have otherwise. What does the length of that feedback loop do for finding and fixing issues in the code? The slow feedback meant that I would have to go work on some other stuff while I waited for a build to finish, which led to some of the inefficiencies of multitasking. Does the same thing happen with long feedback loops for code changes?
I could go on, but I think the point has been made. The pain I felt here applies to code changes as well. Having a long running set of regression tests reduces the value of those tests in many ways! Do we perhaps have too many tests?