We have a problem. There are lots of regression tests in place and they are even automated (in parallel!) so we can run through hundreds or even thousands of tests on every build. But we still have a problem. You see we are in what we call the hardening phase of our release cycle and we are finding many defects. It seems that despite the many regression tests we run, a lot of bugs still get missed. Why is this? With so much automation and with so many good testers working on the product, why do we still struggle with our product quality?
I was recently writing some code and – as I usually do – I wrote some unit tests first. While writing the code I realized that I was using my unit tests to experiment with the implementation. I had a basic idea of what my implementation would look like, but rather than work out all the details of it ahead of time, I quickly tried something that I thought might be what I needed. I ran the tests and got some feedback on what I did wrong. I then went back to the function, made some changes and tried again. After a couple of repeats of this I had a working function that did what I needed it to.
However, at this point I realized something. My function was a bit of mess. It did what it needed to do as defined by the tests that I had in place, but it was kind of ugly and messy. There was a lot special case handling and the function had grown a lot of bloat. Thankfully I had the tests in place that, together with the knowledge I had gaining in the implementation, allowed me to go in and clean things up to make a much more readable and clean implementation.
Retrospecting on it afterwards, I began to wonder if having tests (even the much loved unit tests) in place aren’t a bit of a two edged sword,. There is a lot of power to being able to experiment and learn about what you trying to do, but if you don’t leverage the information gained from that experimentation you may be in a worse place that if you didn’t have the test at all. Thinking about it, I realized that if I hadn’t gone back in and cleaned up the code with the knowledge I had just gained through my experimentation I might have had worse code in place than if I had implemented the function without any tests written. Knowing that I did not have a safety net in place, I would have thought through the implementation more and probably would have written a cleaner initial implementation.
I think the same things happens on a broader scale in our development. We have many regression tests in place that create a safety net for the developers, but do we leverage the information we get from this to ‘make it work somehow’ or do we use the feedback we get to improve the underlying quality of the code we are working on? I fear that far to often the time pressures we have in place lead to a ‘make the tests pass’ approach rather than a ‘what can we learn from this failure’ approach. And so it is that the very thing intended to protect us can turn around and hurt us. Now all I have to do is figure out how to better wield this regression testing sword so that it can help us get better quality code. Should be easy right?
offbeattesting 
2 Comments