Check out my latest article on the Test Project blog.
I wrote about some thoughts on how to test your tests
Check out my latest article on the Test Project blog.
I wrote about some thoughts on how to test your tests
This isn’t the usual kind of post on this blog, but I want to take a couple of minutes to talk about changes in my life. I started my testing career at Ansys right out of university and it has been a good 9.5 years. I have learned a lot a grown in many ways, but the time has finally come to try something new. I started today at D2L. It is a totally different area and will present many new challenges for me as I work with a different technology stack and in a very different area. I’m looking forward to learning and growing in many ways and I’m sure I will continue to share thoughts and experiences as I learn new things and try to work though new learning challenges.
Have you ever been lied to?
I don’t mean those little white lies that smooth social interactions – “You don’t mind if I do this?” “No, not at all” – I’m talking about a full blown lie. You ask your son to take out the garbage and he tells you did it already. The next morning, after the garbage truck has left, you find out he never did it. Or you call customer service and they promise to take care of it as soon as you hang up, but you have to call back two days later because nothing has happened. What happens when you are lied to? You lose trust in that person or organization.
Much of the way we work as a society is build on trust. There are places in the world where stores have armed men with guns by the door to deter theft. Why? lack of trust. In the West we have many blessings and one of them is trust. I can call a dump truck and ask them to bring by a load of sand and they will do it and send me an invoice. Why? They trust that I will pay them. Without that trust the transaction would become much more painful. I would somehow need to get them money ahead of time, but would I really want to if I couldn’t trust they would deliver once they had the money? Transactions become much more difficult in a society with low trust.
Trust in Software Development
In the software development industry we primarily view trust as a financial proposition. “If my customer doesn’t trust my credit processing system they won’t click on buy and I won’t make money.” However, trust is a bigger problem than that. It is a societal problem. In a world with low trust we have high cost. Everything get expensive when there isn’t trust in place.
How do we contribute to lack of trust in software development? Well, how about when our customer’s data gets hacked? Does that build trust? What about when users come to our site and it won’t load? Does that build trust? What about when your ‘click here’ icon doesn’t do anything? Does that build trust?
Let me drive the point home in a way that might hurt.
If you are ok with software that breaks trust, you are not working for the social good.
But I think it’s true. We hear a lot of talk about social entrepreneurship and the importance of social initiatives in corporations. I want to challenge software development teams – and in particular those who think about testing problems – to think about the social responsibilities they hold. The ethics of testing are clear in some domains like medical applications, but what about if you are writing an iPhone app? Do you have ethical considerations then? My argument is that if you are ok with releasing a buggy app that breaks the implicit and explicit promises you are giving your customers you are contributing to a societal problem. Teaching people to expect to be lied to is not a good thing. Don’t have any part in it. Release good quality software.
It’s ok to make mistakes. We will sometime break people’s trust despite our best intentions. Don’t beat yourself up over missing things. The point of this post is that it should be despite our best intentions. We ought be shooting for high quality, trust building applications. Saying that it’s ok to ship with bugs in our product, shouldn’t just be a financial decision. We need to be thinking about other things like the ethics of it as well.
We’ve all been there are some point. It’s crunch time. The deadline is looming. Everyone on the team is frantically working to get the product shipped. Crunch time reveals to us what we really see as important. During crunch time certain things get dropped. Think about the things that get dropped on your team during crunch time and I’ll bet that they all relate to one thing:
We stop checking things in as much detail as we used to. We take some shortcuts in our coding. We do things that we know are not good for the health of our code base. We quickly patch up defects without addressing the underlying issues. Under pressure, the first thing offered on the alter of the Almighty Deadline, is quality. Our real priorities are revealed. We might talk about how we take quality seriously, but our actions show differently. Our actions show that quality is seen as the low guy on the totem pole. Perhaps this is why in some places testers feel like they are lower status members of the team. Testers are concerned with quality. How do you think they are viewed if we see quality as the least important part of the software development process?
But what if it isn’t? What if quality is actually the most important part? What if many of the problems and challenges we face in the software industry are directly related to quality? If that is the case, the way we are approaching our problems is totally upside-down. Instead of the first thing to go, quality should be the last. Instead of a dispensable part of our process it should be the keystone of everything that we do.
Here is the thing: If we want to be able to move more quickly we need to invest in quality first. Quality can’t be an afterthought. It can’t be something we tack on to the end of the process. It needs to be something that is fundamental to the very core of how we go about making software. Until we understand this and until we start to act like we believe it, we are just going to keep spinning around on the same hamster wheel.
One of the challenges we face with quality is that it is often a hidden property of the system. Imagine you go to your local generic chain furniture store and pick up a pretty cheap table. Or imagine you buy a custom made table for 5 or 10 times as much. The two tables might look very similar (initially), and yet which one do you think has higher quality? Quality is often not immediately visible. This is what make it so tempting to cheat on it. At crunch time we can hide sloppy work and shoddy testing, because it won’t be immediately visible. But the fact is, poor quality will be experienced by someone. It won’t stay hidden forever.
As testers we are like the mechanic friend you take along when you are looking at a new car. You see a car that looks nice and has some nice features – it’s even red! – but then your friend crawls under the car and opens the hood and shines his flashlight around to see in the cracks and crevices. He comes back with a list of problems (or potential problems) that you’ll want to consider. He’s slowing down the process. Now you have to go look at another car or negotiate a lower price or get the owner to fix those things. You just want that car now, so you shouldn’t bring your friend along next time right? Right? Or not?
Another challenge when it comes to quality is that we don’t pay the price. If we release a low quality product the company might get hurt and it might loose some money, but oh well, we can always find a job somewhere else right? And besides somebody has to fix up all those problems. The reality is most of the pain of the bad quality products we work on get passed on to someone else.
Invest in Quality now!
But we are ethical – and I really believe we are – so most of us want to do the right thing. We want to make something we are proud of. We want to help people. We want to do right by our customers and our employers. So how do we do this during crunch time? We need to get these features out for the good of the company and the customers. They are important. How do we do that without sacrificing quality?
We do it by investing in quality now.
You know you are going to have crunch times. We live in a world where we can’t see perfectly into the future and so there are going to be times when there is a crunch to get things done, no matter how well we have planned. We also know that during crunch time one of the easiest things to sacrifice is quality. So prepare for this. Invest in your quality well before crunch time comes.
When we are driving for that final push, do you think it will help to have a robust and useful set of automated tests in place? Well, you certainly won’t get that up and running during the crunch time – invest in quality now.
When you are in the middle of a crunch do you want to discover fundamental flaws in your architecture or design? You are only going to find problems like this early if you look for them early – invest in quality now.
When the pressure is on and someone is out sick, do you want to have to feel your way through some tangled, unreadable code in order to fix a bug? Having clean and readable code doesn’t just happen by itself – invest in quality now.
When you are trying to merge that feature in, do you want to have no time for interactive testing because you are waiting for it to make it through the build pipeline? Good build pipelines take time to build and maintain – invest in quality now.
When there is very little time for testing do you want testers to struggle to trace down issues they see? At crunch time it is too late to think about testibility – invest in quality now.
Crunch times will come, and during crunch time we will have to change some of things that we do, but if you want to come out the other side holding your head high and able to be confident that you’ve done a good job, you need to think about quality. Don’t put it off or leave it as an afterthought. You’ll be much more successful as a software development team if you start with quality.
Invest in quality now.
I wasn’t feeling very well.
It had seemed like a good idea at the time, but now I was regretting it. That morning I had woken up early while everyone else in the family was still asleep and quietly crept down the stairs and headed to the kitchen. I opened the refrigerator door and there it was: a full carton of eggnog. I pulled it out of the fridge, pushed a chair over to the cupboard, climbed up and found a cup, and poured myself a glass of delicious, creamy eggnog. After the glass was finished I had another, and then another until the entire carton was gone. It had been so delicious, but now my stomach was telling me I shouldn’t have done it. I will spare you the details, but that day I quickly came to realize that you can have too much of a good thing.
Kind of like automated tests.
They are a good thing. They are very helpful, but have you even been in a situation where you just have too much of a good thing? What happens when you have hundreds or even thousands of end-to-end or integration tests?
You start to realize that automated tests aren’t free.
Too Many Tests
How much time are we spending on these automated tests? Well lets add it up. We spend time fixing tests that fail for expected reasons. We spend time debugging intermittent failures. We spend time patching and updating VMs. We spend time creating and maintaining test builds. We spend time figuring out why things have broken. We spend time dealing with requisitions to get new machines in place. We spend money on buying those machines and operating them.
There are a lot of costs that go into tests and the above list in certainly not exhaustive. A lot of those costs can be hidden, but when you have too much of a good thing you can’t ignore them anymore. Your team is overloaded with work and it forces you to realize that automated tests are not free.
So what can you do about it?
You drank the
cool aid eggnog, and now you have too many tests. How do you go about fixing the problem? Those tests were initially added for a reason – somebody at some point thought it would be a good idea to add each of them – so how do we reduce the number of tests? We don’t want to just arbitrarily delete tests, but we are already overloaded so we don’t really have time to go through and carefully evaluate them to figure out which ones to get rid of.
What we are face to face with here is something known as ‘the real world.’ There may be some perfect level of automation coverage and there may be some way to get there, but the reality is you can’t get to perfection. You just need to get better. Tackle the problem one bite at a time.
Find the most expensive tests
We outlined above some of the expenses that go into tests. All you need to do is find out which tests are the most expensive. Easy right? Well maybe not, but there are some strategies that can help you focus your attention when cleaning up tests:
Long running tests. That test that runs for 20 minutes – It’s expensive. The time it takes to run and debug that test is high. One really quick way to pick which tests to clean up or remove is to sort your tests by run time
Flaky tests. That tests that fails off and on for no reason – also expensive. You need to look into the failure again and again and you start to learn to ignore it. Do you really need that test? Can you isolate some of what it is doing so that you only check the thing you are interested in? This is another example of tests that are easy to identify (just look through the test reports for how often tests pass or fail) and that would be high value tests to cleanup or remove.
High maintenance tests. The test that always needs to be updated is another class of expensive tests. These can also be reasonably easy to detect if you version control your tests. Just look through the version control logs and see which tests are changed the most. Could you remove some of these, or change what they are checking so they don’t fail for things you don’t care about? Or could you perhaps check the things these tests are looking at more easily with a quick manual check of the product (gasp!). Not everything needs to be automated.
There are more strategies that you can use to help you prioritize your test cleanup activities, but we’ll stop here for now. What about you? Do you have too many tests? Have you started to realize that you need to do something about it? What strategies do you use to approach test cleanup?
The other day I was watching a show on TV about how food is made and I started thinking about software testing (because that’s what normal people do right)?
So much of what we have in our society is made on assembly lines and food is no exception. That candy you bought? Produced in an assembly line style environment. The loaf of bread and container of milk? Same thing. We still have artisanal food, but the reality is that most of what he eat is at least partially made in a factory somewhere.
How do you find meaning in what you do in the assembly line world? How do you find meaning in just being a step in a process. Do you actually feel like you have made something when at the end of the day, you’ve rolled out 240 candy canes per hour? I think that we as humans have an innate desire to create, but does a process like this allow you to create? Can we find meaning and fulfillment in a job like this? Can we meet our need to create things when most of what we build as a society requires the input of many different people?
I think we need to think about these questions, and not just in the assembly line context. These kinds of questions matter in the software world as well. It isn’t just physical goods that are too complex for most of us to make on our own. The software systems we build require teams as well. We may call this creative work, but very few of us create entire products. We work on one small piece that fits in with other pieces. Can we fulfill our need to create in this context?
What about those of us who’s primary role on the team is to test? What do we create? How do we scratch that creative itch?
I think this is why a common complaint among testers is that they get left out of decisions about what the product will do or look like. It’s why we talk about the need to shift left. It isn’t just because those things are helpful (although they usually are). It’s because we have a need to create something and to be included in the creative process. If all we are doing is checking someone else’s work to see if they did it correctly, we have no job fulfillment.
Before I started in the testing job I have now, I worked in a couple of QA jobs in factories. I hated them. Take this batch of paint. Measure it for various properties. Compare to the specification table the customer had provided and accept or reject. Boring work with nothing to point to. At least if I worked at a factory that made candy canes, I could look at a candy cane in the store and be proud of the fact that I had twisted two colors together perfectly. If I looked at a can of paint, all I could say was that I had told everyone they did their job correctly. Not only was the job boring, it felt like it wasn’t tied to something I was creating.
Now back to software testing. If my job as a tester is to compare the work someone is doing to a specification and accept or reject it, I’m not going to feel like I am producing something of value. We want to be involved in making something. We want to be able to point to the system proudly and say “I helped make that!” The human need for creativity is why we testers want to be involved in more than just checking if the product does x or y, but – and this is important – it is also why your team needs a great tester. All members of the team want to be able to point to the software and be proud of the fact that we made something great. If we can’t do that we will be demoralized as a team and we aren’t going to produce a lot of value in the long run. Testers help with this! Value your testers. Listen to them. They share the same desires you do. The desire to make something we can all be proud of. Let’s do this together!
David Hume’s Guillotine tells us that we cannot derive an ought from an is.
Abstract philosophical musings you might think, but then again, you work in software, so you know that philosophy matters, right? I want to muse on this for a while, and not in an abstract realm. I want to take this down into the real world. I’m going to take the law at face value. If it really is true that an ought cannot be derived from an is, what should we do differently?
“How can I automate this?”
“Should I automate this?”
Two similar sounding questions, but which one you start with makes a world of difference. How often do we start with the is? We start with the fact that we have a framework in front of us that let’s us easily add tests, or the fact that we are told to automate everything or even the ‘fact’ that manual testing is too slow or too cumbersome for the modern software shop. But aren’t we forgetting something? Aren’t we forgetting the law our friend Hume told us about? Just because we can do something doesn’t mean that we should.
This applies at every level of software development (because what is software development except automation?)
Should you make this product?
A number of years back the company I work at started a new initiative. We had acquired two previously competing products and after the acquisitions we decided to try and combine the best of both products and re-architect everything into a new product. It seemed like a great idea and we put a lot of effort into it for several years, but recently the product was canceled. Why? Because we had decided that this product did not need to be made. Some people were interested in it, but market analysis and other factors indicated that it was not worth further effort. We might have been right or wrong about the market forces at the play and the strategy employed, but this does illustrate something. Could we make this product? Sure. Should we? Not if it won’t help people. Not if it will lose us money.
Just because you can do something doesn’t mean you should.
Should you automate this?
What about an example that is a little more related to test automation?
At one point I worked with a team that did workflow tests. These were scripts that we would run through every morning to check if the nightly builds were ok. As you can imagine they were boring to do and so we decided to automate them. Makes sense right? Automate away the boring stuff. We don’t want to spend half an hour every morning working through the same boring scripts. So we went ahead and did it. We automated them, but should we have?
I’ve talked before about how boredom can be an indicator that you should automate something, but that it can also be an indicator that you are doing something useless. So which is it here? I think a bit of both actually. We automated them all at that time, but since then we have removed or changed many of them. There are still a few that have stayed around because they are valuable, but we certainly didn’t need as many as we had.
This once again illustrates the point that just because you can automate something doesn’t mean you should. If we start from the is – is this possible? Does a framework exist that allows me to do this? – we never get to the ought – does this make sense? Will this be valuable? If we had started with ought questions we would have just gotten rid of half the scripts we had and automated the few that were actually valuable. Instead we started with the is questions and ended up spending a lot of time creating and maintaining tests that we really didn’t need.
We need to start with the ought. In the software world we (understandably) love using software to do things. Sometimes that is the wrong answer. Sometimes there is a non-software solution to something. Sometimes we are just doing the wrong thing and so setting up systems to do it faster isn’t helping anyone.
Start with ought. Know why you do what you do. Take a minute to think things through before you plunge in and start doing. If you know the ought, figuring out what to actually do becomes much easier