In my last post I proposed that it is possible to create bug free software if the context is very limited and the number of stakeholders who determine if something is buggy is small. In the discussion on that post in various forums we fleshed out a bit more the idea that this is limited to a snapshot in time, since new context or stakeholders could change the things that are considered to be bugs.
Can this apply at scale though? If you have a large scale software product with thousands or millions of users and stakeholders can it be bug free?
I think here the answer is no. But I think it is a softer no that I would have used before
No, you can’t simultaneously satisfy the competing demands of many different customers. But there are still some powerful techniques that we can use. If we use the same definition of a bug that we did in the last post. (i.e. something that is a threat to stakeholder value), we can actually get pretty close to a bug free state. If we understand enough about what our customers value and aggressively fix things that get in the way of this, we can make very valuable products. We might not ever get to a fully bug free state, but we can get pretty close.
I guess in way what I am saying is that a 0 bug backlog way of working is a possibility. If you are willing to put in the work. They key here is understanding that a bug involves a relationship. Bugs don’t exist without people. They don’t exist if they impact something no-one cares about or uses. If we think back to that limited scope bug free project in the last post, there could be any number of things that are in there that they programmer did not intend the software to do. But if those things do not affect the stakeholder, are they really bugs? A bug isn’t a bug, until it impacts someone that matters.
Working from this, leads to an important point about being able to achieve quality at scale. We won’t get to bug free, but we can get to high quality if we focus aggressively on fixing actual bugs. A lot of times we end up working on things that we think are bugs, or things that might be bugs, but we don’t consider the relationship aspect of a bug. Who is this bug bugging? Why does it matter?
As a testing profession we have tended to focus on what I’m going to call ‘potential bugs.’ These are issues like logic errors or crashes or really anything that gets filed in a bug report, but they are found before they impact any of the stakeholders. I’m not against finding and fixing potential bugs, but we need to remember that there is importance to finding and fixing actual bugs. Bugs that are currently bugging someone, not just bugs that might bug someone if we release them. We need to find and fix things that are actually, right now, threatening the value of our product. We need to get rid of the things that are currently bugging someone. As the proverb goes, a bird in the hand is worth two in the bush.
This also means that the goal of a nearly bug free product will lead us to prioritize the fixing of these actual bugs over creating new features. Businesses don’t always like this, especially in a highly competitive landscape where time to market for features is important, but if your goal is have a high quality product, you need to prioritize in this way.
This gets expensive. It takes time to find and fix bugs. Time that could be spent on making new features. It also add a lot of interruptions. If we prioritize fixing actual bugs over feature delivery we are going to sometime end up in a responsive mode. Remember the comments about how ‘bug free’ is a time limited snapshot? Our system and users and environments change over time and so even though we might be bug free today, that does not mean we will be so tomorrow. Again this approach is expensive. It has real costs associated with it.
And that is what I want to talk about in the next post in this series. The interplay between the cost and value of creating ‘nearly bug free’ software. Is this cost worth it?
What do you think? I’ll share my thoughts in the next post. Stay tuned!
offbeattesting 
IMO it’s simplistic to write on bugs: “They don’t exist if they impact something no-one cares about or uses. ” What if I’m doing security or pen testing on your team’s product, and I find a security weakness that (as far as we know) nobody else knows about? I would certainly bug it, but by your definition above the bug does not exist.
Another example is localization. Suppose I discover that some displayed strings (labels) are hard-coded, because nobody is planning for localization. I would certainly bug it, although (like the security bug above) it might never get fixed, yet if the product ever does go to different locales, the issue must be fixed.
Nice writing, Dave!
LikeLike
Hey Matt,
I agree that it is a simplistic way to talk about bugs. I’m trying to think through the idea of how there can be a way in which people see software as ‘bug free’ even if it isn’t.
I don’t want to play around too much with semantics as I was using particular definitions to explore the idea of what we ought to fix and not so much what label we ought to use, but in a way filing a security bug ‘creates’ the bug. Let me try to explain what I mean 🙂
The flaw was there in the code, but no one new about it, so it wasn’t really a bug. You find the flaw and create a bug report. Now people in the company know about it and knowing what could happen if the wrong person found this flaw some of these people are very uncomfortable with that flaw being there. This is now a bug. It impacts something people care about (being able to sleep at night without worrying that a hacker will steal customer data). In a sense the act of writing (or talking about) the flaw has created a bug.
I know I am pushing the boundaries of semantics here, but I think it can be a useful thought experiment to think about how a bug isn’t so much a flaw in the code as about how code impacts people that matter. I hope that all makes a bit more sense?
–Dave
LikeLike