In a recent post to his email list, Jonathan Stark talked about giving guarantees for your work. He gave the example of offering a “bug free” guarantee for software development work. All the testing hairs on my arm stood up and I wanted to protest that there is no such thing as bug free software!
But then I let another testing instinct kick in – curiosity. I started thinking about how he can do this. How can he offer a guarantee that he will deliver bug free software? Is there such a thing as bug free software?
I came to a conclusion that yes there is. There really can be bug free software.
The context for that statement has to be just right though. If you are delivering a piece of software that has only one (or a very limited number of) stakeholder that gets to decide if something is bug free, I think you can indeed make the software bug free. There might still be logic errors in the code somewhere. There might even be things that you as a tester could find that are ‘wrong.’ But if the client agrees that the software does what they need it to in a way that is reasonable for them and if they agree that there are no issues that prevent them from getting the value they need out of it, you have indeed delivered bug free software.
I think a bug is something that threatens the value of the product. If you have a limited scope product and limited number of stakeholders, you should be able to make something that has no threats to it’s value. You would need to do this through careful consultation and iteration with the customer. They would have to tell you where they are not getting the value they need and you would need to work closely with them to remove all bugs (that matter to them) from the product, but it should be possible to do.
I want to try a more difficult thought experiment though. Take this into general purpose software with many thousands or millions of users with competing demands, skill levels and expectations. In this context can you create bug free software?
I’m going to take up that thought experiment in the next post. Stay tuned!
Photo by pan xiaozhen on Unsplash
offbeattesting 
Interesting! I like your definition of a bug “something that threatens the value of the product” ☺.
Sam
From: offbeattesting
Reply-To: offbeattesting
Date: Monday, January 14, 2019 at 7:40 AM
To: Sam Chandrashekar
Subject: [New post] How to Create Bug Free Software
CAUTION: This email originated from outside of D2L. Do not respond to, click links or open attachments unless you recognize the sender and know the content is safe.
offbeattesting posted: “In a recent post to his email list, Jonathan Stark talked about giving guarantees for your work. He gave the example of offering a “bug free” guarantee for software development work. All the testing hairs on my arm stood up and I wanted to protest that t”
LikeLike
Hi Sam,
Yeah I think a lot of my argument hinges on that definition of a bug 🙂
Dave
LikeLike
Interesting idea! What happens if/when a problem crops up, that threatens the value of the product, after the relevant stakeholders have agreed the software is “bug-free”?
To my mind, the risk of stating software is bug-free is that it assumes that we can know with at-or-near 100% certainty that there are no behaviours in the software, intentional or not, that have a negative impact on the value to a relevant stakeholder. Maybe in software that isn’t particularly complex, running in environments (both hardware and OS) that are relatively simple as well?
LikeLike
Ahh! Very important question! Just because it is bug free now, doesn’t mean it will stay that way. Environments change. Business needs change. Maybe the software itself changes. I think this is essentially the regression testing problem summarized and I think it’s a really part of this discussion
I have a couple of more posts I want to do on ‘bug free’ software and in one of them I want to talk about this problem, so guess stay tuned for more discussion on this 🙂
Also, as an aside, I love how you phrased it: “The risk of stating software is bug free…”
When providing a guarantee of any sort (including a bug free guarantee) you are essentially taking risk away from the client and on to yourself. This concept of risk (and it’s relation to value) is bouncing around in my head right now too and will probably come out in a future ‘bug free’ post.
LikeLiked by 1 person
I’d argue that just because you THINK it’s bug free, doesn’t mean it actually IS. Bugs can lie undiscovered for a long time, even without any actual changes. Admittedly, they’re often edge cases, but that doesn’t make them any less of a bug, or reduce the potential impact. For example, look at the Heartbleed bug in OpenSSH – it was in the wild for about 2 years before anyone discovered it.
LikeLiked by 1 person
Totally agree. But if we define a bug as something that impacts customer value, is it really a bug if it has not been found (by anyone)? It is a potential bug for sure, and once it has been found it is a bug, but in the meantime if everyone is happy and it is not really causing damage is it really a bug?
I’m playing around a bit with semantics and definitions here, but I guess the point I’m trying to get it is that the only bugs that really matter are those that ‘bug’ somebody who matters.
Now, how important it is to find potential bugs (like heartbleed), is another interesting question. I’m loving the discussion on this post in various forums as it is giving me a lot to think about. Thanks for your interactions!
LikeLiked by 1 person
Good point! If a bug hasn’t been discovered, is it really a bug? Maybe, maybe not. Thanks for the brain exercise 😀
LikeLiked by 1 person