How Should Testers use AI?

How Should Testers use AI?

As a tester I am excited about the possibilities of AI and machine learning.1
I hope that there will be many ways to leverage this technology to level up on our testing powers. As with any tool though, we need to recognize that powerful and magical are two different things. As hardware and machine learning capabilities get more powerful we need to leverage them, but just using them won’t magically produce good results.

If you were smashing up some concrete a jackhammer would be much better than a sledgehammer right?  What if you started using the jackhammer like it was a sledgehammer? Swinging it over your head and hitting the concrete with it. You have a better tool. Are you going to more effective? When we approach new tools we need to use them effectively.  We can’t just use them as if they were the same as the old tools we had.

I recently read an article about how testers could use AI to improve the stability of UI tests. One idea presented was that UI tests could be more robust if we used AI to figure out what a button was.  By using machine learning and image recognition we can figure out if a button is the submit button or the back button even if the way it looks has changed.


Ok, that was a bit rude, but the reality is that if all AI is going to do is make UI automation more robust, I have better things to do than learn how to use it in my testing. There are a lot of easier ways to make UI automation more robust (not least of which is not doing so much of it).  We don’t need AI to help us out here as much as we just need a bit more common sense.  Throwing more technology at a problem that is, at it’s heart, a problem of misunderstanding how to effectively use UI automation won’t help. It will just allow people to add more UI automation without thinking about some other effective ways to test their apps.  To return to the jackhammer metaphor, if someone is smashing up the wrong part of the sidewalk, giving them a better tool won’t help with what matters. They will just smash the wrong thing more effectively.

If you want to stand out from the crowd you’ll need to dig a little deeper. You’ll need to find some uses for AI that are a little more interesting. I’ve just started poking my nose into some of the AI libraries and trying them out, so these are just some brainstorm style ideas I have at this point.  I want to think about this ahead of time and see if it would be something that is worth further investigation.  I’m always on the lookout for new tools to add to my testing toolbox – could machine learning be one of them?

Idea for using AI in my testing

Data Analytics.

The developers need to implement that for me, you might object.  This is true in some areas, but think about it a little longer. What data do most testers have access to? What happens when you run  your test automation? Does it generate any log files? Could it?  Can you cross correlate data in those files with some simple system information? We generate a lot of data during test sessions. Can we get some information out of that data? Do you need the developers to implement advanced telemetry in your app to do this? I think there are a lot of ways machine learning could be used to generate insights into the behavior of your application that do not involve advanced telemetry.

Writing Bug reports

We all know that bugs like company. Where there is one bug there are often more.  What about using machine learning to parse through the bug reports and see if there are patterns to be discerned? Where do the bugs cluster? What kinds of steps/actions frequently show up in bug reports? What phrases are common to bug reports? We have bots that can write a realistic news articles, why shouldn’t we use them to write plausible bug reports?  Will those reports show actual defects? Probably not, but they could generate some great test ideas and ares of focus. One of the biggest challenges in testing is reducing the scope of infinite possibilities in a way that is valuable.  Could we use AI to help us with this?

Writing new scripts

Our current ideas around regression testing involve writing scripts that do the same darn thing every time they are run.  There are reasons we do this, but there are also a lot of problems with this approach. What if we gave a machine learning algorithm the pass/fail data for our tests and started to figure out which ones are likely to find bugs?  What if we took it further and let the AI suggest some new tests?  I think there are a lot of possibilities for automating the finding of regressions in a much more efficient way.


In looking at these things, I think there is a lot of potential for machine learning to help with testing in the future.  However, it seems like most of these things are still too advanced for the individual tester to do.  We will need better machine learning tools, before we will see payoff on investments like this. For now, I intend to learn a bit more about machine learning, but I don’t think it is going to transform too much in my testing in the short term.  I guess we will see again in a year or two where the tools are at.

I really do hope that we see development of creative machine learning tools for testing that break into entirely new and innovative areas. So far much of what I see people talking about using ML for in testing is to do the same things we did in the past, but better – because AI. I’m sure there are some gains to be had in those areas, but I really think we will see AI become a powerful testing tool when we start to use it to do new things that we can’t do at all with our current set of tools.

What do you think? Where is AI going?  How will testers use it in the future?


1. Note that I am being sloppy with my terms in this article and using Machine Learning and AI as interchangeable.


Do Tools Matter?

Do Tools Matter?

Life is different.

I’ve moved from the world of desktop software into the world of web software.  I was working on a team with thousands of integration tests – now I’m on a team that has very few. I was one of the experts on the team – now I’m asking all the newbie questions.  Things have changed a lot. In a new environment there are many new things to learn and it raises a question: is being a good tester about the tools you use, or not?


Being a good tester is not about the tools

I’ve noticed that I was able to very quickly get up to speed on the product and how it worked.  I was able to file several bugs in my first week.  I was able to feel comfortable talking about the different areas of the product. I was able to understand much of what the team was talking about very quickly.

How so?

I have been a tester for almost 10 years. My experience might be in a different world, but there are still many things that are common to software testing regardless of the type of application you are testing.  Software for example. And programmers.  You see, both desktop and web developers are humans. Desktop and web developers also both use this thing called code to help them accomplish things. Programmers and programming languages share things in common with each other that go beyond the boundaries of the kind of application you are building. The ways in which we create and interact with data structures and control flows are constrained by the way coding languages work.  The way that we create and maintain code are constrained by the ways that humans think and work.

I was actually surprised at how much of my skill and experience as a tester was easily transferable to an entirely new domain.  There are many things about being a good tester that go far beyond the tool set being used and the type of application being used.  There is a real sense in which being a good tester is not about the tools you use.

Being a good tester is about the tools

But that isn’t the full story. In the last couple of weeks I have not only been learning a new application, I have also been learning many new tools.  Some of them are internal tools that help make things easier for me.  Some of them are very well known tools that I didn’t need to use before.  Developer tools in Chrome for example. Or JMeter and Webdriver and Postman and the list goes on. As I get a better understanding of the application I get the urge to dig deeper and learn more and for that I turn to tools. Could I quickly get up to speed and contribute without learning new tools? Yes! Am I content with staying there?  No! I want to be a testing Batman. A skilled testers can be effective independent of the tools used, but a skilled tester will use tools to get even better.

I’m re-building my toolbelt now.  I’m like an electrician who has moved from doing residential work to commercial buildings. The underlying skill of knowing and understanding how electricity and wiring works stays the same, but there are some things that need to be done differently.  If the electrician refuses to learn and use new tools, she will be less effective than she could be, no matter how skilled of an electrician she is.

I still use the same underlying skills, but there are some new tools that I need to help me get more effective at my job. What tools do you use? Do you have any recommendations for a tester new to web apps? What tools should I add to my toolbelt?



We shall not cease from exploration
And the end of all our exploring
Will be to arrive where we started
And know the place for the first time.

I’ve been learning a new product and so I have been in exploration mode. I’ve spent the last week poking around and learning many new things about the product. Making maps of it to get a better understand. Trying things out in different ways. Setting up little experiments. Asking questions. Even on occasion reading some documentation.

The team I am a part of is responsible for one particular area of the product, and so I have kept circling back to that area. Every time I come back I find that during my wanderings in other parts of the product I have learned something new. I’ve gained insights that give some new light on the area of interest.  At first glance it seems like this part of the product is pretty simple, but the more I learn and understand how the pieces fit together, the more I see that simplicity is hard to do. It takes a lot of work to make something look and feel simple. There is a lot of hidden complexity. As I understand that complexity better, I get better at understanding the risks.  I get better at seeing where the problems might be and chasing down odd behaviors.

It can be easy to think that you know something, but it is only as you continue to return to it time after time that you really begin to understand it.  Sometimes testers can feel like they are spinning in circles, returning to the same area over and over, but every time you come back to something you are different.  You’ve learned new things and had new experiences. There are biases you have (and you would do well to learn and understand them), but it is still a new you. Use the things you’ve learned. Make the old familiar areas of the product come alive by bringing in your new perspective. Perhaps T.S. Eliot was right. It is only when we return to something that we will be able to know  it for the first time.


New Job!

New Job!

This isn’t the usual kind of post on this blog, but I want to take a couple of minutes to talk about changes in my life. I started my testing career at Ansys right out of university and it has been a good 9.5 years.  I have learned a lot a grown in many ways, but the time has finally come to try something new.  I started today at D2L. It is a totally different area and will present many new challenges for me as I work with a different technology stack and in a very different area. I’m looking forward to learning and growing in many ways and I’m sure I will continue to share thoughts and experiences as I learn new things and try to work though new learning challenges.



Liar Liar

Liar Liar

Have you ever been lied to?

I don’t mean those little white lies that smooth social interactions – “You don’t mind if I do this?”  “No, not at all” – I’m talking about a full blown lie.  You ask your son to take out the garbage and he tells you did it already.  The next morning, after the garbage truck has left, you find out he never did it.  Or you call customer service and they promise to take care of it as soon as you hang up, but you have to call back two days later because nothing has happened.  What happens when you are lied to? You lose trust in that person or organization.

Much of the way we work as a society is build on trust. There are places in the world where stores have armed men with guns by the door to deter theft.  Why? lack of trust.  In the West we have many blessings and one of them is trust. I can call a dump truck and ask them to bring by a load of sand and they will do it and send me an invoice.  Why?  They trust that I will pay them. Without that trust the transaction would become much more painful. I would somehow need to get them money ahead of time, but would I really want to if I couldn’t trust they would deliver once they had the money? Transactions become much more difficult in a society with low trust.

Trust in Software Development

In the software development industry we primarily view trust as a financial proposition. “If my customer doesn’t trust my credit processing system they won’t click on buy and I won’t make money.”  However, trust is a bigger problem than that. It is a societal problem. In a world with low trust we have high cost. Everything get expensive when there isn’t trust in place.

How do we contribute to  lack of trust in software development? Well, how about when our customer’s data gets hacked?  Does that build trust?  What about when users come to our site and it won’t load?  Does that build trust?  What about when your ‘click here’ icon doesn’t do anything? Does that build trust?

Let me drive the point home in a way that might hurt.

If you are ok with software that breaks trust, you are not working for the social good.


But I think it’s true. We hear a lot of talk about social entrepreneurship and the importance of social initiatives in corporations.  I want to challenge software development teams – and in particular those who think about testing problems – to think about the social responsibilities they hold. The ethics of testing are clear in some domains like medical applications, but what about if you are writing an iPhone app?  Do you have ethical considerations then?   My argument is that if you are ok with releasing a buggy app that breaks the implicit and explicit promises you are giving your customers you are contributing to a societal problem. Teaching people to expect to be lied to is not a good thing. Don’t have any part in it.  Release good quality software.


It’s ok to make mistakes. We will sometime break people’s trust despite our best intentions.  Don’t beat yourself up over missing things. The point of this post is that it should be despite our best intentions.  We ought be shooting for high quality, trust building applications. Saying that it’s ok to ship with bugs in our product, shouldn’t just be a financial decision.  We need to be thinking about other things like the ethics of it as well.

Invest in Quality Now!

Invest in Quality Now!

We’ve all been there are some point. It’s crunch time. The deadline is looming. Everyone on the team is frantically working to get the product shipped.  Crunch time reveals to us what we really see as important. During crunch time certain things get dropped.  Think about the things that get dropped on your team during crunch time and I’ll bet that they  all relate to one thing:


We stop checking things in as much detail as we used to. We take some shortcuts in our coding.  We do things that we know are not good for the health of our code base.  We quickly patch up defects without addressing the underlying issues. Under pressure, the first thing offered on the alter of the Almighty Deadline, is quality.  Our real priorities are revealed. We might talk about how we take quality seriously, but our actions show differently.  Our actions show that quality is seen as the low guy on the totem pole.  Perhaps this is why in some places testers feel like they are lower status members of the team. Testers are concerned with quality.  How do you think they are viewed if we see quality as the least important part of the software development process?

But what if it isn’t?  What if quality is actually the most important part?  What if many of the problems and challenges we face in the software industry are directly related to quality? If that is the case, the way we are approaching our problems is totally upside-down. Instead of the first thing to go, quality should be the last. Instead of a dispensable part of our process it should be the keystone of everything that we do.

Here is the thing: If we want to be able to move more quickly we need to invest in quality first. Quality can’t be an afterthought.  It can’t be something we tack on to the end of the process.  It needs to be something that is fundamental to the very core of how we go about making software.  Until we understand this and until we start to act like we believe it, we are just going to keep spinning around on the same hamster wheel.

Quality Challenges

One of the challenges we face with quality is that it is often a hidden property of the system. Imagine you go to your local generic chain furniture store and pick up a pretty cheap table.  Or imagine you buy a custom made table for 5 or 10 times as much.  The two tables might look very similar (initially), and yet which one do you think has higher quality? Quality is often not immediately visible.  This is what make it so tempting to cheat on it. At crunch time we can hide sloppy work and shoddy testing, because it won’t be immediately visible. But the fact is, poor quality will be experienced by someone.  It won’t stay hidden forever.

As testers we are like the mechanic friend you take along when you are looking at a new car.  You see a car that looks nice and has some nice features – it’s even red! – but then your friend crawls under the car and opens the hood and shines his flashlight around to see in the cracks and crevices.  He comes back with a list of problems (or potential problems) that you’ll want to consider. He’s slowing down the process.  Now you have to go look at another car or negotiate a lower price or get the owner to fix those things.  You just want that car now, so you shouldn’t bring your friend along next time right? Right? Or not?

Another challenge when it comes to quality is that we don’t pay the price. If we release a low quality product the company might get hurt and it might loose some money, but oh well, we can always find a job somewhere else right? And besides somebody has to fix up all those problems. The reality is most of the pain of the bad quality products we work on get passed on to someone else.

Invest in Quality now!

But we are ethical – and I really believe we are – so most of us want to do the right thing.  We want to make something we are proud of.  We want to help people. We want to do right by our customers and our employers. So how do we do this during crunch time? We need to get these features out for the good of the company and the customers. They are important.  How do we do that without sacrificing quality?

We do it by investing in quality now.

You know you are going to have crunch times. We live in a world where we can’t see perfectly into the future and so there are going to be times when there is a crunch to get things done, no matter how well we have planned. We also know that during crunch time one of the easiest things to sacrifice is quality. So prepare for this.  Invest in your quality well before crunch time comes.

When we are driving for that final push, do you think it will help to have a robust and useful set of automated tests in place? Well, you certainly won’t get that up and running during the crunch time – invest in quality now. 

When you are in the middle of a crunch do you want to discover fundamental flaws in your architecture or design? You are only going to find problems like this early if you look for them early – invest in quality now. 

When the pressure is on and someone is out sick, do you want to have to feel your way through some tangled, unreadable code in order to fix a bug? Having clean and readable code doesn’t just happen by itself – invest in quality now.  

When you are trying to merge that feature in, do you want to have no time for interactive testing because you are waiting for it to make it through the build pipeline? Good build pipelines take time to build and maintain – invest in quality now.  

When there is very little time for testing do you want testers to struggle to trace down issues they see? At crunch time it is too late to think about testibility – invest in quality now.

Crunch times will come, and during crunch time we will have to change some of things that we do, but if you want to come out the other side holding your head high and able to be confident that you’ve done a good job, you need to think about quality.  Don’t put it off or leave it as an afterthought.  You’ll be much more successful as a software development team if you start with quality.

Invest in quality now.