Release More Defects

Release More Defects

What do you think is the biggest thing that holds back the quality of your product? Is it poor code quality?  insufficient testing? Not enough time for testing? Missing designs? Manger, tester or developer apathy? If you had to pick one thing, what would it be?

I don’t know if this is the biggest thing, but one threat to product quality is an inability to release defects.

Stay with me ok? I’ll explain what I mean!

As a tester I’m about making this world a better place one dead bug at time, and in order to kill those bugs you need a few things. You need to find the bug, you need to know that it is making the world a worse place, and you need to get rid of the bug somehow.

Bugs can be very sneaky. Sometimes it’s easy to find them, but let’s be honest, the sheer number of testers employed in the software industry is a good indicator of how hard it can be to find bugs. They like to hide in the most unexpected places and appear at the most unexpected of times.

And there are a few other things you should know about bugs. For example, not all of them make this world a worse place. You can have a bug (something like a logic error) in some code that doesn’t get used and the world couldn’t care less.  You can also have a bug that doesn’t look like a bug. It might be working as designed, but the designer and the user have different perspectives as evidenced by the number of swear words coming out of the user’s mouth. This boils down to the fact that bugs involve a complex relationship between software and humans. It is very hard to know what kinds of bugs make this world a worse place. In fact, just to make things more complicated consider the fact that sometimes a bug for one person is a feature for another!

Customer

So these things leave testers with a challenge. If we fix (or advocate for fixing) the wrong bugs we aren’t achieving our mission. If we miss finding a bug at all, well pretty clearly we aren’t going to get rid of it.

Enter Stage Left: The Customer.  This is the person who’s problem our software system is solving.  This is person who’s life we want to make better. It is through helping and serving this person that we are making the world a better place. This is the person who is the ultimate arbitrator of what in your system is a bug. A bug your customer doesn’t run into isn’t really a bug at all.  A bug your customer doesn’t care about doesn’t make the world a worse place. The customer is the one who should decide what bugs you spend time fixing.

So how do you improve the quality of your product? You fix the problems that your customers care about. Sometimes it is pretty easy to hypothesize that your customer will care about something. A security vulnerability for example. Other times, it is much harder to know. How are they even going to use the product? Is this issue I found going to be relevant? Part of the calling of a good tester to is be able to come up with reasonable answers to those questions. This means we need to put ourselves in the shoes of our customers.  We need to figure out how to think like them and see the product through their eyes.

Tools

If this is indeed one of the primary challenges of the testing profession (and I think it is), what are some tools we have at our disposal to help us with this? Is it all just guess work, or can we approach it in a bit more of a scientific way?  The truth is there are some really good ways to learn more about what your customers care about.  There is a lot of new technology that can help with this, but don’t forget about some of the tried and true methods. There are ways to learn about out customers that we have had for many years. Customer service representatives for example. Or your competitors products.  Why are people buying them? What do they have that you don’t? Then there are customer surveys and usability studies and external bug logs and customer posts on forums, blogs and social media….and the list goes on.

I already mentioned new technology and so let’s not forget about this. We have tools like telemetry and data analysis that can help us answer questions about customer quality. We can log and analyze errors, and automatically alert based on them. We can use tools build right into our cloud hosting systems that give us real time information on what our customers are doing. There is the potential for reams of data that can help us in this quest to figure out what things are causing issues for our customers.

There are many, many places that we can mine for information about the things that our customers really care about, but how does this all relate back to releasing more bugs? Well, remember those tough questions we have to try and answer as testers? The questions about if a bug is really a bug and if it worth fixing?  Well, we know the customer is the best person to ask those questions to right? and letting them see or experience the bug is the best possible way to figure out if it matters to them.  If we think of it this way a released bug isn’t a horrible blight on your ability as a tester, it is a way of asking the questions you need to ask so that you can figure out things you just don’t or can’t know.

Caveats!

I’ve just proposed a controversial thesis, and maybe you buy it or maybe you don’t, but let’s talk together for a few minutes about how to make this strategy work. I hope I have already made it clear that this strategy isn’t for every bug.  The purpose of it is to help you answer the question on those bugs where there is controversy about the ROI to find or fix it.

There is another important caveat though.  If these are a way of asking your customers a question, you need to be able to respond to them with an answer.  If you ask your customer, is this a bug? and the answer clearly comes back as yes! then you need to respond to that and you need to respond quickly.  This is where the technical side of things comes into play.  You know those boring things called release pipelines? Yeah, they are a really important part of this whole process. You need a release process and build pipeline that allow you to respond effectively to the answers your customer are giving you.

In fact, I think the speed with which you can turn around a change, (I would define this as the amount of time your customer is exposed to the issue) dictates the kinds of bugs you can be ok with releasing. If you can turn around a change in minutes or seconds, you can be ok with more serious bugs getting out there. You can also be ok with it if you can limit the number of people affected by it.

We shouldn’t think of testing as a one size fits all process. Your build pipeline actually influences the kind of testing you need to do. The longer the turnaround time the more effort you need to put in up front in finding issues. If you can quickly detect and remedy issues that you customers find, you can actually take a lot less time up front on testing for certain types of issues.

But why do this?  I think it comes down to efficiency.  We want to more effectively and efficiently add value to our customers and taking an approach like this allows us to do that. We will spend less time finding bugs that our customers don’t care about and we will also be able to more quickly find and know about the ones that do matter. This frees us up to spend more time in adding value in other ways.

Now one last caveat and then I’ll close out this article. This doesn’t come for free. Getting your build pipeline to allow for this kind of fast turnaround can be a long and difficult process. Having the monitoring and alerting in place that allows you to actually know what things are affecting your customers is a challenging and time consuming thing (especially at first).  I don’t want to pretend that what I’m talking about here is some fairy land of magic and rainbows where you don’t have real trade-offs. I am arguing that this effort is worth it in the long run (for many companies).

We can often pursue the latest and greatest thing as if it is some kind of magic pill that will solve all the world’s problems (or at least the ones in our software department). The reality is the things I am talking about in this article can (and do!) work, but they need to be done well. They won’t work if we are trying tack them on top of old ways of working. They require mindset shifts and changes in our processes. This is why I talk about it in terms of being able to release more defects. Using these tools effectively will require counter-intuitive thinking. Embrace that and move forward. Pursue the ability to release more defects!

 

Photo by Ankush Minda on Unsplash

Advertisements

30 Days of API Testing – Browser Dev Tools

30 Days of API Testing – Browser Dev Tools

F12.

It is one of the most used keys on my keyboard. There is just so much power in the developer tools of modern browsers that we should not be ignoring. This is especially true when doing API testing. You can often find out a lot about the API calls your application is making directly from the network tab in dev tools.  This tool gives you insight into all the calls that being made across the network when you load a page and API calls are among them. This means you can often find those calls and see what information was sent and what response was received back. It can give you a lot of insight into how an API works within the full system.

Let’s look at some screenshots and see how this might work.

Here I have loaded the the github page for one of my repositories after having hit F12.  You can see a lot of different network calls that have been made, but just looking at the first one here we can see that it is a GET call. We can see the URL used for this and if we look at the Preview, and Response tabs we can see what this call returns

GetCall

Or take a look at the call we get when we try to change the name of a repository.  We can see in this one a POST request that includes query parameters and form data. All of this information is the kind of thing that is really helpful if you are trying to figure out how an API works and how you might go about doing further testing on it.

postcall

Being able to see what API calls are being sent from a page is very helpful, but there are other ways that the developer console can help you in API testing as well. For example, another powerful thing you can do within the dev console is network throttling. There is a throttling option that lets you simulate being on a slow connection or not having any internet. This can be a great testing tool for seeing how your APIs work in those kind of conditions. Other things you can do on the dev console include seeing console errors, performance monitoring and even blocking particular requests.

All of these tools and many other are available in the developer console and if you haven’t played around with it before you really should! It is a helpful tool for any part of web testing but it is especially powerful when you are looking at API testing.

 

Photo by Kimberleigh Aleksandra on Unsplash

30 Days of API Testing – Performance

30 Days of API Testing – Performance

I think performance testing is a very overloaded term. It can mean a lot of different things ranging from how fast responses come back, to how well things scale with increasing usage, to many other stress or speed related tests.

The way I approach performance testing in an API is similar to the way I approach all my testing, which is to say holistically. There are many great tools that can help with performance testing an API, but you really don’t need to go there right away.

Just the other day, I was testing an API that allowed you set a display of items as selected or not. The API calls were not the fastest, but they really weren’t that slow. That is they weren’t that slow until you looked at how long it took for the user to be able to interact with the list. You see, each item on the list made it’s own API call, and so if you had say, 100 items on the list, you would have to wait for 100 API calls to resolve before you could interact with the last item. This caused a performance issue. Not so much because the API calls were too slow, but because the way the API was being used by the client did not give the necessary responsiveness.

The point being that you can’t just find performance issues, by checking how fast API responses come back or by looking at how many calls per second you can send to the API. Those may be important things to check, but there is a holistic approach that needs to come into play with performance testing as well.  How is the API used and can it support a valuable end user experience? To me this is the crucial question we need to start with and then we can use tools to help us along the way. We can create automated performance tests and use stress and load testing tools to help us keep things where they need to be in terms of the ongoing performance of different aspects of the system.  We can also use tools like these to help us figure out and understand where issues are coming from, but we need to always keep in mind the end goal. The user doesn’t care about how many milliseconds it takes for your API call to return.  They care about how quickly they can do what they want to do. Make sure your API performance testing is in support of that!

 

Photo by Marc-Olivier Jodoin on Unsplash

30 Days of API Testing – Getting Started

30 Days of API Testing – Getting Started

So you want to get started with API testing?

First of all, good for you! It is pretty hard to avoid APIs if you work in software and testing them is an important part of creating high quality software. If you haven’t done stuff like this before it can be pretty scary, but I think you will find that it can also be rewarding. There is something exciting about learning how the pieces of an application come together and being able test and explore and API can help you understand this better.

But how do you get started?

Well, try using an API. There are many, many public APIs that you can look at. Here for example, is a big list of public APIs.  Find something that is pretty simple and that you find interesting. Maybe the cat facts API. Download Postman or another easy to use API tool and put in a URL and press send.

Making your first API call is pretty easy and from there, just try stuff. Try different endpoints. See what happens when you change things. Google for answers when you get stuck. I think you will pretty quickly find that you can do a few different things and are starting to get an understanding of the way APIs work.

The next step might be to find out what APIs your team has that you can test. Do the same thing. Find an endpoint you can get started with and try stuff. Look at what it can do and change things to see what happens. Search for internal documentation. Talk to developers and architects. See what you can figure out about it and how it works.

At first it will be a frustrating experience. You will feel like you have no idea what is going on and you will get stuck frequently.  Just leave it for a little while and then come back again, perhaps the next day. Keep trying things. Look at it from new angles. Grow your understand of the pieces of the product. Ask lots of questions. Over time you will find that things become more and more clear and then you’ll be doing it.  You will be an API tester!

 

Photo by Tim Huyghe on Unsplash

30 Days of API Testing – Security

30 Days of API Testing – Security

Security testing is a huge topic in it’s own right, but the intersection between API and security is an important one to consider. It can be overwhelming since there is a lot of specialist knowledge required. The truth is the majority of us do not have the ability to do a deep dive on security testing when we are testing APIs, but I think there are a few simple things that we can consider as testers.

One of the simplest things is to try out API paths with different credentials. For example, if you have an API path that allows you to edit something that is meant for admins, make sure you can’t access it as a ‘regular’ user. Another similar check to this is to try a different user’s credentials on endpoints that give back private information.

Another quick and dirty test that anyone should be able to do in API testing is try different API paths that give back the same information. Often there are multiple ways to access something in an API. For example you might have a direct endpoint that lets you get information for a particular resource. You might also be able to get to that information by using search functionality on another part of the API. Checking that the search functionality does not give you more information than it should based on your permission is something anyone can do without needing a lot of security training.

There are also system level issues to consider. Does the way your API is setup encourage or make it easy for people to accidentally expose data? Considering API design and how it interacts with human behaviour is another important place where you can provide security insights without needing  security certifications.

There will be the need for security experts who are trained in the intricacies of technical security testing to take a look at your APIs, especially if they are public, but don’t let that stop you from digging in and doing your own security testing!

 

Photo by Bernard Hermant on Unsplash

30 Days of API Testing – Status Codes

30 Days of API Testing – Status Codes

I’m not even sure anymore where I found this, but I have found this image to be a helpful resource when I need to be reminded of what the various API status codes mean

API Request Response codes

On thing I would note about status codes, is that they aren’t always correct. The status codes need to be set by the API and sometimes APIs aren’t setup to give back the correct status code. Most APIs only use a few of these status codes and so you might get a generic error code instead of the specific one related to the issue you are actually hitting. It also happens on occasion that there are bugs in the API that return the incorrect status code. When testing you want to pay attention to the status codes, but don’t just blindly trust them

One other thing to keep in mind is that there is some ambiguity in these codes as well. Sometimes an API programmer has to make a choice as to which status code to return and different APIs will sometimes make slightly different choices in what they return. Once again you will want to pay close attention to how the API you are testing uses them. Does it make sense? Is it helpful to the users?

Status codes are an important part of API testing and are often one of the things we check in API automation, so hopefully this list is helpful for you!

 

Photo by rawpixel on Unsplash

30 Days of API Testing – Frustrations!

30 Days of API Testing – Frustrations!

I really enjoy testing APIs, but let’s face reality here: every job has its frustrations. One of the things that I have found frustrating in API testing is when an API is inadequately documented. There is something fun about digging around and figuring out what paths there are in an API and how it works, but some APIs make this too hard.

When I’ve worked with hypermedia APIs, I’ve found they generally include enough information in the calls themselves, that with a little domain knowledge you can figure them out fairly well. But I’ve also tested APIs that don’t have any self documentation and only partial external documentation.  I’ve had to go digging through the code base and I’ve had to try and figure out who I could ask about stuff and kept banging my head against the API until I eventually figured out how it worked. Doing this kind of thing can get very frustrating.

As with any code, APIs need to be testable and when they are written without documentation and using magic numbers and other anti-patterns, test-ability drops. We don’t need every detail documented, but there should be enough there that someone new to the API can figure it out fairly quickly.

 

Photo by Steve Johnson on Unsplash