Queuing Theory and Testing

Just for fun, I was recently reading about queuing theory (that’s a normal thing people do for fun right?). One of the things that you need to think about in queuing theory is scheduling policy. There are a number of different way that you can pick which object in the queue goes next. One scheduling policy that intuitively makes sense to me in the first-in, first-out policy. Basically, you would pick the thing that has been in the queue the longest as the one that goes next.

But you can also choose to use a last-in, first-out policy. That doesn’t seem very fair to me. Think about waiting in a queue at the grocery store. You’ve been in line for a while, and someone else comes along and stands behind you in line, but when the next chashier is ready, they help the person who joined the line after you. The last one to join the line is the first one out. Why would you ever choose this scheduling policy?

Well, since I was just reading this for fun, I didn’t bother myself about it too much, but then a couple of days later it hit me that we often use a variation of this policy in the way we manage our backlogs. Things that have been recently added to the backlog are more likely to be worked on than things that have been there for a long time.

Why is that?

One reason for it might be availability bias. The newest items added to the backlog are fresh in our minds and so they seem to be more important. This is a bias, but since software is built by humans, for other humans, it might not be so bad to have human biases in the decisions. Things that are fresh in our minds are probably fresh because they are related to new features that we were working on. These things might be ok to work on first as they might be fresh in our clients minds too. Of course, we should still at least give a little bit of thought to that before assuming it is true in every particular situation, but it might not be an all bad thing.

Another reason for using this kind of queue scheduling is that it might be cheaper to work on stuff that is fresh rather than having to get back into an old context. If we have just created the story or filled out the details on a defect, we should have a pretty good understanding of what we need to do for it. This can mean that we can more quickly complete the newer work in the backlog than we can the older.

I don’t want to gloss over a glaring reality though. The last-in, first-out policy means that those poor items at the front of the queue are very unlikely to ever get a turn. Since there is always more work to be done than there are people available to do it, without some other mechanism to bubble up those older tickets they will languish forever. Most teams have mechanisms that do this (even if they are not deliberately put in place). For example, a client might escalate a known issues and hence bump it up in the queue.

But can we be more intentional about them? What do you think? What are your strategies for dealing with those old defects or stories that have been in backlog for years?

I’m an Author?

I love learning new things. One of the reasons I started this blog was so that I could capture some of the things I’ve learned, but also so that I could learn more effectively. I have always found that for me one of the best ways to learn is by explaining the thing I’m learning to others. Blogging has allowed me to hone and deepen my thinking on a lot of things related to testing.

In the last couple of years I have also branched out into learning and teaching with video courses. I’ve made courses on a number of different testing related topics, and in the process of doing that have been able to learn a lot.

Last year Packt Publishing approached me about writing a book on API testing with Postman. I had published a video course about this with them previously, but writing a book is a lot of work so I said I would have to decline. However, after thinking about it a bit more and discussing it with my manager, we worked out an agreement where I would be able to go down to 4 days a week at work so that I would have the time to work on the book.

Writing a book!

I have to say, writing a book really is a LOT of work. I was already fairly familiar with Postman and API testing, but in the course of writing the book I found I had to really dive deep into some areas. When you dive deep into something you start to discover that there is a lot that you don’t know yet. I had to learn new things and struggle through fundamental concepts. I wanted to make sure that I understood what was going on at a deep level so that I could clearly and simply explain it in the book.

I also spent a lot of time coming up with practical examples. Things that the readers can do on their own. Books are great. I read 3 or 4 books a month and I think books are probably the biggest bang for your buck that you can get when it comes to learning. However, as the ancient Greek philosoper Epictetus said “Books are the training weights of the mind. They are very helpful, but it would be a bad mistake to suppose that one has made progress simply by having internalized their contents.” You can’t just read a book and think that you know the subject. You need to internalize it’s contents. With technical books like this one, I think that is best done by working through the subject on your own. To that end, I have numerous challenges and worked examples that help the readers internalize what they are learning.

Writing a book was certainly a tough and challenging endeavor, but it was also a helpful part of my ongoing quest to be a life-long learner. I learned a lot, and I find myself working on some new things at work that come directly out of the things that I learned. I hope that if you read the book, you find that you are able to learn a lot too!

Free Copies?!

I’ve not only been learning new things about API testing, I’ve also been learning about how book publishing works. One of the things that is important with new books is to get some reviews. In light of that, the publisher has a few free copies available in exchange for your honest review of the book. If you are interested in that, feel free to reach out to me on twitter, or LinkedIn or feel free to send me an email at davewesterveld@gmail.com and I’ll see if I can get you a free copy.

The book is called API Testing and Development with Postman and it is coming out on May 7th (2021). If you do pick it up, let me know what you think! I hope it is helpful.

I guess in a way I have been an author for a long time. I’ve been blogging for many years, and I have published a lot of video courses, but there is something about writing a book that makes me feel like I can now actually call myself an author. Damion Synadinos, has a talk called More Than That, where he talks about the different ways that we label ourselves. Well, I feel like now I have applied a new label to myself. I’m an author.

I guess I better go update my LinkedIn profile.

Photo by Aaron Burden on Unsplash

New Post on TestProject Blog

I know I haven’t been writing on here much lately, but if you are looking to read something I wrote, check out my latest post on the TestProject blog.

It’s about getting started with their open source Java SDK. I like learning new things and this was a fun way for me to learn a bit more about Java!

As I say in that post

 When you have a powerful tool, you don’t need to avoid it. Rather, you need to learn how to wield it well.

It was fun diving in to learn how to better use that tool. Check out the full post here.

Creating Schema for an API

From the outside, software development and magic are indistinguishable. Of course, with enough time you can understand how it works and for those of us who have made it through the frustration of learning to code, it’s really not all that magical after all. It’s just the application of a few simple rules.

I kind of felt that way about schema. I’d heard the term before and I had seen errors come up in my code editor saying that I had a schema validation error. As far as could tell though this all came about by some sort of magic. Well, it turns out this kind of magic can also be learned.

In my last blog post, I wrote about what it was like getting started with creating an Open API specification. In that post, I glossed over the part about creating a schema for the API. However, that was actually the hardest part of getting started with Open API for me. I realized that I didn’t really know that much about schema and how testers can use it. If you too are in that boat, I hope this article will help you out a bit!

What is an API Schema?

Well, when it comes to API schema, we are talking about a set of rules that dictate what the API request or response is allowed to look like. Creating an API schema is about finding the patterns in what the API does and creating rules that define those patterns.

At this point you might still be a bit confused and thinking “give me something concrete.” Well who am I to argue with you on this? When the people ask for concrete examples they get them, even if the people are only made up voices in my head.

Suppose you were looking at an API that gives you data about COVID cases in your country. One of the fields in the API response is the number of cases. You know some things about this field. For example, I’m pretty sure you can’t have less than 0 COVID cases (unfortunately). I’m also pretty sure you can’t have partial cases of covid. This means that you know that this field needs to be an integer that is greater than or equal to 0. These are some rules that you can apply to that field in your API response. What an API schema does is gives you a way to encode and document these rules.

How Does Open API use Schema?

Open API uses the JSON Schema as it’s starting point. It doesn’t use all the elements of JSON Schema and it does add some of it’s own elements as well, although the latest 3.1 Open API spec has pretty much synchronized the two. You can define the schema in a couple of ways. You can just do it directly inline with the resource you are trying to define, or you can create it in a shared schema section in the specification file and refer to it from the resource. I will show you an example of the second way of doing it.

In the previous post, I showed an example of defining an endpoint that looked like this:

      description: Makes the world a better place.
          description: You got a totally ok result.
                $ref: '#/components/schemas/SchemaName'

The $ref: part points to where it should be getting the schem from. You can create that section in the specification file and then add your schema into it.


In this section you can then create the rules that you want to enforce. I have found the documentation on this page to be helpful with figuring out how to create the schema rules that I want. If you consider the example of an endpoint that should only have positive integers, you could specify the schema for it like this:

      type: integer
      minimum: 1

By referencing this schema in a response, you would be saying that htis response can only contain integers that are 1 or more. There are of course, many other rules that you might want to specify in a schema, and you can see some great examples in the documentation I referenced above.

I know this is a very quick survey through the subject, but hopefully it helps you to get a grasp on what a schema is and on how and why you would want to use one in an Open API Specification.

Photo by Ivan Aleksic on Unsplash

Getting started with the Open API Specification

Twenty, twenty-one, twenty-two. Twenty-two seconds! Twenty-two seconds to load the data for a simple table. Somehow we had repeated the same mistake we had run into before. A year or two before we were working with an API that used sperate endpoints for each cell in the table we were creating on the page. That works fine until you have a table with 100 rows and 5 or 6 columns. Now you have to send hundreds of API calls and even if each one is small on it’s own, it takes time for the calls to traverse the network. The HTTP protocol limits the number of calls you can make concurrently and so queing theory applies and the load time for the table gets really slow.

And now here we were. We were creating another table in our UI and the API that supplied it had a different endpoint for each cell in the table. How did we get here? During the design we had discussed performance and how we would need to design the API so that we wouldn’t come across this exact issue. Somehow as a team when we go caught up in the work of creating the APIs that would pull together all the data we needed, we missed this piece of the API design.

We are a well functioning, highly productive software development team. In fact, we refactored the API to work as it should have within a week, and yet we had missed this. I suspect we aren’t the only team that has faced something like this. We were building a system that would deal with a lot of complexity and present it in a simple way so that our clients wouldn’t need to deal with it. The challenge of course, is that there are so many details to keep in mind. Domain and functional requirements. Design and presentation of the UI. Security and correct user permissions. Accessibility and performance. It isn’t possible to keep this all our heads at the same time. Thing get missed.

That’s why we have testers right?

I don’t know. That doesn’t feel like the right approach to me. Sure, we can use after-the-fact testing to find those bugs, but we had already identified this bug. We knew that creating an API where each cell in a table needs to call a different endpoint is not scalable. In essence, we had to find this bug twice. Once during design and once after development. The second time we found it, the bug was expensive to fix. Could we have prevented that regression? Can we create regression tests for our designs?

This is one of the problems that API specifications attempt to solve. In reflecting on this issue, I started to wonder if we could have saved ourselves a lot of rework by encoding the things we had discussed and discovered during our design sessions. What if we wrote them down in an API definition that set out what the API should look like and do?

You can’t do much about the past, but you can learn lessons from it and use those lessons to make the future better. I’m sure our team will be writing more APIs in the future and so I want to explore some of the different API specification to see if there is something that would make sense for us to use in the future.

Open API Specification

The most common specification format is the Open API Specification (or OAS). It has in many ways become the default specification for describing how APIs work. It has very broad adoption across many companies and has a large supporting community. It is designed for RESTful APIs, which we use, and it has support for almost anything you would do with these APIs.

I’ll be honest though, I find it overwhelming to get started with. It is a lot of work to document every detail of what your API can or should do. Since we have existing APIs that do not yet have API definitions I thought I would try using the Swagger inspector. It gave me something to get me started, but it still needed a lot to be done by hand. Since at this point, my goal is more around evaluating if this makes sense for future APIs, I decided to instead look at creating a definition “from scratch”. I’d like to use a design first rather than code first approach to this.

So how hard is it to create an API defintion that follows the OpenAPI Specification?

The Tools

Well, let’s start with talking about the tools. There are a lot of Swagger tools that work with Open API Specifications. Confusingly the OAS used to be called Swagger, but once you get past that, there are still many tools out there. It’s easy to get lost in trying to figure out which tool does what, and which one you want to use. I know because I spent too much time trying to figure out which one to use. Eventually I decided that I wouldn’t use any of the custom built tools. OpenAPI defintions are just text files (either yaml or json format) and so they can be edited in any text editor.

I created a file in my trusty Visual Studio Code, installed a couple of OpenAPI formatting and linting extentions and I was finally off the races. I’m sure that if I use this more, I will eventually want tools that are built specificially for the OAS, but I’ll stick with the basics for now and see where the pain points show up as I scale. If you are new to using the OpenAPI Spec, I’d recomend taking this approach too. Not only will it force you to understand what is going on more deeply, but it will also remove a whole layer of confusion that can get in the way of just getting started.

Creating the API Defintion file

A blank file staring at you can be intimidating, but I guess the best way to get something done is to start doing it, so let’s look at what data needs to go into this file.

The First Section

So, how do you actually create an API definition with the OAS? The basics of it are straigtforward. Start with a bit of boilerplate at the top that says what version of the OpenAPI Spec you are using along with some information about the API your are creating the definition for.

openapi: 3.0.1
  title: A cool API
  description: This API does cool stuff
  version: '1.0'
  - url: https://{tenantId}.domain.api.com
        default: tenant-id-guid-123

One simple thing I ran into here was that each instance that we create (and we have at least one for each of our clients) has a different base url. This includes all the testing sites. Pretty easy to take care of though. You can create a parameter in the url. You can read more about how the base urls work here if you want.

Defining an Endpoint

With the boiler-plate out of the way, the next thing to define is an endpoint. Endpoints are defined under the paths section.

      description: Makes the world a better place.
          description: You got a totally ok result.
        - name: pathParameter
          in: path
          required: true
            type: integer
            format: int64
            minimum: 6606

I’m obviously redacting some stuff here, since these are internal APIs, but it’s interesting to note how easy it is to create a parameter in the path. All I needed to do was to include it in the path defition and then create a parameters section under the request to define what kind of parameter it was.

You will notice though, that the responses object only defines the response code. It says nothing about what kind of content this API should give back. That seems like it would be kind of important. Defining this opens up a whole new can of worms. Obviously the API does not always give back an identical response every time you call it, and so you can’t just copy a response and put in in there. Instead you need to specify some general rules that should apply to the response. For example, you could specify that it should be a list and that each item in the list should be a string. These kinds of rules are known as schema. The kinds of objects our API represents are pretty complex and so it takes a bit of work to create a schema for them.

In fact, as I dug into this I realized that this is actually the area that was the most intimidating to me. I’m pretty comfortable with the way RESTful APIs work and so setting up stuff with the paths was pretty intuitive. However, although I have benefited greatly from using schema that other have made to help with validation, I have not had to create schema before. I did find a good schema generation tool here that really helped, but I needed to go through and do a lot of clean up. Everything that went into creating the schema is worthy of it’s own blog post someday, so I won’t go down that rabbit trail right now.

After creating the schema, I could use it to define how the content of the response should look:

      description: Makes the world a better place.
          description: You got a totally ok result.
                $ref: '#/components/schemas/SchemaName'

In the content object I’ve first defined the content type. This is what is returned in the Content-Type header of the response. I then use a $ref object to specify a reference location where it should look for the schema for this response. Schema is often shared between different parts of an API, and so by moving it into it’s own section, you can reference it from multiple places without needing to create it multiple times.

Defining Security

As I said earlier, I won’t get into the details of creating the schema in this post. Instead let’s look at how to define security options. The API I’m investigating uses bearer tokens which is pretty easy to define with the OAS.

      type: http
      scheme: bearer
  - bearerAuth: []

And with that, I have an extremely basic, but functional schema. I’ll leave it there for now, but in future posts I will share how it went when I was creating a schema and also share how I used the definition file to actually do something useful.

Although it was a bit intimidating at first, as I worked my way throught this, it didn’t take long to feel like I could put together something useful. How did you find it getting started with using the Open API Specification?

Photo by Jon Eric Marababol on Unsplash

Getting Started with API Specification Languages

The quest for ever higher levels of abstraction never ends in computing. I used to work at a company that still ran some FORTRAN code. Although out of date now, the invention of FORTRAN in the 1950 was revolutionary in the way it abstracted away the need for humans to work with assembly language and punch cards. Fast forward a couple of decades and object oriented languages like C++ came along with increased levels of abstraction. Then we got rid of the need for humans to do garbage collection and memory allocation in increasingly abstract languages. We have continued to created additional layers of abstraction through widespread use of libraries and I would argue even through the increase use of APIs.

But we didn’t stop there.

Once APIs become widespread, we started looking for ways to simplify our interactions with them. This is where API Specification languages come into play. I want to explore some of the different specification languages that are in common use over a few blog posts. Before diving into specific specifications though, I want to look at what an API specification is.

What are API Specifications?

API specifications provide a structured way to define an API. If you follow the rules of the specification, you can interact with the API at a higher level of abstraction. For example, an API specification could be used to automatically create mock servers that you can use for testing and experimentation during development. You can also do things like automatically generate documentation and contract tests. You can even use these specifications to generate some kinds of code, for both server implementations and for client side code that calls the API.

RESTful APIs follow an architectural style laid out by Roy Fielding in his doctoral dissertation. This means that there are general principles laid out that they follow, but there is not a strict protocol that they must adhere to. This balance between structure and flexibility has been a powerful concept and has contributed to the widespread adoption of this kind of API architecture. There is no such thing as a perfect solution, however, and this one is no different. For example, one of the benefits of strict API protocols like SOAP is that the structure must be specified following strict programmatic rules. This means that the API definition must be written in a way that computers can easily understand. This makes it easy to create a lot of general-purpose automation for these kinds of APIs. If a computer can understand the layout of an API, you can automatically generate some kinds of documentation, tests and even code from that specification.

Without that strict specification, each RESTful API may have some nuances to it that are different from others. As humans, this is generally fine. We can quickly and easily figure out where those differences are and accommodate them. Computers struggle with this. Things have to be very clearly and explicitly laid out for a computer to use it. This challenge with RESTful APIs was recognized pretty early on, and so some API specification formats were proposed.

Should you use an API Specification Language?


And no.

Asking if you should use an API specification language is like asking if you should use a hammer. In order to answer that question, you need to first know if you are trying to get in a nail or a screw. Whether or not to use it depends on what you are trying to do.

API specification languages are good for helping you solve certain kinds of problems. They can help to keep things consisent between documentation and the functionality of your code. They can also be used to help you design APIs and to establish and enforce contracts around how an API should work.

However, there are downsides as well. If you want to use an API specification you need to be willing to maintain it. They often won’t have much value if you let them get out of sync with the underlying code. There are tools that can help with this maintance, but it is an important consideration if you are looking to implement the use of an API specification. Since using API specifications can cause some additional overhead they are usually best suited to scenarios where you have multiple people across multiple teams working on a set of APIs and where you need clear and unambigous ways to communicate how an API works.

So should you use an API specification language? I would say that unless you are working on a very small API with a small team, it is something worth considering. In my experience many development teams could benefit from a little more rigour in their processes and API specification will help with that. Just go in with your eyes wide open and don’t expect it to be some kind of magic pill. As with anything, it take hard work and discipline to use well.

Types of API Specifications

There are three main RESTful API specifications: RAML (https://raml.org/), API Blueprint (https://apiblueprint.org/) and OpenAPI (https://github.com/OAI/OpenAPI-Specification). Previously OpenAPI was called Swagger, so if you hear anyone talking about Swagger, just realize that it is the same thing as OpenAPI.

Over the next couple of weeks, I will share posts diving into these three API specifications and how to use them, but for now I just want you to have an idea of what they are. API specifications can be a powerful testing tool. You can use them to keep documentation, tests and even underlying code all in sync with each other. They can also be a tool that helps to maintain contracts between different parts of an application and can help less technical people get included in the design and testing of APIs. As I go through how to use each of the specifications, I’ll talk about this stuff in more detail. Stay tuned for more posts in this series!

Photo by Tim Gouw on Unsplash

What Problems do your Customers have?

Back in the days before COVID, when we could go to museums en masse, the Louvre in Paris had a problem. They had moved the Mona Lisa to a new room. The problem was, the room was only accessible through a single door and the resulting bottleneck led to many who had bought tickets being turned away or having to wait in line for hours to see the painting.

It was a horrible customer experience for thousands of museum goers. How many of them do think complained? How many requested a refund?

How do you know if you have a good quality product? Well, one of the things you could measure for that is how often you get customer complaints. Surely if there are very few clients complaining, the quality is good right?

Well, those museum goers who had a horrible experience: about 2% complained and less than 1% asked for a refund. I think there are two lessons to learn here.

In the first place, one customer complaint might just mean 50 to 100 other customers have the same pain point and never bothered to complain. Take customer complaints seriously. In the second place, you can’t rely on customer complaints to let you know when you have quality problems.

You need to do testing before releasing, and you need to have other ways of discovering problems that have been released that don’t require relying on customer complaints. When testing ahead of a release, don’t just look for functional bugs. Look for where users might experience pain. What things are going to give the customer a negative experience with the product?

Also, don’t just throw a product out there and assume that you’ll hear about the problems. Invest in some monitoring. Figure how to find those pain points for your customers and improve them without waiting for a complaint. As a software industry we have spent many years teaching people to put up with annoyances and frustrations. We can do better.

Photo by Mika Baumeister on Unsplash

Current Projects

I love creating courses and content for my fellow testers. Over the years I have learned a lot about testing in various ways, and I’m glad that I can help other testers in some small way.

I have a few cool things that I’ve been up to recently, but the big excitement for me right now has to do with Postman. I was contacted by Packt Publishing asking if I would be interested in writing a book about using Postman for API testing and development. The idea was very interesting to me, but as a I have a full time day job and spend a lot of my spare time creating courses, I didn’t think I would be able to do it and so I had to turn down the offer.

However, I love API testing and I think Postman is a great tool and the more I thought about it, the more I wanted to do it. I ended up reaching out to my boss and having a discussion about it, and we decided that I would reduce my hours and take off one day a week so that I would have some time to work on this book. I am grateful to my employer for their flexibility on this. And now? Well, I have some writing to do!

The book won’t be coming out for a while, but if you can’t wait to see some of the things that you might be able to learn, you should check out the Ultimate Postman Tutorial for API Testing that I just published on the TestProject blog. This tutorial will help you get up and running with using Postman and give you a bit of taste for what you might get in my book.

I might not have much time to post on this blog over the next few months, as many of my keystrokes will be dedicated to putting together a book that I can be proud of. I’m nervous and excited about this. Wish me luck!

Photo by Aaron Burden on Unsplash

API Mocking

COVID-19 Note: There is a virus shutting down the world right now. It’s destroying a lot of things including lives and livelihoods. I want to take some of the focus off the destruction this virus is causing, and focus instead on creating something. I’m doing a series of posts on API testing as my way of fighting back against the virus. We can still go on with life even in times like this.

You might have heard about API mocking before but what exactly does that mean? No, it’s not pointing your finger at an API while laughing at it and making fun of it (as tempting as that might be sometimes). Mocking is actually an important test strategy that we can use to control how an API responds to certain calls.

Mocks for Exploring

Let’s imagine you have an application with a front end that calls certain APIs to get the data that it needs and then renders that data to the user. This is a pretty common way to build websites, but what if you want to test what happens when the API returns an error code? It can be hard to do this kind of testing because you would have to somehow trigger an error in the API. The problem is API designers want their API to return good data and so finding situations where it gives back error codes can be hard. You could try things like disconnecting the network or intercepting the network calls to trigger errors. Or you could just create a mock.

A mock API is just a locally hosted version of the API that will return preset calls for given endpoints. If you have a mock API, you can set it up so that the endpoint will return the kind of error you are trying to test for and then point the application to the mock API instead of the real one. This allows you to generate the exact situation that you want to check. 

Essentially when you are mocking out an API you are creating a (usually simplified) version of the API that you can use instead of the real one so that you can have full control over what it does. This makes it much easier to create a wide array of test scenarios that can greatly enhance your ability to explore how a front end will react to different kinds of data in an API call.

Mocks for Automation

Using mocks to help with exploratory testing isn’t the only way they are beneficial. Another way to use API mocking is to help out with test automation. Mocks can help solve a couple of test automation challenges.

One thing they can help with is flaky tests due to network problems. If the network disconnects or slows down during a test run, any network calls being performed could cause test failures that you aren’t interested in. A mock API can eliminate the need to make network calls. If you setup and host a mock API locally you can use it in your tests without needing to do any network calls.

Another potential benefit of mock APIs in test automation is that your tests can run a bit faster. If you are getting your data locally instead of sending it off to a server somewhere and waiting for a reply, your tests can run much faster, depending of course on how many network calls you are making and how slow the network is.

Problems with API mocks

However, using mocks isn’t the ultimate solution to all of the world’s testing problems. There are some challenges that can arise from using them as well. One of the biggest challenges lies in data management. You see, mock APIs aren’t the real deal. I know. Mind blown right?

Despite how obvious that statement might seem, its easy to miss what is right there in front of your face. A mock API is usually a handcrafted affair that is setup to match what the real API does. This gives you the benefits mentioned above, but it also means that the mock API is only as accurate as you have made it to be. If the data and endpoints in the mock API don’t match up with the real API, you may be missing important stuff in your testing. Getting a mock API setup just right can be a challenge in it’s own right, but even after that, we know that software continues to change which means that in all probability the real API will change. If you don’t have some mechanism in place to keep the mock API up to date, it will no longer accurately reflect the system you are testing.

Data Management

This is where data management comes into play. You need to be thinking about the data in your mock API and how you can ensure that it stays up to date. If you don’t, you will end up with problems like an out of date mock API that no longer tests what it should, or a maintenance headache that you need to deal with every time something changes.

So what can you do? Well, test data management is a complex thing on it’s own and this article is already getting long, so we can’t dive into it in detail, but let me share a couple of examples of things I’ve done in the past.

One strategy that I’ve used to reduce the maintenance work of a mock API is to create a script that can update it. This doesn’t work for every kind of API and may not always be perfect, but in my situation it was very helpful. I had a test site that was setup with the data that I wanted and when I wanted to update the mock API, I would execute the script which would make calls to the test site and update the mock API with any changes in the API. This worked well in this particular case because the API I was using was a hypermedia API which included information in the responses that my script could easily use to see if there were any new endpoints added or other changes. Not all APIs give you that kind of information and so this strategy may not work as well in other cases, but I would encourage you in general to consider cases where you might have to do manual data updates like this and to see if you can’t learn enough scripting to automates parts of it.

Another strategy I have used in the past to keep API mocks up to date, is to do some variation of contract testing. The idea of this (at least as I have implemented it), is to have some set of tests that checks the API functionality you are using in your mocks. Any changes found by these tests are changes to the API contract and let you know that you need to update your mocks.

Now, these are just a few ideas and are certainly not perfect solutions, but hopefully they give you something that you can build off of if you are thinking about data management in your API mocks. 

So should you use mock APIs? Well, it depends of course on what problem you are trying to solve, but they certainly are a powerful tool to keep in your testing toolbox.

Photo by Phil Hearing on Unsplash

Testing Third Party APIs

COVID-19 Note: There is a virus shutting down the world right now. It’s destroying a lot of things including lives and livelihoods. I want to take some of the focus off the destruction this virus is causing, and focus instead on creating something. I’m doing a series of posts on API testing as my way of fighting back against the virus. We can still go on with life even in times like this.

APIs great. They are a powerful way for different applications to work together. They allow for the easy integration of one application with another. APIs really have enabled much of the modern web. They are great.


Until the API changes. If it is an internal API, it might not be too much of problem, although in some companies the API team might be totally separate from those that use the APIs and so even internal API changes can cause problems. In fact, in bigger companies with separate API development teams, it often makes sense to treat internal APIs as if they are third party. API developers will also often version APIs so that you can continue to use old versions without breaking your workflows, but what happens when the company decides to deprecate or stop supporting old versions of an API?

Software changes are inevitable. Any piece of software that people care about and that is actively being used is going to change. Somehow though this seems to be a difficult thing for us humans to get our minds around. We want to be able to ‘set it and forget it’ but since we know software is going change we can’t do that. In broad strokes there are two strategies we can take to deal with this

Test First

The first strategy we could take is one where we just fully embrace the fact that software changes will happen and so we don’t just start using an API and hope for the best. Instead we build up a set of tests for the API that verifies that it does everything we need it to do. We then run those tests before every build and use them to ensure that the API still does what we need it to before we start using it in any of our software.

This approach obviously will take quite a bit of work since you will need to create the tests up front and run and maintain them into the future, but there are times when this is a good strategy to employ. If you are using a third party API to provide you with functionality that is core to what you application is doing, you will probably want to verify the API before you consume it. The more critical the information you are getting from the API and the more devastating the effects of failure the more careful you will want to be with verifying it ahead of time.


There is another approach you could take though. Instead of testing the API before you even use it, you could just monitor for failures and then fix things up afterwards.

This idea can sometimes rub testers the wrong way, since we want to be able to prevent failures before they are out in the wild, but there are times when this strategy makes sense. Perhaps the API you are using doesn’t provide critical information and if it stops working you won’t have a huge impact on your clients. Or perhaps the API calls are for internal scripts that workflow that help you with your job, but if they break and those scripts don’t work for a few days it’s no big deal. There are many cases where broken functionality isn’t too big of deal as long as you can fix it up in a timely way. In those situations it makes a lot more sense to just monitor for things breaking and deal with the changes after you see something broken.

I would also remind you that monitoring doesn’t have to be a big complicated high tech thing. For example, I have a course on API testing and in that course I provide a lot of example APIs that people can use to practice their testing with. However, software changes, and so sometimes the examples that I use don’t work anymore due to changes in the API. This has happened in the course and one of the students sent me a message and asked me about why the example wasn’t working. I then went and looked into it and was able to update things to point to the correct API. The student reaching out to me is an example of monitoring. It might not be very efficient monitoring since it depends on the goodwill (or annoyance) of one of my students, but the usage of that API is still being monitored.

Monitoring doesn’t have to be high tech, but you should consider how you would actually know if it fails. Do you have to wait for a client to complain (like in my example), or do you have some internal user or process that you can rely on to let you know if it has failed? You may even want to consider monitoring software of some kind if it is important to know about breaking changes in a timely manner.


Regardless of which approach you take, you should have a way to mitigate the problems when an API fails to do what you want it to. Monitoring that lets you know that your API calls are failing doesn’t do you much good if you don’t know how fix it or if you don’t have some way to work around the problem until it is fixed. The same applies with testing it up front. If you don’t have a plan for how to make the changes that you need to when the API stops working you could end up holding back changes that you want because of a ‘broken’ API. Understanding what you will do if/when the API breaks is an important part of dealing with 3rd part APIs.

Mitigation is also important in another way. No matter how much testing or monitoring you do, things are going to fail in weird ways sometimes. There are some things that testing just can’t deal with. What if the service the API provides you access to goes down or stops working (or maybe even the company providing it goes bankrupt or stops giving access to it). What do you do? I guess one thing you could do is deny it – that will never happen! But if covid-19 has taught us anything it is to never say something like that! Having some kind of fall back strategy that at least informs users that something has gone wrong and preferably also gives them some kind of (perhaps reduced) functionality is good plan when possible.


Testing 3rd party APIs is actually a lot like testing other kinds of software. You need to think carefully about what kind of testing it makes sense to do for the given context you are in.

As to the API calls that break in my course, I have been taking a very low tech (and inefficient) way to monitoring them by waiting for students to report issues. I’m not sure I like that. Who knows how many students just skip past the broken stuff before one finally asks a question and lets me know what is going on. I would like to know sooner if the one of the APIs I’m using breaks or moves to another location. I plan to make a simple script that will ping the APIs I’m using once a day and email me if the endpoint goes away or changes. I want to make my monitoring more efficient. What about you? Are there any places in your application that you are doing your monitoring by waiting for a human to notice something? Could you maybe improve that just a little bit with a simple script?

Third party APIs have enabled a lot of the wonderful things we see on the modern web, but they have their own set of risks. What are you doing to test for and mitigate them?

Photo by Mari Helin on Unsplash