Testing Third Party APIs

COVID-19 Note: There is a virus shutting down the world right now. It’s destroying a lot of things including lives and livelihoods. I want to take some of the focus off the destruction this virus is causing, and focus instead on creating something. I’m doing a series of posts on API testing as my way of fighting back against the virus. We can still go on with life even in times like this.

APIs great. They are a powerful way for different applications to work together. They allow for the easy integration of one application with another. APIs really have enabled much of the modern web. They are great.

Until.

Until the API changes. If it is an internal API, it might not be too much of problem, although in some companies the API team might be totally separate from those that use the APIs and so even internal API changes can cause problems. In fact, in bigger companies with separate API development teams, it often makes sense to treat internal APIs as if they are third party. API developers will also often version APIs so that you can continue to use old versions without breaking your workflows, but what happens when the company decides to deprecate or stop supporting old versions of an API?

Software changes are inevitable. Any piece of software that people care about and that is actively being used is going to change. Somehow though this seems to be a difficult thing for us humans to get our minds around. We want to be able to ‘set it and forget it’ but since we know software is going change we can’t do that. In broad strokes there are two strategies we can take to deal with this

Test First

The first strategy we could take is one where we just fully embrace the fact that software changes will happen and so we don’t just start using an API and hope for the best. Instead we build up a set of tests for the API that verifies that it does everything we need it to do. We then run those tests before every build and use them to ensure that the API still does what we need it to before we start using it in any of our software.

This approach obviously will take quite a bit of work since you will need to create the tests up front and run and maintain them into the future, but there are times when this is a good strategy to employ. If you are using a third party API to provide you with functionality that is core to what you application is doing, you will probably want to verify the API before you consume it. The more critical the information you are getting from the API and the more devastating the effects of failure the more careful you will want to be with verifying it ahead of time.

Monitor

There is another approach you could take though. Instead of testing the API before you even use it, you could just monitor for failures and then fix things up afterwards.

This idea can sometimes rub testers the wrong way, since we want to be able to prevent failures before they are out in the wild, but there are times when this strategy makes sense. Perhaps the API you are using doesn’t provide critical information and if it stops working you won’t have a huge impact on your clients. Or perhaps the API calls are for internal scripts that workflow that help you with your job, but if they break and those scripts don’t work for a few days it’s no big deal. There are many cases where broken functionality isn’t too big of deal as long as you can fix it up in a timely way. In those situations it makes a lot more sense to just monitor for things breaking and deal with the changes after you see something broken.

I would also remind you that monitoring doesn’t have to be a big complicated high tech thing. For example, I have a course on API testing and in that course I provide a lot of example APIs that people can use to practice their testing with. However, software changes, and so sometimes the examples that I use don’t work anymore due to changes in the API. This has happened in the course and one of the students sent me a message and asked me about why the example wasn’t working. I then went and looked into it and was able to update things to point to the correct API. The student reaching out to me is an example of monitoring. It might not be very efficient monitoring since it depends on the goodwill (or annoyance) of one of my students, but the usage of that API is still being monitored.

Monitoring doesn’t have to be high tech, but you should consider how you would actually know if it fails. Do you have to wait for a client to complain (like in my example), or do you have some internal user or process that you can rely on to let you know if it has failed? You may even want to consider monitoring software of some kind if it is important to know about breaking changes in a timely manner.

Mitigation

Regardless of which approach you take, you should have a way to mitigate the problems when an API fails to do what you want it to. Monitoring that lets you know that your API calls are failing doesn’t do you much good if you don’t know how fix it or if you don’t have some way to work around the problem until it is fixed. The same applies with testing it up front. If you don’t have a plan for how to make the changes that you need to when the API stops working you could end up holding back changes that you want because of a ‘broken’ API. Understanding what you will do if/when the API breaks is an important part of dealing with 3rd part APIs.

Mitigation is also important in another way. No matter how much testing or monitoring you do, things are going to fail in weird ways sometimes. There are some things that testing just can’t deal with. What if the service the API provides you access to goes down or stops working (or maybe even the company providing it goes bankrupt or stops giving access to it). What do you do? I guess one thing you could do is deny it – that will never happen! But if covid-19 has taught us anything it is to never say something like that! Having some kind of fall back strategy that at least informs users that something has gone wrong and preferably also gives them some kind of (perhaps reduced) functionality is good plan when possible.

Conclusion

Testing 3rd party APIs is actually a lot like testing other kinds of software. You need to think carefully about what kind of testing it makes sense to do for the given context you are in.

As to the API calls that break in my course, I have been taking a very low tech (and inefficient) way to monitoring them by waiting for students to report issues. I’m not sure I like that. Who knows how many students just skip past the broken stuff before one finally asks a question and lets me know what is going on. I would like to know sooner if the one of the APIs I’m using breaks or moves to another location. I plan to make a simple script that will ping the APIs I’m using once a day and email me if the endpoint goes away or changes. I want to make my monitoring more efficient. What about you? Are there any places in your application that you are doing your monitoring by waiting for a human to notice something? Could you maybe improve that just a little bit with a simple script?

Third party APIs have enabled a lot of the wonderful things we see on the modern web, but they have their own set of risks. What are you doing to test for and mitigate them?

Photo by Mari Helin on Unsplash