Getting started with the Open API Specification

Getting started with the Open API Specification

Twenty, twenty-one, twenty-two. Twenty-two seconds! Twenty-two seconds to load the data for a simple table. Somehow we had repeated the same mistake we had run into before. A year or two before we were working with an API that used sperate endpoints for each cell in the table we were creating on the page. That works fine until you have a table with 100 rows and 5 or 6 columns. Now you have to send hundreds of API calls and even if each one is small on it’s own, it takes time for the calls to traverse the network. The HTTP protocol limits the number of calls you can make concurrently and so queing theory applies and the load time for the table gets really slow.

And now here we were. We were creating another table in our UI and the API that supplied it had a different endpoint for each cell in the table. How did we get here? During the design we had discussed performance and how we would need to design the API so that we wouldn’t come across this exact issue. Somehow as a team when we go caught up in the work of creating the APIs that would pull together all the data we needed, we missed this piece of the API design.

We are a well functioning, highly productive software development team. In fact, we refactored the API to work as it should have within a week, and yet we had missed this. I suspect we aren’t the only team that has faced something like this. We were building a system that would deal with a lot of complexity and present it in a simple way so that our clients wouldn’t need to deal with it. The challenge of course, is that there are so many details to keep in mind. Domain and functional requirements. Design and presentation of the UI. Security and correct user permissions. Accessibility and performance. It isn’t possible to keep this all our heads at the same time. Thing get missed.

That’s why we have testers right?

I don’t know. That doesn’t feel like the right approach to me. Sure, we can use after-the-fact testing to find those bugs, but we had already identified this bug. We knew that creating an API where each cell in a table needs to call a different endpoint is not scalable. In essence, we had to find this bug twice. Once during design and once after development. The second time we found it, the bug was expensive to fix. Could we have prevented that regression? Can we create regression tests for our designs?

This is one of the problems that API specifications attempt to solve. In reflecting on this issue, I started to wonder if we could have saved ourselves a lot of rework by encoding the things we had discussed and discovered during our design sessions. What if we wrote them down in an API definition that set out what the API should look like and do?

You can’t do much about the past, but you can learn lessons from it and use those lessons to make the future better. I’m sure our team will be writing more APIs in the future and so I want to explore some of the different API specification to see if there is something that would make sense for us to use in the future.

Open API Specification

The most common specification format is the Open API Specification (or OAS). It has in many ways become the default specification for describing how APIs work. It has very broad adoption across many companies and has a large supporting community. It is designed for RESTful APIs, which we use, and it has support for almost anything you would do with these APIs.

I’ll be honest though, I find it overwhelming to get started with. It is a lot of work to document every detail of what your API can or should do. Since we have existing APIs that do not yet have API definitions I thought I would try using the Swagger inspector. It gave me something to get me started, but it still needed a lot to be done by hand. Since at this point, my goal is more around evaluating if this makes sense for future APIs, I decided to instead look at creating a definition “from scratch”. I’d like to use a design first rather than code first approach to this.

So how hard is it to create an API defintion that follows the OpenAPI Specification?

The Tools

Well, let’s start with talking about the tools. There are a lot of Swagger tools that work with Open API Specifications. Confusingly the OAS used to be called Swagger, but once you get past that, there are still many tools out there. It’s easy to get lost in trying to figure out which tool does what, and which one you want to use. I know because I spent too much time trying to figure out which one to use. Eventually I decided that I wouldn’t use any of the custom built tools. OpenAPI defintions are just text files (either yaml or json format) and so they can be edited in any text editor.

I created a file in my trusty Visual Studio Code, installed a couple of OpenAPI formatting and linting extentions and I was finally off the races. I’m sure that if I use this more, I will eventually want tools that are built specificially for the OAS, but I’ll stick with the basics for now and see where the pain points show up as I scale. If you are new to using the OpenAPI Spec, I’d recomend taking this approach too. Not only will it force you to understand what is going on more deeply, but it will also remove a whole layer of confusion that can get in the way of just getting started.

Creating the API Defintion file

A blank file staring at you can be intimidating, but I guess the best way to get something done is to start doing it, so let’s look at what data needs to go into this file.

The First Section

So, how do you actually create an API definition with the OAS? The basics of it are straigtforward. Start with a bit of boilerplate at the top that says what version of the OpenAPI Spec you are using along with some information about the API your are creating the definition for.

openapi: 3.0.1
info:
  title: A cool API
  description: This API does cool stuff
  version: '1.0'
servers:
  - url: https://{tenantId}.domain.api.com
    variables:
      tenantId:
        default: tenant-id-guid-123

One simple thing I ran into here was that each instance that we create (and we have at least one for each of our clients) has a different base url. This includes all the testing sites. Pretty easy to take care of though. You can create a parameter in the url. You can read more about how the base urls work here if you want.

Defining an Endpoint

With the boiler-plate out of the way, the next thing to define is an endpoint. Endpoints are defined under the paths section.

paths:
  /endpoint/path/{pathParameter}:
    get:
      description: Makes the world a better place.
      responses:
        '200':
          description: You got a totally ok result.
      parameters:
        - name: pathParameter
          in: path
          required: true
          schema:
            type: integer
            format: int64
            minimum: 6606

I’m obviously redacting some stuff here, since these are internal APIs, but it’s interesting to note how easy it is to create a parameter in the path. All I needed to do was to include it in the path defition and then create a parameters section under the request to define what kind of parameter it was.

You will notice though, that the responses object only defines the response code. It says nothing about what kind of content this API should give back. That seems like it would be kind of important. Defining this opens up a whole new can of worms. Obviously the API does not always give back an identical response every time you call it, and so you can’t just copy a response and put in in there. Instead you need to specify some general rules that should apply to the response. For example, you could specify that it should be a list and that each item in the list should be a string. These kinds of rules are known as schema. The kinds of objects our API represents are pretty complex and so it takes a bit of work to create a schema for them.

In fact, as I dug into this I realized that this is actually the area that was the most intimidating to me. I’m pretty comfortable with the way RESTful APIs work and so setting up stuff with the paths was pretty intuitive. However, although I have benefited greatly from using schema that other have made to help with validation, I have not had to create schema before. I did find a good schema generation tool here that really helped, but I needed to go through and do a lot of clean up. Everything that went into creating the schema is worthy of it’s own blog post someday, so I won’t go down that rabbit trail right now.

After creating the schema, I could use it to define how the content of the response should look:

paths:
  /endpoint/path/{pathParameter}:
    get:
      description: Makes the world a better place.
      responses:
        '200':
          description: You got a totally ok result.
       content:
            application/vnd.siren+json:
              schema:
                $ref: '#/components/schemas/SchemaName'

In the content object I’ve first defined the content type. This is what is returned in the Content-Type header of the response. I then use a $ref object to specify a reference location where it should look for the schema for this response. Schema is often shared between different parts of an API, and so by moving it into it’s own section, you can reference it from multiple places without needing to create it multiple times.

Defining Security

As I said earlier, I won’t get into the details of creating the schema in this post. Instead let’s look at how to define security options. The API I’m investigating uses bearer tokens which is pretty easy to define with the OAS.

components:
  securitySchemes:
    bearerAuth:
      type: http
      scheme: bearer
security:
  - bearerAuth: []

And with that, I have an extremely basic, but functional schema. I’ll leave it there for now, but in future posts I will share how it went when I was creating a schema and also share how I used the definition file to actually do something useful.

Although it was a bit intimidating at first, as I worked my way throught this, it didn’t take long to feel like I could put together something useful. How did you find it getting started with using the Open API Specification?

Photo by Jon Eric Marababol on Unsplash

Getting Started with API Specification Languages

Getting Started with API Specification Languages

The quest for ever higher levels of abstraction never ends in computing. I used to work at a company that still ran some FORTRAN code. Although out of date now, the invention of FORTRAN in the 1950 was revolutionary in the way it abstracted away the need for humans to work with assembly language and punch cards. Fast forward a couple of decades and object oriented languages like C++ came along with increased levels of abstraction. Then we got rid of the need for humans to do garbage collection and memory allocation in increasingly abstract languages. We have continued to created additional layers of abstraction through widespread use of libraries and I would argue even through the increase use of APIs.

But we didn’t stop there.

Once APIs become widespread, we started looking for ways to simplify our interactions with them. This is where API Specification languages come into play. I want to explore some of the different specification languages that are in common use over a few blog posts. Before diving into specific specifications though, I want to look at what an API specification is.

What are API Specifications?

API specifications provide a structured way to define an API. If you follow the rules of the specification, you can interact with the API at a higher level of abstraction. For example, an API specification could be used to automatically create mock servers that you can use for testing and experimentation during development. You can also do things like automatically generate documentation and contract tests. You can even use these specifications to generate some kinds of code, for both server implementations and for client side code that calls the API.

RESTful APIs follow an architectural style laid out by Roy Fielding in his doctoral dissertation. This means that there are general principles laid out that they follow, but there is not a strict protocol that they must adhere to. This balance between structure and flexibility has been a powerful concept and has contributed to the widespread adoption of this kind of API architecture. There is no such thing as a perfect solution, however, and this one is no different. For example, one of the benefits of strict API protocols like SOAP is that the structure must be specified following strict programmatic rules. This means that the API definition must be written in a way that computers can easily understand. This makes it easy to create a lot of general-purpose automation for these kinds of APIs. If a computer can understand the layout of an API, you can automatically generate some kinds of documentation, tests and even code from that specification.

Without that strict specification, each RESTful API may have some nuances to it that are different from others. As humans, this is generally fine. We can quickly and easily figure out where those differences are and accommodate them. Computers struggle with this. Things have to be very clearly and explicitly laid out for a computer to use it. This challenge with RESTful APIs was recognized pretty early on, and so some API specification formats were proposed.

Should you use an API Specification Language?

Yes.

And no.

Asking if you should use an API specification language is like asking if you should use a hammer. In order to answer that question, you need to first know if you are trying to get in a nail or a screw. Whether or not to use it depends on what you are trying to do.

API specification languages are good for helping you solve certain kinds of problems. They can help to keep things consisent between documentation and the functionality of your code. They can also be used to help you design APIs and to establish and enforce contracts around how an API should work.

However, there are downsides as well. If you want to use an API specification you need to be willing to maintain it. They often won’t have much value if you let them get out of sync with the underlying code. There are tools that can help with this maintance, but it is an important consideration if you are looking to implement the use of an API specification. Since using API specifications can cause some additional overhead they are usually best suited to scenarios where you have multiple people across multiple teams working on a set of APIs and where you need clear and unambigous ways to communicate how an API works.

So should you use an API specification language? I would say that unless you are working on a very small API with a small team, it is something worth considering. In my experience many development teams could benefit from a little more rigour in their processes and API specification will help with that. Just go in with your eyes wide open and don’t expect it to be some kind of magic pill. As with anything, it take hard work and discipline to use well.

Types of API Specifications

There are three main RESTful API specifications: RAML (https://raml.org/), API Blueprint (https://apiblueprint.org/) and OpenAPI (https://github.com/OAI/OpenAPI-Specification). Previously OpenAPI was called Swagger, so if you hear anyone talking about Swagger, just realize that it is the same thing as OpenAPI.

Over the next couple of weeks, I will share posts diving into these three API specifications and how to use them, but for now I just want you to have an idea of what they are. API specifications can be a powerful testing tool. You can use them to keep documentation, tests and even underlying code all in sync with each other. They can also be a tool that helps to maintain contracts between different parts of an application and can help less technical people get included in the design and testing of APIs. As I go through how to use each of the specifications, I’ll talk about this stuff in more detail. Stay tuned for more posts in this series!

Photo by Tim Gouw on Unsplash

What Problems do your Customers have?

What Problems do your Customers have?

Back in the days before COVID, when we could go to museums en masse, the Louvre in Paris had a problem. They had moved the Mona Lisa to a new room. The problem was, the room was only accessible through a single door and the resulting bottleneck led to many who had bought tickets being turned away or having to wait in line for hours to see the painting.

It was a horrible customer experience for thousands of museum goers. How many of them do think complained? How many requested a refund?

How do you know if you have a good quality product? Well, one of the things you could measure for that is how often you get customer complaints. Surely if there are very few clients complaining, the quality is good right?

Well, those museum goers who had a horrible experience: about 2% complained and less than 1% asked for a refund. I think there are two lessons to learn here.

In the first place, one customer complaint might just mean 50 to 100 other customers have the same pain point and never bothered to complain. Take customer complaints seriously. In the second place, you can’t rely on customer complaints to let you know when you have quality problems.

You need to do testing before releasing, and you need to have other ways of discovering problems that have been released that don’t require relying on customer complaints. When testing ahead of a release, don’t just look for functional bugs. Look for where users might experience pain. What things are going to give the customer a negative experience with the product?

Also, don’t just throw a product out there and assume that you’ll hear about the problems. Invest in some monitoring. Figure how to find those pain points for your customers and improve them without waiting for a complaint. As a software industry we have spent many years teaching people to put up with annoyances and frustrations. We can do better.

Photo by Mika Baumeister on Unsplash

Current Projects

Current Projects

I love creating courses and content for my fellow testers. Over the years I have learned a lot about testing in various ways, and I’m glad that I can help other testers in some small way.

I have a few cool things that I’ve been up to recently, but the big excitement for me right now has to do with Postman. I was contacted by Packt Publishing asking if I would be interested in writing a book about using Postman for API testing and development. The idea was very interesting to me, but as a I have a full time day job and spend a lot of my spare time creating courses, I didn’t think I would be able to do it and so I had to turn down the offer.

However, I love API testing and I think Postman is a great tool and the more I thought about it, the more I wanted to do it. I ended up reaching out to my boss and having a discussion about it, and we decided that I would reduce my hours and take off one day a week so that I would have some time to work on this book. I am grateful to my employer for their flexibility on this. And now? Well, I have some writing to do!

The book won’t be coming out for a while, but if you can’t wait to see some of the things that you might be able to learn, you should check out the Ultimate Postman Tutorial for API Testing that I just published on the TestProject blog. This tutorial will help you get up and running with using Postman and give you a bit of taste for what you might get in my book.

I might not have much time to post on this blog over the next few months, as many of my keystrokes will be dedicated to putting together a book that I can be proud of. I’m nervous and excited about this. Wish me luck!

Photo by Aaron Burden on Unsplash

API Mocking

API Mocking

COVID-19 Note: There is a virus shutting down the world right now. It’s destroying a lot of things including lives and livelihoods. I want to take some of the focus off the destruction this virus is causing, and focus instead on creating something. I’m doing a series of posts on API testing as my way of fighting back against the virus. We can still go on with life even in times like this.

You might have heard about API mocking before but what exactly does that mean? No, it’s not pointing your finger at an API while laughing at it and making fun of it (as tempting as that might be sometimes). Mocking is actually an important test strategy that we can use to control how an API responds to certain calls.

Mocks for Exploring

Let’s imagine you have an application with a front end that calls certain APIs to get the data that it needs and then renders that data to the user. This is a pretty common way to build websites, but what if you want to test what happens when the API returns an error code? It can be hard to do this kind of testing because you would have to somehow trigger an error in the API. The problem is API designers want their API to return good data and so finding situations where it gives back error codes can be hard. You could try things like disconnecting the network or intercepting the network calls to trigger errors. Or you could just create a mock.

A mock API is just a locally hosted version of the API that will return preset calls for given endpoints. If you have a mock API, you can set it up so that the endpoint will return the kind of error you are trying to test for and then point the application to the mock API instead of the real one. This allows you to generate the exact situation that you want to check. 

Essentially when you are mocking out an API you are creating a (usually simplified) version of the API that you can use instead of the real one so that you can have full control over what it does. This makes it much easier to create a wide array of test scenarios that can greatly enhance your ability to explore how a front end will react to different kinds of data in an API call.

Mocks for Automation

Using mocks to help with exploratory testing isn’t the only way they are beneficial. Another way to use API mocking is to help out with test automation. Mocks can help solve a couple of test automation challenges.

One thing they can help with is flaky tests due to network problems. If the network disconnects or slows down during a test run, any network calls being performed could cause test failures that you aren’t interested in. A mock API can eliminate the need to make network calls. If you setup and host a mock API locally you can use it in your tests without needing to do any network calls.

Another potential benefit of mock APIs in test automation is that your tests can run a bit faster. If you are getting your data locally instead of sending it off to a server somewhere and waiting for a reply, your tests can run much faster, depending of course on how many network calls you are making and how slow the network is.

Problems with API mocks

However, using mocks isn’t the ultimate solution to all of the world’s testing problems. There are some challenges that can arise from using them as well. One of the biggest challenges lies in data management. You see, mock APIs aren’t the real deal. I know. Mind blown right?

Despite how obvious that statement might seem, its easy to miss what is right there in front of your face. A mock API is usually a handcrafted affair that is setup to match what the real API does. This gives you the benefits mentioned above, but it also means that the mock API is only as accurate as you have made it to be. If the data and endpoints in the mock API don’t match up with the real API, you may be missing important stuff in your testing. Getting a mock API setup just right can be a challenge in it’s own right, but even after that, we know that software continues to change which means that in all probability the real API will change. If you don’t have some mechanism in place to keep the mock API up to date, it will no longer accurately reflect the system you are testing.

Data Management

This is where data management comes into play. You need to be thinking about the data in your mock API and how you can ensure that it stays up to date. If you don’t, you will end up with problems like an out of date mock API that no longer tests what it should, or a maintenance headache that you need to deal with every time something changes.

So what can you do? Well, test data management is a complex thing on it’s own and this article is already getting long, so we can’t dive into it in detail, but let me share a couple of examples of things I’ve done in the past.

One strategy that I’ve used to reduce the maintenance work of a mock API is to create a script that can update it. This doesn’t work for every kind of API and may not always be perfect, but in my situation it was very helpful. I had a test site that was setup with the data that I wanted and when I wanted to update the mock API, I would execute the script which would make calls to the test site and update the mock API with any changes in the API. This worked well in this particular case because the API I was using was a hypermedia API which included information in the responses that my script could easily use to see if there were any new endpoints added or other changes. Not all APIs give you that kind of information and so this strategy may not work as well in other cases, but I would encourage you in general to consider cases where you might have to do manual data updates like this and to see if you can’t learn enough scripting to automates parts of it.

Another strategy I have used in the past to keep API mocks up to date, is to do some variation of contract testing. The idea of this (at least as I have implemented it), is to have some set of tests that checks the API functionality you are using in your mocks. Any changes found by these tests are changes to the API contract and let you know that you need to update your mocks.

Now, these are just a few ideas and are certainly not perfect solutions, but hopefully they give you something that you can build off of if you are thinking about data management in your API mocks. 

So should you use mock APIs? Well, it depends of course on what problem you are trying to solve, but they certainly are a powerful tool to keep in your testing toolbox.

Photo by Phil Hearing on Unsplash

Testing Third Party APIs

Testing Third Party APIs

COVID-19 Note: There is a virus shutting down the world right now. It’s destroying a lot of things including lives and livelihoods. I want to take some of the focus off the destruction this virus is causing, and focus instead on creating something. I’m doing a series of posts on API testing as my way of fighting back against the virus. We can still go on with life even in times like this.

APIs great. They are a powerful way for different applications to work together. They allow for the easy integration of one application with another. APIs really have enabled much of the modern web. They are great.

Until.

Until the API changes. If it is an internal API, it might not be too much of problem, although in some companies the API team might be totally separate from those that use the APIs and so even internal API changes can cause problems. In fact, in bigger companies with separate API development teams, it often makes sense to treat internal APIs as if they are third party. API developers will also often version APIs so that you can continue to use old versions without breaking your workflows, but what happens when the company decides to deprecate or stop supporting old versions of an API?

Software changes are inevitable. Any piece of software that people care about and that is actively being used is going to change. Somehow though this seems to be a difficult thing for us humans to get our minds around. We want to be able to ‘set it and forget it’ but since we know software is going change we can’t do that. In broad strokes there are two strategies we can take to deal with this

Test First

The first strategy we could take is one where we just fully embrace the fact that software changes will happen and so we don’t just start using an API and hope for the best. Instead we build up a set of tests for the API that verifies that it does everything we need it to do. We then run those tests before every build and use them to ensure that the API still does what we need it to before we start using it in any of our software.

This approach obviously will take quite a bit of work since you will need to create the tests up front and run and maintain them into the future, but there are times when this is a good strategy to employ. If you are using a third party API to provide you with functionality that is core to what you application is doing, you will probably want to verify the API before you consume it. The more critical the information you are getting from the API and the more devastating the effects of failure the more careful you will want to be with verifying it ahead of time.

Monitor

There is another approach you could take though. Instead of testing the API before you even use it, you could just monitor for failures and then fix things up afterwards.

This idea can sometimes rub testers the wrong way, since we want to be able to prevent failures before they are out in the wild, but there are times when this strategy makes sense. Perhaps the API you are using doesn’t provide critical information and if it stops working you won’t have a huge impact on your clients. Or perhaps the API calls are for internal scripts that workflow that help you with your job, but if they break and those scripts don’t work for a few days it’s no big deal. There are many cases where broken functionality isn’t too big of deal as long as you can fix it up in a timely way. In those situations it makes a lot more sense to just monitor for things breaking and deal with the changes after you see something broken.

I would also remind you that monitoring doesn’t have to be a big complicated high tech thing. For example, I have a course on API testing and in that course I provide a lot of example APIs that people can use to practice their testing with. However, software changes, and so sometimes the examples that I use don’t work anymore due to changes in the API. This has happened in the course and one of the students sent me a message and asked me about why the example wasn’t working. I then went and looked into it and was able to update things to point to the correct API. The student reaching out to me is an example of monitoring. It might not be very efficient monitoring since it depends on the goodwill (or annoyance) of one of my students, but the usage of that API is still being monitored.

Monitoring doesn’t have to be high tech, but you should consider how you would actually know if it fails. Do you have to wait for a client to complain (like in my example), or do you have some internal user or process that you can rely on to let you know if it has failed? You may even want to consider monitoring software of some kind if it is important to know about breaking changes in a timely manner.

Mitigation

Regardless of which approach you take, you should have a way to mitigate the problems when an API fails to do what you want it to. Monitoring that lets you know that your API calls are failing doesn’t do you much good if you don’t know how fix it or if you don’t have some way to work around the problem until it is fixed. The same applies with testing it up front. If you don’t have a plan for how to make the changes that you need to when the API stops working you could end up holding back changes that you want because of a ‘broken’ API. Understanding what you will do if/when the API breaks is an important part of dealing with 3rd part APIs.

Mitigation is also important in another way. No matter how much testing or monitoring you do, things are going to fail in weird ways sometimes. There are some things that testing just can’t deal with. What if the service the API provides you access to goes down or stops working (or maybe even the company providing it goes bankrupt or stops giving access to it). What do you do? I guess one thing you could do is deny it – that will never happen! But if covid-19 has taught us anything it is to never say something like that! Having some kind of fall back strategy that at least informs users that something has gone wrong and preferably also gives them some kind of (perhaps reduced) functionality is good plan when possible.

Conclusion

Testing 3rd party APIs is actually a lot like testing other kinds of software. You need to think carefully about what kind of testing it makes sense to do for the given context you are in.

As to the API calls that break in my course, I have been taking a very low tech (and inefficient) way to monitoring them by waiting for students to report issues. I’m not sure I like that. Who knows how many students just skip past the broken stuff before one finally asks a question and lets me know what is going on. I would like to know sooner if the one of the APIs I’m using breaks or moves to another location. I plan to make a simple script that will ping the APIs I’m using once a day and email me if the endpoint goes away or changes. I want to make my monitoring more efficient. What about you? Are there any places in your application that you are doing your monitoring by waiting for a human to notice something? Could you maybe improve that just a little bit with a simple script?

Third party APIs have enabled a lot of the wonderful things we see on the modern web, but they have their own set of risks. What are you doing to test for and mitigate them?

Photo by Mari Helin on Unsplash

Types of APIs

Types of APIs

COVID-19 Note: There is a virus shutting down the world right now. It’s destroying a lot of things including lives and livelihoods. I want to take some of the focus off the destruction this virus is causing, and focus instead on creating something. I’m doing a series of posts on API testing as my way of fighting back against the virus. We can still go on with life even in times like this.

In order to effectively test an API, you need to know what kind of API you are testing. Often in today’s world when we talk about APIs we are talking about REST APIs, but there are several other important API formats that you might need to test. Let’s look through them and see what we can learn about some of the major API types.

REST APIs

We’ll start with what is probably the most common type of API you’ll come across on the modern web; the RESTful API. REST stands for Representational State Transfer and refers to an architectural style that guides how you should create APIs. I won’t go into the details of the properties that a RESTful API should have (you can look them up on wikipedia here if you want), but there are a few clues that can let you know that you are probably testing a RESTful API.

RESTful APIs are based on a set of guidelines and so they do not all look the same. There is no official standard that defines the exact specifications that a response has to conform to. This means that the following are in general good clues and hints that you are dealing with a REST API, but they are not definitive. It is also worth noting that many APIs that are considered to be RESTful do not strictly follow all the REST guidelines. REST in general has more flexibility than a standards based protocol like SOAP, but this means that there can be a lot of diversity in the way REST APIs are defined and used.

How do I know if the API I’m looking at is RESTful?

So, what are some clues that you are looking at a REST API? Well in the first place, what kind of requests are typically defined? Most REST APIs will defined GET, POST, PUT, and DELETE calls with perhaps a few others. Depending on the needs of the API, it may not define all of these, but those are the common ones.

Another clue, is in the types of requests or responses that are allowed by the API. Often REST APIs will use JSON data in their responses (although the could use text or even xml). Generally speaking if the data in the responses and requests of the API is not xml, there is a good chance you are dealing with a REST based API of some sort.

REST API Example

There are many examples of REST APIs on the web, but let’s take a quick look at just one of them to help cement the idea of what RESTful API is in our minds. It’s the internet, so let’s look at cats, and since this is an article about API testing, let’s also look at http status codes (which are used by REST APIs). And of course, the internet knows that combining cats with http status codes needed to be done, so we can go to https://http.cat/ to see a cat image for each http status code.

This site provides an API and so we can call https://http.cat/200 to get the cat image for the 200 status code. For example, if we were to do this in Postman, we could create a request, set the method to GET and put in the url and click on send.

call cat API in Postman

We don’t have to specify any xml when we make the call and can just send the request directly in Postman (or another API tool) as a GET call, so we are most likely dealing with a REST API here.

SOAP APIs

Before REST there was SOAP. SOAP stands for Simple Object Access Protocol. The SOAP protocol has been around since long before Roy Fileding came up with the concept of REST APIs. It is not as widely used on the web now (especially for smaller applications), but for many years it was the default way to make APIs and so there are still many SOAP APIs around.

SOAP is an actual protocol with a W3 standards definition and so its usage is much more strictly defined than REST which is an architectural guideline as opposed to a strictly defined protocol.

If you want a little light reading, check out this document. It claims to be a:

non-normative document intended to provide an easily understandable tutorial on the features of SOAP Version 1.2

I’ll be honest, I’m not sure how well it delivers on the ‘easily understandable tutorial’ part of that statement. Looking at some of the examples in there may help you understand why REST APIs have become so popular. SOAP APIs require a highly structured xml message to be sent with the request. Being built in xml, these requests are not that easy to read for humans, and require a lot of complexity to build up. There are of course many tools like SoapUI that can help with this, but in general SOAP APIs tend to be a bit more complex to get started with. You need to know more information (like the envelope structure) in order to get started.

How do I know if the API I’m looking at is a SOAP API?

The most important rule of thumb here is: does it require you to specify structure xml in order to work? If it does, it’s a SOAP API. Since these kinds of APIs are required to follow the w3c specification, they must use xml and they must specify things like env:Envelope nodes inside of the xml. If the API you are looking at requires xml to be specified and that xml includes the Envelope node you are almost certainly dealing with a SOAP API.

SOAP API Example:

A SOAP API example is a little bit harder than just sending a GET request to an endpoint. We will use this webservice as an example SOAP API. Let’s look at the first action there, which gives us a list of continents. In order to call this API in Postman we will need to set up a few things. We of course need to create a request in Postman and then we will need to set the request method to POST and put in the url. However we can’t yet click send. Since this is a SOAP API, we need to send some xml information as well. In Postman, this means setting the body type to raw, and choosing XML from the dropdown and then putting in the xml envelope data as indicated by the documentation

SOAP API Information

For this particular API, we would also need to modify the Content-Type header (by adding a new one) at the bottom, so that is is set to application/soap+xml

Content-Type Header

As you can see, there is a lot more complexity to calling SOAP APIs. REST APIs can of course have complex bodies specified as well, but the requirement to do this in xml and the existence of the Envelope node in this, indicates that this API is indeed a SOAP API.

GraphQL APIs

SOAP came before REST and in many ways REST was designed to deal with some of the shortcomings of SOAP. Of course in software we are never done making things better and so along comes GraphQL. GraphQL is a query language and it was designed to deal with some of the situations where REST APIs have shortcomings. RESTful APIs don’t know what specific information you might be looking for and so when you call a REST API endpoint, it gives back all the information it has. This can mean that we are sending extra information that you don’t need, or it can mean that we aren’t sending all the information you need and you need to call multiple endpoints to get what you want. Either of these cases can slow things down and for big applications with many users that can become problematic. GraphQL was designed by Facebook to deal with these issues.

GraphQL is a query language for APIs, and so it requires you to specify in a query what you are looking for. With REST APIs you will usually need to know what the different endpoints are in order to find the information you are looking for, but with a GraphQL API a single ‘endpoint’ will contain most or all of the information you need and you will use queries to filter down that information to only the bits that you are interested in. This means that with GraphQL APIs you will need to know the schema or structure of the data so that you know how to properly query it, instead of needing to know what all the endpoints are.

How do I know if the API I’m looking at is a GraphQL API?

Well, if the documentation is telling you about what kinds of queries you need to write, you are almost certainly looking at a GraphQL API. In some ways a GraphQL API is similar to a SOAP API in that you need to tell the service some information about what you are interested in, but a SOAP API will always use xml and follow a strict definition in the calls, whereas GraphQL APIs will usually a bit simpler and not defined in xml. Also with GraphQL the way the schema is defined can vary from one API to another as it does not need to follow a strictly set standard.

GraphQL API Example

Let’s take a look at a real life example of calling a GraphQL API to understand it a bit better. This example will use the countries api as hosted here. There is information about the schema for it on github here and we can use that to create queries in the provided playground, but for this example let’s look at setting it up in Postman

Similar to calling a SOAP API, we will need to specify the service we want and do a POST request. We will also need to choose the GraphQL option on the body tab, and put in the query that we want

GraphQL query

As you can see in the example I’ve done, I’ve requested the name and languages of Canada. Once I have specified this information I can click Send and I get back some json with the country name and a list of the official languages. If I wanted additional information (Say the name of the capital city), I could just modify the query to include a request for that information and send it off again using the same endpoint.

Conclusion

We’ve looked at three different kinds of APIs. Each of them has it’s own strengths and weaknesses and is used in different applications. Some applications will even support more than one type of API, and as a tester you need to be ready to test any kind of API that you come across. Hopefully this article has at least helped you in that journey and given you some understanding of how the various APIs you might come across work and can be interacted with.

Photo by Markus Spiske on Unsplash

The Simplified Guide to API Testing

The Simplified Guide to API Testing

COVID-19 Note: There is a virus shutting down the world right now. It’s destroying a lot of lives and livelihoods. However, I want to take some of the focus off the destruction this virus is causing, and focus on creating instead of destruction. For my creative act of defiance, I want to do a series of posts on API testing over the next few weeks. Consider it to be my way of fighting back against the covid-19 virus. We can still go on with life even in times like this.

When I started testing APIs, didn’t know anything about them. I’ve learned a lot over the last few years and I now feel pretty comfortable with using and testing APIs, but it wasn’t always that way. When I first started it was hard to even figure out what to do. I can still remember some of that pain I went through.

In my first year of university we had a calculus professor we nicknamed Einstein. Partly because of his hair and partly because he managed to make calculus as confusing as relativity. He would start to solve a problem on the board and then he would say ‘and therefore it is obvious that’ and write down some solution. We would all sit there and look puzzled.  What was obvious to him was not obvious to us as the students. We needed to know the intermediate steps that he had skipped in order to understand what had happened.

This is a common occurrence. As you learn things, the pain of that initial learning curve goes away and it becomes harder and harder to explain things to those who are just starting out. You start to assume things and makes leaps of logic that seem obvious to you but that are confusing to those who are new to the subject.  As I continue to progress in what I am learning about APIs, I want to get down some of my basic thoughts on API testing, hopefully before it is ‘too late’ for me to explain to those that are just starting out.

So with that long preamble, here is my attempt at a simplified guide to getting started with API testing

Figure out the endpoints

I found this to be one of the most difficult parts of API testing. How do I even know what there is to test? What things can the API do?  If you are working with a well documented public API, this isn’t much of a concern, but in my experience most API testing is done on internal APIs that support different parts of your application.  These kinds of APIs tend to be poorly documented (if they are at all). So how do you figure it out?

Well of course any available in house documentation helps, so certainly start there if you have some available.  And don’t forget that documentation can sit in places you might not expect. For example sometimes code comments or even signatures in a method can help you out. You also might also be able to find documentation in stories or requirements that point you in the direction you need to go.

Another great source of information is humans. You see there are people in your company who have created, designed, tested and/or used the API you are looking at. If you can find and talk to those people, they can be an invaluable source of information.

Code bases are another place that contain information about how APIs work. You might not know how to write code, but you can often still find out some information by looking at how an API is called. You might be able to pick up something by the urls used in the caller, or you might be able to find some unit tests that use the API and give you insights into how it was used.

Last, but certainly not least, you might want to look at network calls in the developer console to figure out what API endpoints are available.  Often API calls are sent over the network and by looking at network calls in the developer tools you can figure out a lot about how they work.

One thing you will find with all of these sources of information is that they will give you imperfect and incomplete information.  There will be things you just don’t understand and (especially at first) everything will be very unclear and foggy.  But if you stick with it and continue to ask questions and get feedback, you will suddenly realize one day that you actually get it and have a decent understand of the API.

Auth and Security

Authorization. Authentication. Security. What is it all about and how does it work? I found auth workflows hard to understand. Security is obviously very important to the way an API works, but it can significantly complicate the use of an API.  This article would get far too long if I was to dive deep on this, but I do want to point out a few hints.

First of all look at how your tools can support you. Tools like Postman offer a lot of support for making authorization and authentication easier.  They can also provide you with insight in the security of your API. Dig into those features a bit in whatever tool you use and see what you can figure out.

Another consideration is that, again, when you find yourself frustrated by it, consider it to be a learning experience. Ask questions of your coworkers (and Google).  Try and figure out a bit about how Oauth works. Get a bit of an understanding of authorization workflows.  I have found that digging into authorization really pushed me with respect to understanding stuff about how APIs and HTTP protocols worked and so it was a great way to learn a lot more about API testing

A last comment on the security front, is that many things are ‘security’ issues that are not related directly to technical security testing. Making sure that different users types can only access the information they have permissions for and making sure that all paths give back the correct information are also forms of security testing.  Remember to look at your API holistically

Learn to do by doing

It can be tempting to spend a lot of time learning about things like API testing. We can feel like we need to know what we are doing before we get started, but I really believe that the best way to learn is by doing. Just try making API calls. Figure out one endpoint and try calling it. Start with the smallest thing you know and use it. As you try to do things and get stuck and then figure it out you will learn the things you need to know for the API you are working on.  You will also learn what things you don’t know and where you need further study and so you’ll be able to find resources that can help you with the exact things you need to know.

Figure out some of the basic terminology

I have a glossary of terms I’ve put together here. This is a set of terms I had to learn. You may find it helpful, but I think it will be important to also figure out what terms you hear that you don’t understand and figure them out. It is also important to listen to how your team uses terms and to make sure you are understanding what they mean. Often different teams or companies can use the same or similar terms in different ways. Pay attention to the words and build up a vocabulary.  You will find that this makes it easier to ask questions and to understand what is going on.

Next steps

So where to do from here?

  1. Install an API testing tool (Postman perhaps)
  2. Find an endpoint on the API you are interested and make a call to it
  3. Change something in the API call and try again
  4. Start writing down what you observe and try to build a mental map of what the API does
  5. Try to use the API to answers some questions you have and follow up on figuring out how to get it to do those things
  6. Research and learn the things you need
  7. Try your hand at some API testing challenges (The automation in testing site is a great place to start).  You could also try out some public APIs (here is a list of some you could try)
  8. Ask lots of questions!

Photo by Daniil Silantev on Unsplash

Effective Testing During COVID-19

Effective Testing During COVID-19

Can one blog about something that isn’t corona virus related at this time? This is affecting us all in so many ways. I’m thankful though that I have a job that makes it easy to work from home. The company I work at is also in an industry that will likely benefit from this as we enable online education which is kind of a big thing right now. So as someone who does not have to worry about their job and income and who is at an age where the impact of the virus is unlikely to be serious if I was to get it, I feel blessed.

I have always worked from home on a regular basis (usually 1 or 2 times a week), and so the transition to full time working from home hasn’t been too difficult for me. My wife and I also started homeschooling our kids just this year, so that also hasn’t been too big of an adjustment for us either. I think the biggest thing this pandemic has given me is some perspective. When everything changes you realize what things you might have been taking for granted. Full time remote work is different than going into the office to regularly have face to face contact with your coworkers. As a team we are all working on ways to stay connected and have effective communication in a world where we can’t just pop our heads over the cubicle wall and chat.

What does one do as software tester in this world? How do you work effectively in a fully remote environment? There are a few things I have been thinking about and working on at this time. One important thing (probably for all remote workers) is organization and discipline. There are different kinds of distractions when you work from home. Your family, your pets, even your housework all distract in different ways than your co-workers. You may need to have conversations with kids and your spouse about how to protect your time so that you have the ability to dive deep on your testing sometimes.

I think organization is another important factor to consider. There may be some built in accountability at work where your co-workers or boss can see what you are doing when they walk past your desk. At home you may need to watch your schedule more and make sure you aren’t spending too much time reading the latest news about the virus (it’s probably not a healthy thing to spend too much time reading about anyways).

There are also less clearly defined lines between work life and family life when you work from home, so set some boundaries for yourself that way too. You may have needed to talk to your kids about boundaries while you are working, but don’t forget to set boundaries for when you are not working too. Just because your computer is only 5 feet away in your office, doesn’t mean you need to go do some work at 8 pm. Give yourself and your family some uninterrupted time

Those items above are probably general to anyone working from home. As a tester are there any particular challenges? One thing that may be a challenge is access to your test lab or test devices. This might be the time to investigate Sauce Labs or CrossBrowserTesting or similar services as options to get access to the devices that you need. Another challenge testers may face is in communicating with developers about issues. I will often go over to a developer’s desk to discuss, debug or demonstrate an issue I’ve found. This can be more difficult when working remotely, but certainly using tools like zoom or google hangouts to do these things virtually helps a lot.

I also firmly believe that quality is a team sport. When working from home, testers need to remember that we aren’t working off on our own, finding bugs and throwing them over the wall at the developers. I think as an industry we have been making positive strides towards integrating quality into the entire development life cycle. Let’s not let this set us back to a mindset of developers develop and then give to testers who find all the bugs and give it back to developers. This kind of ping-pong approach of bouncing things back and forth over the wall between test and dev is not an effective way to create high quality software. Quality is a team sport and so in times of ‘isolation’ we need to stay connected as teams to ensure we are focused on this together.

Talk about testing and quality as a team. Meet with developers to talk about things to consider for stories they are working on. Pair up with developers to help grow quality skills throughout the team. Stay involved in design and story definition meetings. Keep working on having quality thinking permeate the entire development process. Working as teams remotely can be challenging, but stay focused on ensuring that you are indeed working as part of a team.

Take care of yourself in these crazy times in which we live, and happy testing!

 

Photo by Dimitri Karastelev on Unsplash

Speed Limits in Software Development

Speed Limits in Software Development

I took the train into Toronto last weekend and without my car spent the weekend walking everywhere. As I walked under the highway, I was struck by how obsessed we are with speed. I could hear the thump, thump of car tires as they rushed by over my head at 100+ km/hr (traffic was good for once). As a species we have gone from being able to move at the speed our legs could carry us, to using animals to help us move faster and then eventually we started using mechanical devices like the bicycle to increase our speed. With the invention of mechanical engines we’ve added even more speed. Our cars will rush us from place to place in almost the blink of an eye. We can listen to an audio book about a pioneer family taking months to cross a part of America and we can’t even finish the audio book before all the country they covered has all rushed passed the windows of our car. What used to take months, now takes hours.

But even with all that speed, there are limits on how fast we can go. Jump on a jet plane and you can get across a country or continent pretty quickly, but it still takes time. Physics still applies. I could wax philosophical on our obsession with speed and what we loose from it – and I may yet do that at some point – but in this article I want to move us into thinking about the limits of speed in software development. It is an axiom of modern software development that faster is better, and like I said that is a philosophical discussion for another day, but how fast can we actually go? What are the limits of software development? The speed a developer can type? I don’t think anyone is advocating instantaneous deployment on each keystroke a developer does. If not that, then perhaps the limit is how fast we can get a minimum viable feature completed. By how do we know if it is viable?

You might be able to guess that I am going to argue that quality matters. What is the fastest we can move in software development? That is, of course, not quite the right question. How fast a can a car go? Well, if it is an F1 race car, pretty fast. If it is a car attempting to set a land speed record, even faster. But those things aren’t really what we are asking about. For the vast majority of us, the answer to how fast a car can go has to do with the posted speed limit and not with the physical limits of our car. Someone engaged in a police chase can go pretty fast, but it certainly wouldn’t be safe to do so. When we are trying to figure out how fast we can go in software development we aren’t trying to figure out they physical limits of it (deploy on every keystroke). We are trying to find how fast we can safely deploy.

For software development there aren’t road signs telling us a safe speed to deploy at, but perhaps we can extend the driving metaphor a bit more to help us think this through. One thing that relates to safe speed is responsiveness. A slick road makes it harder for your car to respond to changes in direction, and slow deployment makes it hard to respond to problems with your application. How easy is it to respond to issue in your application? Don’t forget that an F1 race car with new tires and perfect tuning can respond a lot better than the little commuter car you might have. We can tune our code and deployments and get better at responsiveness over time.

If the road is foggy and you can’t see where you are going when you drive, I hope you slow down. If you can’t see what is going in your application and understand how it is being used, I hope you slow down.

Let’s think a little more about the foggy road analogy though. Sometimes we think of testers as fog lights. We pierce the darkness a bit and show where there might be threats. That is helpful when we are stuck in the fog, but in software we have an option that we don’t have in car driving. We can dispel the fog. We can make our applications more observable. Fog lights are helpful but sunshine is even better.

So how fast can we go in software development? Well, in the ideal case if we know everything and have a smooth path ahead of us, pretty fast. I don’t think we can get to a land speed record since software development doesn’t often involve going in a straight line, but with a bit of work on the code and deployment process and with investment in observability and operations, I think we can go pretty fast, pretty safely. Just be careful. Don’t try to drive your homemade hot rod as if it is an F1 race car. Go fast and push the limits, but know where those limits are for your situations (and then make them better).

 

Photo by Adrien CÉSARD on Unsplash