Deep Testing

When I was working on my API testing book, I noticed that the internet is broken.

The Internet is Broken?

Of course, the internet works just fine for many things, but in one particular way it is broken. The problem is, this broken piece of the internet is making it hard for you to advance in your career as a software tester.

We have long known in the software testing industry that testing skills are not usually taught within the traditional educational institutions. Instead we have managed to use the internet to learn. But what if the internet is broken? Then what do we do?

This article will be longer than most that I post on here. Partly because I think there is a lot to say on this subject, but also because one of the points I want to make is that shallow content isn’t always helpful. I want to explore this topic in a bit of detail. Stick with me and I’ll do my best to make this an interesting ride!

I want to look at this in 2 parts. First of all, I want to look at how and why the internet is broken. As we explore this, we will look at the history of filing cabinets (yes, you read that correctly), tool vendors, and of course social media.

Once we understand how the internet is broken we’ll turn to part 2 where we will look at some ways to fix it.

Part 1 – The Internet is Broken

As testers we often enjoy finding bugs. So maybe it’s not surprising to hear a software tester say the internet is broken. We’re always going on about how things are broken. Don’t just tune it out though. Let’s dig into a why I think this and then explore a few possible cause for it.

I’m a broken record when it comes to testing

How is it broken?

Before writing my book, I had done a lot of API testing and had even put together a couple of introductory level courses on API testing with various tools. I had a pretty good grasp of API testing. I had also used Postman quite a bit and so while not a world leading expert on software testing with Postman, I wasn’t a newbie either.

However, while writing the book, I had to do a lot of research. As I dug into things I realized that my knowledge of a lot of these things did not go very deep. And that is when I discovered that the internet was broken. I started searching for answers to some of the questions I was having, and they just weren’t there. This tweet might best sum up what happened:

All the search results kind of seemed the same. If I was looking to figure out that first surface level “Hello World” style of thing, the internet could give me all kinds of information. But if I was looking to deeply understand how authorization options worked in Postman, or if I wanted to know the nuances around different assert types, I would come up empty handed. More than once, I ended up reading through tests and source code to find the answers I was looking for.

I could find pages of results that would help me get started (although even then, for true beginners it’s pretty confusing), but it was very hard to find detailed information on a particular subject.

The internet had failed me.

Well actually, not really the internet. It was the search engines that were failing me. Let’s see if we can understand a little bit about why this was happening.

Why is it broken?

My dad was a pig farmer, and as farm kids, my brothers and I got really excited when he would take us to the pork producers convention. This was a trade show where various vendors would set up booths showcasing their products. As child I loved the free pens, stickers and candy they would hand out, but one part of the convention that I always found fascinating was a competition they had for inventions that different farmers had come up with.

Any farmer could submit their invention and have a display showing it off, and the attendees at the convention could vote for which one they thought was the best. Of course, part of the fun was to see if the invention that you voted for ended up winning. After a couple of years of following this part of the show I noticed a pattern. Every year without fail, the winning invention was incredibly simple. So simple in fact that you would almost feel like it wasn’t an invention. Of course, you would do that! The benefits were face-slappingly obvious to anyone in the industry. How could we ever have done it differently?

I think the filing cabinet was an invention like that.

The Filing Cabinet

The filing cabinet seems so simple and obvious that we probably don’t even think of it as an invention, but there was a time when they did not exist. Up until around the 1890s information was not stored in filing cabinets. However, it was an idea whose time had come and it soon swept across the business world. The filing cabinet allowed users to almost instantly find the papers they were looking for. No more thumbing through hundreds of pages in a ledger to find the entry you wanted. Now all you had to do was open the correctly labeled drawer and go to the folder you wanted and thumb through a few sheets of paper to find the one you were looking for.

But what does all this have to do with the broken internet? Well, as Craig Robertson argues in his book The Filing Cabinet: A Vertical History of Information the filing cabinet has influenced the way that we think about information. We think of it in terms of efficiency and indexing. It’s not the only way we think about information of course, but it certainly has coloured our view in a lot of ways, not least of which is in our search engines.

Indexing is a multi-billion dollar industry. Google makes it’s money by using software to index the world for us. It’s much faster and more efficient than the lowly filing cabinet, but it serves much the same purpose; allow users to almost instantly find the information they are looking for.

But isn’t that a good thing?

It sure is, if you are trying to find a certain piece of information. Forget about Google for a minute and jump back 100 years with me. Imagine going into an office building and asking a file clerk for information on how those newfangled Model T Fords are made. Thanks to the filing cabinet, the clerk can look up all kinds of things for you. They can quickly find the plans for the car, they can give you stats on how many were made and where they are getting sold. They can tell you all kinds of information, and get it to you quickly too.

You are now ready to create a company that displaces the Ford Motor company right?

Or maybe not.

Jump back into the present again with me. You want to learn how to be a software tester and so you ask your personal file clerk Google how to test software. You get back all kind of data. Blog posts. Tool vendors sites. Stats on how many testing jobs there are. Tutorials on using a tool. Opinions about testing.

You are now ready to be a successful software tester right?

Or perhaps, like the filing cabinet, indexed facts and data aren’t enough. Maybe the skills required to test software are a more complex puzzle then can be solved with a couple of hours of googling. Indexing prioritizes accessibility and speed over complexity and depth. The problem is successful software testing (and indeed all knowledge work) is complex and requires deep knowledge. When our primary information source de-prioritizes complexity and thoughtfulness – well you end up struggling to get the skills that you need.

Tool Vendors

In 1996 Bill Gate wrote an essay titled Content is King. It’s not quite what Bill Gates had in mind, but marketers have taken this to heart and now content marketing is a key pillar of many company’s marketing strategies. Combine that with search engine optimization and many tool vendor’s content can dominate the first page of Google results. This isn’t all bad and I don’t think there is anything wrong with tool vendors doing this. In fact, if done well, it can be a service to the community. I’ve written a few post for tools vendors myself.

The challenge is that this type of content – for very understandable reasons – is pushing you towards purchasing rather than learning. Again, nothing wrong with purchasing tools. Tools are a great way to help you get better at your job, but when content is created for “content marketing” purposes, it may not be helping you get to the kinds of depths that you need to be successful in your career as a software tester.

Social Media

And then there is social media. I think the “social media is destroying us all” horse has been beaten nearly to death, so I won’t go on about this too long, but it would be remiss of me to not mention this elephant in the room. Social media, being what it is, encourages silly debates and shallow engagement. It preys on emotional responses and doesn’t push us towards depth.

I don’t think I’m saying anything new or particularly insightful here (and certainly there are helpful things on social media), but if you are trying to figure out how to have a successful testing career, I don’t think the social media platforms hold the solution.

Part 2 – How to Fix the Problem

Ok, so anyone can complain about all the problems with the world, what about proposing some solutions? Problems like the ones I’ve mentioned above are systemic, cultural and large scale in nature, and so certainly there are no quick fixes for them, but I’ll share some steps that you can take that will help tip the scales in your favour as you seek to grow in your ability as a software tester.

Ground yourself in the basics

The internet encourages us to live in the never ending now. The latest news cycles or Twitter debates take our attention. We don’t have time to think about old things. Everything is always new. The tech industry has also continued to grow and expand dramatically which means that there are many new testers.

There are also a lot of new things to know about and use in testing. There was a time when UI test automation pretty much just consisted of figuring out how to use Selenium. Now we have a variety of UI automation tools that are all popular. We also expect testers to know about API testing. And how to use containers. And integrate with cloud platforms like AWS, Azure or GCP. And use cloud platforms like Sauce for cross browser testing. And accessibility testing. And don’t forget about security and monitoring and telemetry and performance and the list goes on and on. We have so much to know that it gets overwhelming and we end up just scratching the surface on all of these things.

All of these tools and ideas are wonderful and powerful, but we need a framework around them. We need to be able to understand which ones to use and when to use them. We need to understand why they might help us in solving the problems we face as testers. In order to get this, we need to also be grounded in the fundamentals of testing.

How do you decide what to test? How do you document what you’ve tested or are going to test? What are some test techniques you can use? How do you report on the testing that was done? How do you deal with the human part of the human/software relationship?

I was recently looking through my RSS feed and I noticed that many of the testing blogs I follow haven’t posted in a while.

Yes, I use RSS feeds

I went to some of them and tried to figure out what the blog owners were up to. Many of them have moved on to other positions. Managers or directors, or CTOs or stuff like that. It is great to see people moving up in their careers, but it made me feel a bit nostalgic. Maybe I’m just turning into a grumpy old man, but I kind of miss the days when it seemed that learning about basic elements of software testing was the cool thing to do.

There is a lot of richness and depth to the “basics” of software testing, and while we certainly shouldn’t fetishize them or dismiss new tools and ideas, we would do well to build on a solid foundation.

However, I think there is more to it than just learning the fundamentals. I think we need to contextualize them. If you read older books on software testing, they will sometimes use examples of things like testing dot matrix printers or drivers or even monitors and screens. The underlying fundamentals of how to think about and do good testing haven’t changed, but the ways in which the modern tester is required to put them into practice have!

We need to learn these without getting lost in the weeds of learning them in the abstract. There is a kind of person (I am such a person) who can get lost in the idea of things just because they are fun and interesting and learning about them is intellectually stimulating. That’s not what we need. We need testers who know how to test in the modern world of testing. This means we need testers that understand the fundamentals – but more than that – testers that understand how to apply these fundamentals in the daily work they are doing.

Fundamentals of Software Testing

So, what are the basics of software testing? I tried to map out what I thought they were, and the map got pretty big pretty quickly. There are a lot of concepts and ideas that relate to software testing! After a lot of wrestling with it, I distilled it down to the following list. At the most basic level I think we have:

  • Why test? – Understanding why we even invest in testing at all
  • What to test? – How to come up test ideas and how to do test planning and design
  • Communicating what you could test – How to write down test plans
  • How to test – How to actually go about questioning the product and evaluating the answers
  • Communicating what you did test – Test reports and bug reports

I think these boil down the basics of testing, but there are a lot of other pretty basic concepts to explore as well. These include things like different test techniques like boundary value analysis or risk based testing. There are also concepts like heuristics, oracles and mental models that are useful to be aware of when thinking about software testing. Then there are different software development methodologies, non-functional testing, and how to effectively use tools and automation.

As you can see, there is plenty to learn when it just comes to the basics of software testing. So in a world where the internet is broken how do you learn all of these things? Well, this in’t a blog on pedagogical methods, but one of my roles is to teach. I teach some testing courses and I am a homeschool dad, so I have thought about and paid attention to how people learn. I’m not an expert on these things, but I have a few suggestions that I hope will be helpful.

Try things

I am a firm believer in learning by doing. Learn a little bit about a concept I’ve listed above, and then try it out. Put it into practice and see how it works out for you. Until you try it and see where it is hard and where it works and where it doesn’t, you won’t really understand it.

Try things you aren’t ready to do

There is a bit of a chicken and egg problem when it comes to learning new testing skills. In order to learn the skills you need to do the testing work, but in order to do the testing work you need to have the skills. The way out of this dilemma is to do things you aren’t ready for yet. In order to get good at something, you have to first be bad at it. Part of learning by doing, is trying things before you feel ready for them. Don’t get in too deep over your head, but if you only stick to the things that feel comfortable you won’t make a lot of progress. You have to try things that you aren’t ready for yet and fail a few times.

I remember trying to write python code to automate some data generation that I wanted to do. I hadn’t been using python for long at that point and so I got stuck on something that I wanted to do (I don’t even remember what it was now), and I struggled with it for about a day. No matter how much searching I did, I still couldn’t figure it out. I asked some of my colleagues for help, but when I did the things they suggested, I didn’t really understand how it worked and so I would end up getting stuck again. I was in over my head, trying to learn something I wasn’t ready for and so I finally gave up and did the work the boring, slow manual way.

However, a few months later I need to do some similar data generation. With some trepidation I decided to try scripting it again. I ran into some of the same challenges, but this time I was able to figure them out and actually understood why things were working. The struggle of trying to do something that was “too hard” for me had helped me learn.

Longform Learning

Learning by trying is important, but you can’t go in too deep without having at least some grounding. I find the best way to learn is to try something out to figure out where my gaps are. Once I get stuck though it’s time to turn to learning some of the basics. I find that blog posts, articles and short videos (i.e. the things that come up in google searches) are helpful for filling some gaps, but eventually you need to figure out some of the foundations of a subject. Unless you don’t really care that much about getting good at it. If your goal is to just learn the subject at a surface level, Google results are fine. If you want to master a subject, or at least use it professionally, you need to go deeper then the first page of Google results. You need some longform learning

Books

Books are a great way to dig into a subject. Software testing has been around for a long time. Although the world has changed and software testing has (correctly) changed to accommodate it, there are still many foundational principles that you can learn from books. In fact, reading some older books can be a helpful learning exercise if you try to take the principles out of them and figure out how they can apply to the world of modern software testing.

So what books should you read? Well, asking this question made me realize I have some reading to do. I have a lot more books that I would like to read. Here are a couple of books that have been helpful to me on my learning journey:

These are books that deal with a lot of the basics of testing, but there are others that I would like to read as well. I was looking through my list of books that I’ve read and I realized that it has been over a year since I’ve read a book about testing or quality (The last one was Leading Quality since you wanted to know). I certainly don’t want to only read books about quality, but it’s time to get back on that train. Time to start working through my software testing wish list again.

Workshops and Training

There are a lot of free webinars and talks. Those can be helpful, but since we are talking about deep learning, you can probably predict that I’ll say they aren’t enough. A 40 minute talk can’t get you deep enough to dig into learning. Online courses can help you go a bit deeper, especially if they integrate a lot of activities (and you actually do those activities).

In depth learning takes time though. Half day or longer workshops and training sessions have a much better chance of taking you into deep learning than a one hour webinar or conference talk. This type of learning can get expensive, but if you are able to afford it (or to convince your company to pay for it), these things can help you get real depth into your learning journey.

Community

When learning is social it is much more fun. I preach this to myself. I have never been particularly good at keeping up relationships and community and it has only been to my own detriment. There are so many smart, talented and passionate testers. We need to feed off each other’s energy. There has been a lot of great work done around building community by the Ministry of Testing but whether you plug in there, or just find a few other passionate testers at your company to hang out with, community is a great way to invest in deep learning.

But I need to know for myself; how do you become a part of a learning community? Well, let’s think this through together.

Find the watering holes

Where do people hang out and talk about testing? Here’s a few places I’ve found:

Slack groups

I’ve found the slack group built around the AB Testing podcast to be a good place for community. It’s a moderately sized group (Under 1,000 currently) so it has a manageable post rate.

Another helpful slack group is the Ministry of Testing group. At close to 17,000 people, there can be a lot of posts. I actually stopped checking on it for a while because I found it overwhelming (Maybe its just my OCD around not leaving unread messages). It is a good place for building a learning community though, so I’m going to try again. I think I will look for sub-communities on there so that I can get to know some people with shared interests. Join me there. Let’s do this together.

Social Media

I know. I know. I just finished complaining about social media above. However, social media isn’t all bad. It can be used as a way to make new connections and to stay in touch with what others are up to. I find myself craving in person interaction though. Covid has been a long, long slog for me. It has been nice to be able to feel like I have a bit of an idea of what other testers are up to by seeing Twitter or LinkedIn posts, but as time has gone on, I’ve been craving in person interactions. I need to see my fellow testers in person. The passion and energy that drives us to learn and grow is best absorbed in person.

Conferences

So let’s talk about those in person interactions! Conferences are the best place I have found for that. I’ll be honest, unless you are able to go to an in depth workshop or tutorial, you probably won’t get a lot of deep learning from conference talks. However, that doesn’t mean that there is no value in conferences. I think that their real value comes from making connections with others that are passionate about creating good quality software. It comes in the hallway conversations and chats at the pub.

The difficulty is that conferences are expensive. Tickets alone can be expensive. Then you have to layer travel, accommodations and food on top of that. Plus, there is the time investment of going to a conference. As expensive as they are, if you can make it work I think the value is often worth it. As in person conferences open up again, I’m really hoping to go to a few. So far I have only booked to go to a local conference (KWSQA) which I will be speaking at. If you are going to be there let me know. I’d love to hang out!

Teaching

Speaking of speaking (how’s that for a phrase?), teaching others what you know is a great way to both build community and learn. I started blogging as a way to learn. Sharing my thoughts helps me learn. Blogging is also a way to be a part of the community of testers. I know blogging, speaking or other ways of teaching aren’t for everyone, but if you are up for it I would highly recommend it as a way to help you skew towards deep learning.

Short Form Learning

In an effort to contradict myself – a paradox makes me seem smarter right? – I want you to also do some short form reading. Yes, the internet is broken. Yes, it is indexed on shallow content. Yes, you won’t get far in building testing skills if you bounce around from search result to search result.

But that doesn’t mean these things are useless. If you have a particular topic you are trying to learn about, long form learning helps you dig into it. There is a downside to this kind of learning. You can get too influenced by a limited set of inputs. If you are trying to deeply learn something, it can be really helpful to expose yourself to a number of different perspectives and ideas.

Part of deep learning is spending some time doing a shallow survey of different ideas that are out there. I don’t want you to give up on blogs or following people on LinkedIn. Use the shallow nature of most internet content as a stimulant. Take the ideas you get exposed to and try to flesh them out on your own (maybe write your own blog post about them). Play around with the ideas you come across. Combine them with other ideas. Analyze them and reject them. Or accept them.

Use the shallows of the internet as a path into deeper learning and maybe we can save the internet.

Happy Testing!

Photo by Louis Hoang on Unsplash

Published by offbeattesting

I am a highly skilled tester with years of experience in using exploratory testing to find bugs. I am also good at solving automation problems and am a strong team contributor. In addition to the analytical and exploratory skills required by any good tester, I also have strong interpersonal skills and am an avid learner, quickly picking up new things. I blog at offbeattesting.com as a way for me to never stop learning.

Leave a Comment