Working with TestProject

Working with TestProject

Note that this is a sponsored post. Posts like this will only ever be about things that I believe and that I think you would potentially find useful. You can read about my full philosophy on sponsored posts here

I always manage to keep a full plate in life. I have many interests and responsibilities. Sometimes it can feel like life is too busy, but I really do enjoy the many different things I do. One interesting project I have been working on lately is a collaboration I have been doing with TestProject. Their goal is to provide the testing community with a community powered platform that helps with recording, developing and analyzing test automation.

I have been working with them on creating some video tutorials that explain how the tool works and highlights some of its benefits. If you want to check out some of the videos I’ve made, you can go to the TestProject page and click on the “Watch how it works” button to hear me talk about the high level overview of the product, or you can go to the videos section to find my other tutorials. There is also a YouTube playlist that has them all in one place.

As I have been putting these together, I thought I would stop and reflect a little on this process. I want to look at it from two perspectives. In the first place, I want to reflect a little bit on the tool itself and in the second place I want to talk a bit about what it has been like to work with the TestProject team

Ok. So the tool itself. I find that I learn the most about something when I teach others how to use it. In making these tutorials I have been learning a lot about TestProject and I have to say that overall I am pretty impressed by the tool. I have been making some videos, for example on the smart test recorder. UI automation is notoriously hard to do well, but I have to say that the smart test recorder really does a good job of setting you up for success. It makes creating test scripts straightforward and easy, and it also does a good job of creating the necessary levels of abstraction for you. For example it automatically saves elements for you so that you can share them between tests. It also allows you to call one test from another test, which allows you to follow the ideas of the page object model. We usually think of the page object model as something you would do in coded tests, but TestProject makes it easy to take this test design principle and put it into action in high level UI tests as well.

There are also a lot of other helpful things that TestProject does. It takes care of all the dirty work of getting the selenium and appium drivers installed and setup and is honestly the fastest and simplest way I have come across for getting you started with tools like this. I have started using TestProject to automate some of the boring things I have to do. For example, if I want to start doing some exploratory testing on a new build, I have a TestProject script I can run to setup some stuff in the way I want it to, so that I can just dive right into the part of the testing that I am currently interested in. This is something I have done with Selenium in the past, but I found it be a lot easier to get started when using TestProject.

I think this could be especially powerful if you don’t know how to code since you can use the smart test recorder to set this all up for you with very little difficulty (and without needing to know a single line of code). You can of course also use the TestProject SDK to create coded tests or even export your recorded tests to code if you wanted to dive a little deeper.

TestProject also works well for creating regression tests that you run across browsers and platforms. It instantly recognizes any devices you have attached and lets you create and run tests on applications you have on those devices as well. It also does all this in a way that is designed around collaboration. They really are seeking to provide a solution that is team based and you can see that in the way that all tests and run time environments etc. can be shared with your team.

The community driven aspect of TestProject brings me to the second point I wanted to talk about which is what it has been like working with the TestProject team. I don’t have a ton to say on this front, but I did want to mention what a great experience it has been for me. The team members I have interacted with there have really made me feel like my input and expertise is valued and important and I can see in those interactions why they have a free forever plan. They really are committed to making things better for the testing community. Just to be clear the free forever plan isn’t just a ‘give you just enough to make you want to upgrade.’ It is something that would be really valuable just on its own including things like unlimited test execution and access to all important functionality like the addons library, smart test recorder, reporting and much more. I’d encourage you to check it out and sign up.

I really think this a good tool and a welcome addition to the list of testing tools out there. It has been a lot of fun working on creating tutorials that explain what it does and I’m looking forward to continuing to learn and use this tool.

Why Create Bug Free Software?

Why Create Bug Free Software?

I’ve been talking about creating bug free software in my last two  posts. In this article I want to think about why we might want to do this.

Bug free software (as I have narrowly defined it in these posts), is valuable. If you can really deliver software that solves a customer problem in a way that they agree does not create any bugs, you have something that is incredibly valuable to them. However, it will most likely also be expensive. Figuring out what exactly the customer needs and working with them to deliver exactly that, is cost and time intensive. We all know that good testing takes time.

I think there is a place for dollar stores and other providers of low cost goods.  There is also a place for low cost providers of software solutions.  If all you need is a quick and dirty solution that doesn’t last long, a piece of software with a lot of issues that ‘bug’ you might be just fine. But if you are dealing with a long term and important problem that can be solved with software, do you really want a dollar store solution for it? Is it worth it to go back to the dollar store every two day to replace that broken item?  Or is it perhaps worth getting a higher quality item that will last and do exactly what you need it to?

I think this metaphor can be applied to software.  If your software solves a problem well, and does so in a way that doesn’t feel like a dollar store toy that breaks 2 minutes after your kid starts playing with it, you have something that is valuable.

When you have something that is valuable you can get people to give you money for it, and then you have a successful business. So caring about quality, and the effort that goes into it is a great way to create value for the business. Taking time to invest in good automation and learn good coding practices; taking the time to reach out to customers and understand usage patterns; taking the time to keep the code base understandable; taking the time to find and fix things that prevent customers from efficiently solving their problems; all of these things contribute to having valuable software.

I very often see companies getting caught up in the need to release new features and make more exciting stuff that they can market and sell.  Those things matter and are ways to add value and make money, but we need to remember that working on the quality of our existing solutions is also a way to add value.  The better we solve the customer’s problems that more valuable our software will be. And I think it is a pretty simple inference to say that the more valuable our software is, the more potential there is to make money.

So why create ‘bug free’ software?  Well in a nutshell, because it is valuable.

Let me close with one parting thought: How valuable would it be to the company to have someone (say a tester) that can demonstrably help them get closer to that bug free state?

 

Photo by Niels Steeman on Unsplash

Bug Free Software at Scale?

Bug Free Software at Scale?

In my last post I proposed that it is possible to create bug free software if the context is very limited and the number of stakeholders who determine if something is buggy is small.  In the discussion on that post in various forums we fleshed out a bit more the idea that this is limited to a snapshot in time, since new context or stakeholders could change the things that are considered to be bugs.

Can this apply at scale though? If you have a large scale software product with thousands or millions of users and stakeholders can it be bug free?

I think here the answer is no. But I think it is a softer no that I would have used before

No, you can’t simultaneously satisfy the competing demands of many different customers. But there are still some powerful techniques that we can use. If we use the same definition of a bug that we did in the last post. (i.e. something that is a threat to stakeholder value), we can actually get pretty close to a bug free state. If we understand enough about what our customers value and aggressively fix things that get in the way of this, we can make very valuable products. We might not ever get to a fully bug free state, but we can get pretty close.

I guess in way what I am saying is that a 0 bug backlog way of working is a possibility. If you are willing to put in the work. They key here is understanding that a bug involves a relationship. Bugs don’t exist without people. They don’t exist if they impact something no-one cares about or uses. If we think back to that limited scope bug free project in the last post, there could be any number of things that are in there that they programmer did not intend the software to do. But if those things do not affect the stakeholder, are they really bugs? A bug isn’t a bug, until it impacts someone that matters.

Working from this, leads to an important point about being able to achieve quality at scale. We won’t get to bug free, but we can get to high quality if we focus aggressively on fixing actual bugs. A lot of times we end up working on things that we think are bugs, or things that might be bugs, but we don’t consider the relationship aspect of a bug. Who is this bug bugging? Why does it matter?

As a testing profession we have tended to focus on what I’m going to call ‘potential bugs.’ These are issues like logic errors or crashes or really anything that gets filed in a bug report, but they are found before they impact any of the stakeholders. I’m not against finding and fixing potential bugs, but we need to remember that there is importance to finding and fixing actual bugs. Bugs that are currently bugging someone, not just bugs that might bug someone if we release them. We need to find and fix things that are actually, right now, threatening the value of our product. We need to get rid of the things that are currently bugging someone. As the proverb goes, a bird in the hand is worth two in the bush.

This also means that the goal of a nearly bug free product will lead us to prioritize the fixing of these actual bugs over creating new features. Businesses don’t always like this, especially in a highly competitive landscape where time to market for features is important, but if your goal is have a high quality product, you need to prioritize in this way.

This gets expensive. It takes time to find and fix bugs. Time that could be spent on making new features. It also add a lot of interruptions. If we prioritize fixing actual bugs over feature delivery we are going to sometime end up in a responsive mode. Remember the comments about how ‘bug free’ is a time limited snapshot? Our system and users and environments change over time and so even though we might be bug free today, that does not mean we will be so tomorrow.  Again this approach is expensive. It has real costs associated with it.

And that is what I want to talk about in the next post in this series. The interplay between the cost and value of creating ‘nearly bug free’ software. Is this cost worth it?

What do you think?  I’ll share my thoughts in the next post. Stay tuned!

 

Photo by rawpixel on Unsplash

How to Create Bug Free Software

How to Create Bug Free Software

In a recent post to his email list, Jonathan Stark talked about giving guarantees for your work.  He gave the example of offering a “bug free” guarantee for software development work. All the testing hairs on my arm stood up and I wanted to protest that there is no such thing as bug free software!

But then I let another testing instinct kick in – curiosity.  I started thinking about how he can do this. How can he offer a guarantee that he will deliver bug free software? Is there such a thing as bug free software?

I came to a conclusion that yes there is.  There really can be bug free software.

The context for that statement has to be just right though. If you are delivering a piece of software that has only one (or a very limited number of) stakeholder that gets to decide if something is bug free, I think you can indeed make the software bug free. There might still be logic errors in the code somewhere. There might even be things that you as a tester could find that are ‘wrong.’ But if the client agrees that the software does what they need it to in a way that is reasonable for them and if they agree that there are no issues that prevent them from getting the value they need out of it, you have indeed delivered bug free software.

I think a bug is something that threatens the value of the product. If you have a limited scope product and limited number of stakeholders, you should be able to make something that has no threats to it’s value. You would need to do this through careful consultation and iteration with the customer. They would have to tell you where they are not getting the value they need and you would need to work closely with them to remove all bugs (that matter to them) from the product, but it should be possible to do.

I want to try a more difficult thought experiment though. Take this into general purpose software with many thousands or millions of users with competing demands, skill levels and expectations. In this context can you create bug free software?

I’m going to take up that thought experiment in the next post. Stay tuned!

 

Photo by pan xiaozhen on Unsplash

 

How Good Testing Improves Security

How Good Testing Improves Security

Security and data breaches are probably one of the biggest risks any product faces. One public incident can take a company down or at least have a huge impact on it’s bottom line. Not to mention what something like that does to company morale. It would follow then that security testing is very important.

However, I don’t know if you’ve looked into security testing. It is a complex and intimidating world to get into.  You need a lot of specialist knowledge and a high degree of technical competence to be an effective security or penetration tester. Not all of us have the time or inclination to learn what it takes to get into this field, but that doesn’t mean we can’t contribute to security.

Good testing leads to a more a secure product. There are many security concerns like cross site scripting or packet sniffing or SQL injection that you probably won’t find without a least some specialist knowledge, but there are many other things that contribute to security.

Understandable Code

If you have code that you can understand, it will be much easier to see possible security flaws through things like code review. Cleaner code could also make it easier for security analysis tools to find flaws and vulnerabilities.

Testing can of course contribute to understandable code. Even if you don’t know much coding you can contribute. Look at code reviews and even if you don’t know how to write code you should be able to sort of follow along with it.  If the variables, methods and classes are well named and the code is well organized you should be able to least feel like you basic grasp of what is going on.  I’m not talking about finding issues in the code review, I’m just talking about able to kind of get the gist of it. If you can’t even do that, it might be sign that the code is unnecessarily complex or that things are not well named and structured in the code.  You don’t need to be a programmer to be able to contribute to the team having understandable code

Sometimes all you need to do is ask a question.  Why do we do it this way here?  What is happening here?  You would be surprised at how simple questions like that can sometime expose unclear code and even if they don’t you’ve now learned a little more about the system you are testing and that certainly isn’t a bad thing!

Exploratory Testing

Many security breaches boil down to less technical things than something some black hat hacker does.  For example read this article about mining public passwords using google. Doing this required a very minimal amount of technical expertise in using google, and did not involve any of the highly technical things we often think of when we think of security hacking.

Finding ways that we could accidentally expose private information is a creative and exploratory exercise. These are skills every good tester has in abundance and we can use these skills to help prevent security breaches.  Think about creative ways people might accidentally use your project or about ways that people might try to get around limitation you have in place.

Good exploratory testing can expose many security flaws. Just knowing that code will be looked at with this multifaceted approach can help everyone in the development process think more carefully about those less than ideal ‘users’ that our products might have

Monitoring

There are of course many other ways that an emphasis on quality can help reduce security threats, but at the end of the day it is going to be very very hard, if not impossible, to ensure that there are no security holes anywhere. This is where monitoring comes into play. We need to think about how we would detect and quickly shut down any issues that might arise. Thinking about how things might go wrong and what kind of things we could check to know if has happened is also very much a testing activity and so here again having good testing in place on your product helps improve security.

So, don’t leave out the importance of targeted security testing on applications, but at the same time don’t forget that a big part of having a secure product is just having a good quality product and code base. Those things don’t guarantee security, but they can help improve it!

 

Photo by Tomas Sobek on Unsplash

 

Release More Defects

Release More Defects

What do you think is the biggest thing that holds back the quality of your product? Is it poor code quality?  insufficient testing? Not enough time for testing? Missing designs? Manger, tester or developer apathy? If you had to pick one thing, what would it be?

I don’t know if this is the biggest thing, but one threat to product quality is an inability to release defects.

Stay with me ok? I’ll explain what I mean!

As a tester I’m about making this world a better place one dead bug at time, and in order to kill those bugs you need a few things. You need to find the bug, you need to know that it is making the world a worse place, and you need to get rid of the bug somehow.

Bugs can be very sneaky. Sometimes it’s easy to find them, but let’s be honest, the sheer number of testers employed in the software industry is a good indicator of how hard it can be to find bugs. They like to hide in the most unexpected places and appear at the most unexpected of times.

And there are a few other things you should know about bugs. For example, not all of them make this world a worse place. You can have a bug (something like a logic error) in some code that doesn’t get used and the world couldn’t care less.  You can also have a bug that doesn’t look like a bug. It might be working as designed, but the designer and the user have different perspectives as evidenced by the number of swear words coming out of the user’s mouth. This boils down to the fact that bugs involve a complex relationship between software and humans. It is very hard to know what kinds of bugs make this world a worse place. In fact, just to make things more complicated consider the fact that sometimes a bug for one person is a feature for another!

Customer

So these things leave testers with a challenge. If we fix (or advocate for fixing) the wrong bugs we aren’t achieving our mission. If we miss finding a bug at all, well pretty clearly we aren’t going to get rid of it.

Enter Stage Left: The Customer.  This is the person who’s problem our software system is solving.  This is person who’s life we want to make better. It is through helping and serving this person that we are making the world a better place. This is the person who is the ultimate arbitrator of what in your system is a bug. A bug your customer doesn’t run into isn’t really a bug at all.  A bug your customer doesn’t care about doesn’t make the world a worse place. The customer is the one who should decide what bugs you spend time fixing.

So how do you improve the quality of your product? You fix the problems that your customers care about. Sometimes it is pretty easy to hypothesize that your customer will care about something. A security vulnerability for example. Other times, it is much harder to know. How are they even going to use the product? Is this issue I found going to be relevant? Part of the calling of a good tester to is be able to come up with reasonable answers to those questions. This means we need to put ourselves in the shoes of our customers.  We need to figure out how to think like them and see the product through their eyes.

Tools

If this is indeed one of the primary challenges of the testing profession (and I think it is), what are some tools we have at our disposal to help us with this? Is it all just guess work, or can we approach it in a bit more of a scientific way?  The truth is there are some really good ways to learn more about what your customers care about.  There is a lot of new technology that can help with this, but don’t forget about some of the tried and true methods. There are ways to learn about out customers that we have had for many years. Customer service representatives for example. Or your competitors products.  Why are people buying them? What do they have that you don’t? Then there are customer surveys and usability studies and external bug logs and customer posts on forums, blogs and social media….and the list goes on.

I already mentioned new technology and so let’s not forget about this. We have tools like telemetry and data analysis that can help us answer questions about customer quality. We can log and analyze errors, and automatically alert based on them. We can use tools build right into our cloud hosting systems that give us real time information on what our customers are doing. There is the potential for reams of data that can help us in this quest to figure out what things are causing issues for our customers.

There are many, many places that we can mine for information about the things that our customers really care about, but how does this all relate back to releasing more bugs? Well, remember those tough questions we have to try and answer as testers? The questions about if a bug is really a bug and if it worth fixing?  Well, we know the customer is the best person to ask those questions to right? and letting them see or experience the bug is the best possible way to figure out if it matters to them.  If we think of it this way a released bug isn’t a horrible blight on your ability as a tester, it is a way of asking the questions you need to ask so that you can figure out things you just don’t or can’t know.

Caveats!

I’ve just proposed a controversial thesis, and maybe you buy it or maybe you don’t, but let’s talk together for a few minutes about how to make this strategy work. I hope I have already made it clear that this strategy isn’t for every bug.  The purpose of it is to help you answer the question on those bugs where there is controversy about the ROI to find or fix it.

There is another important caveat though.  If these are a way of asking your customers a question, you need to be able to respond to them with an answer.  If you ask your customer, is this a bug? and the answer clearly comes back as yes! then you need to respond to that and you need to respond quickly.  This is where the technical side of things comes into play.  You know those boring things called release pipelines? Yeah, they are a really important part of this whole process. You need a release process and build pipeline that allow you to respond effectively to the answers your customer are giving you.

In fact, I think the speed with which you can turn around a change, (I would define this as the amount of time your customer is exposed to the issue) dictates the kinds of bugs you can be ok with releasing. If you can turn around a change in minutes or seconds, you can be ok with more serious bugs getting out there. You can also be ok with it if you can limit the number of people affected by it.

We shouldn’t think of testing as a one size fits all process. Your build pipeline actually influences the kind of testing you need to do. The longer the turnaround time the more effort you need to put in up front in finding issues. If you can quickly detect and remedy issues that you customers find, you can actually take a lot less time up front on testing for certain types of issues.

But why do this?  I think it comes down to efficiency.  We want to more effectively and efficiently add value to our customers and taking an approach like this allows us to do that. We will spend less time finding bugs that our customers don’t care about and we will also be able to more quickly find and know about the ones that do matter. This frees us up to spend more time in adding value in other ways.

Now one last caveat and then I’ll close out this article. This doesn’t come for free. Getting your build pipeline to allow for this kind of fast turnaround can be a long and difficult process. Having the monitoring and alerting in place that allows you to actually know what things are affecting your customers is a challenging and time consuming thing (especially at first).  I don’t want to pretend that what I’m talking about here is some fairy land of magic and rainbows where you don’t have real trade-offs. I am arguing that this effort is worth it in the long run (for many companies).

We can often pursue the latest and greatest thing as if it is some kind of magic pill that will solve all the world’s problems (or at least the ones in our software department). The reality is the things I am talking about in this article can (and do!) work, but they need to be done well. They won’t work if we are trying tack them on top of old ways of working. They require mindset shifts and changes in our processes. This is why I talk about it in terms of being able to release more defects. Using these tools effectively will require counter-intuitive thinking. Embrace that and move forward. Pursue the ability to release more defects!

 

Photo by Ankush Minda on Unsplash

30 Days of API Testing – Browser Dev Tools

30 Days of API Testing – Browser Dev Tools

F12.

It is one of the most used keys on my keyboard. There is just so much power in the developer tools of modern browsers that we should not be ignoring. This is especially true when doing API testing. You can often find out a lot about the API calls your application is making directly from the network tab in dev tools.  This tool gives you insight into all the calls that being made across the network when you load a page and API calls are among them. This means you can often find those calls and see what information was sent and what response was received back. It can give you a lot of insight into how an API works within the full system.

Let’s look at some screenshots and see how this might work.

Here I have loaded the the github page for one of my repositories after having hit F12.  You can see a lot of different network calls that have been made, but just looking at the first one here we can see that it is a GET call. We can see the URL used for this and if we look at the Preview, and Response tabs we can see what this call returns

GetCall

Or take a look at the call we get when we try to change the name of a repository.  We can see in this one a POST request that includes query parameters and form data. All of this information is the kind of thing that is really helpful if you are trying to figure out how an API works and how you might go about doing further testing on it.

postcall

Being able to see what API calls are being sent from a page is very helpful, but there are other ways that the developer console can help you in API testing as well. For example, another powerful thing you can do within the dev console is network throttling. There is a throttling option that lets you simulate being on a slow connection or not having any internet. This can be a great testing tool for seeing how your APIs work in those kind of conditions. Other things you can do on the dev console include seeing console errors, performance monitoring and even blocking particular requests.

All of these tools and many other are available in the developer console and if you haven’t played around with it before you really should! It is a helpful tool for any part of web testing but it is especially powerful when you are looking at API testing.

 

Photo by Kimberleigh Aleksandra on Unsplash