Asking Questions – A Tester’s Superpower

Asking Questions – A Tester’s Superpower

“Why are we planning to work on this feature?”

It had seemed like an innocent enough question, but there was a moment of silence followed by  “Umm, the objective owner thinks that it will help performance”

Another innocent question:  “Will it?”

Another silence

“Well, we’re not sure that it will. Maybe a little bit…”

We had recently made a similar change in another area of the code (where we knew that it would help the performance), and that change had been quite costly in terms of both development and testing time and had also introduced a limitation to that area.  I brought up these points and also asked if there was any way we could answer the question of how helpful it would be to the performance without investing days or even weeks of work.

This led to a discussion of which profiling tools we had available. We pulled someone from outside the team into this discussion and found out that we had licenses to tools our team didn’t know about.  Also, in the course of  that discussion he was asking why we wanted profiling tools and the limitation we had introduced earlier came up. He then said that his team could make some changes to help support what we needed so that we could remove that limitation.

After 30 minutes of discussion the planned work looked very different that it had before I asked my innocent questions.  Instead of going ahead with this change, we are instead going to be looking at cleaning up the previous limitation and profiling some parts of the finished code that are similar to the new code we would want to add, so that we can have a somewhat data driven answer to the question of how much this would help the performance.

One innocent question led to a lot of change. If you are a tester, asking questions is one of your superpowers.  Don’t be afraid to speak up if something is confusing – you never know how one question might change things!

Practical Steps to Increase Coding Productivity

Practical Steps to Increase Coding Productivity

In my last post I want onto my soapbox and shouted for while about how testers need to consider increasing coding productivity as one of their goals in the work they do.  It’s always fun to go on a rant for a while, and hopefully it was even helpful for some people, but I’ve jumped off of the soapbox now, and here on the ground things look a bit different.  I want to do this.  I want to increase coding productivity – but how?  What are some practical things I can do as a tester to increase the coding productivity of my team?

I want to share some of the ideas I have around this, but before I do that I need to insert a little caveat. These are ideas that I have for my team and my context.  They may or may not make sense for your team and your context.  I share them so that you can see some of my ideas and because ideas generate new ideas.  Maybe you can take some of my ideas and mix them with some of your ideas and produce some cute little idea babies.  But please don’t just blindly take what I’m talking about here and try to do the same thing in your context.

Ok that’s out of the way, so now what are some of the things I want to do to try and improve coding productivity on my team?

Fast Tests

I’ve mentioned this before and so won’t belabor the point here, but in general, faster tests equal more valuable tests.  I want to have fast tests in place.  In the short term this means that I’ll be going through the tests we have and shortening them where I can, but in the longer run this means pushing tests down to lower levels and running them on smaller units of code.  There are so many things we do with high level integration tests right now, that we ought to be able to do with code level, or unit test level tests instead

Fast Run Times

This might seem related to faster tests (and it is in some ways), but there are many other things that also go into fast run times.  We can run tests in parallel or on faster machines so that we can get through them more quickly.  We can also automate more of the steps.

There are two parts to what I am talking about here. One is reducing the wall clock time it takes to get through a set of test, which is a function of the speed of the tests and the ability to run them in parallel. However, another part of what I am talking about is the effort that it takes to get feedback from the tests. Right now we need to manually setup and run certain kinds of builds and then once we are notified of their completion we can go back and parse through the results.  If we were to automate some of those steps so that we could setup and run all tests and compare failures against a known ‘good’ build and then provide a local run queue with the test failures caused by this change – and if we were able to do this without any human intervention, we would be a long way down the path of having faster turnaround on the feedback.

The point of having fast runs times is so that the feedback can come in a timely manner. Having fast running tests doesn’t solve the problem if it takes a long time to set up the actual builds and debug the failures. I want to reduce the amount of time it takes from when a developer says ‘I want feedback on this change’ to the point were that feedback is actually available.

Relevant Feedback

Imagine you are making a change to the way error messages are handled in your app and you run some tests to check that nothing is broken. The run comes back with a number of tests that say that message X can’t be found in location Y.  The point of your changes was to have message X show up in location Z instead so this information isn’t really news to you as a developer.  It lets you know that indeed the messaging has moved away from location Y as you expected, but not much else.  This is an example of feedback that is not relevant.  What you really wanted to know was if there were any other messages unintentionally affected by this, or if there were still messages showing up in location Y.

Irrelevant feedback tends to be a big problem in tests and it comes in different forms. Sometimes it is like the example above, where we are told things we already know or where we don’t get answers to the questions we wanted to ask. But it can also come in more subtle forms as well.  Sometimes a passing test run is a form of irrelevant feedback. This might seem counter-intuitive, so let’s turn to another though experiment.  Imagine that you made a minor change and you run a full suite of tests on it.  However, in that suite only 1% of tests actually execute any code remotely related to your change.  The tests all pass, but how relevant is that feedback?  Think about all those tests you ran that had no chance of failing.  The fact that they passed is in no way relevant to you.  Of course they passed.  Nothing they checked was affected by your change.

So how do you make tests more relevant?  There are a couple of things I want to do with the tests I’m currently working with.  One is consolidation of things that are common.  So for example, if we have several tests checking a certain message we would consolidate those together so that if we change something related to that messaging we would just update in one spot (ahead of the test run) and be able to quickly and easily see if the changes occurred as expected.  Another change I would like to make is to leverage tools that will better show which areas of the codes particular tests exercise.  This way we can make more intelligent decisions about what tests to run for particular changes.  Getting this to an automated state will take us a quite a while, so in the meantime we are working on manually defining standard run queues that are more specifically targeted to certain kinds of changes.  Having a better understanding of which tests map to which areas of the code will help us get more relevant results from our tests runs.

I could go on with this, but it seems I’ve been quite long winded already, so I’ll draw it to a conclusion here.  I have a lot of ideas on how to improve my tests automation.  Maybe that is because there are a lot of problems with my system, or maybe some of these tweaks are the kinds of things you can change on your system as well.  Hopefully this article has given you some ideas for changes you can make to help your automation increase your coding productivity.


Improve Coding Productivity

Improve Coding Productivity

In my last post I talked about coding productivity being one of the considerations for deciding what automation to keep.  I want to dig into this a little more as I’m not sure that it’s something we as testers think about a lot.  We think about things like making sure we don’t let bad builds get out or about “finding all the bugs.” We think about making sure we have good coverage or large automated regression testing suites.  We think about a lot of different things, but in many cases it seems that the stuff we think about is related to ‘test only’ activities.

But aren’t we supposed to be system’s thinkers?  Aren’t we supposed to looking at things from a holistic point of view?  Think about it.  Someone needs to do that right? In most companies testers are hired to help mitigate risk.1 Companies pay good money to testers because they want some assurances that the code being released is going to be valuable to their customers.  However, this isn’t something testers can do on their own. Creating valuable code is a team based activity.  Are we thinking about things in a holistic team based way?

One of the most important things we as testers can do is to help improve coding productivity.  I repeat myself here: the reality is, creating valuable software is a team activity.  Testers need to think about how their actions impact and influence the team.  If we spend too much time find bugs and no time thinking about how to work with the team to introduce less bugs, are we really helping the team?  If we think about getting good coverage of the features and not about how we can figure out what kinds of issues and interactions our customers actually use, are we really helping the team?  If we think about preventing bad builds from getting through the pipeline and not about about how to change our build and deployment process so that the builds don’t get mangled in the first place, are we really helping the team?

We need to think big picture.  Don’t get so caught up in the official tester activities that you forget to look at what is going on. Where are the pain points in getting the code out the door?  What things can you do to help make developers more productive?  How can we as testers reduce the time it takes to get good quality code shipped?

Hint: it probably isn’t by running more regression tests.

Look for things that are slowing down the release of code. There are many things that need fixing.  Some of them you can do something about and some of them you can’t.  Find one that you can and start chipping away at it.  Don’t get so caught up on the hamster wheel of running and maintaining tests and flinging bugs over the cubicle wall that you forget to come up for air and look around. There are pearls to be found, but if the current has pulled you away from the reef, you might need to spend some time swimming back into position. Be a systems thinker.  Zoom out a bit and tackle the problems that are slowing down the system.


1. Whether you agree with this being what testing is about or not, I think the reality is that for many companies this why they hire testers.

ROI of Less Automation

ROI of Less Automation

I want to reduce the size of my automated regression testing suite.  It takes too long to run and it takes too much work to maintain.  It’s a big clunky beast and I want to reduce the size of it.  The problem is I don’t want to just arbitrarily delete tests.  I want to reduce the size in an intelligent way since there is still value to be had in some of the checks we have in there.  To do this requires work – a fair bit of work at that, and since we are paying people to do that work, it requires money to do this.

So how do I know if it is worth it?  What is the return on investment the business will get out of the work it will take to reduce the size of my automated test suite?

It seems like we don’t often ask the question from this angle.  Go google search queries related to the ROI of test automation and almost everything that comes up will be about how you can justify adding tests and ways to calculate the return on investment your business will get from adding test automation.  However, the reality is that not all tests are created equal.  Some tests do give you a positive return on your investment and some tests give a negative return.  In addition, the return vector can change over time. A test that was very valuable when it was made, might now be costing you more than it gives you.  To satisfy yourself on the fact that not all tests are equal, simply turn back to our good friend Google and ask him (her? it? –  I dunno) about test automation problems. The millions of hits you get speak to the fact that there are many ways for test automation to not add value.

We have a disconnect here.  Many people talk about the ROI of using test automation and many also talk about the how to deal with problematic tests, but are we connecting these two together?  How can we figure out (even in just a rough way) when there is a positive return on removing test automation?

Well let’s talk about the return side of things.  What benefits could your company see from having less automation?

Reduced Test Maintenance

Anyone who has worked in test automation will be able to pretty quickly tell you one of the main benefits – less test maintenance time.  Having less test to run, means less time spent on figuring out failures and updating or changing tests. This seems pretty obvious, but let’s not skip over this point too quickly.  There is more to test maintenance than just the time spent reviewing tests.  The more tests you have, the more machines you need to run those tests on.  Those machines also need to be maintained.  You need to apply patches and reboot them after power failures, or you need to buy licenses and time in the cloud. You also need some way to run those tests, which probably means some kind of build system like TeamCity or Jenkins.  More tests and more machines means more complexity in your build system as well. Those setups and script need maintenance too.

Most of us already know that test maintenance can be pretty expensive, but have you ever stopped think about how much it really cost you?  You might be surprised!

Reduced Machine costs

Machine cost have been driving downwards since computers made their debut and so we might think this one doesn’t matter as much in today’s world, but once again let’s not skip too quickly over this one.  There are a number of factors that play into machine costs.  In addition to the hardware purchases, we have the ongoing electricity costs of running that machine and the licensing costs to have an operating system on that machine.  We also often have other more hidden costs like insurance and the time spent setting up the machines in the first place.  Things like installing the OS and virus scanners, and setting up the machine on the proper domain controllers, etc. etc.  All of these costs add up and mean that there are significant savings to the company for each machine we don’t have to have for running automation.

Increased Coding Productivity

This one requires a little more nuance and thought, but I think there is an argument to be made that in many cases reducing your test automation will make developers become more effective.  This is the opposite of most of the sales pitches you’ll hear about automation, but hey you should probably use your critical thinking skills when it comes to sales pitches anyways.

Snarky comments about sales pitches aside, this probably isn’t as obvious as some of the other things I’ve talked about so I want to dig into it a bit more. Let’s say you have a test suite that takes a long time (definition:  more than 1 hour) to run. Let’s also say it needs to run before code gets merged into the release branch.

At this point we can think though a simple workflow: a coder submits code for merging and the test suite starts running.  An hour later, the build fails for some reason. The coder takes a look at it and it ends up being due to some missing edge case that a test caught.  She makes the fix and re-submits the build and things get in.

This is exactly why we run automated tests right?  To catch those sneaky issues. However, if we extend this thought experiment a bit more we can see where things start to go wrong.  What if you add some more tests that add a couple more minutes to that test run?  And now some more.  And some more.  Keep going with that thought experiment until we have it taking 5 hours to run the test suite.  Now when the developer runs the tests they find the same issue, but she doesn’t find out about it until the next day because the run took so long.  Does waiting this long help or hinder the coder’s productivity?

Of course there are many ways that we try to mitigate this kind of problem (parallel test runs, smoke tests vs. daily tests etc.), but at the root of it all is trying to solve this problem: increasing the run time of a test suite slows down developer productivity.  The converse of this is that by reducing the run time of your test suite, you might just be increasing developer productivity.

So how is your test suite?  Is it worth investing in making it smaller?  Could you help the business by running less test automation?  Count the costs of your test automation – it just might surprise you how expensive it really it!




Spurious Failures

Spurious Failures

I have a confession to make.

We don’t like to talk about the ‘bad’ things we do (at least not until those things are distant memories), but that isn’t really helpful is it?  If we don’t talk about our mistakes how are people going to learn from them.  So I have a confession to make.

I have been ignoring certain failures in my regression suite for months now.  I have a couple of tests that fail about 25% of the time and then pass on the next run.  So what have I done?  Nothing.  I just let them run again on the next build and pass.  I haven’t looked into the failures.  I haven’t tried to see if there are certain conditions that reproduce the failures.  I haven’t taken any steps to deal with these problematic tests differently from the other tests.  Nope.  All I’ve done is look the other way, cross my fingers, and go on with my day.  I’ve been too busy and I haven’t had time and I’ve….*mumbles more excuses*

Well today it nearly bit me in the back.  I was looking through the build history and realized that the build had been failing for a few days.  That seemed odd as usually the spurious failures would only fail one build, so I took a closer look.  When I did this I realized that the failures were actually happening on similarly named tests and were due to a recent code change.  It didn’t end up being a defect (I just needed to make a minor tweak to a couple tests), but it made me realize that my ignoring those spurious failures could have easily allowed other issues to slip through unnoticed.  By being ok with failing tests, I had trained myself to not trust failing build statuses.

Also today I saw an email from one of the customer engagement staff asking about why he was seeing a certain error when trying to set up something for a potential sale.  The error message looked familiar.  I don’t think it is the exact same error I’m seeing (and in his case it is consistent), but it is probably in a similar area of the code.  Maybe if I had spent more time on tracking down the root cause of the errors I’m seeing we could have found a fix for this already.

I’m sure most of us have been there.  Spurious failures are part of the job when it comes to writing test automation, but don’t make my mistake.  Don’t just treat it as an unimportant thing you can ignore.  The reality is, as I have said elsewhere, your test automation changes your behaviour.  If you have tests that fail sporadically, those failures will train you.  They’ll train you to pay less attention to build failures.  Is that really what you want from your automation?

Is it easy to deal with issues like this?  No.  Why do you think I’ve been ignoring it for so long?  It’s really hard to figure out what to do and how to handle these kinds of failures, but isn’t that what you’re getting paid to do?  Figure out the hard problems so that you can help your team release high quality software.

Alright, time to get back to work – I have a test failure to investigate.

Trustworthy Tests

Trustworthy Tests

I was working on a script to use for an automated test and one of the checks in it was comparing two similar things with different inputs.  I decided to change one of them to make the difference between them more obvious, but to my surprise, when I ran the test everything passed.  What gives?  I had made a pretty significant change to the test – it should be failing.  I tried out some stuff interactively and it all looked ok, so I ran the test again – Still passing.

Puzzled I reviewed the test again.  Everything looked fine, but clearly something was going wrong.  I started adding some debug output into the tests and after a couple of tries I found out that I had accidentally switched the order of some commands.  I was checking the output before updating the engine.  A simple switch later and everything was as I expected.

This is just a normal story in the life of a tester.  I’m sure anyone who has written automation scripts can relate to this, but let’s take a minute to think about the implications of it.  What if I hadn’t tried that change?  I would have just merged in the test and it would have been passing but telling us nothing. The passing state of the test was due to a bug in the test and not due to the code we were trying to check.  The code could have changed quite dramatically and this test would have happily kept on reporting green.

I”ll just get straight to the point here. Your tests are only as good as you make them and since you are a human, you are going to have bugs in them sometimes.  A good rule of thumb is to not trust a test that has never failed.  I usually try to deliberately do something that I expect to cause a failure just to make sure my test is doing what I expect it to be doing.  Try it yourself.  You might be surprised at how often you find bugs in your tests just by doing this simple action.

Waiting too Long for Reviews

Waiting too Long for Reviews

As we continue to refactor our automated test scripts, I have been a part of a number of pull request reviews.  I have also been involved in some code reviews for large changes that are coming.  Code (or test) review is a very important and helpful strategy for a team to participate in and I am thankful for the many tools that have enabled this and made it a common practice.  To be honest, I can’t really imagine a world where we didn’t have easy and frequent code reviewing.

But. (Could you sense that coming?)

Just because something is good and helpful, doesn’t mean it is perfect and certainly doesn’t mean we can use it in every situation without putting on our critical thinking caps.  Participating in so many reviews has made me put that hat on.  What kinds of things are code reviews good for, and more importantly, what are their limitations?

I think code reviews are very very helpful for fine tuning.  ‘Don’t use that data type here.’ ‘Should you return this instead?’ ‘It look’s like you are missing a check here.’ Comments like this help greatly on catching mistakes and tweaking things to get the code into a good readable state.

Code reviews are great at helping us make those little adjustments that are needed to keep up the quality and maintainability, but what happens if the entire approach is wrong?  What if you are reviewing something and you want to just shout out, NO you’re doing it completely backwards?  We all do silly things sometimes and need to be called out on it, but do code reviews help with this?  I guess in some ways they do.  The fact that someone else will see the code makes us think about it differently and by putting it up for review someone might let us know we are approaching it from the wrong angle.  But think about it;  how likely are you to, in effect, tell someone ‘you just wasted 3 days doing something you need to delete.’  And even if you do, how effective is this form of feedback?

If we rely on code review as the first time someone else sees major new work we are doing, we are waiting too long.  We need much earlier feedback on it.  Does my approach make sense? Am I solving to correct problem here?  Are there any considerations I’m missing? These kinds of questions need to be asked long before the code review goes up. Many of them should be asked before any code is even written.  We need early communication on these kinds of things, so that we can get the feedback before we spend too much time going down a dead end road.  Another consideration is that code reviews are usually text based and to have a discussion on direction and approach via a text based medium is likely to be an exercise in frustration.  Some things work better with face to face discussions.  The powerful tools that we have which enable easy code reviews can sometimes push us towards inefficient communication about the code.  Stay aware of how your tools affect you and seek to do what make sense and not just what is easy.

I have been able to participate in more code reviews lately. Code reviews are an important part of testing and so I’m glad to be included in these, but there is still lots of testing work to do before the code review and I will be trying to get more testing input happening then. I guess it is my lot in life to celebrate the small victories (more code reviews – yay!) and then push on to the next thing.  Never stop improving!