When Trivial Changes Aren’t

When Trivial Changes Aren’t

We are working on an additional tweak to one of our features so that it will automatically add an object. I briefly met with the developer to discuss it and get a better understanding and then I put together a checklist of testing items I would be considering as part of this change.  I shared this document with the developer to get his feedback and so he could see some of the things the would be considered.  He put what I thought was an interesting comment on the document:

Thanks for this Dave! On second thoughts maybe defaulting is not such a trivial thing 🙂

This reveals a lot.  A lot about what collaboration is good for.  The developer had been thinking about how to put together code that would do what the story owner wanted, but after a short conversation and reviewing a few items in a checklist, he had a very different view of what was involved in making this happen.  Our short interaction will lead to a different level of baked in quality when it comes time to actually use the code.  I will still check the items on the checklist, but now the difference between what he thinks he has to do and what he actually has to do is much smaller. This means less surprises while testing this code which means less defects and less churn. At the end of the day, this means we can get this feature into shippable condition more quickly.

Sometimes it just takes a second perspective to see that the trivial thing is more complex than we thought.



Automation That Teaches

Automation That Teaches

I think learning is important.  In fact I think it is probably vital to continue to learn new things throughout your career if you want to be a successful knowledge worker. In particular as a software tester one of the primary ways I add value to the team is by the things I learn about the software.  I learn where things don’t work well and where they do.  I learn about how certain approaches to developing and testing tend us toward certain outcomes.  I learn about when things go horribly wrong and try to figure out why, and I try to use all this knowledge to work with my team so that we get better at  more rapidly producing valuable software.

One of things that I spend a lot of time learning about is test automation and how to use it effectively.  I’ve spent a lot of time learning about it, but I’ve come to realize that I can also learn things from it.  My automation can teach me things. By changing my mindset, I have been able to turn my automation into a teacher. Instead of thinking about automation as something that is primarily used to make sure nothing has changed since the last time we ran it (what I like to call ‘change detection’ automation), I see automation as a powerful way to learn things about the product that I would not (or could not) otherwise know,

I still use change detection automation in many places (and I think it is helpful in those places) but even with these types of tests I have asked myself what it can teach me.  With that change in mindset I have been able to leverage my automation in new ways.  For example, I have a small tool I wrote that will perturb my tests in very specific ways.  By making minor changes to the tests and then running them and parsing through the results I can very easily gain new insights about the behavior of the system.  Another example is parsing through the logs after the fact. I have many gigs of log data from years of running tests (I have a large hard drive and don’t delete) and so I wrote a script that can parse through that data and search for some specific things, put them into a database, and allow me to do some statistical analysis on them.  By doing this I have discovered new and interesting behavior that I probably would not have ever noticed without seeing it in the aggregate.

There are other ways that I have and continue to use automation as a teacher, but I won’t share them all.  Instead I want you to think about your automation.  Do you learn anything from it?  If you shift to thinking about it as a teacher are there any simple changes or additions you can make that would turn it into a better teacher?  What can you learn from your automation?

Your Automation is Making You Less Agile

Your Automation is Making You Less Agile

Does your automation help you move faster, or does it help you change directions more easily?  For many of us I suspect the answer is that or automation is about speed and not agility.  I wrote an article for the Agile Testing Days blog about some indicators that your automation might be making your team less agile.  Go ahead and read the full article here

The Mythical Developer – A Parable

The Mythical Developer – A Parable

I saw one once.  At least I’m pretty sure I did.  I saw something across the office and squinted my eyes but didn’t quite get a good look before she disappeared around the corner.  If I didn’t know better I would have thought I’d imagined it, but I knew the rumors.  I had heard it from Bob who had heard it from Sally who had been told from a very good source that it was true.  You’ve probably heard the legends too.  They do exist. They’re just not seen very often (or so I have been told).

One day she coded an entire app.  Literally one day.  And it had no bugs in it.  It worked perfectly the first time and everyone loved it.  She might have bragged about it a little, but hey, let’s just admit that she deserved it.  That was Monday.  We won’t even mention what had been accomplished by Friday afternoon.

And then there is the guy in the cubicle across from me.  He usually writes pretty good code, but some days …. well I could just about …  I need him to get something done, and he says  he’ll have it done in one day, but it somehow ends up taking three. After three days you’d think the quality would be high, but no, it ends up having a bunch of bugs. Plus he keeps bothering me with questions and so by the time Friday comes around – well, it’s kind of hard to close out the sprint.

It can be pretty frustrating working with someone like this.  If only I had run a bit faster that time, I might be able to actually get stuff done.

That would be the best way to fix the problems I have right?


Unit Test!

Unit Test!

I’ve been working on an automation related project for the last couple of days.  As the first stage in the process, I quickly threw together some code to figure out if the idea I had would even be feasible.  Once I had proved the concept and had a pretty good idea that it was something I could do without too much work (and hence would probably have a worthwhile payoff), I started to clean it up and put into a maintainable structure.  This meant adding unit tests.  The exercise of doing this has re-opened my eyes to the power of unit testing.

I haven’t been writing a lot of code lately, but I had noticed in the past that writing unit tests changed the way I wrote my code.  It magically became more modular and better encapsulated and different in so many ways. It had to so that I could actually run unit tests against it.  However, today I was struck by a few other insights in the value of using unit tests.

Bug finding

One of the first things I noticed was how many bugs can be found by unit testing.  I had used the original proof of concept code in a few different ‘black box’ scenarios to see that it could do what I needed it to, but once I start putting tests around it I immediately started finding bugs.  The first test I added revealed a bug with inconsistent data types I was using.  The very first test.  Talk about return on investment.  A couple of tests later I found a bug where I was inadvertently overwriting some of my data. This same pattern continued and just by adding some tests to the existing code I found and fixed several more bugs.  These bugs probably would have been found eventually, but it would have taken a lot more time than the 1/2 hour or so of work that I put in to these.

Quick Feedback

I also quickly realized how much easier it was to debug those failures and figure out what was going wrong.    When I was working on the code without having unit tests in place it had consistently taken me a lot longer to track down bugs, simply because it would take me longer to run each experiment.  If I suspected that something was wrong in module X I might put a print in there and then run the code to see what happened.  When I had the code unit tested that would take a fraction of  second and so I could quickly find out if my hunch about the problem was correct.  On the non unit tested code it would take much long to see if I was right or not.  I also noticed that once I had written a unit test I felt an almost physical desire to kick it off and see what would happen.  It was like getting that immediate feedback was giving me a dopamine kick and I needed to have more of it.  That quick feedback loop made finding and fixing problems much easier (almost fun you might say).  There is nothing more frustrating than not being able to figure out why your code doesn’t work.  Unit tests help take away much of that frustration

Trust your tests

I was trying to learn some new unit testing tricks related to stubs and mocks and so twice today I thought I had done something wrong in my test only to realize, that no, my test was actually pointing to an error in the code.  Don’t trust tests implicitly of course as they are written by humans and as we all know to err is human, but don’t forget that your code is written by humans as well (in my case by the same human).  Often the problem is indeed a bug in the code.  As I learned today, your code might have a lot more bugs in it than you think it does.  I knew my original code was put together pretty quickly and probably had a few minor issues, but I was quite surprised by how many problems I found while adding unit tests.

The moral of the story

Unit tests are awesome.  Do more unit testing if you can or push for it’s use if you aren’t in a position or technical skill level to add them yourself.  My experience in the last couple of days has once again reminded me of the power and usefulness of this testing tool.


Don’t only do unit testing.

The End

Don’t be an Automation Miser

Don’t be an Automation Miser


I don’t know if you are the kind of person who flips to the end of the book to see what happens, but I’m going to do that in this post.  I’ll start with the end of story.  Here it is: Embrace automation, not because it’s automation, but because of what it can do for you. Now for the story.

I was working on a script to do some fuzz style testing of some of our UI input fields and I got to thinking (because writing code has a way of making my brain wander off into the metaphysical realm) about why I was writing this.  Why bother automating?  This particular script is going to be very different from what we usually think of when we think of automated tests (i.e. regression tests), and so does it have a different goal or is the purpose of it the same as any other automated test?

Once I started off down the rabbit trail of the purpose of automation – oh my!  Things got out of hand quickly.  I had to start scribbling things down and making notes and before long I had a tangled mesh of ideas bouncing around in my head.  Sigh.  This happens a lot.  It’s why I have a blog.  Somewhere to force me to slow down and put some more structure into my thinking. So below are some of my (semi) structured thoughts on why we bother with automation.

Let’s start with something obvious.  We use automation to achieve some kind of goal.  That goal varies from team to team and company to company.  Perhaps it is so that we can ship code more quickly.  Perhaps it is so that we can hire less testers.  Perhaps it is so that I’m not bored. Or perhaps it is so that we can be sure of the quality of the code.  Whatever the reason(s) we have for it and however good or bad they may be, we can be sure of one thing, there is a reason for doing it.  Somebody, somewhere expects to see benefit from it or we would never bother with it in the first place.  If we start with an understanding of the goals we have for the automation we can think about it in two ways.  First of all we can think about if that goal makes sense and is achievable.  Sometime the goals that we have for our automation just aren’t reasonable.  For example if the goal of your automation is to be able to talk about how you have X% of your tests automated, you may need to rethink the goal itself.  That goal isn’t really tied to anything that has value to the company.

Once we have a reasonable goal for our automation we can start to figure out ways to evaluate how well it is meeting that goal.  This becomes a metrics problem and is actually much harder that you might think on the face of it, but for this post I want to focus in on the goals side of thing.  What are some of the goals I have for different automation projects I’m working on?

Automated Regression Tests

The current goal of these tests is to give us confidence that most of the stuff our customers depend on continues to work and to give us quick feedback when existing workflows are broken.  I have some additional goals for these as well that I have been thinking about recently, including using them to increase our agility and using them to more clearly define and establish the interfaces in our product.

Automated ‘Convergence’ Tests

The purpose of these tests is to allow us to experiment with various tweaks to the code in order to evaluate how well they enhance one part of the problem (the convergence) across a huge variety of cases.  The primary goal of these tests is to help us evaluate if a given hypothesis about how a change will affect convergence is true or not.

Automated ‘Fuzzing’ Tests

These tests are a way to help when testing new UI elements.  They can help check that the field limits are set in a way that match what the underlying engine can support.  The goal of these is to help make sure we don’t miss checking for certain things (zero, one, many, none, invalid, on the bounds, near the bounds, outside the bounds, etc. etc.)

So with just this simple survey of a couple of different automation projects, we can see that there are vastly different goals and purposes for each of them.  Automation is an incredibly powerful testing tool and we need to use it to help us achieve goals.  Sometimes we can start thinking like misers when it comes to automation.  A miser wants to have money so that he can have more money.  Of course a more reasonable approach to money is to use it to help you achieve certain goals.  Similarly an automation miser is someone who wants to have automation so that they can have automation.  Don’t be that person. Don’t just use automation because ‘we need too’ or so that you can have a certain number of automated tests. Use it to help you achieve your goals.

Intuitions of Testing

Intuitions of Testing

The other day I was working on my taxes.  The online tax preparation software that I used tries to guide you through the process in a certain way.  However, when I started on it I didn’t have all my slips yet so I had to jump around in the product a bit. There were controls that allowed this to happen, but it was pretty clear from the overall approach and feel of the software that this was not what they expected typical users to do.

When I was doing this, I noticed something:  I was looking very carefully for issues in a way that I rarely do while using software.  It was like there was a warning bell going off in my head.  Without thinking about it explicitly, I intuitively knew I was doing something ‘dangerous.’  This was software I knew very little about, but this warning bell was going off in my head so I paid attention to it and checked things closely.  In doing this I noticed that the software had indeed failed to save some of the data that I had already put in.  I’m not sure if I would have noticed this or not without the warning bell in my head, but I’m sure glad it was there.

It also got me thinking about what makes a good tester.  Without explicitly thinking about it, I knew that this was somewhere there might be bugs, and I think that speaks to the importance of experience in testing.  There are many things that can be taught and learned, but there are also skills and abilities that only come from years of experience.  You begin to develop a sense for where bugs like to hide and what kinds of things to look for in finding them.  If you want to get to be a great tester, there is no substitute for just doing a lot of testing.  Build up that intuition and then pay close attention to it.  Embrace the intuitions of testing.

When to not look for Bugs

When to not look for Bugs

I love finding bugs.  I realized this again recently while testing an area of the code that was pretty buggy.  As a tester there is something thrilling about having thought of ways that the code might not work well and then having your ideas validated as you watch the application start to self-destruct in front of you.  When I find a particularly bad defect I find that I want to go talk to some other testers and ‘brag’ for a while about the way I found it, but is this healthy?  Should I be celebrating the finding of bugs, or am I like an addict seeking the next thrill as I try to hunt down bugs in the software?

I don’t think there is an easy answer to that question, but I want to explore the idea that testers should not want to find bugs.  This is of course not true in the absolutist sense, but thinking about the reasons we don’t want to find bugs can be a very helpful exercise and so today I want to think about one of reasons you might not want to focus on finding bugs.

We are part of a team

Finding bugs is often about shattering illusions.  When we find a bug we are demonstrating that someone’s idea of working software is wrong.  Illusions (or biases) are some of the biggest problems we have in software development.  Our lack of ability to make the software do the things that add value for people is at the heart of what makes software development so hard.

But let me ask this question:  Who’s job is it to get a good quality product to the customer?

Let me take you to story from outside of software engineering.  After dinner in the evening, I usually clean up the dishes.  I clear the table and load the dishwasher etc.  One day, however, my wife said something interesting to me.  She told me that she would appreciate it if I would also wipe down the counter when I was done.  Now here’s the funny thing: I was wiping down the counter!  It was just that my wife and I had different standards as to what constituted a well wiped counter. At this point the illusion of adequate counter wiping had been broken.  Now, let’s imagine that the next day I did nothing different.  I wiped the counter the same way I always had.  And so once again my wife tells me that the counter is dirty.  I quickly clean it up but then the next day it happens again.  I don’t do a good job, my wife points it out and I fix it up.  No harm done right? At the end of day the counter is cleaned to the quality standard we want, so what does it matter? Well I think any of you in a relationship understand that there is harm done.  There are a couple of healthy choices here for the relationship (have a conversation to agree on what will be considered a clean counter, who will do it etc.), but to have a relationship where one person keeps giving feedback to the other person who ignores that feedback is only going to create a wedge and distance between the two parties.

Probably by now the moral of the story is pretty clear.  If we as testers keep pointing out the issues and developers keep doing the same things over and over, are we in the kind of relationship with each other that is growing together or apart?  The answer to the question of who’s job it is to get a good quality product to the customer is everyone!  The whole team is responsible for this. But how are we going to do this effectively if we can’t work together as a team?   Finding bugs is not wrong or bad, but when we find ourselves repeating the same feedback over and over again without things changing, the relationship might not be optimal.  It might be time to pursue the kinds of conversations that can bring consensus to the team on what constitutes a well wiped counter and how to make sure it gets properly wiped from the start. The effective shipping of high quality software takes a team working together.  If it is consistently easy to find ‘big’ bugs in your product, it might be sign that your team needs you to focus more on building relationships and a bit less on finding bugs.