Author Archives: offbeattesting

Have you ever been lied to?

I don’t mean those little white lies that smooth social interactions – “You don’t mind if I do this?”  “No, not at all” – I’m talking about a full blown lie.  You ask your son to take out the garbage and he tells you did it already.  The next morning, after the garbage truck has left, you find out he never did it.  Or you call customer service and they promise to take care of it as soon as you hang up, but you have to call back two days later because nothing has happened.  What happens when you are lied to? You lose trust in that person or organization.

Much of the way we work as a society is build on trust. There are places in the world where stores have armed men with guns by the door to deter theft.  Why? lack of trust.  In the West we have many blessings and one of them is trust. I can call a dump truck and ask them to bring by a load of sand and they will do it and send me an invoice.  Why?  They trust that I will pay them. Without that trust the transaction would become much more painful. I would somehow need to get them money ahead of time, but would I really want to if I couldn’t trust they would deliver once they had the money? Transactions become much more difficult in a society with low trust.

Trust in Software Development

In the software development industry we primarily view trust as a financial proposition. “If my customer doesn’t trust my credit processing system they won’t click on buy and I won’t make money.”  However, trust is a bigger problem than that. It is a societal problem. In a world with low trust we have high cost. Everything get expensive when there isn’t trust in place.

How do we contribute to  lack of trust in software development? Well, how about when our customer’s data gets hacked?  Does that build trust?  What about when users come to our site and it won’t load?  Does that build trust?  What about when your ‘click here’ icon doesn’t do anything? Does that build trust?

Let me drive the point home in a way that might hurt.

If you are ok with software that breaks trust, you are not working for the social good.

Ouch.

But I think it’s true. We hear a lot of talk about social entrepreneurship and the importance of social initiatives in corporations.  I want to challenge software development teams – and in particular those who think about testing problems – to think about the social responsibilities they hold. The ethics of testing are clear in some domains like medical applications, but what about if you are writing an iPhone app?  Do you have ethical considerations then?   My argument is that if you are ok with releasing a buggy app that breaks the implicit and explicit promises you are giving your customers you are contributing to a societal problem. Teaching people to expect to be lied to is not a good thing. Don’t have any part in it.  Release good quality software.

Post-script:

It’s ok to make mistakes. We will sometime break people’s trust despite our best intentions.  Don’t beat yourself up over missing things. The point of this post is that it should be despite our best intentions.  We ought be shooting for high quality, trust building applications. Saying that it’s ok to ship with bugs in our product, shouldn’t just be a financial decision.  We need to be thinking about other things like the ethics of it as well.

We’ve all been there are some point. It’s crunch time. The deadline is looming. Everyone on the team is frantically working to get the product shipped.  Crunch time reveals to us what we really see as important. During crunch time certain things get dropped.  Think about the things that get dropped on your team during crunch time and I’ll bet that they  all relate to one thing:

Quality

We stop checking things in as much detail as we used to. We take some shortcuts in our coding.  We do things that we know are not good for the health of our code base.  We quickly patch up defects without addressing the underlying issues. Under pressure, the first thing offered on the alter of the Almighty Deadline, is quality.  Our real priorities are revealed. We might talk about how we take quality seriously, but our actions show differently.  Our actions show that quality is seen as the low guy on the totem pole.  Perhaps this is why in some places testers feel like they are lower status members of the team. Testers are concerned with quality.  How do you think they are viewed if we see quality as the least important part of the software development process?

But what if it isn’t?  What if quality is actually the most important part?  What if many of the problems and challenges we face in the software industry are directly related to quality? If that is the case, the way we are approaching our problems is totally upside-down. Instead of the first thing to go, quality should be the last. Instead of a dispensable part of our process it should be the keystone of everything that we do.

Here is the thing: If we want to be able to move more quickly we need to invest in quality first. Quality can’t be an afterthought.  It can’t be something we tack on to the end of the process.  It needs to be something that is fundamental to the very core of how we go about making software.  Until we understand this and until we start to act like we believe it, we are just going to keep spinning around on the same hamster wheel.

Quality Challenges

One of the challenges we face with quality is that it is often a hidden property of the system. Imagine you go to your local generic chain furniture store and pick up a pretty cheap table.  Or imagine you buy a custom made table for 5 or 10 times as much.  The two tables might look very similar (initially), and yet which one do you think has higher quality? Quality is often not immediately visible.  This is what make it so tempting to cheat on it. At crunch time we can hide sloppy work and shoddy testing, because it won’t be immediately visible. But the fact is, poor quality will be experienced by someone.  It won’t stay hidden forever.

As testers we are like the mechanic friend you take along when you are looking at a new car.  You see a car that looks nice and has some nice features – it’s even red! – but then your friend crawls under the car and opens the hood and shines his flashlight around to see in the cracks and crevices.  He comes back with a list of problems (or potential problems) that you’ll want to consider. He’s slowing down the process.  Now you have to go look at another car or negotiate a lower price or get the owner to fix those things.  You just want that car now, so you shouldn’t bring your friend along next time right? Right? Or not?

Another challenge when it comes to quality is that we don’t pay the price. If we release a low quality product the company might get hurt and it might loose some money, but oh well, we can always find a job somewhere else right? And besides somebody has to fix up all those problems. The reality is most of the pain of the bad quality products we work on get passed on to someone else.

Invest in Quality now!

But we are ethical – and I really believe we are – so most of us want to do the right thing.  We want to make something we are proud of.  We want to help people. We want to do right by our customers and our employers. So how do we do this during crunch time? We need to get these features out for the good of the company and the customers. They are important.  How do we do that without sacrificing quality?

We do it by investing in quality now.

You know you are going to have crunch times. We live in a world where we can’t see perfectly into the future and so there are going to be times when there is a crunch to get things done, no matter how well we have planned. We also know that during crunch time one of the easiest things to sacrifice is quality. So prepare for this.  Invest in your quality well before crunch time comes.

When we are driving for that final push, do you think it will help to have a robust and useful set of automated tests in place? Well, you certainly won’t get that up and running during the crunch time – invest in quality now. 

When you are in the middle of a crunch do you want to discover fundamental flaws in your architecture or design? You are only going to find problems like this early if you look for them early – invest in quality now. 

When the pressure is on and someone is out sick, do you want to have to feel your way through some tangled, unreadable code in order to fix a bug? Having clean and readable code doesn’t just happen by itself – invest in quality now.  

When you are trying to merge that feature in, do you want to have no time for interactive testing because you are waiting for it to make it through the build pipeline? Good build pipelines take time to build and maintain – invest in quality now.  

When there is very little time for testing do you want testers to struggle to trace down issues they see? At crunch time it is too late to think about testibility – invest in quality now.

Crunch times will come, and during crunch time we will have to change some of things that we do, but if you want to come out the other side holding your head high and able to be confident that you’ve done a good job, you need to think about quality.  Don’t put it off or leave it as an afterthought.  You’ll be much more successful as a software development team if you start with quality.

Invest in quality now.

I wasn’t feeling very well.

It had seemed like a good idea at the time, but now I was regretting it. That morning I had woken up early while everyone else in the family was still asleep and quietly crept down the stairs and headed to the kitchen.  I opened the refrigerator door and there it was: a full carton of eggnog.  I pulled it out of the fridge, pushed a chair over to the cupboard, climbed up and found a cup, and poured myself a glass of delicious, creamy eggnog. After the glass was finished I had another, and then another until the entire carton was gone. It had been so delicious, but now my stomach was telling me I shouldn’t have done it. I will spare you the details, but that day I quickly came to realize that you can have too much of a good thing.

Kind of like automated tests.

They are a good thing.  They are very helpful, but have you even been in a situation where you just have too much of a good thing?  What happens when you have hundreds or even thousands of end-to-end or integration tests?

You start to realize that automated tests aren’t free.

Too Many Tests

How much time are we spending on these automated tests? Well lets add it up.  We spend time fixing tests that fail for expected reasons.  We spend time debugging intermittent failures. We spend time patching and updating VMs.  We spend time creating and maintaining test builds. We spend time figuring out why things have broken.  We spend time dealing with requisitions to get new machines in place.  We spend money on buying those machines and operating them.

There are a lot of costs that go into tests and the above list in certainly not exhaustive.  A lot of those costs can be hidden, but when you have too much of a good thing you can’t ignore them anymore. Your team is overloaded with work and it forces you to realize that automated tests are not free.

So what can you do about it?

You drank the cool aid eggnog, and now you have too many tests.  How do you go about fixing the problem? Those tests were initially added for a reason – somebody at some point thought it would be a good idea to add each of them – so how do we reduce the number of tests? We don’t want to just arbitrarily delete tests, but we are already overloaded so we don’t really have time to go through and carefully evaluate them to figure out which ones to get rid of.

What we are face to face with here is something known as ‘the real world.’  There may be some perfect level of automation coverage and there may be some way to get there, but the reality is you can’t get to perfection.  You just need to get better. Tackle the problem one bite at a time.

Find the most expensive tests

We outlined above some of the expenses that go into tests. All you need to do is find out which tests are the most expensive.  Easy right? Well maybe not, but there are some strategies that can help you focus your attention when cleaning up tests:

Long running tests. That test that runs for 20 minutes – It’s expensive. The time it takes to run and debug that test is high.  One really quick way to pick which tests to clean up or remove is to sort your tests by run time

Flaky tests.  That tests that fails off and on for no reason – also expensive.  You need to look into the failure again and again and you start to learn to ignore it.  Do you really need that test?  Can you isolate some of what it is doing so that you only check the thing you are interested in?  This is another example of tests that are easy to identify (just look through the test reports for how often tests pass or fail) and that would be high value tests to cleanup or remove.

High maintenance tests. The test that always needs to be updated is another class of expensive tests.  These can also be reasonably easy to detect if you version control your tests. Just look through the version control logs and see which tests are changed the most.  Could you remove some of these, or change what they are checking so they don’t fail for things you don’t care about? Or could you perhaps check the things these tests are looking at more easily with a quick manual check of the product (gasp!). Not everything needs to be automated.

There are more strategies that you can use to help you prioritize your test cleanup activities, but we’ll stop here for now. What about you? Do you have too many tests?  Have you started to realize that you need to do something about it?  What strategies do you use to approach test cleanup?

The other day I was watching a show on TV about how food is made and I started thinking about software testing (because that’s what normal people do right)?

So much of what we have in our society is made on assembly lines and food is no exception.  That candy you bought?  Produced in an assembly line style environment.  The loaf of bread and container of milk?  Same thing.  We still have artisanal food, but the reality is that most of what he eat is at least partially made in a factory somewhere.

How do you find meaning in what you do in the assembly line world?  How do you find meaning in just being a step in a process. Do you actually feel like you have made something when at the end of the day, you’ve rolled out 240 candy canes per hour? I think that we as humans have an innate desire to create, but does a process like this allow you to create? Can we find meaning and fulfillment in a job like this? Can we meet our need to create things when most of what we build as a society requires the input of many different people?

I think we need to think about these questions, and not just in the assembly line context. These kinds of questions matter in the software world as well. It isn’t just physical goods that are too complex for most of us to make on our own.  The software systems we build require teams as well.  We may call this creative work, but very few of us create entire products. We work on one small piece that fits in with other pieces. Can we fulfill our need to create in this context?

What about those of us who’s primary role on the team is to test? What do we create?  How do we scratch that creative itch?

I think this is why a common complaint among testers is that they get left out of decisions about what the product will do or look like. It’s why we talk about the need to shift left. It isn’t just because those things are helpful (although they usually are). It’s because we have a need to create something and to be included in the creative process. If all we are doing is checking someone else’s work to see if they did it correctly, we have no job fulfillment.

Before I started in the testing job I have now, I worked in a couple of QA jobs in factories. I hated them. Take this batch of paint. Measure it for various properties.  Compare to the specification table the customer had provided and accept or reject. Boring work with nothing to point to. At least if I worked at a factory that made candy canes, I could look at a candy cane in the store and be proud of the fact that I had twisted two colors together perfectly. If I looked at a can of paint, all I could say was that I had told everyone they did their job correctly. Not only was the job boring, it felt like it wasn’t tied to something I was creating.

Now back to software testing. If my job as a tester is to compare the work someone is doing to a specification and accept or reject it, I’m not going to feel like I am producing something of value. We want to be involved in making something. We want to be able to point to the system proudly and say “I helped make that!” The human need for creativity is why we testers want to be involved in more than just checking if the product does x or y, but – and this is important – it is also why your team needs a great tester. All members of the team want to be able to point to the software and be proud of the fact that we made something great.  If we can’t do that we will be demoralized as a team and we aren’t going to produce a lot of value in the long run. Testers help with this! Value your testers. Listen to them. They share the same desires you do.  The desire to make something we can all be proud of. Let’s do this together!

David Hume’s Guillotine tells us that we cannot derive an ought from an is.

Abstract philosophical musings you might think, but then again, you work in software, so you know that philosophy matters, right? I want to muse on this for a while, and not in an abstract realm.  I want to take this down into the real world.  I’m going to take the law at face value. If it really is true that an ought cannot be derived from an is, what should we do differently?

“How can I automate this?”

“Should I automate this?”

Two similar sounding questions, but which one you start with makes a world of difference.  How often do we start with the is? We start with the fact that we have a framework in front of us that let’s us easily add tests, or the fact that we are told to automate everything or even the ‘fact’ that manual testing is too slow or too cumbersome for the modern software shop.  But aren’t we forgetting something?  Aren’t we forgetting the law our friend Hume told us about?  Just because we can do something doesn’t mean that we should.

This applies at every level of software development (because what is software development except automation?)

Should you make this product?

A number of years back the company I work at started a new initiative. We had acquired two previously competing products and after the acquisitions we decided to try and combine the best of both products and re-architect everything into a new product. It seemed like a great idea and we put a lot of effort into it for several years, but recently the product was canceled.  Why?  Because we had decided that this product did not need to be made.  Some people were interested in it, but market analysis and other factors indicated that it was not worth further effort. We might have been right or wrong about the market forces at the play and the strategy employed, but this does illustrate something.  Could we make this product? Sure.  Should we?  Not if it won’t help people. Not if it will lose us money.

Just because you can do something doesn’t mean you should.

Should you automate this?

What about an example that is a little more related to test automation?

At one point I worked with a team that did workflow tests.  These were scripts that we would run through every morning to check if the nightly builds were ok.  As you can imagine they were boring to do and so we decided to automate them. Makes sense right? Automate away the boring stuff. We don’t want to spend half an hour every morning working through the same boring scripts. So we went ahead and did it. We automated them, but should we have?

I’ve talked before about how boredom can be an indicator that you should automate something, but that it can also be an indicator that you are doing something useless.  So which is it here?  I think a bit of both actually.  We automated them all at that time, but since then we have removed or changed many of them.  There are still a few that have stayed around because they are valuable, but we certainly didn’t need as many as we had.

This once again illustrates the point that just because you can automate something doesn’t mean you should.  If we start from the is – is this possible?  Does a framework exist that allows me to do this? – we never get to the ought –  does this make sense?  Will this be valuable?  If we had started with ought questions we would have just gotten rid of half the scripts we had and automated the few that were actually valuable. Instead we started with the is questions and ended up spending a lot of time creating and maintaining tests that we really didn’t need.

We need to start with the ought.  In the software world we (understandably) love using software to do things.  Sometimes that is the wrong answer.  Sometimes there is a non-software solution to something. Sometimes we are just doing the wrong thing and so setting up systems to do it faster isn’t helping anyone.

Start with ought.  Know why you do what you do.  Take a minute to think things through before you plunge in and start doing.  If you know the ought, figuring out what to actually do becomes much easier

I recently read through Systemantics by John Gall (or the Systems Bible as the newer editions are called). Although presented in a very humorous and entertaining way, this book is packed with ideas that make you stop and think.

Why don’t things work the way you expect them to? Well, this book will tell you. It might seem discouraging to know that a “Complex System cannot be ‘made’ to work. It either work’s or it doesn’t,” but when you think it about, it is easier to (principle 31) “align your system with human motivational vectors,” than it is to keep banging your head against fundamental systems laws.

And never forget that “systems will display antics.” Don’t be surprised when the system doesn’t do what it is designed to do.

This book might make you a little cynical, but you’ll probably get further ahead in the world if you understand why you are feeling frustrated by the systems you are in. Sometimes a does of realism is good for us right? I would highly recommend this book to anyone who works with systems (i.e. all of us).

“It’s easy to write automation”

As the thought jumped through my mind, I had to tip my chair back, look up at the ceiling and ponder.  Is that true?  Is it easy to write automation? As I stared at the grey painted ceiling and chewed on the thought, some examples came to mind that illustrated the truth of this.  It really is easy to write automation.

I had put together a new test in a half an hour and even while I was doing it, I was thinking about how much of the test creation work could be automated.  With a simple template and a few shared functions we should be able to make it even easier to create an automated test.  Creating a test is easy

I had a new co-op student and after a bit of training on our automation system he was able to add several new tests in a day.  Creating a test is easy.

We have thousands of tests in our automation system.  How did we get so many? Well, creating tests is easy.

The Struggle

But, if it is so easy, why do so many people struggle with test automation?  Why is it an ever popular subject of books, blogs and conference talks?  Why do we spend so much time and effort on this.

Well, the answer is simple.  Test automation is really hard.

Wait, what?  What kind of weird world do I live in.  Didn’t I just say it was easy?  Well, not quite.  Adding a new test is pretty straightforward.  Adding a test that is going to be both valuable and low cost – that’s a whole different story.  Adding tests is easy.  Adding good tests?  Not so much.

Remember that test I added in a half an hour.  Easy right?  Well, I ended up making several major edits and changes to it, based on review feedback.  Why?  because adding a test is easy, but adding a good test is hard.

Those tests the co-op student put in?  We spent a lot of time together on figuring out what kinds of things to check in these and how to make them work well.  Adding the tests was easy, making them good tests was hard.

Our automation system does indeed have thousands of tests and we are struggling with how to keep up with them.  How do we change and tweak them so that they are more valuable?  Once again, adding tests is easy, but adding good tests is hard.

How do we fix automation problems?  Do we need to hire more automation engineers? Do we need a new framework? Do we need to get more developers involved?  Maybe. But there is one thing that we really need.  We need skill. Changing tools and processes can help, but to get good automation you need skilled automators.

We need automation craftsmen and women.  We need people who know and understand how to write high quality automation.  People who know how to use automation to actually lower the costs of  testing.  People who know how to write the kinds of tests that actually add value.

You want to stand out from the crowd?  Don’t just write automation – write good automation. Study automation.  Learn how to leverage it.  Learn how to clean up automation messes. Learn how to move automated tests from mediocre to great. Figure out what makes for good automation and then find the tools you need to help you achieve that.  If you are a student of automation and you can craft useful and valuable tests, you’ll have nothing to fear. Machine learning and AI are just additional tools in your toolbox.  If all you do is turn test cases into automated scripts – well, I’m sorry, but that’s easy and your job security is low. Learn how to write good automation before the machines come for you.

Writing Automation is easy. Writing good automation is hard.  Do hard things!

Note that this post is part of a series where I am ‘live blogging’ my way through the ministry of testing’s 30 days of Agile Testing challenge.

What actions do we take when there is a red build?  Well, what a timely question!  I’ve just spent the last couple of days trying to figure out a few different build issues.  The story illustrates one set of responses to a red build, but it also shows that there isn’t just one answer to question of what we do when there is a red build.

Thursday I noticed that one of my builds used to check on a lower level package was red. It wasn’t just one or two tests failing.  Every single test was failing. Clearly something was going badly wrong.  I spent some time digging into it and finally realized that part of the package wasn’t getting extracted correctly during the setup.  After some more time (and frustration), I finally figured out that the issue boiled down to the regression VM using an older version of 7zip.  Apparently the build machine creating the package had been updated to a newer version and so now the old version on the machine I was using couldn’t properly extract the package.  I updated the version of 7zip and re-ran the build. Everything was passing, so I posted an artifact to get picked up in the final build process. Everything is good now right?

Wrong.

Friday morning I came in to find that instead of the build picking up an artifact from Thursday (as it should have), it had picked one up from May?!?! Stop the presses! More sleuthing required!  We stopped the build from progressing and starting digging into it. The problem ended up being another machine that had the wrong version of 7zip installed. This machine had also had not cleaned up properly at some point and so had an old file hanging around that it could (and did) use.  We fixed the 7zip version and updated the scripts to make sure they were correctly cleaning things up and now *touch wood* everything in running smoothly again.

The point of this story is to show that the things we do to deal with red builds varies. Normally we wouldn’t stop all other work and  focus all energy on fixing the build, but in this case the red build was of the ‘Nothing Works!’ category and so the steps taken were more drastic.  In the ‘normal’ day to day red build where a test or two is failing our approach to it would be different.  We would look into it and follow up, but if the issue was small enough we would let the build pipeline continue and just follow up with a fix. Or if we caught the issue early enough, we might just quickly revert a change and things could continue on as expected.  The approach to a red build can’t be strictly prescribed and often requires exploration to figure out.

The lesson? Even when it comes to red builds, the context matters!

Note that this post is part of a series where I am ‘live blogging’ my way through the ministry of testing’s 30 days of Agile Testing challenge.

Today’s challenge asks what columns we use on our work tracker or kanban board.  To be honest we don’t use columns at all….

I know, I know, bad us right? Perhaps so. This probably would be something worth trying, but for some reason we have never gone down this road.  I’m not sure why, but it hasn’t risen up as a high priority thing to try.  Perhaps those of you who do use a kanban style of work management could share: does this transform the way your work? Please leave a comment with your experiences!  We are trying to move towards a more agile way of working from a process that is, frankly, quite waterfall in many ways.  Is this something that would be helpful for us in this journey? Trying new things takes time and energy.  Is this something that is worth the time and energy it would take?

Note that this post is part of a series where I am ‘live blogging’ my way through the ministry of testing’s 30 days of Agile Testing challenge.

Today’s challenge is about contributing to the learning culture in my company.  I work at a pretty big company that is split into a lot of different divisions so I don’t know that I can speak to the learning culture of the company as whole.  Instead I will focus in on the learning culture of the testing team I am a part of.

Testing Team Learning Culture

On our testing team, we approach learning in a few different ways.  We are currently a distributed team spread across 4 offices and so we try to foster learning in ways that accommodate this distributed nature. For example, occasionally during team meeting we will discuss an article that we have all read.  This helps us think about things that might be outside of what we usually do and lets us discuss different viewpoints and approaches to testing.

We also use retrospectives as a learning opportunity to try and see how we can grow and learn from shared problems.  Looking back on problems we have faced as individuals and discussing together ways to address them or think about them is a very helpful learning tactic

Another things we have tried recently is sharing ‘tips and tricks’ at our weekly team meeting.  This is a way to share a quick little tip or tool that you have come across that might be helpful to other testers.

One other, very important way we foster a learning culture is through group testing sessions.  We use these sessions as an opportunity to learn new things about the product and to help each other get better at interacting with and testing the product in various ways.  This also gives us the opportunity to observe other people testing (we use screen sharing during these sessions) and thus to learn from their actions in that way as well.

As a testing team we realize that an ongoing commitment to learning is an important part of become an ever better tester and so we invest time into this.  Don’t get complacent with where you are.  Keep on learning!