Old Year, New Year

As we come to the end of 2016 – and for my company – the end of another release cycle, we have been looking at what we accomplished in the last year.  We just had a meeting where various team leads from our division went through their team’s accomplishments in the past year.  The slide shown for the quality team got me thinking about what we accomplished in the past year and what I would want us to accomplish in the next year.

If I could write next year’s end of year accomplishments right now, what would they look like?  What things do I want to see happen in the coming year?  When I looked at this year’s slide there wasn’t anything that stood out to me as something that would make me stand up with my head held high for what we had done.  It was mostly the normal stuff of the testing life, but there was nothing that would make people perk up and raise their eyebrows.  Nothing that would give a ‘hmm that’s interesting’ reaction.  Part of the reason for that is a communication problem in that we haven’t effectively communicated on up the chain what it is that we are doing, but part of it could be that we just don’t have the goals in place that will keep us on track towards a more impactful role.

So what do I want to see on the 2017 Quality Accomplishments slides?

Slide 1 – Automation 

  • Effective and efficient automation.
    • Effective – More regression defects are caught by automation
    • Effective – Tests run very ‘close’ to code changes (most tests run every hour)
    • Effective – A shift to more property based tests
    • Efficient – Reduction in time spent on regression test maintenance
    • Efficient – less duplication in tests
    • Efficient – Shorter run times due to pushing testing down the test pyramid
  • Interesting new applications of automation
    • Test suite that allows us to get a holistic view of the effect of one area’s changes on the overall workflow performance
    • Development of a set of tests that helps us pinpoint which area of the code errors originate from
    • Cleanup of an existing test suite to get more accurate information about which kinds of changes will help customers get an accurate answer faster
    • Custom data generator to rapidly give a high level of coverage to UI input field testing

Slide 2 – Interactive Testing

  • Increased adoption of group exploratory test sessions
    • Developers included in some of them leading to an increased familiarity and collaboration between testers and developers
    • Increased collaboration with other related testing teams
  • Formalized ‘definition of done’ and staged releasing helps push defect discovery closer to the time of introduction
  • Lighter weight and more targeted test documentation techniques reducing the amount of time spent on test case preparation
    • Dynamic test plan and checklist generation
    • Collaborative ‘10 minute test plans’
    • Test cases produced as an artifact of the interactive testing process
  • Better formalization and capture of interactive testing effort via peer review process and improved note taking processes

And there you have it.  Even before everyone starts posting about their New Years resolutions and 2017 goals, I’ve got mine ready to go.  Ahead of the ball for once!

Preventing Bugs

Bug are expensive.  I think we can all agree on that.  Having them in your product impacts the quality of the product and can hurt sales and customer satisfaction.  We recognize that, and so we put a lot of effort (and by implication, money) into finding them.  We hire testers and invest in test infrastructure.  We do root cause analysis and have bug bashes. We explore and attack our software in many different ways, but how often do we stop and think about preventing bugs?  Are they just inevitable or can we actually prevent bugs from appearing in the first place?

The answer is yes.  Yes they are inevitable and yes we can prevent them from appearing.  I don’t think anyone will ever be able to release ‘bug free’ software, but at the same time, there are many bugs that could be prevented in the first place.

Where do bugs come from?  Well many different places of course.  Miscommunication on the part of customers in what they want.  Misinterpretation of what customers said they wanted by product managers or coders.  Mistakes in coding.  Misunderstanding of the platforms the product will be used on.  There are many ways to miss the mark when it comes to software development, and some ambiguity will always be there.  Nevertheless, we can still prevent some bugs.  For example, by following some good practices in coding you will make it far less likely that you introduce certain classes of bugs.  Or for another example, spending a few extra minutes getting clarification on what exactly a feature is needed for and what contexts it will be used in could prevent a whole raft of bugs.

There are many ways that bugs can be prevented, but what can a tester (working in the testing role) do to help with this?  Testing usually happens after the bug has already been introduced and then it is too late to prevent it right?  Or is it?  One of the things that has been helpful to me is to recognize that the activity of testing and the results that we report out of it can be used to influence the way that a team thinks about some things.  For example, something as ‘innocent’ as an email saying: ‘I’m planning some testing of feature X and I wanted to make sure I’m not duplicating work on this. What kind of unit tests have you done on this?’ can gently nudge coders to think about unit-testing their code (and the bug prevention benefits that come from writing code that is unit-testable in the first place).  As Maaret pointed out in this post, there are many ways we can nudge things (or speed up movement) in the right direction.

So as a tester, don’t just focus on finding bugs.  Think about how you could help prevent them as well.

Infinite Testing

On the face of it testing seems like such a simple thing.  You just need to check that everything works right?  Seems straightforward.  However, I think anyone who has tested for any period of time knows that the reality is much more complex than this.  The longer I work in testing and the more I try to learn and grow in my skills, the more I come to recognize this.  The skills required to be a good tester are vast and varied.  Not only are there a myriad of technical skills that one can (should?) learn, but there are also many skills that are not specific to testing itself but that are general to knowledge work.  We need to know how to navigate relationships and politics in a company.  We need to be effective and skillful communicators.  We need to understand process and change management.  We need to know how to work well with those that we like and those that we don’t.  We need to understand how business need and financial pressures affect what we do.

In my current work, we make simulation software and there is one interesting thing that happens.  The more powerful computers and our software get, the bigger the problems our customers will try to solve.  “Instead of just looking at each component in isolation why don’t we just look at all of them together and see how they interact?”  I feel like we are doing the same thing in much of our knowledge work (testing included).  Instead of looking at each piece on its own we are trying to put everything together into one big system and work with and understand the whole.  What this ends up doing is meaning that no longer do I have ‘just’ worry about testing, but I need to figure out how to fit in and understand all these many other functions and roles.  It is a more realistic approach to what is really happening – knowledge work has always been about relationships and understanding and not just you doing your simple task on your own – but it is a hard world to understand and navigate.  We still have so much to learn.

The previous model for shrinking our world down to a manageable size was to give each person a clearly defined role and action that they had to do and then pass on to the next person (think of an assembly line), but in knowledge work we are coming to see that this way of shrinking the world doesn’t work.  However, our response so far seems to be to make the world infinite and hope that we stumble on the right things to focus on. So the question is how do we shrink down the world when everything is so interconnected and interdependent?  To paraphrase G.K. Chesterton, I though I was getting into testing, but I find myself struggling with gigantic angels and demons.1

One of the things I have been doing with this is putting limits on which areas I want to focus on when learning testing skills, but I wonder if there is more I can do to help shrink down this infinite world into something I can manage? How can I navigate what I ought to be doing and learning – not in the sense of what my job description says, but in the sense of doing the things that are best for the company?

One place where I can see the answer to this is to shrink down the deliverables instead of the roles.  Instead of giving each person a defined role and having them work within that role and not have to worry about the why of doing it, we need to give each team a small set of deliverables.  Within that context we have a limiting of what to do and learn, but we are not constraining each person to only do a particular thing.  What I need to know and learn and work on is defined by ‘whatever we as a team need to be done so that we can get this deliverable completed.’  Maybe, what I need is to learn how to do security testing, or maybe what I need is to learn how to help facilitate people in bringing different parts of the project together, or maybe what I need to learn is how to fix a defect. Whatever it is, what I need to do and learn is defined by the project and the deliverables and the skills needed by the team as a whole and not by something that is heavily and specifically tied to a particular role that I might have as a tester.

We keep going down deeper here, but of course the next question to ask is what are my deliverables as a tester on the team I am on?  We have certain feature deliverables that are pretty obvious, but I think there is another deliverable that I have some unique skills to help with and that is the long term quality deliverable. I’m not thinking here of the testing role in terms of finding bugs during testing, but I’m thinking here of working with the team as a whole to help produce good quality, durable code that will help us continue to produce value in the future.  This post is getting long, so I’ll wrap up here, having given myself (and hopefully you as well) something to chew on.


1. The actual quote is actually a little more profound “anyone who makes himself responsible for one small baby, as a whole, will soon find that he is wrestling with gigantic angels and demons” In Defense of Sanity – page 167