Author Archives: offbeattesting

Bug are expensive.  I think we can all agree on that.  Having them in your product impacts the quality of the product and can hurt sales and customer satisfaction.  We recognize that, and so we put a lot of effort (and by implication, money) into finding them.  We hire testers and invest in test infrastructure.  We do root cause analysis and have bug bashes. We explore and attack our software in many different ways, but how often do we stop and think about preventing bugs?  Are they just inevitable or can we actually prevent bugs from appearing in the first place?

The answer is yes.  Yes they are inevitable and yes we can prevent them from appearing.  I don’t think anyone will ever be able to release ‘bug free’ software, but at the same time, there are many bugs that could be prevented in the first place.

Where do bugs come from?  Well many different places of course.  Miscommunication on the part of customers in what they want.  Misinterpretation of what customers said they wanted by product managers or coders.  Mistakes in coding.  Misunderstanding of the platforms the product will be used on.  There are many ways to miss the mark when it comes to software development, and some ambiguity will always be there.  Nevertheless, we can still prevent some bugs.  For example, by following some good practices in coding you will make it far less likely that you introduce certain classes of bugs.  Or for another example, spending a few extra minutes getting clarification on what exactly a feature is needed for and what contexts it will be used in could prevent a whole raft of bugs.

There are many ways that bugs can be prevented, but what can a tester (working in the testing role) do to help with this?  Testing usually happens after the bug has already been introduced and then it is too late to prevent it right?  Or is it?  One of the things that has been helpful to me is to recognize that the activity of testing and the results that we report out of it can be used to influence the way that a team thinks about some things.  For example, something as ‘innocent’ as an email saying: ‘I’m planning some testing of feature X and I wanted to make sure I’m not duplicating work on this. What kind of unit tests have you done on this?’ can gently nudge coders to think about unit-testing their code (and the bug prevention benefits that come from writing code that is unit-testable in the first place).  As Maaret pointed out in this post, there are many ways we can nudge things (or speed up movement) in the right direction.

So as a tester, don’t just focus on finding bugs.  Think about how you could help prevent them as well.

On the face of it testing seems like such a simple thing.  You just need to check that everything works right?  Seems straightforward.  However, I think anyone who has tested for any period of time knows that the reality is much more complex than this.  The longer I work in testing and the more I try to learn and grow in my skills, the more I come to recognize this.  The skills required to be a good tester are vast and varied.  Not only are there a myriad of technical skills that one can (should?) learn, but there are also many skills that are not specific to testing itself but that are general to knowledge work.  We need to know how to navigate relationships and politics in a company.  We need to be effective and skillful communicators.  We need to understand process and change management.  We need to know how to work well with those that we like and those that we don’t.  We need to understand how business need and financial pressures affect what we do.

In my current work, we make simulation software and there is one interesting thing that happens.  The more powerful computers and our software get, the bigger the problems our customers will try to solve.  “Instead of just looking at each component in isolation why don’t we just look at all of them together and see how they interact?”  I feel like we are doing the same thing in much of our knowledge work (testing included).  Instead of looking at each piece on its own we are trying to put everything together into one big system and work with and understand the whole.  What this ends up doing is meaning that no longer do I have ‘just’ worry about testing, but I need to figure out how to fit in and understand all these many other functions and roles.  It is a more realistic approach to what is really happening – knowledge work has always been about relationships and understanding and not just you doing your simple task on your own – but it is a hard world to understand and navigate.  We still have so much to learn.

The previous model for shrinking our world down to a manageable size was to give each person a clearly defined role and action that they had to do and then pass on to the next person (think of an assembly line), but in knowledge work we are coming to see that this way of shrinking the world doesn’t work.  However, our response so far seems to be to make the world infinite and hope that we stumble on the right things to focus on. So the question is how do we shrink down the world when everything is so interconnected and interdependent?  To paraphrase G.K. Chesterton, I though I was getting into testing, but I find myself struggling with gigantic angels and demons.¹

One of the things I have been doing with this is putting limits on which areas I want to focus on when learning testing skills, but I wonder if there is more I can do to help shrink down this infinite world into something I can manage? How can I navigate what I ought to be doing and learning – not in the sense of what my job description says, but in the sense of doing the things that are best for the company?

One place where I can see the answer to this is to shrink down the deliverables instead of the roles.  Instead of giving each person a defined role and having them work within that role and not have to worry about the why of doing it, we need to give each team a small set of deliverables.  Within that context we have a limiting of what to do and learn, but we are not constraining each person to only do a particular thing.  What I need to know and learn and work on is defined by ‘whatever we as a team need to be done so that we can get this deliverable completed.’  Maybe, what I need is to learn how to do security testing, or maybe what I need is to learn how to help facilitate people in bringing different parts of the project together, or maybe what I need to learn is how to fix a defect. Whatever it is, what I need to do and learn is defined by the project and the deliverables and the skills needed by the team as a whole and not by something that is heavily and specifically tied to a particular role that I might have as a tester.

We keep going down deeper here, but of course the next question to ask is what are my deliverables as a tester on the team I am on?  We have certain feature deliverables that are pretty obvious, but I think there is another deliverable that I have some unique skills to help with and that is the long term quality deliverable. I’m not thinking here of the testing role in terms of finding bugs during testing, but I’m thinking here of working with the team as a whole to help produce good quality, durable code that will help us continue to produce value in the future.  This post is getting long, so I’ll wrap up here, having given myself (and hopefully you as well) something to chew on.

Footnotes

1. The actual quote is actually a little more profound “anyone who makes himself responsible for one small baby, as a whole, will soon find that he is wrestling with gigantic angels and demons” In Defense of Sanity – page 167

Yesterday and today, I had the privilege of attending some of the talks of the first ever 100% online testing conference – the online test conf.  It was an interesting experience.  I was unable to to fully commit to the entire experience as I was not given the day off and so had to work between (and sometimes during) the talks I attended.  Overall while I can see the appeal of something like this as it much cheaper to not have to have many people (and speakers) travel to one location, there are some downsides as well.

One of may favorite aspects of conferences is meeting other testers in other contexts and getting excited about testing together, and it seems that the online experience does not do this as well as an in person conference (although the slack channel helped with that).  One of the other things is the ease of dropping in and out of a talk. This may seem like a good thing as you can easily (without social pressures) drop out of a talk that isn’t relevant for you, but it also means that you can easily get distracted or have something ‘important’ come up which causes you to drop of talks that could be very helpful.

In terms of the actual talks I learned some really great things from some of them.  My favorite talks were Rob Lambert’s talk on the traits of employable people – I really got a lot of actionable items out of that one – and Adam Knight’s talk on risk.  Both of those talks gave me a lot to chew on and think about and apply in my life as a tester.

I’m glad I could (kind of) attend this conference and I’m thankful for Joel Montvelisky and his group for putting this together.  It’s exciting to see and try new things, but I’ll still be asking my boss for money to attend some ‘in person’ conferences as well 🙂

Our team was already tight.  Several members had been shifted to work on other projects and so we were pretty well down to a bare bones team.  And then it happened.  One of the testers took a job in another city since he wife had a job there.  At this point in time it does not look likely that we will get approval to replace him, and so we are faced with the question of what to do.  We have lost 1 tester on a team that had 4 full time and one part time tester.  At first glance this seems like a pretty serious problem.  We just lost almost 25% of the team.  How are we going to be able to keep up on the work that needs to be done?  However, after some reflection, I think this is actually an opportunity in disguise.  It is a new constraint for us and and constraints lead to creativity.  We have already seen some helpful and healthy shifts where developers will be taking on more of the regression maintenance work and where we are being forced to look carefully at our process from a full team perspective to see what things we can spread around the team more.

We all agree that having high quality is an important factor, and this constraint is forcing us to ask the questions about how we are going to get high quality.  It is making us as a team face the fact that quality is indeed something that has to belong to the team as a whole and not something we can just rely on testing to take care of.  So while it is certainly hard to lose a team member and while it can be very painful to have to transition to different ways of thinking, there are certainly some benefits to be had here as well if we are only willing to grab them. We need to let the creative juices flow on how we can deal with this constraint and how we can work together as a team to achieve our goals.

How do you harness serendipity in your testing?  Recently I have been migrating a number of tests from one regression system to another, and in the process of doing this I have a found several defects that highlighted areas we were not testing at all.  When I started this migration, I was not really expecting to find any issues, as we should have been just ‘copying’ tests from one system into the other.  This exercise has highlighted some of the limitations of automated testing as compared to more human involved testing – which is  topic for another day – but one of the biggest takeaways for me in this is the serendipity of finding defects.

I wasn’t looking for bugs in particular and yet I found many important bugs and it seems that this is a common pattern in my testing.  Often many important bugs are found while trying to do something else.  Sometimes that is finding bugs that are unrelated to the feature or area I’m testing.  Sometimes that is finding bugs while trying to do a demo or to walk someone else through the product.  Sometimes that is finding bugs while doing something like this migration that seems incidental to bug hunting.

In thinking about this I wonder if there are ways we can harness this serendipity?  Are there things I can do as a tester to make it more likely that I find these things?  Can I deliberately put myself in contexts that make it more likely that I will run into this stuff?

I think they answer is that yes I can!  I’m reading through Elisabeth Hendrickson’s book Explore It! and she has lots of ideas in there about changing your context or way of thinking about the system so as to make it more likely to run into these serendipitous bugs.  Exploring things from different angles and with deliberate context shifts will certainly help to find these things.  One thing I wonder though is if I can do this for a particular feature.  Many of the approaches and tactics for exposing ourselves to this type of serendipity seem to apply better to the product holistically and less to checking a particular feature.  We do of course want the holistic approach, but when I’ve been assigned to work on testing a particular feature, how do I expose myself to finding these totally unexpected insights about this feature?

At this point, I’m just thinking out loud some ideas that I will need to try in the next weeks and months, but perhaps one of the things I can try is situating this feature inside workflows.  This would mean starting with working through several ‘likely’ workflows in which users might exercise the feature and then using those workflows to work out the states the feature might go through and then using those states to help me dive deep into the feature in those particular areas.

I usually ask developers what their thoughts are on how to test a feature before I get started, so perhaps I could flesh that out a bit more and instead of just asking them an open ended question, I could come to them and say ‘this is my understanding of how the feature works.  Does that match your understanding of it?’  By exposing my ignorance I will hopefully be able to gain better information of areas I don’t understand as well and I might also gain insight into areas that are less clear for the developers as well.

One final idea (for today) is that I could create artificial constraints on the feature.  For example only allowing myself to interact with it using the underlying command line interface, or only interacting with it via right click options in the UI.  By creating these constraints for myself I’ll end up forcing myself to come up with creative ways of doing things that I would not otherwise think of.

I think there are many ways to increase exposure to this kind of serendipitous discovery, and I guess it mostly boils down to breaking out of the ruts and habits of approaches to testing the product.  In thinking about this now, I think even the ideas given here won’t work in the long run either if they become habits.  The only way to keep stumbling onto stuff like this is to keep changing the things that I do and the ways that I approach testing.  I can of course, have my favorite approaches, but to stick exclusively to them, will mean that I will fail to find out many interesting things about the product.  The way to harnessing serendipity is to become a shape shifter.

I have been trying to draw out everything I can – even things that don’t seem like that would make sense to draw – just so that I can get a feel for where it might be useful to use more visual tools.  One of the things that I have discovered is that visualization can be used in many many different situations and ways. Almost everything that I have tried this with has given me positive results, and I have also discovered that there are a lot of different ways to communicate visually.  Words themselves can be turned into helpful visual communications just by playing around with their layout or by adding in connectivity between them etc.  The classic model of this is of course the mind map, but I have found it helpful to use other combinations of text and visual elements at well.  For example, checklists with lines drawn between some of the bullets on the list can add a layer of meaning that would not otherwise be there.  Another thing that I have found really helpful is to make changes to ‘typical’ formats that we are more familiar with.  An example of that would be creating a table that has some cells with text and some with images.

In my experience so far it would seem that you can use visual elements to enhance the way you think about, execute and communicate almost any task you do, but I have noticed that some types of visual communication are better suited to certain types of task that others.  For example, mind maps are very helpful in thinking about feature interactions when planning some testing, but they don’t work as well when trying to understand and think about the workflows a user might do or the states of the product as we execute that workflow. My current experimentation is geared around figuring out if there are some general principles I can use to help in deciding which types of doodling to include in what am doing.

As I’m writing this I realized I didn’t even try any visual thinking exercises for this post.  Hmm, maybe I’ll have to  experiment with that next time!

So I have been practicing doodling and thought I would give an update on my progress

First a picture

As you can see, I am no artist, but I’m not going to let that stop me.  I have already been looking for ways to practically use doodling, and the reality is, at lot of the stuff that I have tried to use doodling for has ended being even simpler than the stuff in this picture.

I have already been able to use it in test plans where a simple doodle can illustrate far better what I am thinking than a large block of text.  I have also used visual communication in several other areas as well, including bug reports, test coverage reports and in preparing presentations I had to give.  I am also looking at using story boarding and other techniques to help describe and understand user workflows.

So far this has been a lot of fun and I would certainly recommend others to try their hand at more visual approaches to thinking and communicating as a tester.  Up until now I have been trying it out everywhere I can, even in place where I don’t think it would make sense, just to see where it might be effective.  As I continue this experiment I will be focusing in on areas where it seems to be the most effective.  Stay posted for more thoughts on the most powerful times, solutions and ways to use visual approaches.

Inspired by Danny Dainton’s recent post about focusing his learning on a particular area, I have decided to do something similar.  I too have a concentration problem.  I love learning things and I get excited when I hear a new ideas and before I know it I have 5 different things I’m trying to learn at the same time and I end up having a very shallow understanding of any of them.  It’s kind of like that last sentence – I’m all into and.  I want to learn this, and this, and this and this.  That approach might work well when your starting out on learning something new and you need a quick survey of the topics there are to learn, but if I want to really become a testing craftsman, I need to be able to focus my learning and go deep on some things.

So I’ve decided to do something similar to what Danny is doing, and have focused time spent on learning new topics and skills that will help me be a better tester.  I’m not necessarily going to do a month per topic, but I am going to have a current focus on the go with some goals around it.  Once I meet those goals or I feel I have exhausted the topic or I don’t think that going deeper is going to add any more benefits, I’ll move on to the next topic.

I’m hoping this will give me discipline in my learning and help me to actually focus in on a topic until I have a reasonable mastery of it. I’m also hoping that this will help get rid of some of FOMO.  By knowing I will get to a particular topic in its due time, I can be free from the fear of missing out on some important thing that I think I need to know.  I can bookmark things for later and feel confident that I will get to them as part of my ‘learning plan.’

So no more shotgun learning.  It’s time to be a sharpshooter.

My first topic was inspired by a talk that Karen Johnson gave at the KWSQA conference here in Waterloo, Canada (You can view a recording of the talk here).  At one point in her talk she spoke about using visual thinking to help with problem solving.  I’m not very good at drawing, but I think it would be very helpful for me to be able to think and communicate more visually so my first project is going to be around learning to draw/doodle and how to use that skill effectively as a tester. We will see how it goes. 🙂

Today I found 3 bugs without even opening the program I was testing.

You might not believe me, but it is actually true.  I found three bugs without even using the software being tested.  And it gets better.  I didn’t have to file defects for them and they got fixed within a couple of hours.  This sounds like a tester’s dream doesn’t it?  Finding bugs with very little work, not having to file defects and getting them fixed very quickly.  How did I do this?

Well, I had a task to plan some testing for a story, so I set up a meeting with the developer working on the story to go over what the feature did and to see if there were any things to be aware of as a tester.  The developer gave me a demo of what he had done on the feature, but during the demo I pointed out that he had a filter applied which was hiding some parts of the panel we were looking at.  When he removed that filter he realized there were some things he had not thought of.  There were also a couple of things that he had implemented according to what had been defined in the story, but after discussion we agreed that for these items the way the story said it should be done, would not do what users needed and so after a quick check with the story owner, we had two more bugs that needed to be fixed.

And so that’s how I did it.  I had a quick meeting with a developer and just by asking a few questions found three bugs. Pretty good start to the day 🙂

I work on a distributed team.  The testers are spread across two offices (one in Canada and one in the UK) and the developers are spread across 6 offices in Canada, the UK, Germany and the US.  As you might imagine, this leads to some difficulties with product development.  Communication is one of the things that is difficult both due to the logistics of different time zones and locations, as well as the tendency to revert to poor methods of communication (like email).  We have communication tools like Google Hangouts and other video conferencing and screen sharing services that can be really helpful, but there is still a barrier to communication in those mediums that is not seen in face to face communication.

But this is the team I’m on.  There are obviously challenges that come from being a distributed team, so what are the ways we can overcome them, and perhaps even leverage them to use their power?  One of the things I have heard said, and seen confirmed in my observations of various team and products, is that your product tends to look like the team that build it.  So for example if each member of your team is holed up in their own office and only talks to other members of the team when absolutely necessary, your product is going to reflect that.  You’ll end up with a disjointed product were each part feels separate from the others.  Now a distributed team obviously doesn’t have to look like this.  There are some additional hurdles that come from being distributed, but there are also many teams that have produced products that show the advantages of working in this way  I think the reality of it comes down to understanding what your are trying to do and what kind of team you want to have and then utilizing the tools you have to help you work towards seeing that team come about.  

One of the biggest challenges I’ve seen is what I would call the interrupt challenge. It is is tricky thing to balance out when you should or shouldn’t interrupt someone with a question.  On the one hand we know that in order to get deep work done we (testers and developers) need significant periods of uninterrupted time, and yet on the other hand, it is easy to waste hours trying to figure out something that could be solved with a simple question to someone that knows.  So finding that balance is hard.  I think that in co-located scenarios we can tend to interrupt too often.  It’s just too easy to walk up someone’s desk or pop your head over their cubicle wall and interrupt them.  

However, in the distributed team setting, I think you end up with not enough interruption.  For some reason it feels more intrusive to call someone on Skype than it does to stop by their desk and so we interrupt less often.  This ends up causing several issues.  For example this can sometimes lead to duplication of work.  I don’t want to bother you with a question and so instead I start working on something that will solve my problem and end up doing so, but at the expense of having duplicated work you already did someplace else.  It can also lead to poorly defined interaction points, where I make assumptions about what your code does based on reading it or observing it, instead of asking you about it.  Now, hopefully most of those things are flushed out in code reviews but it is still likely that some things slip through which can only hurt the quality.  

So what is a tester to do about it?  How can a tester work in a way that makes a distributed team more effective?  I think one of the important things to remember is that testers are often the glue between the various parts of the system.  Although we may not understand the inner workings of the code, we use all the parts of the code and we have a pretty good idea of what will and will not work together.  Getting the tester’s input early in the project is usually helpful, and so as a tester I need to the know that usually the most valuable time to ‘interrupt’ is right at the start.  While the story is still being defined or as soon as possible after the developer has started working on it.  Interruptions at that stage may save many headaches later on. This is especially important to remember when working on distributed teams were the desire to not interrupt can become stronger.

So what about you?  What interrupt balance to you try for?