Finding bugs without starting the program

Today I found 3 bugs without even opening the program I was testing.

You might not believe me, but it is actually true.  I found three bugs without even using the software being tested.  And it gets better.  I didn’t have to file defects for them and they got fixed within a couple of hours.  This sounds like a tester’s dream doesn’t it?  Finding bugs with very little work, not having to file defects and getting them fixed very quickly.  How did I do this?

Well, I had a task to plan some testing for a story, so I set up a meeting with the developer working on the story to go over what the feature did and to see if there were any things to be aware of as a tester.  The developer gave me a demo of what he had done on the feature, but during the demo I pointed out that he had a filter applied which was hiding some parts of the panel we were looking at.  When he removed that filter he realized there were some things he had not thought of.  There were also a couple of things that he had implemented according to what had been defined in the story, but after discussion we agreed that for these items the way the story said it should be done, would not do what users needed and so after a quick check with the story owner, we had two more bugs that needed to be fixed.

And so that’s how I did it.  I had a quick meeting with a developer and just by asking a few questions found three bugs. Pretty good start to the day 🙂

The Interrupt Challenge

I work on a distributed team.  The testers are spread across two offices (one in Canada and one in the UK) and the developers are spread across 6 offices in Canada, the UK, Germany and the US.  As you might imagine, this leads to some difficulties with product development.  Communication is one of the things that is difficult both due to the logistics of different time zones and locations, as well as the tendency to revert to poor methods of communication (like email).  We have communication tools like Google Hangouts and other video conferencing and screen sharing services that can be really helpful, but there is still a barrier to communication in those mediums that is not seen in face to face communication.

But this is the team I’m on.  There are obviously challenges that come from being a distributed team, so what are the ways we can overcome them, and perhaps even leverage them to use their power?  One of the things I have heard said, and seen confirmed in my observations of various team and products, is that your product tends to look like the team that build it.  So for example if each member of your team is holed up in their own office and only talks to other members of the team when absolutely necessary, your product is going to reflect that.  You’ll end up with a disjointed product were each part feels separate from the others.  Now a distributed team obviously doesn’t have to look like this.  There are some additional hurdles that come from being distributed, but there are also many teams that have produced products that show the advantages of working in this way  I think the reality of it comes down to understanding what your are trying to do and what kind of team you want to have and then utilizing the tools you have to help you work towards seeing that team come about.  

One of the biggest challenges I’ve seen is what I would call the interrupt challenge. It is is tricky thing to balance out when you should or shouldn’t interrupt someone with a question.  On the one hand we know that in order to get deep work done we (testers and developers) need significant periods of uninterrupted time, and yet on the other hand, it is easy to waste hours trying to figure out something that could be solved with a simple question to someone that knows.  So finding that balance is hard.  I think that in co-located scenarios we can tend to interrupt too often.  It’s just too easy to walk up someone’s desk or pop your head over their cubicle wall and interrupt them.  

However, in the distributed team setting, I think you end up with not enough interruption.  For some reason it feels more intrusive to call someone on Skype than it does to stop by their desk and so we interrupt less often.  This ends up causing several issues.  For example this can sometimes lead to duplication of work.  I don’t want to bother you with a question and so instead I start working on something that will solve my problem and end up doing so, but at the expense of having duplicated work you already did someplace else.  It can also lead to poorly defined interaction points, where I make assumptions about what your code does based on reading it or observing it, instead of asking you about it.  Now, hopefully most of those things are flushed out in code reviews but it is still likely that some things slip through which can only hurt the quality.  

So what is a tester to do about it?  How can a tester work in a way that makes a distributed team more effective?  I think one of the important things to remember is that testers are often the glue between the various parts of the system.  Although we may not understand the inner workings of the code, we use all the parts of the code and we have a pretty good idea of what will and will not work together.  Getting the tester’s input early in the project is usually helpful, and so as a tester I need to the know that usually the most valuable time to ‘interrupt’ is right at the start.  While the story is still being defined or as soon as possible after the developer has started working on it.  Interruptions at that stage may save many headaches later on. This is especially important to remember when working on distributed teams were the desire to not interrupt can become stronger.

So what about you?  What interrupt balance to you try for?

Generating Test Data

One of the difficult problems of testing is coming up with good test data to use.  In my world that often means being able to find particular files with particular types of data in them that I can use and so I have written a utility that searches for these kinds of files.  However, I have been thinking recently about how to find or generate data that I might not otherwise think of.  Humans are notorious for having certain cognitive biases and as a human, I guess I’m subject to this as well.  Mindfulness and other techniques can help to reduce the effect of cognitive biases, but they will never fully take them away and so I’ve been thinking about how we can leverage and use computers to generate useful input data.

There are a couple of ideas I have had so far:

There are some random generators on the web that will generate random data of a particular type (ex. floats or strings), which you could then use in your test fields.  In general the ones I have been able to find are a bit too ‘nice’ though. For example, floats like 0.0 or 1.0/3.0 are much more likely to cause issues than other random float we might generate and so its seems that it would make sense to bias your float generator towards those kinds of numbers.  Randomness in itself can help you find some issues you might not otherwise, but coupling that with built in biases that might counteract or complement your normal biases is helpful as well.  I’ve been looking into writing some of my own data generators here to help me out with these kinds of issues as this seems to be an area the current test tooling options are weak on.

Another idea, is to let a random number generator choose my input files for me.  As a creature of habit I tend to have my favorite set of input files that I use.  By creating  a script that will randomly pick a file from the database, I could increase the coverage of the characteristics in the files while also increasing the likelihood of running into some totally unexpected issues.  By adding randomness into the test, I may increase the probability of running into a black swan event.

Those are just a couple of ideas off the top of my head.  What about you? What do you use when generating input data for your tests?  Have you been able to use data generators to help overcome some of your natural biases?

Do I Own Quality?

I have been noticing lately that in my job I’ve somehow shifted towards more of what could be described as a gatekeeper role.  I’ve built some very good relationships within the team, and developers are now much more willing to get my feedback and input which is great, but somehow this has morphed into me being a quality gatekeeper for the product.  Quite often now I will have a developer approach me with a branch that they would like feedback on before it gets merged into the main release branch.  This is great as it allows us to find bugs sooner and keep the release branch in much better shape.  

However, in doing this I have noticed that I’ve become a gatekeeper.  I’m the guy who declares “this is ready for merging into the release branch.”  And so, although the gate may have moved downstream, it is still there and I’m not sure if that is what I ought to be doing as a tester.

There are a few things about being a gatekeeper that make me uncomfortable.  For one thing, it means that in a lot of ways I’m now the one primarily responsible for making sure that the quality is high.  If bugs show up the developers can wash their hands of them and say that they gave it to the tester to look at.  Even though the developer a very willing to work on fixing bugs when I find them, it seems like this is fostering an environment where quality is the tester’s problem and not the team’s problem.  Another thing about it that makes me uncomfortable is that where there is a gate there tends to be a narrowing effect, which means it is very easy for testing to become a bottleneck (and we have already seen that happen on the team).  

But what to do?  We had a lot of issues before with developers merging stuff in whenever they felt it was ready which was causing broken builds, so the process we have now is actually something we as testers advocated for to help fix some of these problems.  And yet now we face another set of unintended consequences.  What can or should we do to change this?

I’m not sure that I have or know the answer at this point.  I think the big picture idea that we need to get into place is making it so that the whole team owns quality in a real and meaningful way.  The question is how do I do that?  Some companies have gone the unified engineering route to deal with this issue, but I don’t see something like that happening in our company any time soon and the team I am on does not want to voluntarily move in that direction.  So what else can I do to help make it so that the whole team owns quality?  What practical steps can I take as a test-practitioner to move the team towards producing high quality code in a way that does not rely too heavily on testers?  

Perhaps what I need to do is to work with the developers so that they become better testers.  Even if we have an independent test team, there are still many advantages to the developers if they write better code and maybe I need to sell them on those advantages and work with them to help them get better at it.  I think one of the biggest reasons we’ve become gatekeepers is that we as testers don’t trust the developer’s testing (and they don’t always trust their own). This is based on the history of finding so many issues after developers said they were done with code and so is certainly not an unreasonable thing, but if we could get rid of this mistrust of developer’s ability to produce good quality code, we could remove the need for gatekeeper-like activity and become much more efficient as a whole (test and dev) team.  

Another thing I could do, is to work with developers on improving the automation that we have available to run.  If we can have a better automation suite, we can have far more trust that new code is not breaking things and reduce the need for as much tester time put into it.  On a related note, having much faster automation would help as well since we would be able to run more tests more frequently.  Hmm, maybe working with developers on adding and enhancing unit test suites would be helpful here too.

Sounds like I have plenty to do.  How about you?  Do you ever feel like a gatekeeper?  What things have you done to help make quality something the whole team owns?