Test Links Thursday

Exploratory Testing Self-Management – Good exploratory testing is hard. Managing it can sometimes be even harder.  Some really good tips here.

Test Tradeoffs – Think about your tests in terms of what you want them to do, instead of just what kind of test they are.  This one has really got me thinking.

Mentally Owning a book – Sounds like a lot of work (maybe more than it is worth), but do you mentally own your books?

Password Managers (video) – What password manage do you use?

Usability First


The job of a tester is not just to find issues, but it is also to present the case for why those issues matter.  This is much the same thing as saying that we want to find relevant issues and see the quality of the product improve.  There is another side of the equation though.  As with most things in life, trade-offs have to be made and the trade-off here is the balance between how important and relevant an issue is and how much work it will take to fix it.  Some things may be important to fix, but may end up being too expensive to fix and so don’t get addressed.  This can be very frustrating for a tester who sees the importance of the issues found but perhaps has less exposure to the cost of fixing it.

However, something that struck me recently is that there are certain kinds of feedback that are more important to give early in the process.  Some types of issues become more and more expensive to fix as you get later in the release cycle. In my experience workflow and usability problems typically fall in this category.  Early in the development cycle of a feature many things are changing and so it is actually much easier to change workflows or usability issues.  As the feature gets more mature, other things start to depend on it being a certain way and APIs are defined and there is upgrade code in place for older versions, etc.  Making workflow changes at this point in the process is often much more painful and so the cost side of the equation goes up.  As a result, we often have these kinds of issues dismissed as not worth fixing or as a system limitation.  However, I have noticed that if we find these kinds of issues early in the cycle they are much more likely to be addressed.  The benefit side of the equation stays the same, but the cost side has gone down and so the issue gets corrected.

Based on this experience I have been wondering if I need to shift my testing approach a bit and really focus on finding workflow and usability issues early on.  Typically my approach has been to bug hunt at the early stages and try to give the developers feedback about bugs early so they can fix them early. I still think this is a good approach, but what I think I need to do is shift my search to be for certain types of bugs.  I need to look for the usability and confusing workflow issues early and leave the testing for coding and logic errors and crashes and other things that are clearly defects until later in the cycle.  These other items are things that usually have less cost increases as development progresses and so it is ok to find them later in the process.  We need to ask questions early, but we need to make sure we are asking the right kinds of questions as well.

Test Links Thursday

Read more books  – Most advice on how to read more books is just silly, but here is some actually helpful advice on how to  become more of a book reader.  I love reading blogs, but I think that to really advance in an area you have to get into the books.

Discomfort as a tool for change – A lot of interesting thoughts and insights in this one.  The biggest takeaway for me was the whole mindset towards testing.  What are we really trying to achieve and is the testing we are doing working towards that?

Including this just for this quoteAt first it was done so that nothing worked. Then it was done so that simple and straightforward things worked. Then it was done so that most things worked” – Sounds like the definitions of done I hear a lot 🙂

Pouring Molten salt into water (Video) – Just for fun.  Always fun to watch an explosion and it’s even better if you can think that you are learning something along the way.

When your workflow is a bug

Today I was trying to add something onto my cell phone plan, and I found a bug on the website.  To be fair, I think everything was working ‘as designed’ but it was still an annoying bug for me.  I have had similar experiences the last few times I have tried to do things on their website and this bug even had me toying with the idea of finding another phone provider.

So what was the bug?  Well, I was on the ‘plan summary’ page and there are three things you can do. (1) change your plan – I’m not interested in this so we’ll leave it aside. (2) a Manage link shows up beside the monthly add-ons and (3) an Add now link shows up beside something called Roaming add-ons.   What I want to do is add a long distance add-on to my plan, so I decide to click on the Add now link.  It takes me to a page that has different available add-ons, including the one I am interested in.  I click on the link to ‘add to my plan’ and it takes me back to the plan summary page I started from.  What the @#$%?!  After a few paroxysms and trying on different browsers etc.  I realize that I need to use the link to Manage my monthly add-on instead, because I am trying to add a monthly add-on and not a temporary roaming one.

Now my guess is that everything is working as expected.  When I used the correct link, I had no problem getting the add-on I wanted, but there are some serious workflow issues here.

  1. If I click on the Add roaming add-ons link and I can’t add monthly add-ons from the resulting page, they should not show up on that page.
  2. The guidance on the initial page should be improved to help me pick the right link in the first place
  3. And most importantly of all, there shouldn’t be a workflow that takes me in a circle.  If I started on a page, I should not be taken back to that same page with nothing at all changed.  Either take me directly to another page that will allow me to do what I want, or, at a minimum, give me a message letting me know why I ended up back here and suggesting some possible alternatives for me to try.

So, before this turns into a rant, let’s stop and think about some lessons we can learn.  The first lesson, of course, is that workflows matter.  Everything can be technically working, but still be very frustrating and difficult for the users.  Another lesson is a new hueristic  that I found: don’t go in circles. Are there any circular workflows in your product that might frustrate people?  I will be reviewing my product for this!

A third lesson, which I hope every tester knows, is that people will do things you don’t think they will.  If you provide two paths and you expect that users who want x will use path (a) and that users who want y will use path (b), don’t forget to think about the feelings and perspectives of users who want y but go down path (a).  Just because it ‘doesn’t make sense’ for a user to go down path (a) to achieve y  doesn’t mean your users won’t do that. Anticipate that possibility and help them out that situation and you’ll have a much better quality product.

Looking this post over it is almost a bug report.  Maybe I should just apply for a job at this company 😉


Test links Thursday

Not Right now –  I would add if someone frequently interrupts you, maybe you need to schedule some time with them so that you can pass on the information they need to know

Detroit’s Patterns of Growth – fascinating look at the development of city streets over time.  Do you see any parallels here to the development of your software framework?

Everything – (Testing tool recommendation) – Best tool I have found for easily searching for and finding files on my computer.

Because we all need to look up and contemplate the world sometimestumblr_nufkppjms31t5dwujo1_1280

Keep it Simple


I am always amazed by how many bugs can be found with the simplest of setups. For our (physics simulation) software, I would estimate that half of the bugs I find are on a simple cube without doing anything fancy.  However, in trying to think of testing as more than just finding bugs and also as something that can help drive quality in a holistic way, the challenge of finding good (interesting) input data becomes more important. For this kind of testing the important thing is variety and variation (as well as getting as close to customer data as possible).  So how do I find this kind of input data?

There are a few approaches I take to this.  In the first place I can always just create it, and over the last 9 years of testing I have of course created many different kinds of input data that I can use.

This naturally leads to the second approach which is finding data.  One of the challenges of having a lot of different files and sets of input is being able to find the one(s) you need in your current situation.  This is a search problem and I have tried a lot of different tools and strategies over the years to address this.  I have written custom searching tools that dig into the various data sets I have and provide searchable metadata.  I have also tried indexing my machine with google search and other third party tools, but over the years I’ve found the simplest solutions are often the best.  We now have a simple database that searches for files of certain types, opens them and saves an image of them.  This image along with the file location and a very limited set of metadata about the file is displayed on a internal web page.  There is some very limited text searching available, but the main power of this database is to give a rapid visual idea of the what you might expect a particular data set to be good for.

It is quick, dirty and simple, but therein lies it’s power.  It’s leveraging something the human brain is great at (rapid processing of images) to help us testers accomplish something.  Sometimes I spend too much time on trying to come up with elegant complete solutions when a quick and simple solution will give most of value in a fraction of the time.  I was struck by this again yesterday when trying to find some customer input data.  The first thing I wanted to do was to construct some complex queries to find stuff in the support system, but instead I decided to whip together a quick and dirty script that would search through the shared drive for files with a particular set of keywords in their names.  Will this miss a lot of potential matches?  Of course!  People often don’t name things as I would expect.  But did I quickly find a bunch of data that I needed with a minimum investment of effort?  Yup.

So I guess the moral of the story is that although there may be times when the complex solution is the answer, you might just be able to get by with a simple approach or tool instead.  Keep it simple.  Be efficient.

Testlinks Thursday

Nobody Wants to use software: – It’s not features that add value to your software.  It’s how well you help users achieve their desired outcome

Frustrations on system test automation efforts – I have felt like this!  Now to figure out how to change it.

Should developers test their own work? – This was discouraging.  The points raised weren’t bad in themselves, but the idea that the only way to ‘fix’ it is to throw testers at it isn’t something I can get behind. Almost everybody in this thread seems to agree that there isn’t much developers can do to get good (or at least better) at testing their own work.  I want to hope that isn’t the case.

Be part of an Awesome team – Some great ideas here on how to be instrumental in making your team awesome.