Author Archives: offbeattesting

One of the difficult problems of testing is coming up with good test data to use.  In my world that often means being able to find particular files with particular types of data in them that I can use and so I have written a utility that searches for these kinds of files.  However, I have been thinking recently about how to find or generate data that I might not otherwise think of.  Humans are notorious for having certain cognitive biases and as a human, I guess I’m subject to this as well.  Mindfulness and other techniques can help to reduce the effect of cognitive biases, but they will never fully take them away and so I’ve been thinking about how we can leverage and use computers to generate useful input data.

There are a couple of ideas I have had so far:

There are some random generators on the web that will generate random data of a particular type (ex. floats or strings), which you could then use in your test fields.  In general the ones I have been able to find are a bit too ‘nice’ though. For example, floats like 0.0 or 1.0/3.0 are much more likely to cause issues than other random float we might generate and so its seems that it would make sense to bias your float generator towards those kinds of numbers.  Randomness in itself can help you find some issues you might not otherwise, but coupling that with built in biases that might counteract or complement your normal biases is helpful as well.  I’ve been looking into writing some of my own data generators here to help me out with these kinds of issues as this seems to be an area the current test tooling options are weak on.

Another idea, is to let a random number generator choose my input files for me.  As a creature of habit I tend to have my favorite set of input files that I use.  By creating  a script that will randomly pick a file from the database, I could increase the coverage of the characteristics in the files while also increasing the likelihood of running into some totally unexpected issues.  By adding randomness into the test, I may increase the probability of running into a black swan event.

Those are just a couple of ideas off the top of my head.  What about you? What do you use when generating input data for your tests?  Have you been able to use data generators to help overcome some of your natural biases?

I have been noticing lately that in my job I’ve somehow shifted towards more of what could be described as a gatekeeper role.  I’ve built some very good relationships within the team, and developers are now much more willing to get my feedback and input which is great, but somehow this has morphed into me being a quality gatekeeper for the product.  Quite often now I will have a developer approach me with a branch that they would like feedback on before it gets merged into the main release branch.  This is great as it allows us to find bugs sooner and keep the release branch in much better shape.  

However, in doing this I have noticed that I’ve become a gatekeeper.  I’m the guy who declares “this is ready for merging into the release branch.”  And so, although the gate may have moved downstream, it is still there and I’m not sure if that is what I ought to be doing as a tester.

There are a few things about being a gatekeeper that make me uncomfortable.  For one thing, it means that in a lot of ways I’m now the one primarily responsible for making sure that the quality is high.  If bugs show up the developers can wash their hands of them and say that they gave it to the tester to look at.  Even though the developer a very willing to work on fixing bugs when I find them, it seems like this is fostering an environment where quality is the tester’s problem and not the team’s problem.  Another thing about it that makes me uncomfortable is that where there is a gate there tends to be a narrowing effect, which means it is very easy for testing to become a bottleneck (and we have already seen that happen on the team).  

But what to do?  We had a lot of issues before with developers merging stuff in whenever they felt it was ready which was causing broken builds, so the process we have now is actually something we as testers advocated for to help fix some of these problems.  And yet now we face another set of unintended consequences.  What can or should we do to change this?

I’m not sure that I have or know the answer at this point.  I think the big picture idea that we need to get into place is making it so that the whole team owns quality in a real and meaningful way.  The question is how do I do that?  Some companies have gone the unified engineering route to deal with this issue, but I don’t see something like that happening in our company any time soon and the team I am on does not want to voluntarily move in that direction.  So what else can I do to help make it so that the whole team owns quality?  What practical steps can I take as a test-practitioner to move the team towards producing high quality code in a way that does not rely too heavily on testers?  

Perhaps what I need to do is to work with the developers so that they become better testers.  Even if we have an independent test team, there are still many advantages to the developers if they write better code and maybe I need to sell them on those advantages and work with them to help them get better at it.  I think one of the biggest reasons we’ve become gatekeepers is that we as testers don’t trust the developer’s testing (and they don’t always trust their own). This is based on the history of finding so many issues after developers said they were done with code and so is certainly not an unreasonable thing, but if we could get rid of this mistrust of developer’s ability to produce good quality code, we could remove the need for gatekeeper-like activity and become much more efficient as a whole (test and dev) team.  

Another thing I could do, is to work with developers on improving the automation that we have available to run.  If we can have a better automation suite, we can have far more trust that new code is not breaking things and reduce the need for as much tester time put into it.  On a related note, having much faster automation would help as well since we would be able to run more tests more frequently.  Hmm, maybe working with developers on adding and enhancing unit test suites would be helpful here too.

Sounds like I have plenty to do.  How about you?  Do you ever feel like a gatekeeper?  What things have you done to help make quality something the whole team owns?  

I’ve been thinking lately about the idea of craftsmanship in our work.  In the consumer driven society we live in, we have embraced the idea of planned obsolescence for our physical products and I think in some ways our cultural viewpoints come into play in software development as well.  Software is made by people, and people bring their cultural views into everything they do.  Perhaps this is part of the reason we see so many shoddy examples of software. If it sells for a while and isn’t sustainable, oh well.  We’ll just make something new to sell you.

I think though, that there is a bit of a cultural change on the way.  We have tried rampant consumerism as worldview for a few generations and found it wanting.  Younger generations seem to want to see something change, to find some rootedness and depth in something.  I think we are not yet sure what that is, but we are looking and trying to find the answers to the question of what we find our value in if it isn’t in being stuff accumulating, resource hogging, consumers. One of the things that we have started to see in response to this is a resurgence of the idea of craftsmanship.  Hand crafted, custom made things are all the rage.  

While there are things to be careful of here (I don’t think craftsmanship on its own can fill that need for meaning we are looking for), there is also something to consider here as well.  What is the balance between long and short term value?  And this isn’t just something to think about in the physical world.  This is something to consider in the digital world as well.  What is the balance we should have between quickly getting something out the door, and releasing something that has quality baked in from the start?  Do we have a responsibility to create something that has inherent beauty and durability, or is our goal just to make sure company profits are maximized?  I actually don’t mean these as rhetorical questions. I think we need to grapple with where on these continuums we fit.  As employees we have responsibilities to our employers, but as ethical humans we also have responsibilities for how our behavior affects others as well.

As I think about it, I would like to be a craftsman.  I don’t want to contribute to the cultural zeitgeist of a disposable, transient, ungrounded, consuming-as-meaning, society.  I want to be a part of something that brings stability, groundedness, beauty and quality into our societies.  Maybe as a software tester my dreams are too lofty, but there they are.  That is what I would like to see.  However, in thinking about this, I’ve been struck by one thing in particular.  As a software tester, can I really be a craftsman?  I don’t (usually) create things. I’m helping developers and others see where there might be quality issues and I’m helping to smooth out processes etc. so that we can see better quality products going out the door, but at the end of the day I don’t make things.  So can I be a craftsman?  What does it even mean to be a craftsman as a software tester?

I think the answer lies in the team.  As a team of coders, testers, business analyst, writers, and others, we produce a product.  Whether this products is a disposable throw-away product, or a high quality craftsman like product, is a reflection of the attitudes on the team. When I focus on changing myself and becoming a better tester without explicit reference to the team and how my growth in skills will fit into the team and change the team culture for the better, I’m missing out on what it really means to be a tester. If I want to be a testing craftsman, I need to let go of the identity of a tester to some degree and I need to embrace the idea of being a part of a team that works together as a team to achieve long lasting high quality software.

If you think about it, this has some pretty big practical applications.  It means thinking of myself as more of an advocate on the team for cultural changes that help enhance code quality even at levels that are not customer visible.  It means less of an emphasis on being only a bug finder and more of an emphasis on being an agent for cultural change on the team. It means helping others to see the importance of having clean code and a commitment to quality.  It means pushing and advocating for things that might be expensive in the short term but with long term payoffs.  It means leading and teaching, and not just nagging.  It means working with developers and getting to know them.  It means leaving my desk more and engaging with others on the team. It mean having hard conversation and fun conversations, and intense conversations.  It means all this and so much more.  It means refocusing and redirecting toward the team instead of towards myself.

With the size of most modern development projects, we can’t work as a lone craftsman in the workshop lovingly making beautiful products.  But this doesn’t mean we need to become just a cog in the production process, a mindless worker on the assembly line.  We can still embrace craftsmanship as part of a workshop.  As part of a team that makes and produces beautiful, high quality, well crafted software.  That is my goal as a testing craftsman.

As with many companies, my company has a performance review program in place. As part of this program we have to work with our managers to set goals for the year ahead.  This seems like a good idea in theory, but sometimes the way it worked in practice didn’t seem to be adding value.  In trying to make it more valuable for myself I have worked towards setting SMART goals. Here’s the problem though.  It didn’t work.

I would struggle to come up with specific, measurable, attainable, relevant and timely goals and after all the work put into coming up with them, I would still find they didn’t really help me.  Things would change too quickly and the goal would become irrelevant, or I just wouldn’t have a connection to the goal and so would be forcing myself to try and work towards it without any motivation, which works when your willpower is high, but not when it is low.  All in all, it just wasn’t working, and so I have been thinking about how to make better goals.

Originally the goal setting process for me was task related.  I would have goals that said something like, “spend x percentage of my time doing exploratory testing.”  That didn’t really feel like a goal at all. It was just a record of what I would be doing with my time, so I tried to re-write those as SMART goals into something like: “research and use at least 5 new exploratory testing techniques within 3 months.”  That seems like it would be a lot better, but it still didn’t work for me.  I had to really force myself to do it and I ended up giving lip service to it rather than really engaging with what I was doing.  Part of what was missing here was the intention behind the goal.  Why am I even doing this anyways?

But now what? If SMART goals aren’t working for me and if I want to try and get something useful out of the goal setting I need to do, what kind of goals should I set?  I could try another goal setting paradigm – perhaps HARD goals or BHAGs or something else – or I could try to come up with my own way of setting goals.

At this point it doesn’t seem worth it to me to go research and figure out a new goal setting method, and in some ways it feels like trying to find a magic bullet solution that will tell me how to make good goals (and by implication how to improve).  Rather than spending too much time and energy creating and polishing and measuring and tracking different goals, I will just create goals that put down in paper what I am trying to do in a way that I can refer to and use to help drive myself to change.  So to revisit the exploratory testing goal I could phrase it something like: “I want to become better at exploratory testing”  and leave it at that.  Sure it’s vague.  Sure it’s hard to measure.  Sure there are no time frames to do this in. I think I’m ok with that.  It might not be as ‘helpful’ as a more specific goal, but if it actually drives me to change something and do something and learn something it will really end up being much more helpful.  And on top of that it will be far less expensive.  I don’t have to put thought and effort into measuring and tracking it.  I can use whichever subjective experience I want to use (Am I less bored when doing it?  Do I find better bugs?  Do I find more bugs? – whatever things ‘works’) to track it.  It also took me about 5 seconds to come up with and so doing goals like this could save me a lot of time and energy to focus on – I don’t know – actually improving, instead of writing down things that might help me improve.  

Maybe SMART goals (or one of those other goal setting systems) works well for you.  If it does, wonderful!  I guess I just need something different.  Something that works well for me.  I’ll keep experimenting and maybe someday goal setting will be a time of the year I look forward too.

It has been said that a good heuristic for when to automate it when you are bored.  So then, I have a question:  what do you do when your automation bores you?  I’ve spent years working on and tweaking our automated regression system to that point that it runs very smoothly.  There are occasional glitches and breakages, but in general it is a very reliable system.  It fails when there are problems (most of the time) and it passes when things are ok (most of the time).  Adding new tests is easy. Sometimes I create one off tests that are not intended to be re-run directly in the automation framework just because of the useful debug and analysis tools we have built into our automation framework.

There is very little interesting work to do anymore, so what do we do?  Do we keep adding tests to the system and let ourselves get into a comfortable rut?  Add tests.  Occasionally remove an out of date test.  Clean up some test scripts for a development change.  Debug a failing test.  Rinse and repeat.  Sounds easy and comfortable, but it is also boring.  Where is the challenge? If boredom is good heuristic for automating, maybe I need to automate some of this. There are a couple of ways I can think of to automate the automation:

1. I could increase/improve the workflows around automatically updating test script cleanup or debugging.  However, most of the low hanging fruit has already been done in this area, so that is probably not a great use of my time
2. I could try to tweak the automation framework to help discover new issues instead of having it largely focused on checking for regressions.  This does sound interesting, but has some significant challenges which might require forking the system to a creating a separate framework for this type of testing
3. I could focus on new automation initiatives that could replace or radically enhance the framework to where it is doing something totally new and different.  For example, I could try to push testing down, by replacing these tests with lower level unit tests. Or I could try to add in traceability so that we can automatically know which sets of test to run against which code changes.  I would love to do this, but my skill level isn’t high enough yet to do this on my own.  I could of course partner with developers on this, so it might still be an option
4. I could create ways to measure the value of the tests that we have and/or new tests that we want to add.  A smooth running system can sometimes hide the real cost of things (because it’s just a half hour here and and a few minutes there) and and in the long run lead to more and more time spent on the boring work of maintaining the tests and system. We do try to remove tests that are not useful anymore, but it is hard to know which tests are in this category.  Having a way to measure the cost and value of a test would be helpful.  This a subjective and difficult thing to do, but the very exercise of thinking about a what make an automated test valuable and what the expenses are for automated tests would be useful in itself.

I think the way forward for me is to focus in on point 4.  I’m somewhat leery of metrics, but it will be useful to spend time thinking of heuristics that can indicate the cost and value of automated tests.  Hopefully by having these heuristics explicitly defined I will be able to use them to start conversations that will lead to more radical changes in the way we approach automation on our team.  

Testing is both easy and hard.  When I start testing a new build or feature, I often find 4 or 5 defects very quickly and without much effort (Whether it should happen like this or not is an article for another day).  However, once I have flushed out the easy to find bugs, I tend to get stuck and stop finding bugs, and once I stop finding bugs I get bored and move on.  This isn’t always a bad strategy as boredom can tell you that there isn’t anything interesting to find here, but sometimes others will come and look at something that I’ve tested and quickly find issues that I could have been able to find, or we will mark the story as done and move on, only to find some fundamental issues a few weeks or months later (or course, usually just as we are getting ready to release).  Often this issue will open my mind to whole new avenues of testing and I will end up finding several more issues based on the insights of that issue.

Many times finding defects earlier in the process saves a lot of time¹ and part of the point of testing is to find important defects early in process (at least before the customers find them) so I want to look at ways that I can get the inspiration to think of the product in new ways.  I have been reading through Elisabeth Hendrickson’s book Explore It! and have found some good insights in there that have been helpful.  However, there still is something more that I think I need to do.  

One of the things I have realized is that reporting on the status of a feature or story isn’t just something that is helpful for my manager or for the product owner, but it is something that is helpful for me.By having the structure and discipline of creating a test report, I think of things I might have forgotten (i.e. I am more strict about running through the checklist of things to consider).  As part of a test report I also try to give feedback on areas of risk, which means that the very act of preparing it makes me think about and consider what those areas are and helps to generate ideas for further risk.  It has has also helped me in communication with others as I follow up on information needed to fill out the report.  I continue to tweak the report template to make it present useful and timely information, but I will also be considering tweaks that will help make it a more useful tool for generating good test ideas.  One of the hardest things in testing is establishing a ‘done’ criterion and perhaps a report template is one of the tools that can help with this.

¹ I think

What is testing?  There are almost as many definitions of that as there are software testers and I don’t think it can be contained in just one definition (at least not succinct one). However, today I’m going to put forward a definition that helps to point out and emphasize some of what testing is.  This is not intended to be a definitive definition of testing, but is merely something that I’ve been thinking about recently.

Testing is figuring out how to align our judgement with the judgement of the customer and advocating for the product to change in ways customers judge as valuable

I think much of what we do as testers boils down to this.  We are trying to figure out what customers will want and care about and trying to find any areas in the product where it doesn’t match these expectations.  One of the hardest parts of our jobs is figuring out how to do this.  How do we know what the mythical ‘customer’ wants?  What do we do when two different customers want conflicting things?  How do we distinguish between what I want and what a customer would want?  How do we figure out the importance of the things customers don’t even know that they want or those things that have an indirect impact on customers (clean code, or testability features etc.)?

I’ve just raised a lot of questions that I have no intention of answering in the blog post, but come to think of it, these same things are also one of the hardest parts of a coder’s job as well.  Perhaps test and dev really are in the same game.  Maybe as testers a big part of our job is to take what we have learned about what customers want (and the ways we have managed to figure that out) and share that with developers so that they can better write code that meets customer needs the first time around.  Maybe being a ‘tester’ or a ‘developer’ isn’t what is important so much as being part of team that can produce software that meets the customer’s needs.  

Maybe we don’t need to sweat too much about what testing is, but should instead focus on what effective software development is.  If effective software development efficiently produces software that is valuable to the customers, where does testing fit into there?  I guess it depends on your team and how you work and what your skills are.  

For me I think it means learning how to pass on much of what I have learned to the developers.  I have skills that would help my team become more efficient, but sometime those skills are wasted on things like finding bugs.  Finding bugs is helpful of course, but wouldn’t it be far better to work with developers on preventing those bugs?  When I start testing a new feature and find 5 serious bugs in the first hour of testing, is that because I am skilled tester? – it may be, but does this use my skills in a way that maximizes their usefulness to the team?  Could I have helped developers gain a perspective on the feature such that 4 of those bugs were fixed before the developer marked the coding as done?

Food for my thoughts. Let me know yours.

There are so many good blogs out there on software testing so why start another one?  What value do I think I can really add here that isn’t covered by others?  Why bother with this?

I used to blog, back before Facebook and Twitter.  In those days it was what we used as social media.  Things have changed (of course! – we are talking about software here) and I have changed as well.  I’ve been working as a software tester for 8 years now and hopefully I have  learned a few things along the way.  However, I am also discovering that I’m getting to the point in my career where the learning curve has slowed down. The easy stuff has been covered.  What is left now is the stuff that takes a lot more work and effort to learn.  This is the stuff that really stretches me beyond my natural talents and that requires digging deep on different topics.  

In highschool I was homeschooled and this meant that when I was stuck on something I had to figure out my own way out of it.  Sometimes this meant finding someone who could help me (for example, finding a family friend who was an engineer and could help me with my algebra) but one of the most useful skills I found was to explain it to my younger siblings.  I learned very early that one of the best ways to learn something difficult is to explain it to others.  Doing this revealed the flaws in my own thinking and showed me where I wasn’t fully understanding things.  

I think it is time for me to get back into that mode.  It’s time to start explaining things to others so that I can come to clarity for myself.  It’s time to put into words what I am trying to do so I can see where I’m missing things and where I don’t understand concepts that are crucial for me to get if I want to continue to advance in my abilities as a tester.  I’ve been reading and taking in a lot of information on how to be effective at software testing in the past few years, and now it is time to see how well I really know it.

So come on and join me.  If you’re confused, let me know.  If you disagree, let me know.  If you like it, let me know.  Maybe we can learn a few things together!