Our company recently went through a restructuring and so I’ve ended up on a new team. This team is going to be taking over the testing and regression scripts for parts of a number of other teams. For the last couple of weeks I’ve been working on moving test runs over so that they can run on the machine resources we have in our business unit. As part of this I have had the chance to see a number of different build scripts and test running scripts as well as a number of different ways of setting up builds in TeamCity.
One of the things I noticed was that some builds and scripts were a lot easier to understand and convert than others. Overall most teams were trying to do very similar things but each group ended up using TeamCity in different ways. This got me thinking about code hygiene and how this applies to builds and build tools as well. It seems that often the build process gets approached from the ‘git er done’ perspective. It is often a high pressure area since we need to keep the build pipeline alive and moving or people get pretty upset, and this can sometimes lead to hacking things in to make it work. One little thing follows another and soon the build scripts and processes are so complicated they can hardly be understood by an outsider.
It might seem that this doesn’t matter, but I think it does – and not just because I’m going through a conversion process right now. You could (probably rightly) argue that moving a bunch of tests and builds to another team is not a common occurrence, but the reality is that in the builds as in everything else in software development, change is the only constant. There are always going to be things that need to be changed and especially in the build process, these changes are often going to need to happen quickly. What happens if you are out and someone else has to maintain the builds? What happens if your build setups and scripts get to large and fragile? You start to threaten the ability to quickly make the changes that you need too to keep the build chain running smoothly. Just like with your code, it is important to practice build hygiene. Take some time to clean things up once in a while and make sure that you aren’t building up technical debt in your build process.
In an earlier post I wrote about the importance of learning from your automation and in that post I mentioned some of the tools that I have to help me learn from my automation. I was asked on about these in twitter and while it will be tough to answer in a general way since we use a custom test automation platform, I thought it would be worth trying to explain.
This post will probably be a slightly more technical than most of my posts are but hopefully it will be helpful even to those who are less technically minded (I won’t be posting code here).
In this post I want to talk about my test variation tool. How it works and how I use it to give me new insights. To do that let’s start at the begging with our test automation framework. The framework was custom built for the product we were using, but when we were writing it we didn’t want it to be too tightly coupled to the actual product. There were a couple of reasons for this, not least of which was a desire to not have to modify the testing framework when making changes to the product code. As a result the framework was built in a layered approach that ended being very helpful in many ways. The testing framework itself merely required you to specify a config file for each test with a few items in it (Test description, keywords etc.). The config file then needed to define a Run() function (using python syntax) which could call any arbitrary code as long as it returned back certain status codes (pass/fail/timeout etc.) and messages.
This meant that the actual work of running the product under test and pointing to the run scripts used etc. was done in a set of ‘helper’ functions that we could import and use in any given test. This gave a high degree of customization to the tests and allowed anyone to easily write their own additional functions to use in any particular set of tests.
You can probably figure out by now how I managed to make my test variation tool work. I wrote a function to modify the test scripts which could be used in any of the tests. This function would find the scripts that were going to be run as part of that test. It would also look for a predefined file that contained the commands we wanted to add. It would then parse through the test scripts and modify them to add the requested commands at a point in the scripts immediately before we asked the engine to solve. After that, control would be given back to the functions that were used to startup and run the system under test, but now instead of running the original scripts we would be running modified copies of the scripts.
This allowed us to do a lot of interesting things. For example we could force every test in the system to use one particular option. This would let us see how the option would work in a wide range of settings and we could find potential feature combination issues in any tests that crashed. We could also use it to evaluate what might happen if we changed the default on an option. We could just force that option to use the new default in all the tests and see what happened. In many ways this allowed us to better explore our product and in fact when the developers saw some of the cool things that we could do they started to pull some of the ideas into the development code itself making it even easier to experiment with these things.
Now, my particular framework – and the fact that I had complete access to it since I was one of the people who wrote it – made it pretty easy for me to do something like this, but can we do this in general? I’ve moved to another team now so I guess I’ll be able to find out how easy it is to generalize this, but I’ll close with a few thoughts on how you might be able to implement something like this. If you don’t have a nicely modular testing framework you could still do something like this fairly easily. The nice part about what I did was that I could integrate the test mutations right into the run itself and I only needed to define commands in a particular file and then turn on a flag to tell the tests to use the test variation tool. But the modifications could easily be done in a way external to the test system itself. You could write a script that would traverse your tests and modify your test scripts before you even start your test run. This might be slightly less convenient but in theory it should be pretty straightforward to do.
If I end up doing something like this on my new team I’ll post the results of that here as well, but in the meantime maybe you can give it a try with your tests. Who knows, your automation might just be able to teach you something!
We only ship our product three times a year and while we do fairly frequent builds, we don’t have a continuous delivery deployment system. Since we don’t do it, it might seem that I shouldn’t bother with learning about continuous delivery and integration, but I try to keep up with new developments of this sort and so I’ve read and learned a bit about these things. The interesting thing is that, as with many things that don’t seem directly applicable to my day to day work, I have been able to use ideas from this to help improve things. A lesson or skill learned can be very helpful even if it isn’t directly about the work you are doing.
For example, due to a recent reorganization effort in our company, I am now part of a new team which is combining together some of the automation work of several previously separate teams. I have been thinking about how we should go about consolidating and running these various tests and one of the ideas that I will be using is drawn from the continuous delivery world. This is the idea of what is sometimes called rings of deployment or flighting. The (very) quick summary of this is that you expose new code to a small group of people first and then if everything goes well you gradually roll it out to larger and larger groups of your customers.
Since we only ship once every few months it might seem the idea of rings of deployment doesn’t apply, but in fact I think the idea will fit in very nicely with out development process. However, instead of thinking about gradually larger groups of customers we’ll structure it in terms of gradually exposing parts of the code to larger and larger rings of code. So for example if we have changes in a particular component we will run a set of tests particular to that component. These tests check if anything has broken directly within the component itself. If they pass we will then build that component against the latest ‘certified’ build and run some integration tests. These tests check if the component changes broke anything the larger system relies on. If those pass we can then check if everything works when the component is integrated with the latest combined builds to check that the build pipeline won’t break. At that point we can start to consume those changes in all the developer builds and start certifying a new package which is the final integration step.
Once you add in the ability to use feature flags and beta features we can actually have a pretty sophisticated deployment chain that gives us a lot of information on where failures are coming from and that also allows us to limit the number of people affected by breaking changes. If there is a breaking issue in a component, that will only affect teams working on that component and if there is a breaking issue in the integration that will only affect the people consuming the latest version of the component for their own integration testing. By gradually exposing the changes to integrate with more and more of the code base, we are able to reduce the impact on the number of developers and testers affected by breakages.
I know this isn’t rings of deployment in a strict sense and the purpose of this article isn’t to define that clearly. The point I want to make is that I probably would not have taken the approach that I am to the overall structure of the automation if I hadn’t heard of the idea of rings of deployment. By learning new things (even things that might seem like they don’t really apply to my context) I have been able to make translations in my head and let ideas create new ideas and come up with something better.
Never stop learning!
We are working on an additional tweak to one of our features so that it will automatically add an object. I briefly met with the developer to discuss it and get a better understanding and then I put together a checklist of testing items I would be considering as part of this change. I shared this document with the developer to get his feedback and so he could see some of the things the would be considered. He put what I thought was an interesting comment on the document:
Thanks for this Dave! On second thoughts maybe defaulting is not such a trivial thing 🙂
This reveals a lot. A lot about what collaboration is good for. The developer had been thinking about how to put together code that would do what the story owner wanted, but after a short conversation and reviewing a few items in a checklist, he had a very different view of what was involved in making this happen. Our short interaction will lead to a different level of baked in quality when it comes time to actually use the code. I will still check the items on the checklist, but now the difference between what he thinks he has to do and what he actually has to do is much smaller. This means less surprises while testing this code which means less defects and less churn. At the end of the day, this means we can get this feature into shippable condition more quickly.
Sometimes it just takes a second perspective to see that the trivial thing is more complex than we thought.
I think learning is important. In fact I think it is probably vital to continue to learn new things throughout your career if you want to be a successful knowledge worker. In particular as a software tester one of the primary ways I add value to the team is by the things I learn about the software. I learn where things don’t work well and where they do. I learn about how certain approaches to developing and testing tend us toward certain outcomes. I learn about when things go horribly wrong and try to figure out why, and I try to use all this knowledge to work with my team so that we get better at more rapidly producing valuable software.
One of things that I spend a lot of time learning about is test automation and how to use it effectively. I’ve spent a lot of time learning about it, but I’ve come to realize that I can also learn things from it. My automation can teach me things. By changing my mindset, I have been able to turn my automation into a teacher. Instead of thinking about automation as something that is primarily used to make sure nothing has changed since the last time we ran it (what I like to call ‘change detection’ automation), I see automation as a powerful way to learn things about the product that I would not (or could not) otherwise know,
I still use change detection automation in many places (and I think it is helpful in those places) but even with these types of tests I have asked myself what it can teach me. With that change in mindset I have been able to leverage my automation in new ways. For example, I have a small tool I wrote that will perturb my tests in very specific ways. By making minor changes to the tests and then running them and parsing through the results I can very easily gain new insights about the behavior of the system. Another example is parsing through the logs after the fact. I have many gigs of log data from years of running tests (I have a large hard drive and don’t delete) and so I wrote a script that can parse through that data and search for some specific things, put them into a database, and allow me to do some statistical analysis on them. By doing this I have discovered new and interesting behavior that I probably would not have ever noticed without seeing it in the aggregate.
There are other ways that I have and continue to use automation as a teacher, but I won’t share them all. Instead I want you to think about your automation. Do you learn anything from it? If you shift to thinking about it as a teacher are there any simple changes or additions you can make that would turn it into a better teacher? What can you learn from your automation?
Does your automation help you move faster, or does it help you change directions more easily? For many of us I suspect the answer is that or automation is about speed and not agility. I wrote an article for the Agile Testing Days blog about some indicators that your automation might be making your team less agile. Go ahead and read the full article here
I saw one once. At least I’m pretty sure I did. I saw something across the office and squinted my eyes but didn’t quite get a good look before she disappeared around the corner. If I didn’t know better I would have thought I’d imagined it, but I knew the rumors. I had heard it from Bob who had heard it from Sally who had been told from a very good source that it was true. You’ve probably heard the legends too. They do exist. They’re just not seen very often (or so I have been told).
One day she coded an entire app. Literally one day. And it had no bugs in it. It worked perfectly the first time and everyone loved it. She might have bragged about it a little, but hey, let’s just admit that she deserved it. That was Monday. We won’t even mention what had been accomplished by Friday afternoon.
And then there is the guy in the cubicle across from me. He usually writes pretty good code, but some days …. well I could just about … I need him to get something done, and he says he’ll have it done in one day, but it somehow ends up taking three. After three days you’d think the quality would be high, but no, it ends up having a bunch of bugs. Plus he keeps bothering me with questions and so by the time Friday comes around – well, it’s kind of hard to close out the sprint.
It can be pretty frustrating working with someone like this. If only I had run a bit faster that time, I might be able to actually get stuff done.
That would be the best way to fix the problems I have right?