The Dutch Software Testing Championship
Last year, a Dutch team won the Software Testing World Cup. The newly crowned Software Testing World Cup Champions were so energized by the experience they thought, “why not have a Dutch version as well?” Co-organized and hosted by Dutch Software Testing Association TestNet, the championship took place on May 1st. Expecting a fierce competition, Ronald Nikkels and I took on the challenge and it turned out to be a very nice experience.
Collaboratively trying to break a real-life application
Having a lot of people in one room who do the same work as you is always fun (as you can see in the event's photo gallery). The basic activities and daily headaches of testing create a bond, but you also wonder how they approach testing and organize work. Unfortunately, there wasn't a lot of time to interact with other teams before the contest. Everyone was busy preparing and setting up their work area. Several teams were writing out theoretical frameworks on how to approach the software to be tested. Some even set up physical scrum boards with pre-defined phases to organize work. Tension was mounting. Everyone was ready and raring to start the 3-hour contest.
An old Dutch Proverb says: “Good preparation is half the work” but I must say I sometimes felt overwhelmed by the effort the teams had put into this. As for us, we'd created a document to write down our findings and advice. It's not that we didn't come prepared, but I do think we'll up our preparation next time.
The contest actually felt like a mini sprint during which we had to test a piece of software as thoroughly and efficiently as possible within a set time frame.
The System Under Test (SUT) was a real-life program that was still under construction. According to the two-man development team, about 80-85% of the application was done. One of the developers (who also pitched the product to other companies) lent the application to be tested during the event and acted as the Product Owner for the evening. Over 88 professional testers examining your software product (almost) for free ... pretty cool opportunity, right?
Put to the test
The assignment was to co-operate within your team, properly describe found bugs and create a test report. This report had to contain a test approach, conclusions concerning the state of the software and advice on shipping the product. To be honest, when the competition ended I didn’t think the application was fit for production. I felt it hadn't even been completed for 85%. We'd collectively logged about 500 issues (overlapping of course) and delivered 27 test reports.
The jury paid particular attention to team co-operation. Although our team consisted of only 2 members (most teams had 4) we co-operated smoothly and efficiently. We were forced to clarify who was focusing on what part of the software and diligently did so. Once again I realized the importance of working together and a proper division of tasks to achieve a common goal. Discussing and exploring stuff with another tester (someone who explicitly has the same role as you) is challenging and broadens your horizon. It made me realize I sometimes miss that atmosphere in my team at work, being the only tester of the team.
An interesting thing I recognized is how important it is to communicate with the business throughout the testing process. Engrossed in testing activities, trying to find as many as bugs as possible, you could make too many assumptions on what is important. You could end up taking a wrong turn along the way. During the contest, the Product Owner was sitting in the front of the room and was available for questions. A lot of teams consulted him and his feedback reshaped their idea of the application. Communicating well with the business allowed the teams to set new test directions accordingly. A couple of times, I checked with the Product Owner how important the thing I was testing really was. Such short feedback loops really help your testing progress.
To automate or not to automate
Participants were judged on co-operation, the test report quality and the description and clarity of logged issues. The jury didn't urge testers to study the technical components of the software nor think about test automation upfront. Given the setting of the event (large application, 3 hours) one could expect this had no priority.
This is not the case at bol.com. Testers are investing in becoming more technically aware and taking action on this. Should a Dutch championship testing also be about automation? Or is it test automation not necessarily one of the basic competences of a good tester? Intuitively, test automation feels a bit peculiar to me; you build software and then you build more software to check if the original software is working. However, I do think that once a technology has proven itself over time, one could use it to automate tests so that not everything has to be done manually. Test automation skills expected from a tester probably also differ per organization and the tester’s personal competences.
Measuring our capabilities at bol.com?
Because of the intensity of the event and the tons of questions that popped up during it, time flew by. Although we didn't win the match (ending on a worthy place though!) we learned a lot and had a lot of fun. I would definitely join the next round again, hopefully with a bigger bol.com team!
On a concluding note, having a competition can bring out the best in people and collaboration is indispensable. Asking the right questions will do wonders! It might even be interesting to have some kind of competition like this at bol.com to measure our testing capabilities, one-to-one collaboration and enthusiasm.