Software QA: Time to pull the goalie?

Even casual Minnesota hockey fans remember the 2015 Minnesota Wild season. Long-suffering Minnesota sports fans finally had a team to be excited about. The Wild had assembled perhaps the most talented team in its history. The pieces finally seemed to be in place for a deep playoff run.

But the team did not live up to expectations. The Wild struggled through the first half of the season, hampered by inconsistent play at nearly every position. By mid-January, the Wild had dropped six straight games and were eight games out of playoff contention.

Enter Devan Dubnyk.

Minnesota, desperate to stop the bleeding, acquired goaltender Devan Dubnyk in a trade with Arizona on January 14. The very next night, Dubnyk led the Wild to a 7-0 shutout win against the Buffalo Sabres. Dubnyk went on to start a franchise record 38 straight games, logging an unbelievable 27-9-2 record and a 94% save percentage. The Wild made the playoffs, and won their first playoff series against the St. Louis Blues in six games, with Dubnyk allowing only six goals in the entire series.

Enter the Chicago Blackhawks.

In the second round series, the powerful Blackhawks offense scored four goals in each of the first two games, rocking Dubnyk’s confience and ultimately sending Minnesota home with four straight losses.

What happened?

The truth is this: Dubnyk’s spectacular goaltending, while giving the team a better chance to win hockey games, was also hiding other fundamental team problems, such as sloppy defensive play and an offense that was not taking enough shots on goal. These weaknesses were only exposed when Dubnyk wasn’t playing well. The team continued to struggle with consistency, leading to the firing of head coach Mike Yeo in 2016.

A similar scenario is playing out on software teams every day. Our quality analysts are nearly entirely focused on testing. If we allow a goal (production defect), the first question we ask is how the defect got past our goalie (our software testers). We add new test cases to our test case library. We incorporate test automation to build a bigger and better goalie. Our effort (along with our money) is spent preventing goals from getting past our goalie.

This approach can work in a stable, established system that is not changing much. But what about in an environment that is constantly changing, where we must react fast to market changes in order to survive? What about an environment where a single production defect could be fatal to our business, or even to human life? Do we continue to invest more and more in our goalie?

Manual testing and test automation certainly have an important place, and they can quickly stop the bleeding, but in order for us to fundamentally improve our software products and processes, the role of the Quality Analyst must change from goalie to coach.   I’m not suggesting that we completely pull the goalie (at least not yet), but we certainly must become less dependent. Rather than using our quality analysts to insulate us from poor products and processes, we need our quality analysts to measure, analyze, report on, and help our teams fix what is broken. Only then will our teams reach their full potential.