The case against detailed tests cases (part two)

We recently read an article on the QA Revolution website, titled 7 Great Reasons to Write Detailed Test Cases, which claims to give “valid justification to write detailed test cases” and goes as far as to “encourage you to write more detailed test cases in the future.” We strongly disagree with both the premise and the “great reasons” and we’ll argue our counter position in a series of blog posts.

We now turn our attention to the claims made about offshore testing in the article.

Offshore: If you have an offshore team, you know how challenging that can be. It is really important to write everything out in detail when you communicate offshore so that everyone understands. It is critical to write detailed test cases is no different. Without those details, the offshore team will really struggle to understand what needs to be tested. Getting clarifications on a test case can often take a few days of back and forth and that is extremely time consuming and frustrating.

Let’s start by breaking out statements and examining them.

It is really important to write everything out in detail when you communicate offshore so that everyone understands

This recommendation appears to be conflating additional detail with improved understanding. Our joint experience of working with offshore testing teams suggests that the exact opposite is true. Instead of focusing so much effort on detailed test cases, we’ve preferred to spend time providing support as required to these testers to help them understand. When working with offshore teams for whom English is not their first language, Lee found that detailed documents (including test cases) were very slow for the testers to consume and it was more efficient to use lighter weight documents (in terms of their text content) such as mind maps. Paul has had similar experiences and also finds lightweight documentation and support to be a far more effective method to generate understanding.

It is also interesting that the author claims that “everyone understands”. Think of yourself in your workplace, perhaps in a meeting, and you are discussing the system under test. All attendees have the same native language. Now ask yourself, when was the last time you issued directions and everybody understood? Can you remember an instance? We are struggling to recall an occasion where we were completely clear and not questioned. Now convert this from spoken word to written word. How often do you get questions about statements you have written? It’s pretty frequent, right? So we are more than a little curious about why detailed test cases would be immune to a lack of clarity.

Without those details, the offshore team will really struggle to understand what needs to be tested.

It sounds like the idea is for someone to write detailed test cases with the intent of a tester in the offshore team executing them. There are a couple of assumptions in play in this case. Firstly, it is implied that the test case is clear and without ambiguity – trying to write test cases that achieve this is a fool’s errand (since it’s impossible). Secondly, there is an assumption that what makes sense to the test case author makes sense to the executor of the test case. This implies that the author and the executor model information in the same way and they have the same background knowledge. Again, this is very unlikely to be true. There is plenty of room for differences in modeling and assumptions that could result in the testing by the executor being quite different from that imagined by the author. Having a detailed test case doesn’t necessarily mean it is followed to the letter during execution and, even if it was, the testing will most likely be different from that envisaged by the author.

We also find this statement disrespectful to the intelligence of the testers in the offshore team, who are most likely quite capable of thinking for themselves and coming up with excellent test ideas if given the chance outside the confines of simply being test case executors. We strongly recommend that testers learn about oracles and heuristics as ways to generate test ideas, rather than solely relying on documentation and detailed test cases. The testers – be they in offshore teams or otherwise – will likely find much deeper problems by broadening out their thinking using these tools than if they rely on prescriptions of exactly what to test.

The statement implies that the primary (or maybe only?) form of communication between the onshore and offshore team is the detailed written test case. This is a recipe for disaster. Collaboration through discussion is much more likely to aid with understanding than trying to communicate only in written form.

Getting clarifications on a test case can often take a few days of back and forth and that is extremely time consuming and frustrating.

We find it quite interesting that writing detailed test cases is time consuming (and for many testers, incredibly frustrating) but the article uses those very factors as a reason for writing detailed test cases. Amazingly, it is promoting the notion that quantity of detail provides superb clarity. 

We’ve yet to see a set of test cases that didn’t need some level of clarification. That we can be so precise as to leave no doubt is a fallacy that simply does not hold in day to day communication. Wiio’s Law tells us, if broadly summarised, that “Human communications usually fail except by accident”. While intended to be humorous, there are also recognised “truths” through observation. Wiio’s Law was commented on by  Korpela, Jukka Kalervo  who noted with regards to why human communication fails:

• Language differences. On the Internet, for example, the lingua franca is badly written and poorly understood English. Some people use it as their native language; others learned some of it from various sources. In any case, whatever you say will be interpreted in a myriad of ways, whether you use idiomatic English or not.
• Cultural differences. Whatever you assume about the recipients of your message, the wider the audience, the more of them will fail to meet your assumptions. On the Internet, this virtually guarantees you will be misunderstood. What you intend to say as a neutral matter of fact will be interpreted (by different people) as a detestable political opinion, a horrendous blasphemy, and a lovely piece of poetry.
• Personal differences. Any assumption about the prior knowledge on the subject matter fails for any reasonably large audience. Whatever you try to explain about the genetics of colors will be incomprehensible to most people, since they have a very vague idea of what “genes” are (in written communication you might just manage to distinguish them from Jeans).

Summing up our views

While we both have experience in working with offshore testing teams, we have not seen the use of detailed test cases as either being mandatory or effective in reducing some of the inevitable communication challenges involved in this model. We have instead focused on collaboration and learning to help on both sides of the communication, finding that the use of exploratory testing has been incredibly valuable in effectively working in an offshore model.

Our suggestions for further reading:

• James Bach/Aaron Hodder “Test Cases Are Not Testing” https://www.satisfice.com/download/test-cases-are-not-testing
• Michael Bolton “The Test Case Is Not The Test” https://www.developsense.com/blog/2017/02/the-test-case-is-not-the-test/
• Michael Bolton’s “Breaking The Test Case Addiction” blog series, e.g. https://www.developsense.com/blog/2019/12/breaking-the-test-case-addiction-part-8/
• Michael Bolton’s “Oracles from the Inside Out” blog series
  https://www.developsense.com/blog/2015/09/oracles-from-the-inside-out/
• Wiio’s Laws
  https://en.wikipedia.org/wiki/Wiio%27s_laws
• Michael Bolton “Testing Without A Map”
  https://www.developsense.com/articles/2005-01-TestingWithoutAMap.pdf 

Thanks to Brian Osman for his review of our blog post.

Advertisement