Last month I posted here about the false positive problem.
Here is a specific example that I learned from a friend at Microsoft, where the false positive caused a defect escape:
Imagine a shopping cart application. The simple linear list steps of a check are this:
- Go to shopping app page
- Log in
- Find widget
- Add widget to cart
- Complete purchase
- Verify purchase complete
Here are two sources of problems with this approach:
First, a linear list of steps is limiting because the steps are expected to be close to the same level of abstraction, relative to SUT behavior or to business logic. But what if you need more detail? One can add more detail in more granular steps, but then the sequence gets confusing and it’s easy to lose the context for any one step. The context can be lost for a person reading the steps, or for automation processing the artifact of the check run.
Second, there is the habit of the QA role (or, whoever is writing the automation) to economize by thinking of the steps as applying to themselves, rather than to the product (and the technology that implements the product) or the business (including the business requirements and behavior of the app). Who, they might reason, other than QA, would actually look at the steps? Maybe nobody, so might as well document it for ourselves at authoring time or at runtime.
Step 6 is the verification: verify purchase complete. So, the automation author reasons, verify that the correct objects appear in the GUI of the page to indicate that the purchase is complete.
But, what about the back end? Does the automation know that the purchase has entered the state machine of the purchase with credit card, initiating the finding of the purchased widget in a warehouse and shipping it to the correct address, etc.? No, it does not. But, from QA’s perspective, this is correct, and there was no error in leaving out a verification at the back end.
Meanwhile, the product owner (or, other representative of the business) might see the steps and conclude that the check is complete and has the correct verification “verify purchase complete.”
There is a mismatch between the code and the business language. The result can be a defect escape. The end-user (or whoever is dogfooding the app) might experience “Hey, the web site said my purchase is complete, but nothing happened with the order and my credit card wasn’t charged!”
The false positive is in the check that was meant to verify “purchase complete;” the check passed, but actually the purchase failed.
The escaped defect is that, although the web page appeared to show purchase complete, the purchase is lost.
Applying the Hierarchical Steps pattern to the check, and making the check code self-documenting at runtime, solves this problem neatly:
According to the schema I use in the software samples linked from the MetaAutoamation.net site to GitHub, the steps expressed in the check run artifact might look like this:
Some things to notice:
- Some details are left out
- All timeouts in MS can be adjusted in the artifact of a check run, for future runs to use
- The verification step is an explicitly self-documenting cluster of two verifications, one on the front end and one on the back end
If this looks like a lot of work to define all the details, the work is actually all in implementing the check, including calling other methods or libs for components of the check. If authors write the code as self-documenting, the check will document itself at runtime. See samples on MetaAutomation.net for more information and working open-source implementations that do exactly this.
The beauty of approaching the check this way, following the Hierarchical Steps pattern, is that it is clear both in the check code and in the artifact of the check that the purchase verification is both on the web page and on the back end. Failure in translation from automation code to the business is now extremely unlikely, because the details of what they check does (and, by omission, what it doesn’t do) documents itself at runtime for everybody to see. This information is accessible because anybody can drill down the hierarchy from the root check step called “VerifyHappyPathPurchase” to the details in child steps.
The “false positive” failure of the example can’t happen anymore, nor can the defect escape. It’s completely clear, in trustworthy detail, what the check does (and, what it does not do). The usual layer of obscurity behind check methods or keywords is gone!
This is how teams get clarity on what the SUT is actually doing, and how fast.