Log Statements in Automated Checks? There is a Much Better Way

Tags: Hierarchical Steps, MetaAutomation, hierarchy, steps, automation

This post is part two of two. The first part is here.

When driving a non-real-time software product, the overhead of the quality system driving the product and dependencies is not critical. It is much less important than with the original log scenarios of web servers and hardware instrumentation, because quality systems are not involved with end-user-facing scenarios or on end-user hardware. Cheap and ubiquitous computing power make a more appropriate, more powerful, and sophisticated solution possible for getting data on driving and measuring the system under test (SUT).

 

I discovered and wrote down the Hierarchical Steps pattern as a natural, extensible way to record, describe and consume a repeatable set of steps of any complexity. We all use this pattern, even if we don’t think about it that way. Hierarchical Steps models how we follow procedures, communicate them, modify them, do error handling, and even how we think about them!

 

This page explains more about the pattern and how it fits into MetaAutomation: http://www.metaautomation.net/the-pattern-language

 

Example: You plan to take a date to the movies, and might discuss with your date a) where to meet b) when to meet c) who’s going to drive d) what movie to see, etc. What vehicle will take you there? That detail is implied by, and subordinate to, item c) who’s going to drive. Your mind lumps items a) where and b) when to meet together in your mind into the meeting event with your love interest, with separate details to be considered as needed. Planning your evening together as a long, linear list of equivalently-weighted events is neither easy nor natural.

 

Example: If you have installed a home dishwashing machine recently, or even assembled some complex furniture from instructions, you have followed the Hierarchical Steps pattern (probably without knowing it) because installation manuals of any complexity typically use this pattern to manage and communicate all the details required.

 

If the artifact of a check run is a pure-data hierarchical structure, e.g., a grammar of XML, then it is more compact to store and available for complex, robust, fast analysis with much less need for searches through raw text. This supports patterns of MetaAutomation, e.g., Smart Retry, Automated Triage and Quality over Time. With performance data at each check step (“how many milliseconds did this step take to complete?”) there is no need for a separate set of performance checks.

 

This graphic shows a hierarchy as it applies to the steps of an automated check, driving the product for a quality measurement.

Hierarchical Steps, and how they relate to quality automation 

This highlights some more advantages to using the Hierarchical Steps pattern in this context:

  • The root step (root node) is the natural place to hold the overall purpose of the check. The node could hold, for example, a Gherkin statement that describes the check procedure.
  • The business-facing steps describe the steps of the procedure in a “ubiquitous language” or language that is meaningful to the business of the larger team, not just the people who created the code of the procedure.
  • The technology-facing steps (the leaf nodes in the diagram) drive the system under test or dependencies. With the Hierarchical Steps pattern, and if the check is written well (see the Atomic Check pattern of MetaAutomation) then all these steps can be atomic* so the point of failure is uniquely found in the hierarchy.
  • The structured hierarchy supports a single check running across any number of tiers, with a check artifact that shows unambiguously what happened on what tier, including any measurements and performance.

If there is a failure, the failure propagates naturally up the hierarchy to show which business-facing step failed, and of course fail the whole check:

Error handling within a procedure

If the check run discovers added or changed steps, at a lower level of abstraction than the business-facing steps, then the business-facing steps need not change at all, simplifying analysis that spans the procedure change in time and does not hinge on the more technology-facing steps:

Adding steps to a procedure 

To emphasize an important point: when I wrote above “If the check run discovers added or changed steps…” I mean that the step hierarchy of the check run is self-documenting. Business risks resulting from any discrepancy between a written test case (in Gherkin, or whatever) and what happens at check runtime simply go away.

 

On the Samples page here http://www.metaautomation.net/metaautomation-samples , there are three runnable samples which show these benefits of Hierarchical Steps in action, implemented in XML and C#. One example in all three sample projects checks a very simple SUT with pseudo-random failures built in: the sample with the “… web page built to fail randomly...” on the MetaAutomation.net site. This sample illustrates the diagrams above with failures, in the XML artifact of the check as the illustrated hierarchical structure.

 

For doing quality on software projects of any significant size, and for software that matters to people, Hierarchical Steps shows a much more productive approach to returning robust, analyzable data on driving and measuring the system under test.

 

*A step is “atomic” when it is indivisible. This is a useful concept for applying the Hierarchical Steps pattern to a check, because when each atomic step is represented by a different node in the hierarchy (a leaf step, i.e., a technology-facing step) then an SUT failure at such a step is precisely identified by the hierarchy and this information goes into the artifact of the check run. All such steps provide good placeholders for measurements on and instrumentation from the SUT.

 

An atomic check represents a complete procedure to measure a functional requirement of the product, so it contains at least one atomic step and probably many.

 

For a running code example, see the Example check 2 in Sample 1 here.

No Comments

Add a Comment

Sign up here for emails with bites of wisdom on quality automation and MetaAutomation

Recent Blog Posts

  • The differences: Manual Test vs. quality automation

    In my last post I describe out the two kinds of automation that fit in the quality automation space.

    People who do quality automation (at least, the part of quality automation that drives and … more

  • The two halves of quality automation

    Quality automation is the domain (or problem space) of driving the SUT, measuring and recording data on SUT behavior and communicating that data to the business. I also use “quality automation” to … more

  • Fixing the false negative problem

    False negatives happen when these three things happen in order:

    Operations (ops) promotes the software to the next level, or ships it to end-users

    Someone (or, some automated process) discovers a … more

  • Fixing the false positive problem

    With all the quality automation that is your responsibility, a run of a check failed. It is your job to check it out.

    After 30 minutes or so of investigation, you find that the failure happened … more