Test Passes and the Big Button

---

A set of test cases can be run before every developer check-in to form Basic Verification Tests (BVTs). These help to prevent code changes from breaking major functionality. I try to set a goal for these to be 100% automated, run very reliably, and complete in under five minutes. In the most basic implementation, they are a generic set of tests all developers run and that is a good starting point but longer term an organization should aspire to implementing a BVT system which runs five minutes of reliable automated tests which have been intelligently selected to execute the code being changed and code heavily integrated with it. The best systems will also report back to the developer not only the pass/fail info but what code coverage of the changes have been executed and this can inform the developer of what additional test automation needs to be added to the testing suite to cover these gaps. This advanced system may also recommend a set of manual test cases which execute additional code coverage the automated tests do not. I refer to these advanced types of systems as a “Big Button” system and co-hold a patent for the implementation of one we have created at Microsoft.

A “Big Button” system goes beyond developer check-in tests and can be used for all other test passes. A nightly run of Comprehensive Verification Tests (CVTs) can use the Big Button system to intelligently execute tests based on all developer’s code churn in that build. It can be parameterized to select redundant tests which add additional coverage as time permits. Part of a Big Button system is that data is collected that allows it to keep getting smarter. For example, it may be that the automated tests written for “Feature A” area have most often reported failures in code written in a section of common code that has changed so bias the selection of “Feature A’s” tests when that common code churns. 

Every so often a Full Test Pass (FTP) should be run. Ideally, the Full Test Pass has multiple goals. It can uncover issues in the product, it can refresh the code coverage of the Big Button system, it can re-check the test automation reliability and it can report what coverage gains manual test cases bring to guide automation investments.

Although there is a role for formal test passes and they can become quite targeted and efficient through a Big Button approach, there is also value for an occasional “Bug Bash”. Microsoft has historically used Bug Bashes and made them fun with accolades for “Best Bug Found” or “Most Unusual Bug Found”. Bug Bashes are mostly ad hoc testing of the product although some engineers will use tools like a fuzzer, static code analysis, or Smart Monkey. A Best Practice for hosting a Bug Bash, is to provide “Focus Scenarios”. For example, test using our product with a low-privilege user logged in account… or test simulating users that tend to bookmark and return to web pages… These focusses can help especially in areas feared not to have great formal test pass coverage.

A Bug Bash helps to connect the team to the product through dedicated hands on usage. It helps uncover subjective perception of quality issues not found by automation. Bug Bashes help engineers learn more about the product and features other than ones they directly work on. Insights from the Bug Bash help drive quality and give a double check on the thoroughness of the test automation. Ideally a Bug Bash uncovers nothing new and when it does, it helps point to Formal Test Pass coverage holes.

I have a great passion for building intelligent Big Button systems and would be glad to help your company set a road map to get there.