Sunday, April 15, 2012

Reducing noise in failing test cases

During the past week Adam has looked at test case minimization, also known as shrinking, and how to apply it to test cases generated from external specifications. Shrinking aims to make debugging easier by removing noise, that is information which is not relevant to the error which the a failing test case provokes. Given a failing test case and the specification that was used to generate it, Adam investigated ways to obtain shrunk testcases which are still valid with respect to the specification. This included a simple approach, where the specification was only used to validate shrunk test case variants, and a more sophisticated method where the specification was used to guide the shrinking. In both cases the number of cases to consider is very large, as an optimal solution really requires that all possible shrunk variants are enumerated. In the latter case, however, the task is reduced since variants which are invalid with repsect to the specification can be omitted.