A framework for ranking vacuity results

Vacuity detection is a method for finding errors in the model-checking process when the specification is found to hold in the model. Most vacuity algorithms are based on checking the effect of applying mutations on the specification. It has been recognized that vacuity results differ in their significance. While in many cases such results are valued as highly informative, there are also cases where a vacuity result is viewed by users as "interesting to know" at the most, or even as meaningless. As of today, no attempt has been made to formally justify this phenomenon. We suggest and study a framework for ranking vacuity results, based on the probability of the mutated specification to hold on a random computation. For example, two natural mutations of the specification G(req → F ready) are G(¬req) and GF ready. It is agreed that vacuity information about satisfying the first mutation is more alarming than information about satisfying the second. Our methodology formally explains this, as the probability of G(¬req) to hold in a random computation is 0, whereas the probability of GF ready is 1. From a theoretical point of view, we study of the problem of finding the probability of LTL formulas to be satisfied in a random computation and the existence and use of 0/1-laws for fragments of LTL. From a practical point of view, we propose an efficient algorithm for approximating the probability of LTL formulas and provide experimental results demonstrating the usefulness of our approach as well as the suggested algorithm.