On Fri, Sep 23, 2011 at 11:16:37AM -0400, Richard Hipp wrote: > Opinions vary on the exact meaning of MC/DC for a language (such as C) that > has short-circuit boolean operators. You are advocating a more rigorous > view of MC/DC that what I have heard before. This is not to say it is > wrong, only different.
I think you are describing (plain) Condition/Decision Coverage, not Modified Condition/Decision Coverage (see below). > For a decision of the form: > > if( A && B )... > > The test suite for SQLite tries at least three cases: > > (1) A false > (2) A true and B false > (3) A true and B true > > You seem to be saying that this is only MC/DC if we split case (1) into two > subcases: > > (1a) A false and B false > (1b) A false and B true Hmm, no, not quite, although you do have a point here that in shortcutting languages it's different. I would concede that in a shortcutting language you do not need to test for (1a) and (1b). I think it means that for the decision of the form if( A && B ), you additionally need to show that A and B both can independently affect the outcome; that is, For A: Either (2) or (3) results in different branch being taken from (1) (which in this case would be true for (3), but the analysis is not always so straightforward) For B: The branches taken in (2) and (3) are different (which is the case here). So in case of if( A && B ), I think you are correct that these three test cases suffice for MC/DC. But you *have* to consider the outcome too for it to be MC/DC. Taking a look at the Wikipedia page[1] for MC/DC, I believe what you describe is plain Condition/Decision Coverage, not MC/DC: ------------------------------------------------------------ * Condition/Decision Coverage Every point of entry and exit in the program has been invoked at least once, every condition in a decision in the program has taken all possible outcomes at least once, and every decision in the program has taken all possible outcomes at least once. * Modified Condition/Decision Coverage Every point of entry and exit in the program has been invoked at least once, every condition in a decision in the program has taken on all possible outcomes at least once, and each condition has been shown to affect that decision outcome independently. A condition is shown to affect a decision’s outcome independently by varying just that condition while holding fixed all other possible conditions. The condition/decision criterion does not guarantee the coverage of all conditions in the module because in many test cases, some conditions of a decision are masked by the other conditions. Using the modified condition/decision criterion, each condition must be shown to be able to act on the decision outcome by itself, everything else being held fixed. The MC/DC criterion is thus much stronger than the condition/decision coverage. ------------------------------------------------------------ > Your argument is that there might be redundancy; that if B is always false > when A is false, then it would be possible to simplify the decision to just: > > if( B )... > > True enough. But there are countless ways to refactor a boolean > expression. I don't think that MC/DC has any requirement that the number of > conditions be minimized, does it? I think that's one of the central ideas of MC/DC. Well, I think it doesn't require *minimizing* per se, only that each condition independently is enough to flip the outcome. (I now think I was wrong to talk about subexpressions in my original post; The "conditions" are the atomic conditions, not subexpressions.) I certainly cannot imagine of another plausible reading of the fourth condition. Perhaps you have a different plausible interpretation? In fact nearly all definitions of MC/DC I can find contain this paragraph, which explains it quite well: "A condition is shown to affect a decision's outcome independently by varying just that decision while holding fixed all other possible conditions." > One of the key benefits we derive from testing SQLite to 100% branch > coverage is that we end up testing the object code, not the source code. So > even if the compiler makes a mistake, we will still likely catch it. I > don't think your more rigorous definition of MC/DC is necessary to achieve > that goal, is it? I certainly think that even 100% branch coverage (also called Decision Coverage) is quite impressive in itself without MC/DC, and if what the SQLite test harness does is plain Condition/Decision Coverage, I'm sure that already places SQLite within the best tested 1% of software. I'm happy and impressed with how tested it is, I just don't think it's MC/DC :) As far as I can tell 100% MC/DC coverage is only required in highly safety critical systems, most notably the DO-178B standard for avionics for the most critical (Class A) software only, where failure would be "catastrophic", i.e. result in planes falling from the sky. I don't know how much it would make sense to strive for such a standard in SQLite, unless someone really wants to use it in such a highly critical system. MC/DC testing is often criticized for its high cost, after all[2]. [1] http://en.wikipedia.org/wiki/Modified_Condition/Decision_Coverage [2] http://sunnyday.mit.edu/papers/dupuy.pdf Sami _______________________________________________ sqlite-users mailing list sqlite-users@sqlite.org http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users