> I guess I’m just stating the obvious, but does that mean we agree > (generally) not to make this any worse then? > i.e., no more generative tests, and instead focused small tests.
It seems people believe that full-coverage testing is somehow bad. Exactly the opposite is true: Full-coverage testing is the point against which all other test methods have to measure up. The long years of test research have all tried to find ways to attain the same quality as full-coverage testing without the (sometimes nearly infinite) costs. But when full-coverage testing is possible, it is the way to go. Of course, agreeing on when it is *possible* to do full-coverage testing is an economical question. And as I don't know the economics of NHibernate development, I will not argue about the bearable length of test suite runs. Still, I'd like to add a few not really connected thoughts - some not so constructive, some (the last one) hopefully so: a. As far as I know (but I'd have to look it up), for tools like compilers, generated tests are a standard; they are e.g. used to create many different expressions from grammars. For ways to test grammar-based programs, see e.g. also Boris Beizer's "Black Box Testing" the section on "syntax testing" (which, however, implicitly assumes manual testing; but aims at wide coverage). b. Hibernate and NHibernate have bugs (or at least very wide behavioral deviations from previous versions - I count NH-2583 and NH-2648/HHH-6151, which I posted, among them) that could have been easily found by a medium amount of standard syntax testing even in the first version of that huge HQL rewrite. For someone dealing with ORMs - or even with SQL; or predicate calculus -, the first few "complex" tests to be written are obvious: "not" operators around an "or"; as well as "not" around "or" inside "any" are standard problems where SQL novices and older ORMs (TopLink comes to mind) often stumble - and which, therefore, should be used as test cases for an ORM. I don't know why this wasn't done. c. If one reads the code dealing with JOINs in HqlSqlWalker and SqlGenerator alone, it is obvious that this is a *very complicated design*, with at least one risky design decision (parts of join information are kept redundantly both in the HQL result AST tree and member variables of AST nodes). Each change to this code runs the risk of destroying working behavior - I personally would assume, in an uncontrolled way (but this is only my impression - I have not talked to the designers of that code). d. I do not see that even the standard test concepts of white-box testing relying on coverage to identify "hot spots to test" are enough to guarantee a standard quality for such a nearly-compiler tool as the N/Hib query part - see e.g. Brian Marick's texts on using coverage and their impliciations (I don't know whether the NHib and Hib development crews employ such techniques ). e. I would very much expect that correctness is tantamount to performance, and more so performance of the test suite. f. That does *not* mean that we shouldn't make tests as fast as possible. However, using all my practical experience and my theoretical background, I'm still at a loss to define a suitably clear *and* sufficient class of "more focused tests" than the ones I provided. So that at least one candidate is provided: "Pairwise" testing of the value vectors I provided could be a candidate. That would mean that at least all combinations that contain two null assignment would occur in the tests; and as null assignments are the troublesome events with inner and outer joining, this might be a sufficiently covering set of tests. On the other hand, it might create only test cases with *more* than two null assignments, which might cover up problems that occur only with two (specific) null assignments. Could anyone of you - especially the test gurus? - provide any ideas of an analysis on why this type of test data selection (pairwise) is or is not sufficient for finding joining errors? Regards Harald M. -- NEU: FreePhone - kostenlos mobil telefonieren und surfen! Jetzt informieren: http://www.gmx.net/de/go/freephone
