I'll try the last change, over my changes, Herald had in those tests, then I'll send here my conclusion (and I'm pretty sure that it will be short).
-- Fabio Maulo El 27/04/2011, a las 07:50, "Richard Brown \(gmail\)" <[email protected]> escribió: >> And as I don't know the economics of NHibernate development, I will not >> argue about the bearable length of test suite runs. > > I'm not sure what's acceptable either. James Shore suggested 10 minutes as > a maximum (http://jamesshore.com/Agile-Book/ten_minute_build.html) with a > preference towards 5 minutes, and I agree with that. > > I personally think around 5 minutes is the point at which development will > start to get painful. If we end up having to wait 15 minutes for a > successful build (or hours if we have to wait in suspense for the nightly > run), then I suspect we'll run into much larger problems (e.g., people not > running the build before check-in, broken builds for days, if not weeks, > ...) > > >> It seems people believe that full-coverage testing is somehow bad. > > I didn't say that (I'll sit uncomfortably on the fence instead until I have > to handle a failure in that test fixture myself). However ... I will say > long builds are bad (or more importantly slow feedback). Nothing that > hasn't been said already I guess. > > > -----Original Message----- > From: Harald Mueller > Sent: Wednesday, April 27, 2011 11:12 AM > To: [email protected] > Subject: Re: [nhibernate-development] Re: My feeling of our Tests > >> I guess I’m just stating the obvious, but does that mean we agree >> (generally) not to make this any worse then? >> i.e., no more generative tests, and instead focused small tests. > > It seems people believe that full-coverage testing is somehow bad. > > Exactly the opposite is true: Full-coverage testing is the point against > which all other test methods have to measure up. The long years of test > research have all tried to find ways to attain the same quality as > full-coverage testing without the (sometimes nearly infinite) costs. But > when full-coverage testing is possible, it is the way to go. > > Of course, agreeing on when it is *possible* to do full-coverage testing is > an economical question. And as I don't know the economics of NHibernate > development, I will not argue about the bearable length of test suite runs. > > Still, I'd like to add a few not really connected thoughts - some not so > constructive, some (the last one) hopefully so: > > a. As far as I know (but I'd have to look it up), for tools like compilers, > generated tests are a standard; they are e.g. used to create many different > expressions from grammars. For ways to test grammar-based programs, see e.g. > also Boris Beizer's "Black Box Testing" the section on "syntax testing" > (which, however, implicitly assumes manual testing; but aims at wide > coverage). > > b. Hibernate and NHibernate have bugs (or at least very wide behavioral > deviations from previous versions - I count NH-2583 and NH-2648/HHH-6151, > which I posted, among them) that could have been easily found by a medium > amount of standard syntax testing even in the first version of that huge HQL > rewrite. For someone dealing with ORMs - or even with SQL; or predicate > calculus -, the first few "complex" tests to be written are obvious: "not" > operators around an "or"; as well as "not" around "or" inside "any" are > standard problems where SQL novices and older ORMs (TopLink comes to mind) > often stumble - and which, therefore, should be used as test cases for an > ORM. I don't know why this wasn't done. > > c. If one reads the code dealing with JOINs in HqlSqlWalker and SqlGenerator > alone, it is obvious that this is a *very complicated design*, with at least > one risky design decision (parts of join information are kept redundantly > both in the HQL result AST tree and member variables of AST nodes). Each > change to this code runs the risk of destroying working behavior - I > personally would assume, in an uncontrolled way (but this is only my > impression - I have not talked to the designers of that code). > > d. I do not see that even the standard test concepts of white-box testing > relying on coverage to identify "hot spots to test" are enough to guarantee > a standard quality for such a nearly-compiler tool as the N/Hib query part - > see e.g. Brian Marick's texts on using coverage and their impliciations (I > don't know whether the NHib and Hib development crews employ such > techniques ). > > e. I would very much expect that correctness is tantamount to performance, > and more so performance of the test suite. > > f. That does *not* mean that we shouldn't make tests as fast as possible. > > However, using all my practical experience and my theoretical background, > I'm still at a loss to define a suitably clear *and* sufficient class of > "more focused tests" than the ones I provided. > > So that at least one candidate is provided: > > "Pairwise" testing of the value vectors I provided could be a candidate. > > That would mean that at least all combinations that contain two null > assignment would occur in the tests; and as null assignments are the > troublesome events with inner and outer joining, this might be a > sufficiently covering set of tests. > On the other hand, it might create only test cases with *more* than two null > assignments, which might cover up problems that occur only with two > (specific) null assignments. > Could anyone of you - especially the test gurus? - provide any ideas of an > analysis on why this type of test data selection (pairwise) is or is not > sufficient for finding joining errors? > > Regards > Harald M. > > -- > NEU: FreePhone - kostenlos mobil telefonieren und surfen! > Jetzt informieren: http://www.gmx.net/de/go/freephone >
