Herald, please... Nobody want discard those tests today and the matter is not today but tomorrow or aftertomorrow. Are you 100% sure that you and Patrick will be available to give an hand to the guy will see a broken test in NH2583 in the year 2013 or 2014 ?
In these years (just 4 to be exact) I've heard "don't worry I'll be there" many times. Again... we all have really appreciated your and Patrick's effort around this issue. Now, take all this matter with calm and just try to do all your best for the future of NHibernate. Thanks for your effort. On Tue, Apr 26, 2011 at 10:10 AM, Harald Mueller <[email protected]>wrote: > Hi - > > > As an example, if/when any of these tests fail, it doesn't seem as if it > > would be trivial (or even straightforward) to determine the *meaning* of > > that failure without an inordinate amount of time spent in the debugger > (or > > much worse IMO, spent staring at the test data) to understand just what > > exactly failed. > > No, that's no so. I know this because some of the tests filed "quite > nicely" during development (I don't know about Patrick's experience, > however). The full input coverage creates quite a nice output. I number all > the objects per class with a static variable; so if you run a single test > (that failed in a regression run), you get a failure like > > Expected: <1, 3, 4, 5, 7> > But was: <1, 4, 5> > > So, you knwo that the condition did not work as expected on object 3. But > if you now look at the "Setters" definition, you see e.g. that it is > > Setters<TK, TBO1_I>(SetK1, SetBO1_I1) > > you know that there are 3 * 4 = 1 records in the primary table ("MyBO"), > which will have the values > > K1 BO1.I1 > ---------------- > 0 (null) null > 0 (null) null > 0 (null) 0 > 0 (null) 1 > 0 (zero) null > 0 (zero) null > 0 (zero) 0 > 0 (zero) 1 > etc. > > So you know that the first difference in the 3rd line, which in this case > is where K1 is 0 and BO1.I1 is also 0. > After a few failing tests, you can almost read the "failure data" off the > test output. > > HOWEVER: My introduction of ValueNull for all possible values creates > multiple absolutely identical object graphs. The reason is that - as you can > see above - for a non-nullable int property (like K1), both TK.ValueNull and > TK.Zero produce the same value. Reason: I cannot assign null to K1, so I > assign 0. A better implementation would throw an exception when trying to > assign null to K1; and by this, tell the data generator that this value does > not make sense --> then, these objects would not be generated at all. I'll > try to change this shortly - and check how much smaller the number of > objects then gets. > > > Some of these tests seem to lack clear assertions. > > All of them have the assertion that the Linq2Objects result is equal to the > NHib.Linq result, except for exceptional objects. That's just the - current > - definition of the semantics of NHib.Linq, except for special cases which > have to be handwritten anyway. > > > Presumably > > these are testing for no-exceptions-thrown(?) but this kind of test > failure > > isn't going to result in clarity-of-reason-for-failure. > > Yes, that would be too little to get out of tests. I avoid such trivial > tests as much as possible (if one has slipped through, it can only be a > hand-written one I forgot to upgrade to a useful one). > > > It could be that "this is in fact the best way to go about this" but > > intuitively it just doesn't feel 'right' to me for these tests to survive > as-is > > in the project. > > I agree - that would be useless. > > > Generative tests seem to have their purpose in re: their being a tool to > > 'prove out' via probing to find the proper behavior of the codebase (all > > edge cases, etc) but I'm honestly ambivalent about their long-term > maintenance > > prospects. > > The problem here seems to be to understand the right *space* to be tested: > It is *not* the states of objects, but the possible variations of > conditions. So each of these tests is, like all other unit tests, > hand-crafted to a special condition construct (e.g. doubly nested plus in > the most recent tests; something Patrick wanted because he found an error > there). The assertion to be fulfilled by the NHib.Linq machine is then that > the SQL is equivalent (in a defined sense) to some behavior. The only > alternative I can see is to write a prover that shows that the resulting SQL > is equivalent in that sense to the Linq query - something no-one would try; > and would accept as a "test", IMHO. > > > In a sense, this feels like the results of PEX where you look > > at the generated test suite and say "this was a valuable exercise, but > > there's little value in hanging on to it long-term". > > Well, did you try to "failurize" (or whatever you'd call that - "change the > code so that it fails in a way you designed") any test, e.g. by modifiyng > the condition is a semantically interesting way - e.g. adding existence > guards or the like) and then scrutinize the result? I cannot believe that > you then would have this "feeling" ... but I may be wrong if you did it! > Then, please, tell us about it! > > > > > As an example, I'd wonder what would happen in the following scenario: we > > decide to change the semantic behavior of some part of the LINQ provider > > (and thus, presumably, a number of these tests would then be broken). > Would > > the next step be modifying all the now-broken tests + dependent test data > > to get them to pass or would the next step be to discard all the existing > > tests and regenerate them all again? > > As I did just that weekend (look into the changeset!), you look at each > failing test; decide why it fails - in these cases, the semantics definition > of NHib.Linq is different from Linq2Objects in cases that we (at least > Patrick and I) habve not discussed before; and then either change the > implementation (if the test is right) or change the test (if you want the > implementation as is; and have a good explanation of why the test is wrong - > this is why I sent all of you the modified tests with the comments that > explain the complexity of the change). > > So, as far as I see it, you do exactly what you do with other tests. > > Why should you discard all these well-understood tests?? > > > > > I honestly don't have a well-formed opinion on this, but it sort of seems > > to me as if the answer to this Q would lead us to a better-informed > choice > > about the long-term viability of this generated test suite. > > > > Does this make any sense --? > > Do my answers make any? (Patrick, may I ask what you say ...?) > > Regards > Harald M. > > > -- > NEU: FreePhone - kostenlos mobil telefonieren und surfen! > Jetzt informieren: http://www.gmx.net/de/go/freephone > -- Fabio Maulo
