* Akim Demaille wrote on Fri, Oct 17, 2008 at 11:12:45AM CEST: > Le 16 oct. 08 à 22:02, Ralf Wildenhues a écrit :
>>> Actually I never meant to have hard error stop the whole test suite. >>> The point of hard-errors as they were defined in check.mk was to make >>> them *non* ignorable. For instance our test suite raises a hard- >>> error if the program make a segmentation fault, which we never want >>> to tolerate. >> >> I don't understand. What is the difference to a normail FAIL then, >> i.e., to the process exiting with 1? > > By non-ignorable, I meant cannot be XFAILed. It was meant for error > that should never be accepted. > At some point we were developping more-or-less test driven: many tests > were written way before it was possible to pass them, so we knew that > the tests would fail and they were XFAIL'ed. Yet, it was not acceptable > for the executable being tested to die with a SEGV. So simple failures > are just exit 1, and hard-errors (or 'unacceptable errors') had an exit > status that could not be saved by XFAIL. Hmm. Ah, ok. So the desired semantics are: we have a test that is marked as XFAIL currently, but if it does this and that, then that should be a FAIL because it is worse than the expected failure. Yes, that sounds interesting. And it is different from my proposed semantics: We have a failure that should prevent further test from even being tried. Now, the implementation of my proposed semantics /almost/ allows to have your proposed semantics, too: you can run make -k check and the driver will run all tests, even after hard failures. In the presence of hard failures, however, it will fail to produce a summary and the test-suite.log file. Could be a bit problematic when output is very long. I'm not sure whether I want to introduce both semantics. I see yours as reasonable, I'm not sure how applicable mine would be in practice. Opinions appreciated. [ Thanks section ] > Well, it was developped by me for Vaucanson, and we usually hide our > names behind "The Vaucanson Group". That was part of my work at EPITA. > So I would completely drop the Vaucanson group from there. How about > this? > >> ## This code is adapted from check.mk which was originally >> ## written at EPITA/LRDE, further developer at Gostai, then made >> ## its way from GNU coreutils to end up, largely rewritten, >> ## in Automake. Thanks, that's good. Cheers, Ralf
