On Mon, Feb 01, 2010 at 08:58:19AM +0100, Jim Meyering wrote: > Bruno Haible wrote: > > Jim Meyering wrote: > >> Imagine that the first 10 tests pass, then each of the remaining ones is > >> killed via e.g., SIGHUP. ... > >> a naive search for "FAIL:" in the build output would find nothing. > > > > Yes, and it should be this way, IMO. Each time a user sees a "FAIL:", he > > should be encouraged to investigate. > > > > Whereas in the gettext test suite, often when I sent SIGINTs, I saw some > > tests fail without explanation. (This was due to a missing 'exit' statement > > in the trap handler, but it would be the same if there was an 'exit 1' in > > the trap handler.) I guessed that the FAIL report was due to the SIGINT and > > did not investigate. But I don't think this attitude should be encouraged. > > > > Similarly, when I get reports from Nelson Beebe with lots of failing tests, > > I don't want to spend time on fake failures that were due to, maybe, a > > shutdown of his virtual machine or something like this. > > > >> The final result would be highly misleading: > >> > >> ======================== > >> All 10 tests passed > >> (300 tests were not run) > >> ======================== > > > > But before this final result, you would see 300 times > > > > Skipping test: caught fatal signal > > SKIP: test-foo1 > > Skipping test: caught fatal signal > > SKIP: test-foo2 > > Skipping test: caught fatal signal > > SKIP: test-bar > > ... > > > > That should be enough of an explanation, no? And it will tell us that > > there's > > no gnulib bug to investigate. > > I'm sure it's enough if you are alert and watching all > of the output go by, but that is not reliable. > > More often, I redirect to a file and search for > traces of unexpected behavior (usually failure) by running > "grep FAIL:" on the output. > > Letting automake report "all 10 tests passed" is misleading. > > I want my tests to FAIL whenever something unexpected happens, > be it user interrupt via control-C or a SIGHUP sent by some > other application. If a user interrupts "make check", there > is little risk s/he will report that as a legitimate test failure. > However, if something is misbehaving and killing my shells when > it should not, I don't want to overlook that because some test > framework decided to classify that test as merely "SKIPPED".
I just want to remind you that the undefined behaviour still hasn't been fixed. Please make a decision what kind of fix suits you better. -- ldv
pgp1kpfh2UuS7.pgp
Description: PGP signature