Bruno Haible wrote: > Jim Meyering wrote: >> Imagine that the first 10 tests pass, then each of the remaining ones is >> killed via e.g., SIGHUP. ... >> a naive search for "FAIL:" in the build output would find nothing. > > Yes, and it should be this way, IMO. Each time a user sees a "FAIL:", he > should be encouraged to investigate. > > Whereas in the gettext test suite, often when I sent SIGINTs, I saw some > tests fail without explanation. (This was due to a missing 'exit' statement > in the trap handler, but it would be the same if there was an 'exit 1' in > the trap handler.) I guessed that the FAIL report was due to the SIGINT and > did not investigate. But I don't think this attitude should be encouraged. > > Similarly, when I get reports from Nelson Beebe with lots of failing tests, > I don't want to spend time on fake failures that were due to, maybe, a > shutdown of his virtual machine or something like this. > >> The final result would be highly misleading: >> >> ======================== >> All 10 tests passed >> (300 tests were not run) >> ======================== > > But before this final result, you would see 300 times > > Skipping test: caught fatal signal > SKIP: test-foo1 > Skipping test: caught fatal signal > SKIP: test-foo2 > Skipping test: caught fatal signal > SKIP: test-bar > ... > > That should be enough of an explanation, no? And it will tell us that there's > no gnulib bug to investigate.
I'm sure it's enough if you are alert and watching all of the output go by, but that is not reliable. More often, I redirect to a file and search for traces of unexpected behavior (usually failure) by running "grep FAIL:" on the output. Letting automake report "all 10 tests passed" is misleading. I want my tests to FAIL whenever something unexpected happens, be it user interrupt via control-C or a SIGHUP sent by some other application. If a user interrupts "make check", there is little risk s/he will report that as a legitimate test failure. However, if something is misbehaving and killing my shells when it should not, I don't want to overlook that because some test framework decided to classify that test as merely "SKIPPED".