On Monday 18 July 2011, Ralf Wildenhues wrote: > * Stefano Lattarini wrote on Fri, Jul 15, 2011 at 12:37:07AM CEST: > > * lib/am/check.am ($(TEST_SUITE_LOG)): Recognize a new test result > > `ERROR'. Use it when encountering unreadable test logs (previously > > a simple `FAIL' was used in this situations). > > * lib/test-driver: Set the global test result to `ERROR' when the > > test exit status is 99. When doing colorized output, color `ERROR' > > results in magenta. > > * doc/automake.texi (Log files generation and test results > > recording): Update by also listing `ERROR' among the list of valid > > `:test-results:' arguments. > > * NEWS: Update. > > * tests/trivial-test-driver: Update. > > * tests/parallel-tests.test: Likewise. > > * tests/parallel-tests-harderror.test: Likewise. > > * tests/parallel-tests-no-spurious-summary.test: Likewise. > > * tests/test-driver-global-log.test: Likewise. > > * tests/test-driver-recheck.test: Likewise. > > * tests/test-driver-custom-multitest-recheck.test: Likewise. > > * tests/test-driver-custom-multitest-recheck2.test: Likewise. > > * tests/test-driver-custom-multitest.test: Likewise. > > * tests/test-driver-custom-no-html.test: Likewise. > > * tests/test-driver-end-test-results.test: Likewise. > > * tests/color.test: Likewise. Also, make stricter, and also test > > from VPATH. > > * tests/color2.test: Likewise, and improve syncing with color.test. > > * tests/parallel-tests-exit-statuses.test: New test. > > * tests/parallel-tests-console-output.test: Likewise. > > * tests/Makefile.am (TESTS): Update. > > > --- a/NEWS > > +++ b/NEWS > > @@ -13,6 +13,10 @@ New in 1.11a: > > > > * Changes to Automake-generated testsuite harnesses: > > > > + - The parallel-tests harness has a new test result 'ERROR', that can be > > + used to signal e.g., unexpected or internal errors, or failures to set > > + up test case scenarios. > > The NEWS entry could be a bit more precise in that the ERROR state was > actually used before already with exit status 99, just that it is named > ERROR now. > What about this?
- Test scripts that exit with status 99 to signal an "hard error" (e.g., and unexpected or internal error, or a failure to set up the test case scenario) have their outcome reported as an 'ERROR' now. Previous versions of automake reported such an outcome as a 'FAIL' (the only difference with normal failures being that hard errors were counted as failures even when the test originating them was listed in XFAIL_TESTS). > (Please check whether the semantics of 99 already were in a stable release, > Well, an exit status of `99' has been triggering an hard error since at least automake 1.11 (assuming `parallel-tests' was in use). Is this what you were asking, or have I misunderstood? > otherwise this NEWS entry could of course advertise it > as new.) > > A description of ERROR semantics needs to be part of automake.texi as > well. > Doing that properly will require a small reorganization of the "Support for test suites" chapter, so I'd rather do that in a follow-up patch (which I will start working on shortly). > Also, are you going to followup with Autoconf to rename Autotest's hard > failure state ERROR as well? We should agree on common naming and > semantics, so the two systems are not harder to learn than necessary. > Good idea, consistency is always nice. > I'm OK with this change once these issues are resolved. > > As a minor detail however, please remove the '====' greps, I've already > mentioned why I consider '=' in the output not a good idea. > But those greps are there to ensure that the testsuite summary is not unduly colorized; and since the summary at this point is still looks like: ==================================================== All 2 tests behaved as expected (1 expected failure) ==================================================== those grepping checks have to stay IMNSHO. > Thanks, > Ralf > Regards, Stefano