tags 11791 + patch thanks Hi Peter.
On 06/26/2012 01:56 PM, Peter Johansson wrote: > Hello automakers, > > I've noticed the following behavior from 'make recheck'. > > If I run 'make check' and one test fails > > $ make check > ... > FAIL: foo.test > > I modify the test > > $ edit foo.c > > and rerun the test suite with > > $ make recheck -k > foo.c:1: error: parse error before '}' token > make[1]: *** [foo.o] Error 1 > make[1]: Target `test-suite.log' not remade because of errors. > make: *** [recheck] Error 2 > > which is now interrupted as I introduced a compilation error. > If I now try edit the test again and rerun test suite with > > $ make recheck -k > > I get > > # TOTAL: 0 > # PASS: 0 > # SKIP: 0 > # XFAIL: 0 > # FAIL: 0 > # XPASS: 0 > # ERROR: 0 > > that is, the failed test is not rerun. I suppose the reason is that it > was never run last time and that there exist no trs file, but this is > IMHO suboptimal and unexpected. > I would expect that make recheck also includes tests that failed due > to compilation problems or at least tests that failed last time they > were run. > I agree. The attached patch should solve the issue. It's more ugly than I'd like, but this bug is annoying enough to make such ugliness acceptable. I will push (to maint) in a couple of days if there are no objections. > Please find a script attached illustrating the behavior. > > This is with automake 1.12. > > Cheers, > Peter > Thanks, Stefano
>From 2a3e45d6f3083c5152d94156f8b4e21366090c0f Mon Sep 17 00:00:00 2001 Message-Id: <2a3e45d6f3083c5152d94156f8b4e21366090c0f.1340800913.git.stefano.lattar...@gmail.com> From: Stefano Lattarini <stefano.lattar...@gmail.com> Date: Wed, 27 Jun 2012 12:47:17 +0200 Subject: [PATCH] parallel-tests: "recheck" behaves better in case of compilation failures With this change, the "recheck" target behaves better in the face of build failures related to previously failed tests. For example, if a test is a compiled program that must be rerun by "make recheck", and its compilation fails, that test will still be rerun by further "make recheck" invocations. Previously, its '.log' and '.trs' would have both been lost, so that the test would have not been re-run. This change fixes automake bug#11791. * NEWS: Update. * lib/am/check.am (recheck, check-TESTS): Adjust to cater to scenario described above. * t/parallel-tests-recheck-pr11791.sh: New test. * t/list-of-tests.mk: Add it. Signed-off-by: Stefano Lattarini <stefano.lattar...@gmail.com> --- NEWS | 10 ++++ lib/am/check.am | 43 +++++++++++++---- t/list-of-tests.mk | 1 + t/parallel-tests-recheck-pr11791.sh | 87 +++++++++++++++++++++++++++++++++++ 4 files changed, 133 insertions(+), 8 deletions(-) create mode 100755 t/parallel-tests-recheck-pr11791.sh diff --git a/NEWS b/NEWS index 347287e..0b3004e 100644 --- a/NEWS +++ b/NEWS @@ -82,6 +82,16 @@ New in 1.12.2: to a shell conditional that can be used in recipes to know whether make is being run in silent or verbose mode. +Bugs fixed in 1.12.2: + +* Long-standing bugs: + + - The "recheck" targets behaves better in the face of build failures + related to previously failed tests. For example, if a test is a + compiled program that must be rerun by "make recheck", and its + compilation fails, it will still be rerun by further "make recheck" + invocations. See automake bug#11791. + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ New in 1.12.1: diff --git a/lib/am/check.am b/lib/am/check.am index ba4de46..6816398 100644 --- a/lib/am/check.am +++ b/lib/am/check.am @@ -418,7 +418,7 @@ check-TESTS recheck: ## cannot use '$?' to compute the set of lazily rerun tests, lest ## we rely on .PHONY to work portably. @test -z "$(TEST_SUITE_LOG)" || rm -f $(TEST_SUITE_LOG) - @$(am__set_TESTS_bases); \ + @set +e; $(am__set_TESTS_bases); \ if test $@ = recheck; then \ ## If running a "make recheck", we must only consider tests that had an ## unexpected outcome (FAIL or XPASS) in the earlier run. @@ -434,15 +434,42 @@ check-TESTS recheck: ## be problematic. In this particular case, trailing white space is known ## to have caused segmentation faults on Solaris 10 XPG4 make: log_list=`echo $$log_list`; trs_list=`echo $$trs_list`; \ -## Under "make recheck", remove the .log and .trs files associated -## with the files to recheck, so that those will be rerun by the -## "make test-suite.log" recursive invocation below. But use a proper -## hack to avoid extra files removal when running under "make -n". +## Under "make recheck", move the '.log' and '.trs' files associated +## with the tests to be re-run out of the way, so that those tests will +## be re-run by the "make test-suite.log" recursive invocation below. +## Two tricky requirements: +## - we must avoid extra files removal when running under "make -n"; +## - in case the test is a compiled program whose compilation fails, +## we must ensure that any '.log' and '.trs' file referring to such +## test are preserved, so that future "make recheck" invocations +## will still try to re-compile and re-run it (automake bug#11791). +## The extra contortions below cater to such requirements. + am_backupdir=.am-recheck; \ if test $@ != recheck || $(am__make_dryrun); then :; else \ - test -z "$$log_list" || rm -f $$log_list; \ - test -z "$$trs_list" || rm -f $$trs_list; \ + if test -n "$$trs_list$$log_list"; then \ + { test ! -d $$am_backupdir || rm -rf $$am_backupdir; } \ + && $(MKDIR_P) $$am_backupdir || exit 1; \ + test -z "$$log_list" \ + || mv -f $$log_list $$am_backupdir 2>/dev/null; \ + test -z "$$trs_list" \ + || mv -f $$trs_list $$am_backupdir 2>/dev/null; \ + fi; \ + fi; \ + $(MAKE) $(AM_MAKEFLAGS) $(TEST_SUITE_LOG) TEST_LOGS="$$log_list"; \ + st=$$?; \ + if test $@ != recheck || $(am__make_dryrun) || test ! -d $$am_backupdir; then :; else \ + for f in $$log_list $$trs_list; do \ + test -f $$f || mv $$am_backupdir/$$f . || exit 1; \ + done; \ + rm -rf $$am_backupdir || exit 1; \ fi; \ - $(MAKE) $(AM_MAKEFLAGS) $(TEST_SUITE_LOG) TEST_LOGS="$$log_list" +## Be sure to exit with the proper exit status. The use of "exit" below +## is required to work around a FreeBSD make bug (present only when +## running in concurrent mode). See automake bug#9245: +## <http://debbugs.gnu.org/cgi/bugreport.cgi?bug=9245> +## and FreeBSD PR bin/159730: +## <http://www.freebsd.org/cgi/query-pr.cgi?pr=159730>. + exit $$st; ## Recheck must depend on $(check_SCRIPTS), $(check_PROGRAMS), etc. ## It must also depend on the 'all' target. See automake bug#11252. diff --git a/t/list-of-tests.mk b/t/list-of-tests.mk index e16dd1d..ba27137 100644 --- a/t/list-of-tests.mk +++ b/t/list-of-tests.mk @@ -758,6 +758,7 @@ t/parallel-tests8.sh \ t/parallel-tests9.sh \ t/parallel-tests10.sh \ t/parallel-tests-recheck-depends-on-all.sh \ +t/parallel-tests-recheck-pr11791.sh \ t/parallel-tests-exeext.sh \ t/parallel-tests-suffix.sh \ t/parallel-tests-suffix-prog.sh \ diff --git a/t/parallel-tests-recheck-pr11791.sh b/t/parallel-tests-recheck-pr11791.sh new file mode 100755 index 0000000..bfc55fa --- /dev/null +++ b/t/parallel-tests-recheck-pr11791.sh @@ -0,0 +1,87 @@ +#! /bin/sh +# Copyright (C) 2012 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. + +# parallel-tests: "make recheck" "make -k recheck" in the face of build +# failures for the test cases. See automake bug#11791. + +required='cc native' +. ./defs || Exit 1 + +cat >> configure.ac << 'END' +AC_PROG_CC +AC_OUTPUT +END + +cat > Makefile.am << 'END' +TESTS = $(EXTRA_PROGRAMS) +EXTRA_PROGRAMS = foo +END + +echo 'int main (void) { return 1; }' > foo.c + +$ACLOCAL +$AUTOCONF +$AUTOMAKE -a + +./configure + +$MAKE check >stdout && { cat stdout; Exit 1; } +cat stdout +count_test_results total=1 pass=0 fail=1 xpass=0 xfail=0 skip=0 error=0 + +$MAKE -k recheck >stdout && { cat stdout; Exit 1; } +cat stdout +count_test_results total=1 pass=0 fail=1 xpass=0 xfail=0 skip=0 error=0 + +# Introduce an error in foo.c, that should cause a compilation failure. +$sleep +echo choke me >> foo.c + +$MAKE recheck >stdout && { cat stdout; Exit 1; } +cat stdout +# We don't get a change to run the testsuite. +$EGREP '(X?PASS|X?FAIL|SKIP|ERROR):' stdout && Exit 1 +# These shouldn't be removed, otherwise the next make recheck will do +# nothing. +test -f foo.log +test -f foo.trs + +st=0; $MAKE -k recheck >stdout || st=$? +cat stdout +# Don't trust the exit status of "make -k" for non-GNU makes. +if using_gmake && test $st -eq 0; then Exit 1; fi +# We don't get a change to run the testsuite. +$EGREP '(X?PASS|X?FAIL|SKIP|ERROR):' stdout && Exit 1 +test -f foo.log +test -f foo.trs + +# "Repair" foo.c, and expect everything to work. +$sleep +echo 'int main (void) { return 0; }' > foo.c + +$MAKE recheck >stdout || { cat stdout; Exit 1; } +cat stdout +count_test_results total=1 pass=1 fail=0 xpass=0 xfail=0 skip=0 error=0 +test -f foo.log +test -f foo.trs + +$MAKE recheck >stdout || { cat stdout; Exit 1; } +cat stdout +count_test_results total=0 pass=0 fail=0 xpass=0 xfail=0 skip=0 error=0 +test -f foo.log +test -f foo.trs + +: -- 1.7.9.5