Hi Bob, sorry for the delay. On 02/19/2012 07:55 PM, Bob Friesenhahn wrote: > I am again bit by automake not being able run the test suite on systems > with bounded command line length. Up to automake 1.11.2 I was able to > apply a patch by Ralf Wildenhues. > Which patch are you referring to exactly?
> With 1.11.3 the implementation is totally different and the patch has no hope > of applying. > How does this later patch of mine fares? <http://debbugs.gnu.org/cgi/bugreport.cgi?bug=7868#31> Caveat: it works for GNU make only, and it's *really ugly*. So ugly that I don't feel like applying it to automake proper. > What is the prospect that Automake will be able to run large test > suites again? > "Again" is not the correct word here -- automake, either patched or not, has *never* been able to run arbitrarily large test suites. Still, recent changes might have unintentionally reduced the maximal "testsuite size" automake is able to handle, and we'd want to fix that. How "big" is the testsuite you are referring to concretely? Anyway, for the moment, I've applied the patch (available from the link above) that introduces a testcase for this automake limitation. Better to keep the issue exposed. The exact version of the applied patch is attached, for reference. Regards, Stefano
>From 0f99c10d3f7087f51c84b01e33784d57194afcc4 Mon Sep 17 00:00:00 2001 Message-Id: <0f99c10d3f7087f51c84b01e33784d57194afcc4.1329903945.git.stefano.lattar...@gmail.com> From: Stefano Lattarini <stefano.lattar...@gmail.com> Date: Wed, 28 Dec 2011 22:37:44 +0100 Subject: [PATCH] coverage: expose automake bug#7868 Expose the command-line length limit issue that can affect the Automake-generated parallel-tests harness, especially on systems where this limit is smaller (e.g., MinGW/MSYS). Suggestion by Bob Friesenhahn. * tests/parallel-tests-many.test: New test. We have verified that it actually exposes the bug#7868, as it passes when we opportunely reduce the number of test cases in $(TESTS). Checked on NetBSD 5.1 64bit, Debian unstable 32bit, Solaris 10 64bit and Cygwin 1.5 32bit. * tests/list-of-tests.mk (handwritten_TESTS): Add it. * tests/Makefile.am (XFAIL_TESTS): Likewise. --- tests/Makefile.am | 1 + tests/list-of-tests.mk | 1 + tests/parallel-tests-many.test | 176 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 178 insertions(+), 0 deletions(-) create mode 100755 tests/parallel-tests-many.test diff --git a/tests/Makefile.am b/tests/Makefile.am index bb7bece..8a77681 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -41,6 +41,7 @@ gcj6.test \ override-conditional-2.test \ java-nobase.test \ objext-pr10128.test \ +parallel-tests-many.test \ pr8365-remake-timing.test \ lex-subobj-nodep.test \ remake-am-pr10111.test \ diff --git a/tests/list-of-tests.mk b/tests/list-of-tests.mk index f19c930..04ade17 100644 --- a/tests/list-of-tests.mk +++ b/tests/list-of-tests.mk @@ -726,6 +726,7 @@ parallel-tests-exit-statuses.test \ parallel-tests-console-output.test \ parallel-tests-once.test \ parallel-tests-trailing-bslash.test \ +parallel-tests-many.test \ tests-environment.test \ am-tests-environment.test \ tests-environment-backcompat.test \ diff --git a/tests/parallel-tests-many.test b/tests/parallel-tests-many.test new file mode 100755 index 0000000..0f2d732 --- /dev/null +++ b/tests/parallel-tests-many.test @@ -0,0 +1,176 @@ +#! /bin/sh +# Copyright (C) 2012 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. + +# Check that the parallel-tests harness does not hit errors due to +# an exceeded command line length when there are many tests. +# For automake bug#7868. This test is currently expected to fail. + +am_parallel_tests=yes +. ./defs || Exit 1 + +set -e + +cat >> configure.in << 'END' +AC_OUTPUT +END + +cat > Makefile.am <<'END' +# Sanity check that the $(TESTS) is going to exceed the system +# command line length. +# Extra quoting and indirections below are required to ensure the +# various make implementations (e.g, GNU make or Sun Distributed Make) +# will truly spawn a shell to execute this command, instead of relying +# on optimizations that might mask the "Argument list too long" error +# we expect. +this-will-fail: + @":" && ":" $(TEST_LOGS) +TEST_LOG_COMPILER = true +include list-of-tests.am +# So that we won't have to create a ton of dummy test cases. +$(TESTS): +END + +# The real instance will be dynamically created later. +echo TESTS = foo.test > list-of-tests.am + +$ACLOCAL && $AUTOCONF && $AUTOMAKE -a \ + || framework_failure_ "unexpected autotools failure" +./configure \ + || framework_failure_ "unexpected configure failure" + +# We want to hit the system command-line length limit without hitting +# the filename length limit or the PATHMAX limit; so we use longish +# (but not too long) names for the testcase, and place them in a nested +# (but not too deeply) directory. +# We also prefer to use the minimal(ish) number of test cases that can +# make us hit the command-line length limit, since the more the test +# cases are, the more time "automake" and "make check" will take to run +# (especially on Cygwin and MinGW/MSYS). + +tname="wow-this-is-a-very-long-name-for-a-simple-dummy-test-case" +dname="and-this-too-is-a-very-long-name-for-a-dummy-directory" + +deepdir=. +depth=0 +for i in 1 2 3 4 5 6 7 8 9 10 12 13 14 15 16 17 18 19 29 21 22 23 24; do + new_deepdir=$deepdir/$dname.d$i + mkdir $new_deepdir || break + tmpfile=$new_deepdir/$tname-some-more-chars-for-good-measure + if touch $tmpfile; then + rm -f $tmpfile || Exit 99 + else + rmdir $new_deepdir || Exit 99 + fi + deepdir=$new_deepdir + unset tmpfile new_deepdir + depth=$i +done + +cat <<END +********************************************************************* +Our tests will be in the following directory (depth = $depth) +********************************************************************* +$deepdir +********************************************************************* +END + +setup_data () +{ + awk " + BEGIN { + print \"TESTS = \\\\\" + for (i = 1; i < $count; i = i + 1) + print \" $deepdir/$tname-\" i \".test \\\\\" + print \" $deepdir/$tname-\" i \".test\" + } + " > list-of-tests.am || Exit 99 + $AUTOMAKE Makefile \ + || framework_failure_ "unexpected automake failure" + ./config.status Makefile \ + || framework_failure_ "unexpected config.status failure" +} + +for count in 1 2 4 8 12 16 20 24 28 32 48 64 96 128 E_HUGE; do + test $count = E_HUGE && break + count=`expr $count '*' 100` || Exit 99 + setup_data + if $MAKE this-will-fail; then + continue + else + # We have managed to find a number of test cases large enough to + # hit the system command-line limits; we can stop. But first, for + # good measure, increase the number of tests of some 20%, to be + # "even more sure" of really tickling command line length limits. + count=`expr '(' $count '*' 12 ')' / 10` || Exit 99 + setup_data + break + fi +done + +if test $count = E_HUGE; then + framework_failure_ "system has a too-high limit on command line length" +else + cat <<END +********************************************************************* + Number of tests we will use: $count +********************************************************************* +END +fi + +env TESTS=$deepdir/$tname-1.test $MAKE -e check \ + && test -f $deepdir/$tname-1.log \ + || framework_failure_ "\"make check\" with one single tests" + +rm -f $deepdir/* || Exit 99 + +$MAKE check > stdout || { cat stdout; Exit 1; } +cat stdout + +grep "All $count tests" stdout + +grep "^PASS: .*$tname-[0-9][0-9]*\.test" stdout > grp +ls -1 $deepdir | grep '\.log$' > lst + +sed 20q lst # For debugging. +sed 20q grp # Likewise. + +test `cat <grp | wc -l` -eq $count +test `cat <lst | wc -l` -eq $count + +# We need to simulate a failure of two tests. +st=0 +env TESTS="$deepdir/$tname-1.test $deepdir/$tname-2.test" \ + TEST_LOG_COMPILER=false $MAKE -e check > stdout && st=1 +cat stdout +test `grep -c '^FAIL:' stdout` -eq 2 || st=1 +test $st -eq 0 || fatal_ "couldn't simulate failure of two tests" +unset st + +$MAKE recheck > stdout || { cat stdout; Exit 1; } +cat stdout +grep "^PASS: .*$tname-1\.test" stdout +grep "^PASS: .*$tname-2\.test" stdout +test `LC_ALL=C grep -c "^[A-Z][A-Z]*:" stdout` -eq 2 +grep "All 2 tests" stdout + +# "make clean" might ignore some failures, so we prefer to also grep its +# output to ensure that no "Argument list too long" error was encountered. +$MAKE clean >output 2>&1 || { cat output; Exit 1; } +cat output +grep -i 'list.* too long' output && Exit 1 +ls $deepdir | grep '\.log$' && Exit 1 + +: -- 1.7.9
