Another testsuite patch, based off of maint, but to be merged in master only.
I will wait the customary 72 hours before pushing. Regards, Stefano -*-*-*- Extend and improve tests on DejaGnu support. * tests/dejagnu.test: Do not create useless dummy test script. Add trailing `:' command. In heading comments, add reference to ... * tests/check12.test: ... this new "semantic" test, covering concurrent use of dejagnu tests, simple tests and `check-local' target. * tests/dejagnu2.test: Make test more reliable, by avoid weak grepping of make output. Prefer `cat' over `echo' to append to configure.in. Quote literal dots in grep regexps. Prefer `grep -c ...' over `grep ... | wc -l'. Make grepping of automake stderr slighty stricter. Add trailing `:' command. * tests/dejagnu3.test: Prefer `cat' over `echo' to append to configure.in. Check stderr of expected-to-fail "make" call. Remove extra blank lines from Makefile.am. * tests/dejagnu4.test: Prefer `cat' over `echo' to append to configure.in. Prefer `mv -f' over plain `mv' when the target file already exists. Avoid extra mkdir calls by creating more directories at once. Better use of blank lines. Check that the `*.log' and `*.sum' files are created by runtest also when "make check" fails. * tests/dejagnu7.test: Prefer `cat' over `echo' to append to configure.in. Better use of blank lines. Add a trailing `:' command. * tests/dejagnu6.test: Likewise, and give the dejagnu test a more descriptive name. * tests/dejagnu5.test: Likewise. Also, simply define package name to `$me' rather than using a non-obvious sed script to extract it from `AC_INIT', and write the Makefile.am with only one command. * tests/Makefile.am (TESTS): Updated. --- ChangeLog | 34 +++++++++ tests/Makefile.am | 1 + tests/Makefile.in | 2 + tests/check12.test | 201 +++++++++++++++++++++++++++++++++++++++++++++++++++ tests/dejagnu.test | 7 +- tests/dejagnu2.test | 21 +++--- tests/dejagnu3.test | 17 +++-- tests/dejagnu4.test | 19 +++-- tests/dejagnu5.test | 19 +++--- tests/dejagnu6.test | 16 ++-- tests/dejagnu7.test | 10 ++-- 11 files changed, 298 insertions(+), 49 deletions(-) create mode 100755 tests/check12.test
From 39ab915725c5eb5fe055dd3e09116982b1abcce1 Mon Sep 17 00:00:00 2001 From: Stefano Lattarini <stefano.lattar...@gmail.com> Date: Sat, 16 Oct 2010 17:03:03 +0200 Subject: [PATCH] Extend and improve tests on DejaGnu support. * tests/dejagnu.test: Do not create useless dummy test script. Add trailing `:' command. In heading comments, add reference to ... * tests/check12.test: ... this new "semantic" test, covering concurrent use of dejagnu tests, simple tests and `check-local' target. * tests/dejagnu2.test: Make test more reliable, by avoid weak grepping of make output. Prefer `cat' over `echo' to append to configure.in. Quote literal dots in grep regexps. Prefer `grep -c ...' over `grep ... | wc -l'. Make grepping of automake stderr slighty stricter. Add trailing `:' command. * tests/dejagnu3.test: Prefer `cat' over `echo' to append to configure.in. Check stderr of expected-to-fail "make" call. Remove extra blank lines from Makefile.am. * tests/dejagnu4.test: Prefer `cat' over `echo' to append to configure.in. Prefer `mv -f' over plain `mv' when the target file already exists. Avoid extra mkdir calls by creating more directories at once. Better use of blank lines. Check that the `*.log' and `*.sum' files are created by runtest also when "make check" fails. * tests/dejagnu7.test: Prefer `cat' over `echo' to append to configure.in. Better use of blank lines. Add a trailing `:' command. * tests/dejagnu6.test: Likewise, and give the dejagnu test a more descriptive name. * tests/dejagnu5.test: Likewise. Also, simply define package name to `$me' rather than using a non-obvious sed script to extract it from `AC_INIT', and write the Makefile.am with only one command. * tests/Makefile.am (TESTS): Updated. --- ChangeLog | 34 +++++++++ tests/Makefile.am | 1 + tests/Makefile.in | 2 + tests/check12.test | 201 +++++++++++++++++++++++++++++++++++++++++++++++++++ tests/dejagnu.test | 7 +- tests/dejagnu2.test | 21 +++--- tests/dejagnu3.test | 17 +++-- tests/dejagnu4.test | 19 +++-- tests/dejagnu5.test | 19 +++--- tests/dejagnu6.test | 16 ++-- tests/dejagnu7.test | 10 ++-- 11 files changed, 298 insertions(+), 49 deletions(-) create mode 100755 tests/check12.test diff --git a/ChangeLog b/ChangeLog index a356958..4fd15f1 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,37 @@ +2010-11-14 Stefano Lattarini <stefano.lattar...@gmail.com> + + Extend and improve tests on DejaGnu support. + * tests/dejagnu.test: Do not create useless dummy test script. + Add trailing `:' command. In heading comments, add reference + to ... + * tests/check12.test: ... this new "semantic" test, covering + concurrent use of dejagnu tests, simple tests and `check-local' + target. + * tests/dejagnu2.test: Make test more reliable, by avoid weak + grepping of make output. Prefer `cat' over `echo' to append + to configure.in. Quote literal dots in grep regexps. Prefer + `grep -c ...' over `grep ... | wc -l'. Make grepping of + automake stderr slighty stricter. Add trailing `:' command. + * tests/dejagnu3.test: Prefer `cat' over `echo' to append to + configure.in. Check stderr of expected-to-fail "make" call. + Remove extra blank lines from Makefile.am. + * tests/dejagnu4.test: Prefer `cat' over `echo' to append to + configure.in. Prefer `mv -f' over plain `mv' when the target + file already exists. Avoid extra mkdir calls by creating more + directories at once. Better use of blank lines. Check that + the `*.log' and `*.sum' files are created by runtest also when + "make check" fails. + * tests/dejagnu7.test: Prefer `cat' over `echo' to append to + configure.in. Better use of blank lines. Add a trailing `:' + command. + * tests/dejagnu6.test: Likewise, and give the dejagnu test a + more descriptive name. + * tests/dejagnu5.test: Likewise. Also, simply define package + name to `$me' rather than using a non-obvious sed script to + extract it from `AC_INIT', and write the Makefile.am with only + one command. + * tests/Makefile.am (TESTS): Updated. + 2010-11-07 Stefano Lattarini <stefano.lattar...@gmail.com> Fix a bug in variable concatanation with `+='. diff --git a/tests/Makefile.am b/tests/Makefile.am index da81c49..9d8cd81 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -133,6 +133,7 @@ check8.test \ check9.test \ check10.test \ check11.test \ +check12.test \ checkall.test \ clean.test \ clean2.test \ diff --git a/tests/Makefile.in b/tests/Makefile.in index eb461a9..8e14e1e 100644 --- a/tests/Makefile.in +++ b/tests/Makefile.in @@ -283,6 +283,7 @@ parallel_tests = \ check-p.test \ check10-p.test \ check11-p.test \ +check12-p.test \ check2-p.test \ check3-p.test \ check4-p.test \ @@ -400,6 +401,7 @@ check8.test \ check9.test \ check10.test \ check11.test \ +check12.test \ checkall.test \ clean.test \ clean2.test \ diff --git a/tests/check12.test b/tests/check12.test new file mode 100755 index 0000000..8be6123 --- /dev/null +++ b/tests/check12.test @@ -0,0 +1,201 @@ +#! /bin/sh +# Copyright (C) 2010 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. + +# Test to make sure dejagnu tests, automake-style tests, and check-local +# target can coexist. + +. ./defs || Exit 1 + +set -e + +cat >> configure.in << 'END' +AC_OUTPUT +END + +cat > Makefile.am << 'END' +AUTOMAKE_OPTIONS = +EXTRA_DIST = +CLEANFILES = +END + +## Simple tests. + +cat > a.test << 'END' +#!/bin/sh +exit ${A_EXIT_STATUS-0} +END +cat > b.test << 'END' +#!/bin/sh +exit ${B_EXIT_STATUS-0} +END +chmod +x a.test b.test + +cat >> Makefile.am << 'END' +TESTS = a.test b.test +EXTRA_DIST += $(TESTS) +END + +A_EXIT_STATUS=0; export A_EXIT_STATUS +B_EXIT_STATUS=0; export B_EXIT_STATUS + +## DejaGnu tests. + +cat >> Makefile.am << 'END' +AUTOMAKE_OPTIONS += dejagnu +DEJATOOL = hammer spanner +AM_RUNTESTFLAGS = HAMMER=$(srcdir)/hammer SPANNER=$(srcdir)/spanner +EXTRA_DIST += hammer.test/hammer.exp +EXTRA_DIST += spanner.test/spanner.exp +END + +cat > hammer << 'END' +#! /bin/sh +echo "Everything looks like a ${NAIL-nail} to me!" +END + +NAIL=nail; export NAIL + +cat > spanner << 'END' +#! /bin/sh +echo "I'm a right spanner!" +END +chmod +x hammer spanner + +mkdir hammer.test spanner.test + +cat > hammer.test/hammer.exp << 'END' +set test test_hammer +spawn $HAMMER +expect { + "Everything looks like a nail to me!" { pass "$test" } + default { fail "$test" } +} +END + +cat > spanner.test/spanner.exp << 'END' +set test test_spanner +spawn $SPANNER +expect { + "I'm a right spanner!" { pass "$test" } + default { fail "$test" } +} +END + +## User-defined extra tests. + +cat >> Makefile.am <<'END' +check-local: + case $$CHECKLOCAL_EXIT_STATUS in \ + 0) echo 'check-local succeded :-)';; \ + *) echo 'check-local failed :-(';; \ + esac >local.log + exit $$CHECKLOCAL_EXIT_STATUS +CLEANFILES += local.log +END +CHECKLOCAL_EXIT_STATUS=0; export CHECKLOCAL_EXIT_STATUS + +## Go with the testcase execution. + +$ACLOCAL +$AUTOCONF +$AUTOMAKE --add-missing + +for vpath in : false; do + + if $vpath; then + srcdir=.. + mkdir build_ + cd build_ + else + srcdir=. + fi + + $srcdir/configure + + $MAKE check + test -f hammer.log + test -f hammer.sum + test -f spanner.log + test -f spanner.sum + # This checks will be run only by the autogenerated `check12-p.test'. + if test x"$parallel_tests" = x"yes"; then + cat test-suite.log + grep '^PASS: a\.test (exit: 0)' a.log + grep '^PASS: b\.test (exit: 0)' b.log + else + : + fi + grep 'check-local succeded :-)' local.log + + cp -f config.status config-status.sav + + $MAKE distclean + test ! -r hammer.log + test ! -r hammer.sum + test ! -r spanner.log + test ! -r spanner.sum + test ! -r test-suite.log + test ! -r a.log + test ! -r b.log + test ! -r local.log + + mv -f config-status.sav config.status + ./config.status + + NAIL=screw $MAKE check && Exit 1 + test -f hammer.log + test -f hammer.sum + test -f spanner.log + test -f spanner.sum + grep 'FAIL: test_hammer' hammer.sum + grep 'FAIL:' spanner.sum && Exit 1 + + B_EXIT_STATUS=1 $MAKE check && Exit 1 + # This checks will be run only by the autogenerated `check12-p.test'. + if test x"$parallel_tests" = x"yes"; then + cat test-suite.log + grep '^PASS: a\.test (exit: 0)' a.log + grep '^FAIL: b\.test (exit: 1)' b.log + grep '^FAIL: b\.test (exit: 1)' test-suite.log + else :; fi + + CHECKLOCAL_EXIT_STATUS=1 $MAKE check && Exit 1 + grep 'check-local failed :-(' local.log + + # Do not trust the exit status of `make -k'. + NAIL=screw B_EXIT_STATUS=23 CHECKLOCAL_EXIT_STATUS=1 $MAKE -k check || : + test -f hammer.log + test -f hammer.sum + test -f spanner.log + test -f spanner.sum + grep 'FAIL: test_hammer' hammer.sum + grep 'FAIL:' spanner.sum && Exit 1 + # This checks will be run only by the autogenerated `check12-p.test'. + if test x"$parallel_tests" = x"yes"; then + cat test-suite.log + grep '^PASS: a\.test (exit: 0)' a.log + grep '^FAIL: b\.test (exit: 23)' b.log + grep '^FAIL: b\.test (exit: 23)' test-suite.log + else :; fi + grep 'check-local failed :-(' local.log + + $MAKE distcheck + $MAKE distclean + cd $srcdir + +done + +: diff --git a/tests/dejagnu.test b/tests/dejagnu.test index a273770..b3e58e7 100755 --- a/tests/dejagnu.test +++ b/tests/dejagnu.test @@ -14,7 +14,8 @@ # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. -# Test to make sure dejagnu tests and automake-style tests can coexist. +# Superficial test to check that dejagnu tests and automake-style +# tests can coexist. See also related deeper test `check12.test'. . ./defs || Exit 1 @@ -25,10 +26,10 @@ AUTOMAKE_OPTIONS = dejagnu TESTS = frob.test END -: > frob.test - $ACLOCAL $AUTOMAKE grep '^check-TESTS' Makefile.in grep '^check-DEJAGNU' Makefile.in + +: diff --git a/tests/dejagnu2.test b/tests/dejagnu2.test index 531f744..121f6d8 100755 --- a/tests/dejagnu2.test +++ b/tests/dejagnu2.test @@ -1,5 +1,5 @@ #! /bin/sh -# Copyright (C) 2001, 2002, 2003 Free Software Foundation, Inc. +# Copyright (C) 2001, 2002, 2003, 2010 Free Software Foundation, Inc. # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -20,26 +20,29 @@ set -e -echo 'AC_OUTPUT' >> configure.in +cat >> configure.in <<'END' +AC_OUTPUT +END cat > Makefile.am << 'END' AUTOMAKE_OPTIONS = dejagnu site.exp: - @echo foo + echo :GREP:ME: > $@ END $ACLOCAL $AUTOCONF $AUTOMAKE -Wno-override -grep site.exp Makefile.in -test `grep '^site\.exp:' Makefile.in | wc -l` -eq 1 +grep 'site\.exp' Makefile.in +test `grep -c '^site\.exp:' Makefile.in` -eq 1 ./configure -$MAKE site.exp >stdout -cat stdout -grep foo stdout +$MAKE site.exp +grep ':GREP:ME:' site.exp AUTOMAKE_fails -grep 'Makefile.am:3:.*site.exp' stderr +grep '^Makefile\.am:3:.*site\.exp' stderr + +: diff --git a/tests/dejagnu3.test b/tests/dejagnu3.test index d812ac4..2c5d375 100755 --- a/tests/dejagnu3.test +++ b/tests/dejagnu3.test @@ -1,5 +1,5 @@ #! /bin/sh -# Copyright (C) 2003 Free Software Foundation, Inc. +# Copyright (C) 2003, 2010 Free Software Foundation, Inc. # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -28,15 +28,14 @@ END chmod +x hammer -echo AC_OUTPUT >> configure.in +cat >> configure.in << 'END' +AC_OUTPUT +END cat > Makefile.am << 'END' AUTOMAKE_OPTIONS = dejagnu - DEJATOOL = hammer - AM_RUNTESTFLAGS = HAMMER=$(srcdir)/hammer - EXTRA_DIST = hammer hammer.test/hammer.exp END @@ -65,5 +64,9 @@ $MAKE distcheck # Ensure that the envvar RUNTESTFLAGS is used. # Report from Mark Mitchell. -RUNTESTFLAGS=--unknown-option $MAKE check && Exit 1 -Exit 0 +RUNTESTFLAGS=--unknown-runtest-option $MAKE check >output 2>&1 \ + && { cat output; Exit 1; } +cat output +$FGREP 'unknown-runtest-option' output + +: diff --git a/tests/dejagnu4.test b/tests/dejagnu4.test index 5fdda10..fa64a31 100755 --- a/tests/dejagnu4.test +++ b/tests/dejagnu4.test @@ -1,5 +1,5 @@ #! /bin/sh -# Copyright (C) 2003, 2006 Free Software Foundation, Inc. +# Copyright (C) 2003, 2006, 2010 Free Software Foundation, Inc. # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -26,17 +26,17 @@ cat > hammer << 'END' #! /bin/sh echo "Everything looks like a nail to me!" END - chmod +x hammer cat > spanner << 'END' #! /bin/sh echo "I'm a right spanner!" END - chmod +x spanner -echo AC_OUTPUT >> configure.in +cat >> configure.in << 'END' +AC_OUTPUT +END cat > Makefile.am << 'END' AUTOMAKE_OPTIONS = dejagnu @@ -49,8 +49,7 @@ EXTRA_DIST = hammer hammer.test/hammer.exp EXTRA_DIST += spanner spanner.test/spanner.exp END -mkdir hammer.test -mkdir spanner.test +mkdir hammer.test spanner.test cat > hammer.test/hammer.exp << 'END' set test test @@ -86,8 +85,14 @@ $MAKE distcheck # Test for PR 488. sed 's/E\(verything\)/Not e\1/' hammer > thammer -mv thammer hammer +mv -f thammer hammer chmod +x hammer +rm -f hammer.log hammer.sum spanner.log spanner.sum $MAKE check && Exit 1 +test -f hammer.log +test -f hammer.sum +test -f spanner.log +test -f spanner.sum + : diff --git a/tests/dejagnu5.test b/tests/dejagnu5.test index 0095958..0237a08 100755 --- a/tests/dejagnu5.test +++ b/tests/dejagnu5.test @@ -1,5 +1,5 @@ #! /bin/sh -# Copyright (C) 2003 Free Software Foundation, Inc. +# Copyright (C) 2003, 2010 Free Software Foundation, Inc. # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -22,30 +22,27 @@ required=runtest set -e -# Get the package name from configure.in. -package=`$FGREP AC_INIT configure.in | sed -e 's:^[^[]*\[\([^]]*\)\].*$:\1:'` +package=$me cat > $package << 'END' #! /bin/sh echo "Ah, we have been expecting you, Mr. Blond." END - chmod +x $package -echo AC_OUTPUT >> configure.in +cat >> configure.in << 'END' +AC_OUTPUT +END cat > Makefile.am << END AUTOMAKE_OPTIONS = dejagnu - EXTRA_DIST = $package $package.test/$package.exp +AM_RUNTESTFLAGS = PACKAGE=\$(srcdir)/$package END -echo "AM_RUNTESTFLAGS = PACKAGE=\$(srcdir)/$package" >> Makefile.am - mkdir $package.test - cat > $package.test/$package.exp << 'END' -set test "test" +set test "a_dejagnu_test" spawn $PACKAGE expect { "Ah, we have been expecting you, Mr. Blond." { pass "$test" } @@ -64,3 +61,5 @@ test -f $package.log test -f $package.sum $MAKE distcheck + +: diff --git a/tests/dejagnu6.test b/tests/dejagnu6.test index 25a1f1f..662ca28 100755 --- a/tests/dejagnu6.test +++ b/tests/dejagnu6.test @@ -1,5 +1,5 @@ #! /bin/sh -# Copyright (C) 2003 Free Software Foundation, Inc. +# Copyright (C) 2003, 2010 Free Software Foundation, Inc. # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -25,23 +25,21 @@ cat > faildeja << 'END' #! /bin/sh echo whatever END - chmod +x faildeja -echo AC_OUTPUT >> configure.in +cat >> configure.in << 'END' +AC_OUTPUT +END cat > Makefile.am << 'END' AUTOMAKE_OPTIONS = dejagnu - DEJATOOL = faildeja - AM_RUNTESTFLAGS = FAILDEJA=$(srcdir)/faildeja END mkdir faildeja.test - cat > faildeja.test/faildeja.exp << 'END' -set test test +set test failing_deja_test spawn $FAILDEJA expect { default { fail "$test" } @@ -57,4 +55,6 @@ $AUTOMAKE --add-missing $MAKE check && Exit 1 test -f faildeja.log test -f faildeja.sum -$FGREP 'FAIL: test' faildeja.sum +$FGREP 'FAIL: failing_deja_test' faildeja.sum + +: diff --git a/tests/dejagnu7.test b/tests/dejagnu7.test index 552c2ca..0be4d1f 100755 --- a/tests/dejagnu7.test +++ b/tests/dejagnu7.test @@ -29,21 +29,19 @@ cat > failtcl << 'END' #! /bin/sh echo whatever END - chmod +x failtcl -echo AC_OUTPUT >> configure.in +cat >> configure.in << 'END' +AC_OUTPUT +END cat > Makefile.am << 'END' AUTOMAKE_OPTIONS = dejagnu - DEJATOOL = failtcl - AM_RUNTESTFLAGS = --status FAILTCL=$(srcdir)/failtcl END mkdir failtcl.test - cat > failtcl.test/failtcl.exp << 'END' set test test spawn $FAILTCL @@ -62,3 +60,5 @@ $MAKE check && Exit 1 test -f failtcl.log test -f failtcl.sum $FGREP 'missing close-brace' failtcl.sum + +: -- 1.7.1