bug#8969: improve synchronization between examples in the manual and test cases
Severity: wishlist thanks Reference: http://lists.gnu.org/archive/html/automake-patches/2011-06/msg00220.html On Tuesday 28 June 2011, Ralf Wildenhues wrote: * Stefano Lattarini wrote on Tue, Jun 28, 2011 at 08:24:13AM CEST: On Monday 27 June 2011, Ralf Wildenhues wrote: Sure. Well, do they use some format already? Well, there are only two of them, and both follows this format: @c The test case for the setup described here is @c test/subdircond2.test @c Try to keep it in sync. (which is wrongish BTW, as the testsuite subdirectory is `tests/', not `test/') I'd go with one of these simple formats: @c Keep in sync with subdircond2.test I like this one (with a trailing dot ;-) ;-) I'll soon post a patch that improves the synchronization between some examples from the manual and some test cases. In the process of writing it, I've noticed that various non-trivial examples from the manual are not tested; and while I've remedied to part of this problem with my patch, a lot remains to be done; so I'm opening this bug to avoid forgetting about the issue. Regards, Stefano
bug#8969: improve synchronization between examples in the manual and test cases
On Friday 01 July 2011, Stefano Lattarini wrote: Severity: wishlist thanks Reference: http://lists.gnu.org/archive/html/automake-patches/2011-06/msg00220.html On Tuesday 28 June 2011, Ralf Wildenhues wrote: * Stefano Lattarini wrote on Tue, Jun 28, 2011 at 08:24:13AM CEST: On Monday 27 June 2011, Ralf Wildenhues wrote: Sure. Well, do they use some format already? Well, there are only two of them, and both follows this format: @c The test case for the setup described here is @c test/subdircond2.test @c Try to keep it in sync. (which is wrongish BTW, as the testsuite subdirectory is `tests/', not `test/') I'd go with one of these simple formats: @c Keep in sync with subdircond2.test I like this one (with a trailing dot ;-) ;-) I'll soon post a patch that improves the synchronization between some examples from the manual and some test cases. In the process of writing it, I've noticed that various non-trivial examples from the manual are not tested; and while I've remedied to part of this problem with my patch, a lot remains to be done; so I'm opening this bug to avoid forgetting about the issue. Regards, Stefano And here it is the promised patch. I will push in 72 hours if there is no objection by then. Regards, Stefano From 2dd820b423bea12eac7b94fd8ed2563ee21cf9e1 Mon Sep 17 00:00:00 2001 Message-Id: 2dd820b423bea12eac7b94fd8ed2563ee21cf9e1.1309473910.git.stefano.lattar...@gmail.com From: Stefano Lattarini stefano.lattar...@gmail.com Date: Thu, 30 Jun 2011 20:05:25 +0200 Subject: [PATCH] docs, tests: synchronize examples from docs to tests * tests/README (Writing test cases): Give suggestions on how to keep test cases and examples in the documentation synchronized. * doc/automake.texi: Improve or fix existing testcase-referencing comments, and add many new ones. * HACKING (Administrivia): Suggest to test complex examples and idioms from the manual. * tests/specflg8.test: Improve synchronization with the example in the manual. * tests/output11.test:Likewise. * tests/txinfo21.test:Likewise. * tests/interp.test: Likewise. Since we are at it, and enable the `errexit' shell flag, do related changes, and add trailing `:'command. * tests/amhello-cflags.test: New test. * tests/amhello-cross-compile.test: Likewise. * tests/amhello-binpkg.test: Likewise. * tests/tests-environment-backcompat: Likewise. * tests/parallel-tests-log-compiler-example.test: Likewise. * tests/Makefile.am (TESTS): Update. --- ChangeLog | 23 HACKING|7 ++- doc/automake.texi | 54 ++ tests/Makefile.am |5 ++ tests/Makefile.in |5 ++ tests/README |7 ++ tests/amhello-binpkg.test | 44 +++ tests/amhello-cflags.test | 49 tests/amhello-cross-compile.test | 54 ++ tests/interp.test | 18 -- tests/parallel-tests-log-compiler-example.test | 71 tests/specflg8.test| 35 ++-- tests/tests-environment-backcompat.test| 65 ++ tests/txinfo21.test|4 +- 14 files changed, 402 insertions(+), 39 deletions(-) create mode 100755 tests/amhello-binpkg.test create mode 100755 tests/amhello-cflags.test create mode 100755 tests/amhello-cross-compile.test create mode 100755 tests/parallel-tests-log-compiler-example.test create mode 100755 tests/tests-environment-backcompat.test diff --git a/ChangeLog b/ChangeLog index f1e10c2..c77326d 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,26 @@ +2011-07-01 Stefano Lattarini stefano.lattar...@gmail.com + + docs, tests: synchronize examples from docs to tests + * tests/README (Writing test cases): Give suggestions on how to + keep test cases and examples in the documentation synchronized. + * doc/automake.texi: Improve or fix existing testcase-referencing + comments, and add many new ones. + * HACKING (Administrivia): Suggest to test complex examples and + idioms from the manual. + * tests/specflg8.test: Improve synchronization with the example + in the manual. + * tests/output11.test:Likewise. + * tests/txinfo21.test:Likewise. + * tests/interp.test: Likewise. Since we are at it, and enable + the `errexit' shell flag, do related changes, and add trailing + `:'command. + * tests/amhello-cflags.test: New test. + * tests/amhello-cross-compile.test: Likewise. + * tests/amhello-binpkg.test: Likewise. + * tests/tests-environment-backcompat: Likewise. + * tests/parallel-tests-log-compiler-example.test: Likewise. + * tests/Makefile.am (TESTS): Update. + 2011-06-23 Stefano Lattarini stefano.lattar...@gmail.com docs:
[PATCH] {master} coverage: new test on parallel-tests TESTS runtime overriding
I'd like to add this new test case to master. OK?
I'll push in 72 hours if there is no review by then.
Regards,
Stefano
From b1531a90bf07d3343dda5111f4789e3b6066e480 Mon Sep 17 00:00:00 2001
Message-Id: b1531a90bf07d3343dda5111f4789e3b6066e480.1309444834.git.stefano.lattar...@gmail.com
From: Stefano Lattarini stefano.lattar...@gmail.com
Date: Thu, 30 Jun 2011 16:37:19 +0200
Subject: [PATCH] coverage: new test on parallel-tests TESTS runtime overriding
* tests/parallel-tests-cmdline-override.test: New test, check that
we can use indirections when overriding TESTS and TEST_LOGS from
the command line.
* tests/Makefile.am (TESTS): Update.
---
ChangeLog |8 +++
tests/Makefile.am |1 +
tests/Makefile.in |1 +
tests/parallel-tests-cmdline-override.test | 84
4 files changed, 94 insertions(+), 0 deletions(-)
create mode 100755 tests/parallel-tests-cmdline-override.test
diff --git a/ChangeLog b/ChangeLog
index 1436482..345d41f 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,11 @@
+2011-06-30 Stefano Lattarini stefano.lattar...@gmail.com
+
+ coverage: new test on parallel-tests TESTS runtime overriding
+ * tests/parallel-tests-cmdline-override.test: New test, check that
+ we can use indirections when overriding TESTS and TEST_LOGS from
+ the command line.
+ * tests/Makefile.am (TESTS): Update.
+
2011-06-29 Stefano Lattarini stefano.lattar...@gmail.com
docs: explain why AM_TESTS_ENVIRONMENT must be semicolon-terminated
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 3883dc0..0409258 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -727,6 +727,7 @@ parallel-tests-harderror.test \
parallel-tests-log-override-1.test \
parallel-tests-log-override-2.test \
parallel-tests-log-override-recheck.test \
+parallel-tests-cmdline-override.test \
parse.test \
percent.test \
percent2.test \
diff --git a/tests/Makefile.in b/tests/Makefile.in
index 7453863..c501819 100644
--- a/tests/Makefile.in
+++ b/tests/Makefile.in
@@ -1000,6 +1000,7 @@ parallel-tests-harderror.test \
parallel-tests-log-override-1.test \
parallel-tests-log-override-2.test \
parallel-tests-log-override-recheck.test \
+parallel-tests-cmdline-override.test \
parse.test \
percent.test \
percent2.test \
diff --git a/tests/parallel-tests-cmdline-override.test b/tests/parallel-tests-cmdline-override.test
new file mode 100755
index 000..cd8f82f
--- /dev/null
+++ b/tests/parallel-tests-cmdline-override.test
@@ -0,0 +1,84 @@
+#! /bin/sh
+# Copyright (C) 2011 Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see http://www.gnu.org/licenses/.
+
+# Check that we can use indirections when overriding TESTS and
+# TEST_LOGS from the command line.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat configure.in 'END'
+AC_OUTPUT
+END
+
+cat Makefile.am 'END'
+TEST_EXTENSIONS = .test .t
+TEST_LOG_COMPILER = cat
+T_LOG_COMPILER = cat
+TESTS = bad.test
+var1 = b.test $(var2)
+var2 = c.test
+var3 = d.d
+var4 = e
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE -a
+
+./configure
+rm -f config.log # Not to create false psotives below.
+
+LC_ALL=C sort exp-log 'END'
+a.log
+b.log
+c.log
+d.log
+e.log
+test-suite.log
+END
+
+LC_ALL=C sort exp-out 'END'
+PASS: a.t
+PASS: b.test
+PASS: c.test
+PASS: d.t
+PASS: e.test
+END
+
+do_check ()
+{
+ env $@ $MAKE -e check stdout || { cat stdout; Exit 1; }
+ cat stdout
+ grep '^PASS:' stdout | LC_ALL=C sort got-out
+ cat got-out
+ ls . | grep '\.log$' | LC_ALL=C sort got-log
+ cat got-log
+ st=0
+ diff exp-out got-out || st=1
+ diff exp-log got-log || st=1
+ return $st
+}
+
+tests='a.t $(var1) $(var3:.d=.t) $(var4:=.test)'
+test_logs='a.log $(var1:.test=.log) $(var3:.d=.log) $(var4:=.log)'
+
+touch a.t b.test c.test d.t e.test
+
+do_check TESTS=$tests
+do_check TEST_LOGS=$test_logs
+
+:
--
1.7.2.3
[FYI] New public temporary branch, about improved handling and parsing of test results
On Wednesday 29 June 2011, Stefano Lattarini wrote: Reference: http://lists.gnu.org/archive/html/automake-patches/2011-06/msg00199.html The new code for parsing the testsuite-generated `.log' files, as introduced in commit `v1.11-872-gc96b881', considers each `:test-result:' field anywhere in a `.log' file as a declaration of a test result, and accounts for it as such in the testsuite summary. Unfortunately this could easily cause spurious test failures being reported in the testsuite summary. This happened in practice with the Automake's own testsuite; for example: $ make check TESTS='check12-p.test'; echo exit: $? ... PASS: check12-p.test = 4 of 5 tests failed See tests/test-suite.log Please report to bug-autom...@gnu.org = make[2]: *** [test-suite.log] Error 1 make: *** [check-am] Error 2 exit: 2 This change introduces a new special `:test-result:' END, that, when seen, prevents the rest of the log file from being parsed. OK for the 'test-protocols' branch? Regards, Stefano For the moment, since I need to write more patches on the top of this, I've pushed this patch to a new temporary public branch: - GSoC/experimental/test-results-work Notice that this branch is bound to be amended and rebased, and finally deleted (it won't be merged in the 'test-protocols' branch, rather its changes will be re-applied on the top of that). As usual, reviews are welcome. Regards, Stefano
[PATCH] parallel-tests: new recognized test result 'ERROR'
I've applied the attached patch to the 'GSoC/experimental/test-results-work' branch Note that this is not an FYI, since that branch is temporary, and can thus be amended and modified. Reviews are welcome! Thanks, Stefano From 2df72384ab57d165c7e1da0f88e513171a0035e8 Mon Sep 17 00:00:00 2001 Message-Id: 2df72384ab57d165c7e1da0f88e513171a0035e8.1309445709.git.stefano.lattar...@gmail.com From: Stefano Lattarini stefano.lattar...@gmail.com Date: Thu, 30 Jun 2011 14:55:43 +0200 Subject: [PATCH] parallel-tests: new recognized test result 'ERROR' * lib/am/check.am ($(TEST_SUITE_LOG)): Recognize a new test result `ERROR'. Use it when encountering unreadable test logs (previously a simple `FAIL' was used in this situations). * lib/test-driver: Set the global test result to `ERROR' when the test exit status is 99. When doing colorized output, color `ERROR' results in magenta. * doc/automake.texi (Log files generation and test results recording): Update by also listing `ERROR' among the list of valid `:test-results:' arguments. * tests/trivial-test-driver: Update. * tests/parallel-tests.test: Likewise. * tests/parallel-tests-harderror.test: Likewise. * tests/parallel-tests-no-spurious-summary.test: Likewise. * tests/test-driver-global-log.test: Likewise. * tests/test-driver-recheck.test: Likewise. * tests/test-driver-custom-multitest-recheck.test: Likewise. * tests/test-driver-custom-multitest-recheck2.test: Likewise. * tests/test-driver-custom-multitest.test: Likewise. * tests/test-driver-custom-no-html.test: Likewise. * tests/test-driver-end-test-results.test: Likewise. * tests/color.test: Likewise, and make stricter. * tests/color2.test: Likewise, and improve syncing with color.test. * tests/parallel-tests-exit-statuses.test: New test. * tests/parallel-tests-console-output.test: New test. * tests/Makefile.am (TESTS): Update. --- ChangeLog| 29 ++ doc/automake.texi|8 +- lib/am/check.am |8 +- lib/test-driver |5 +- tests/Makefile.am|2 + tests/Makefile.in|6 +- tests/color.test | 48 ++ tests/color2.test| 51 ++ tests/parallel-tests-console-output.test | 104 + tests/parallel-tests-exit-statuses.test | 108 ++ tests/parallel-tests-harderror.test |6 +- tests/parallel-tests-no-spurious-summary.test|2 + tests/parallel-tests.test|9 +- tests/test-driver-custom-multitest-recheck.test | 27 +++--- tests/test-driver-custom-multitest-recheck2.test | 56 --- tests/test-driver-custom-multitest.test | 33 --- tests/test-driver-custom-no-html.test|2 +- tests/test-driver-end-test-results.test |1 + tests/test-driver-global-log.test|4 +- tests/test-driver-recheck.test |2 +- tests/trivial-test-driver| 11 +- 21 files changed, 419 insertions(+), 103 deletions(-) create mode 100755 tests/parallel-tests-console-output.test create mode 100755 tests/parallel-tests-exit-statuses.test diff --git a/ChangeLog b/ChangeLog index ab7ba2f..22e7087 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,32 @@ +2011-06-30 Stefano Lattarini stefano.lattar...@gmail.com + + parallel-tests: new recognized test result 'ERROR' + * lib/am/check.am ($(TEST_SUITE_LOG)): Recognize a new test result + `ERROR'. Use it when encountering unreadable test logs (previously + a simple `FAIL' was used in this situations). + * lib/test-driver: Set the global test result to `ERROR' when the + test exit status is 99. When doing colorized output, color `ERROR' + results in magenta. + * doc/automake.texi (Log files generation and test results + recording): Update by listing `ERROR' too among the list of valid + `:test-results:' arguments. + * tests/trivial-test-driver: Update. + * tests/parallel-tests.test: Likewise. + * tests/parallel-tests-harderror.test: Likewise. + * tests/parallel-tests-no-spurious-summary.test: Likewise. + * tests/test-driver-global-log.test: Likewise. + * tests/test-driver-recheck.test: Likewise. + * tests/test-driver-custom-multitest-recheck.test: Likewise. + * tests/test-driver-custom-multitest-recheck2.test: Likewise. + * tests/test-driver-custom-multitest.test: Likewise. + * tests/test-driver-custom-no-html.test: Likewise. + * tests/test-driver-end-test-results.test: Likewise. + * tests/color.test: Likewise, and make stricter. + * tests/color2.test: Likewise, and improve syncing with color.test. + * tests/parallel-tests-exit-statuses.test: New test. + * tests/parallel-tests-console-output.test: New test. + * tests/Makefile.am (TESTS): Update. + 2011-06-29 Stefano Lattarini
[FYI] {maint} cosmetics: fix typos in recent ChangeLog entries
--- ChangeLog |8 1 files changed, 4 insertions(+), 4 deletions(-) diff --git a/ChangeLog b/ChangeLog index af1c66d..f1e10c2 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,11 +1,11 @@ -2011-06-23 Stefano Lattarini stefano.lattar...@gmail.com +2011-06-23 Stefano Lattarini stefano.lattar...@gmail.com docs: avoid a footnote, some related rewordings and improvements * doc/automake.texi (Dist): Reword the part about automatically distributed files to avoid a footnote. Since we are at it, extend a bit, and add an example and a reference to a relevant test case. -2011-06-23 Stefano Lattarini stefano.lattar...@gmail.com +2011-06-23 Stefano Lattarini stefano.lattar...@gmail.com docs: minor cosmetic fixes * doc/automake.texi: Break few overly long lines, throughout the @@ -18,7 +18,7 @@ (Other things Automake recognizes @item AM_C_PROTOTYPES): Use @pxref instead of @ref. -2011-06-23 Stefano Lattarini stefano.lattar...@gmail.com +2011-06-23 Stefano Lattarini stefano.lattar...@gmail.com help: improve text about automatically-distributed files This change fixes automake bug#7819. @@ -31,7 +31,7 @@ * tests/autodist.test: Likewise. (configure.in): Remove useless call to AM_MAINTAINER_MODE. -2011-06-23 Stefano Lattarini stefano.lattar...@gmail.com +2011-06-23 Stefano Lattarini stefano.lattar...@gmail.com refactor: split 'usage' subroutine in automake This change is related to automake bug#7819. -- 1.7.2.3
[MERGE REQUEST] merge patches for AM_TESTSUITE_ENVIRONMENT into maint
Hello automakers. I think that the patches introducing support for AM_TESTSUITE_ENVIRONMENT should be merged into maint, as they'd make a nice addition for 1.11.2. I've verified that almost all these patches are based off of maint, and the few that aren't (e.g., `v1.11-867-g1e005df') are just testsuite additions (and could probably be cherry-picked anyway, but I'm not sure whther that would be worthwhile). My proposal is to push the last relevant commit (should be `v1.11-351-g3e334a2') to a new public branch 'am-tests-environment', tie up the last loose ends (if any), and finally merge that branch into maint. Objections? Regards, Stefano
Re: [PATCH] {master} coverage: new test on parallel-tests TESTS runtime overriding
* Stefano Lattarini wrote on Thu, Jun 30, 2011 at 04:40:51PM CEST:
I'd like to add this new test case to master. OK?
Sure, note typo below.
This could actually find bugs in older makes, but hey, we'd want to know
about them.
Thanks,
Ralf
Subject: [PATCH] coverage: new test on parallel-tests TESTS runtime overriding
* tests/parallel-tests-cmdline-override.test: New test, check that
we can use indirections when overriding TESTS and TEST_LOGS from
the command line.
* tests/Makefile.am (TESTS): Update.
--- /dev/null
+++ b/tests/parallel-tests-cmdline-override.test
+# Check that we can use indirections when overriding TESTS and
+# TEST_LOGS from the command line.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat configure.in 'END'
+AC_OUTPUT
+END
+
+cat Makefile.am 'END'
+TEST_EXTENSIONS = .test .t
+TEST_LOG_COMPILER = cat
+T_LOG_COMPILER = cat
+TESTS = bad.test
+var1 = b.test $(var2)
+var2 = c.test
+var3 = d.d
+var4 = e
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE -a
+
+./configure
+rm -f config.log # Not to create false psotives below.
positives
and s/Not to/Do not/ I guess.
+
+LC_ALL=C sort exp-log 'END'
+a.log
+b.log
+c.log
+d.log
+e.log
+test-suite.log
+END
+
+LC_ALL=C sort exp-out 'END'
+PASS: a.t
+PASS: b.test
+PASS: c.test
+PASS: d.t
+PASS: e.test
+END
+
+do_check ()
+{
+ env $@ $MAKE -e check stdout || { cat stdout; Exit 1; }
+ cat stdout
+ grep '^PASS:' stdout | LC_ALL=C sort got-out
+ cat got-out
+ ls . | grep '\.log$' | LC_ALL=C sort got-log
+ cat got-log
+ st=0
+ diff exp-out got-out || st=1
+ diff exp-log got-log || st=1
+ return $st
+}
+
+tests='a.t $(var1) $(var3:.d=.t) $(var4:=.test)'
+test_logs='a.log $(var1:.test=.log) $(var3:.d=.log) $(var4:=.log)'
+
+touch a.t b.test c.test d.t e.test
+
+do_check TESTS=$tests
+do_check TEST_LOGS=$test_logs
Re: [FYI] {master} docs: fix unportable example of AM_TESTS_ENVIRONMENT usage
Hi Stefano, * Stefano Lattarini wrote on Thu, Jun 23, 2011 at 10:56:31PM CEST: On Thursday 23 June 2011, Ralf Wildenhues wrote: -AM_TESTS_ENVIRONMENT = exec 92; warn_fileno=9; export warn_fileno; This example served two purposes: use of AM_TESTS_ENVIRONMENT, and showing how to produce output while the test is running. Your patch removes the second feature. Can we have it back, ideally in a portable fashion? Alas, no. The only way to do that portably is to either: 1. add a trailing `; 2' to AM_TESTS_ENVIRONMENT, *and* force the user not to define TESTS_ENVIRONMENT; or 2. use TESTS_ENVIRONMENT, not AM_TESTS_ENVIRONMENT, in the Makefile.am (still with a trailing `; 2' obviously). Actually, it used to be nonportable to put the redirection before the command. I'm fairly sure that's only with historic shells and not an issue in practice any more (Sven Mascheck's pages will have the details for the curious). In each case, the user's namespace is invaded, and we go against our own advice (in [1] we violate the always terminate AM_TESTS_ENVIRONMENT with a semicolon rule, in [2] we violate don't define TESTS_ENVIRONMENT in the Makefile.am, it's user-reserved rule). Also, the example above belongs IMHO more in a FAQ rather than in a reference manual. Are you volunteering to update the FAQ? ;-) Thanks, Ralf
Re: [MERGE REQUEST] merge patches for AM_TESTSUITE_ENVIRONMENT into maint
* Stefano Lattarini wrote on Thu, Jun 30, 2011 at 05:12:38PM CEST: I think that the patches introducing support for AM_TESTSUITE_ENVIRONMENT should be merged into maint, as they'd make a nice addition for 1.11.2. I've verified that almost all these patches are based off of maint, and the few that aren't (e.g., `v1.11-867-g1e005df') I don't have v1.11-867-g1e005df nor v1.11-351-g3e334a2 in my tree. Can you make a branch available that you'd like to merge? Or alternatively, links to the patch mails in the archive? are just testsuite additions (and could probably be cherry-picked anyway, but I'm not sure whther that would be worthwhile). My proposal is to push the last relevant commit (should be `v1.11-351-g3e334a2') to a new public branch 'am-tests-environment', tie up the last loose ends (if any), and finally merge that branch into maint. Objections? Thanks, Ralf
Re: [PATCH] parallel-tests: new recognized test result 'ERROR'
* Stefano Lattarini wrote on Thu, Jun 30, 2011 at 04:58:21PM CEST:
I've applied the attached patch to the 'GSoC/experimental/test-results-work'
branch Note that this is not an FYI, since that branch is temporary, and can
thus be amended and modified. Reviews are welcome!
Just a couple of quick thoughts:
- is ERROR the same term other testing frameworks use for similar
semantics? Let's strive to be consistent.
- will users confuse FAIL and ERROR and try to use the latter where the
former is appropriate (or vice versa)?
Let's try to avoid inventing different terms where and when we can.
Changing semantics requires a NEWS entry. It's actually an
incompatibility, but a very minor one, so it's not a big problem.
Thanks,
Ralf
Subject: [PATCH] parallel-tests: new recognized test result 'ERROR'
* lib/am/check.am ($(TEST_SUITE_LOG)): Recognize a new test result
`ERROR'. Use it when encountering unreadable test logs (previously
a simple `FAIL' was used in this situations).
* lib/test-driver: Set the global test result to `ERROR' when the
test exit status is 99. When doing colorized output, color `ERROR'
results in magenta.
* doc/automake.texi (Log files generation and test results
recording): Update by also listing `ERROR' among the list of valid
`:test-results:' arguments.
* tests/trivial-test-driver: Update.
* tests/parallel-tests.test: Likewise.
* tests/parallel-tests-harderror.test: Likewise.
* tests/parallel-tests-no-spurious-summary.test: Likewise.
* tests/test-driver-global-log.test: Likewise.
* tests/test-driver-recheck.test: Likewise.
* tests/test-driver-custom-multitest-recheck.test: Likewise.
* tests/test-driver-custom-multitest-recheck2.test: Likewise.
* tests/test-driver-custom-multitest.test: Likewise.
* tests/test-driver-custom-no-html.test: Likewise.
* tests/test-driver-end-test-results.test: Likewise.
* tests/color.test: Likewise, and make stricter.
* tests/color2.test: Likewise, and improve syncing with color.test.
* tests/parallel-tests-exit-statuses.test: New test.
* tests/parallel-tests-console-output.test: New test.
* tests/Makefile.am (TESTS): Update.
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -9248,10 +9248,10 @@ leading whitespace will not be ignored.
@c Keep this in sync with lib/am/check-am:$(TEST_SUITE_LOG).
The only recognized test results are currently @code{PASS}, @code{XFAIL},
-@code{SKIP}, @code{FAIL} and @code{XPASS}. These results, when declared
-with @code{:test-result:}, can be optionally followed by text holding
-the name and/or a brief description of the corresponding test; the
-@option{parallel-tests} harness will ignore such extra text when
+@code{SKIP}, @code{FAIL}, @code{XPASS} and @code{ERROR}. These results,
+when declared with @code{:test-result:}, can be optionally followed by
+text holding the name and/or a brief description of the corresponding
+test; the @option{parallel-tests} harness will ignore such extra text when
ERROR semantics are not actually described.
generating @file{test-suite.log} and preparing the testsuite summary.
Also, @code{:test-result:} can be used with a special ``pseudo-result''
@code{END}, that will instruct the testsuite harness to stop scanning
--- a/lib/am/check.am
+++ b/lib/am/check.am
@@ -66,7 +66,7 @@ include inst-vars.am
## appended.
##
## In addition to the magic exit 77 means SKIP feature (which was
-## imported from automake), there is a magic exit 99 means FAIL feature
+## imported from automake), there is a magic exit 99 means ERROR feature
## which is useful if you need to issue a hard error no matter whether the
## test is XFAIL or not. You can disable this feature by setting the
## variable DISABLE_HARD_ERRORS to a nonempty value.
@@ -153,7 +153,7 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
## Readable test logs.
list2=`for f in $$list; do test ! -r $$f || echo $$f; done`; \
## Each unreadable test log counts as a failed test.
- results1=`for f in $$list; do test -r $$f || echo FAIL; done`; \
+ results1=`for f in $$list; do test -r $$f || echo ERROR; done`; \
## Now we're going to extract the outcome of all the testcases from the
## test logs.
results2=''; \
@@ -187,6 +187,10 @@ $(TEST_SUITE_LOG): $(TEST_LOGS)
skip=`echo $$results | grep -c '^SKIP'`; \
xfail=`echo $$results | grep -c '^XFAIL'`;\
xpass=`echo $$results | grep -c '^XPASS'`;\
+ error=`echo $$results | grep -c '^ERROR'`;\
+## FIXME: for the moment, we count errors as failures, otherwise the code
+## that display the testsuite summary will become too much complicated.
grammar nits: s/display/s/; s/much //
+ fail=`expr $$fail + $$error`; \
failures=`expr $$fail + $$xpass`; \
all=`expr $$all - $$skip`; \
if test
Re: [MERGE REQUEST] merge patches for AM_TESTSUITE_ENVIRONMENT into maint
* Ralf Wildenhues wrote on Thu, Jun 30, 2011 at 10:52:16PM CEST: * Stefano Lattarini wrote on Thu, Jun 30, 2011 at 05:12:38PM CEST: I think that the patches introducing support for AM_TESTSUITE_ENVIRONMENT should be merged into maint, as they'd make a nice addition for 1.11.2. I've verified that almost all these patches are based off of maint, and the few that aren't (e.g., `v1.11-867-g1e005df') I don't have v1.11-867-g1e005df nor v1.11-351-g3e334a2 in my tree. D'oh. EWRONGREPO. I still don't have the latter commit in my tree, and the former has many many commits on top of maint. So no, you cannot merge that into maint. That would require more than just a cursory review, and I don't see the gain right now. are just testsuite additions (and could probably be cherry-picked anyway, but I'm not sure whther that would be worthwhile). My proposal is to push the last relevant commit (should be `v1.11-351-g3e334a2') to a new public branch 'am-tests-environment', tie up the last loose ends (if any), and finally merge that branch into maint. Objections? Sorry, Ralf
Re: [FYI] {master} docs: fix unportable example of AM_TESTS_ENVIRONMENT usage
On Thursday 30 June 2011, Ralf Wildenhues wrote: Hi Stefano, * Stefano Lattarini wrote on Thu, Jun 23, 2011 at 10:56:31PM CEST: On Thursday 23 June 2011, Ralf Wildenhues wrote: -AM_TESTS_ENVIRONMENT = exec 92; warn_fileno=9; export warn_fileno; This example served two purposes: use of AM_TESTS_ENVIRONMENT, and showing how to produce output while the test is running. Your patch removes the second feature. Can we have it back, ideally in a portable fashion? Alas, no. The only way to do that portably is to either: 1. add a trailing `; 2' to AM_TESTS_ENVIRONMENT, *and* force the user not to define TESTS_ENVIRONMENT; or 2. use TESTS_ENVIRONMENT, not AM_TESTS_ENVIRONMENT, in the Makefile.am (still with a trailing `; 2' obviously). Actually, it used to be nonportable to put the redirection before the command. Ouch. I'm fairly sure that's only with historic shells and not an issue in practice any more (Sven Mascheck's pages will have the details for the curious). Good to know (and I can take back my ouch ;-) Anyway, I do think that Eric's original proposal of extending the parallel-tests harness to allow arbitrary and portable fd redirections is the way to go eventually; here are my reasons: - it would be pretty easy to implement and document; - the testsuite harness is being revamped anyway; - that change would also benefit all the future custom and built-in test drivers; - the current hack with TESTS_ENVIRONMENTS = ...; 92, while continuing to work, will prevent the use of AM_TESTS_ENVIRONMENT :-( In each case, the user's namespace is invaded, and we go against our own advice (in [1] we violate the always terminate AM_TESTS_ENVIRONMENT with a semicolon rule, in [2] we violate don't define TESTS_ENVIRONMENT in the Makefile.am, it's user-reserved rule). Also, the example above belongs IMHO more in a FAQ rather than in a reference manual. Are you volunteering to update the FAQ? ;-) Oh no ;-) Not this summer at least. Regards, Stefano
Re: [FYI] {master} docs: fix unportable example of AM_TESTS_ENVIRONMENT usage
* Stefano Lattarini wrote on Thu, Jun 30, 2011 at 10:45:02PM CEST: On Thursday 30 June 2011, Ralf Wildenhues wrote: I'm fairly sure that's only with historic shells and not an issue in practice any more (Sven Mascheck's pages will have the details for the curious). Anyway, I do think that Eric's original proposal of extending the parallel-tests harness to allow arbitrary and portable fd redirections is the way to go eventually; Link? here are my reasons: - it would be pretty easy to implement and document; - the testsuite harness is being revamped anyway; - that change would also benefit all the future custom and built-in test drivers; - the current hack with TESTS_ENVIRONMENTS = ...; 92, while continuing to work, will prevent the use of AM_TESTS_ENVIRONMENT :-( In each case, the user's namespace is invaded, and we go against our own advice (in [1] we violate the always terminate AM_TESTS_ENVIRONMENT with a semicolon rule, in [2] we violate don't define TESTS_ENVIRONMENT in the Makefile.am, it's user-reserved rule). Also, the example above belongs IMHO more in a FAQ rather than in a reference manual. Are you volunteering to update the FAQ? ;-) Oh no ;-) Not this summer at least. That's a pretty strong statement, and when accompanied by a documentation regression (if I may exaggerate a bit), I'm not too happy when I read this. Please do not consider documentation work as second class citicen. Actually, writing (at least a sketch of) the documentation before implementing a feature is a very good way to get more organized and better design, IMVHO. Thanks, Ralf
Re: [MERGE REQUEST] merge patches for AM_TESTSUITE_ENVIRONMENT into maint
On Thursday 30 June 2011, Ralf Wildenhues wrote: * Ralf Wildenhues wrote on Thu, Jun 30, 2011 at 10:52:16PM CEST: * Stefano Lattarini wrote on Thu, Jun 30, 2011 at 05:12:38PM CEST: I think that the patches introducing support for AM_TESTSUITE_ENVIRONMENT should be merged into maint, as they'd make a nice addition for 1.11.2. I've verified that almost all these patches are based off of maint, and the few that aren't (e.g., `v1.11-867-g1e005df') I don't have v1.11-867-g1e005df nor v1.11-351-g3e334a2 in my tree. D'oh. EWRONGREPO. I still don't have the latter commit in my tree, Weird. The commit is present in the official automake repo: $ hash=`git log -1 --format=%H v1.11-351-g3e334a2` $ wget --server-response -O/dev/null http://git.savannah.gnu.org/gitweb/?p=automake.git;a=commit;h=$hash; --2011-06-30 23:17:14-- http://git.savannah.gnu.org/gitweb/?p=automake.git;a=commit;h=3e334a272fa601bebb5896e25cfb63f34822a275 Resolving git.savannah.gnu.org... 140.186.70.72 Connecting to git.savannah.gnu.org|140.186.70.72|:80... connected. HTTP request sent, awaiting response... HTTP/1.1 200 OK Date: Thu, 30 Jun 2011 21:17:14 GMT ... 2011-06-30 23:17:15 (70.4 KB/s) - “/dev/null” saved [7893] and the former has many many commits on top of maint. Yes, because it's based off of master. As I've said, it shouldn't be merged in maint, at most cherry-picked; and I've also question whether that would be worthwhile (see my quoted answer below). So no, you cannot merge that into maint. That would require more than just a cursory review, and I don't see the gain right now. Is this still true in light of the above? are just testsuite additions (and could probably be cherry-picked anyway, but I'm not sure whther that would be worthwhile). My proposal is to push the last relevant commit (should be `v1.11-351-g3e334a2') to a new public branch 'am-tests-environment', tie up the last loose ends (if any), and finally merge that branch into maint. Objections? Sorry, Ralf Regards, Stefano
improve synchronization between examples in the manual and test cases
Severity: wishlist thanks Reference: http://lists.gnu.org/archive/html/automake-patches/2011-06/msg00220.html On Tuesday 28 June 2011, Ralf Wildenhues wrote: * Stefano Lattarini wrote on Tue, Jun 28, 2011 at 08:24:13AM CEST: On Monday 27 June 2011, Ralf Wildenhues wrote: Sure. Well, do they use some format already? Well, there are only two of them, and both follows this format: @c The test case for the setup described here is @c test/subdircond2.test @c Try to keep it in sync. (which is wrongish BTW, as the testsuite subdirectory is `tests/', not `test/') I'd go with one of these simple formats: @c Keep in sync with subdircond2.test I like this one (with a trailing dot ;-) ;-) I'll soon post a patch that improves the synchronization between some examples from the manual and some test cases. In the process of writing it, I've noticed that various non-trivial examples from the manual are not tested; and while I've remedied to part of this problem with my patch, a lot remains to be done; so I'm opening this bug to avoid forgetting about the issue. Regards, Stefano
Re: bug#8969: improve synchronization between examples in the manual and test cases
On Friday 01 July 2011, Stefano Lattarini wrote: Severity: wishlist thanks Reference: http://lists.gnu.org/archive/html/automake-patches/2011-06/msg00220.html On Tuesday 28 June 2011, Ralf Wildenhues wrote: * Stefano Lattarini wrote on Tue, Jun 28, 2011 at 08:24:13AM CEST: On Monday 27 June 2011, Ralf Wildenhues wrote: Sure. Well, do they use some format already? Well, there are only two of them, and both follows this format: @c The test case for the setup described here is @c test/subdircond2.test @c Try to keep it in sync. (which is wrongish BTW, as the testsuite subdirectory is `tests/', not `test/') I'd go with one of these simple formats: @c Keep in sync with subdircond2.test I like this one (with a trailing dot ;-) ;-) I'll soon post a patch that improves the synchronization between some examples from the manual and some test cases. In the process of writing it, I've noticed that various non-trivial examples from the manual are not tested; and while I've remedied to part of this problem with my patch, a lot remains to be done; so I'm opening this bug to avoid forgetting about the issue. Regards, Stefano And here it is the promised patch. I will push in 72 hours if there is no objection by then. Regards, Stefano From 2dd820b423bea12eac7b94fd8ed2563ee21cf9e1 Mon Sep 17 00:00:00 2001 Message-Id: 2dd820b423bea12eac7b94fd8ed2563ee21cf9e1.1309473910.git.stefano.lattar...@gmail.com From: Stefano Lattarini stefano.lattar...@gmail.com Date: Thu, 30 Jun 2011 20:05:25 +0200 Subject: [PATCH] docs, tests: synchronize examples from docs to tests * tests/README (Writing test cases): Give suggestions on how to keep test cases and examples in the documentation synchronized. * doc/automake.texi: Improve or fix existing testcase-referencing comments, and add many new ones. * HACKING (Administrivia): Suggest to test complex examples and idioms from the manual. * tests/specflg8.test: Improve synchronization with the example in the manual. * tests/output11.test:Likewise. * tests/txinfo21.test:Likewise. * tests/interp.test: Likewise. Since we are at it, and enable the `errexit' shell flag, do related changes, and add trailing `:'command. * tests/amhello-cflags.test: New test. * tests/amhello-cross-compile.test: Likewise. * tests/amhello-binpkg.test: Likewise. * tests/tests-environment-backcompat: Likewise. * tests/parallel-tests-log-compiler-example.test: Likewise. * tests/Makefile.am (TESTS): Update. --- ChangeLog | 23 HACKING|7 ++- doc/automake.texi | 54 ++ tests/Makefile.am |5 ++ tests/Makefile.in |5 ++ tests/README |7 ++ tests/amhello-binpkg.test | 44 +++ tests/amhello-cflags.test | 49 tests/amhello-cross-compile.test | 54 ++ tests/interp.test | 18 -- tests/parallel-tests-log-compiler-example.test | 71 tests/specflg8.test| 35 ++-- tests/tests-environment-backcompat.test| 65 ++ tests/txinfo21.test|4 +- 14 files changed, 402 insertions(+), 39 deletions(-) create mode 100755 tests/amhello-binpkg.test create mode 100755 tests/amhello-cflags.test create mode 100755 tests/amhello-cross-compile.test create mode 100755 tests/parallel-tests-log-compiler-example.test create mode 100755 tests/tests-environment-backcompat.test diff --git a/ChangeLog b/ChangeLog index f1e10c2..c77326d 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,26 @@ +2011-07-01 Stefano Lattarini stefano.lattar...@gmail.com + + docs, tests: synchronize examples from docs to tests + * tests/README (Writing test cases): Give suggestions on how to + keep test cases and examples in the documentation synchronized. + * doc/automake.texi: Improve or fix existing testcase-referencing + comments, and add many new ones. + * HACKING (Administrivia): Suggest to test complex examples and + idioms from the manual. + * tests/specflg8.test: Improve synchronization with the example + in the manual. + * tests/output11.test:Likewise. + * tests/txinfo21.test:Likewise. + * tests/interp.test: Likewise. Since we are at it, and enable + the `errexit' shell flag, do related changes, and add trailing + `:'command. + * tests/amhello-cflags.test: New test. + * tests/amhello-cross-compile.test: Likewise. + * tests/amhello-binpkg.test: Likewise. + * tests/tests-environment-backcompat: Likewise. + * tests/parallel-tests-log-compiler-example.test: Likewise. + * tests/Makefile.am (TESTS): Update. + 2011-06-23 Stefano Lattarini stefano.lattar...@gmail.com docs:
Re: [PATCH] {master} coverage: new test on parallel-tests TESTS runtime overriding
Hi Ralf, thanks for the lightspeed review.
On Thursday 30 June 2011, Ralf Wildenhues wrote:
* Stefano Lattarini wrote on Thu, Jun 30, 2011 at 04:40:51PM CEST:
I'd like to add this new test case to master. OK?
Sure, note typo below.
This could actually find bugs in older makes, but hey, we'd want to know
about them.
Thanks,
Ralf
Subject: [PATCH] coverage: new test on parallel-tests TESTS runtime
overriding
* tests/parallel-tests-cmdline-override.test: New test, check that
we can use indirections when overriding TESTS and TEST_LOGS from
the command line.
* tests/Makefile.am (TESTS): Update.
--- /dev/null
+++ b/tests/parallel-tests-cmdline-override.test
+# Check that we can use indirections when overriding TESTS and
+# TEST_LOGS from the command line.
+
+parallel_tests=yes
+. ./defs || Exit 1
+
+cat configure.in 'END'
+AC_OUTPUT
+END
+
+cat Makefile.am 'END'
+TEST_EXTENSIONS = .test .t
+TEST_LOG_COMPILER = cat
+T_LOG_COMPILER = cat
+TESTS = bad.test
+var1 = b.test $(var2)
+var2 = c.test
+var3 = d.d
+var4 = e
+END
+
+$ACLOCAL
+$AUTOCONF
+$AUTOMAKE -a
+
+./configure
+rm -f config.log # Not to create false psotives below.
positives
Oops (I've probably disabled my spellchecker by mistake *again*).
and s/Not to/Do not/ I guess.
Strictly speaking, the Not to was meant, standing as an abbreviation
for: (This is needed) not to create Admittedly, not a great
example of clarity; thus I've done your suggested change.
+
+LC_ALL=C sort exp-log 'END'
+a.log
+b.log
+c.log
+d.log
+e.log
+test-suite.log
+END
+
+LC_ALL=C sort exp-out 'END'
+PASS: a.t
+PASS: b.test
+PASS: c.test
+PASS: d.t
+PASS: e.test
+END
+
Also, the two uses of sort above are redundant: cat is enough. I've
adjusted them.
+do_check ()
+{
+ env $@ $MAKE -e check stdout || { cat stdout; Exit 1; }
+ cat stdout
+ grep '^PASS:' stdout | LC_ALL=C sort got-out
+ cat got-out
+ ls . | grep '\.log$' | LC_ALL=C sort got-log
+ cat got-log
+ st=0
+ diff exp-out got-out || st=1
+ diff exp-log got-log || st=1
+ return $st
+}
+
+tests='a.t $(var1) $(var3:.d=.t) $(var4:=.test)'
+test_logs='a.log $(var1:.test=.log) $(var3:.d=.log) $(var4:=.log)'
+
+touch a.t b.test c.test d.t e.test
+
+do_check TESTS=$tests
+do_check TEST_LOGS=$test_logs
Below is what I've squashed in. Pushed now.
Regards,
Stefano
-*-*-*-
diff --git a/tests/parallel-tests-cmdline-override.test
b/tests/parallel-tests-cmdline-override.test
index cd8f82f..fa47032 100755
--- a/tests/parallel-tests-cmdline-override.test
+++ b/tests/parallel-tests-cmdline-override.test
@@ -40,9 +40,9 @@ $AUTOCONF
$AUTOMAKE -a
./configure
-rm -f config.log # Not to create false psotives below.
+rm -f config.log # Do not create false positives below.
-LC_ALL=C sort exp-log 'END'
+cat exp-log 'END'
a.log
b.log
c.log
@@ -51,7 +51,7 @@ e.log
test-suite.log
END
-LC_ALL=C sort exp-out 'END'
+cat exp-out 'END'
PASS: a.t
PASS: b.test
PASS: c.test
Re: libtool relink problem
On Wed, Jun 29, 2011 at 05:14:19PM +0200, Emmanuel Engelhart wrote:
From: Pippijn van Steenhovenpip8...@gmail.com
Date: Mon, Jun 20, 2011 at 4:55 PM
Subject: Re: libtool relink problem
To: automake@gnu.org
On Sun, Jun 19, 2011 at 03:40:06PM +0530, Santhosh Thottingal wrote:
Hi,
I am facing a problem with libtool. While installing it complains a
relink required and while doing so it tries to link against the
shared library instead of static library.
Well, the problem is better explained in this thread
http://sourceware.org/ml/automake/2004-07/msg00127.html with example
code.
I am facing exactly the same problem.
I saw a similar problem in this thread also
http://www.mail-archive.com/libtool@gnu.org/msg00782.html
Is there a solution available for this?
Thanks
Santhosh
Hi Santhosh,
I suggest writing your *_LTLIBRARIES with a comment in front, perhaps
with # to differ between disabled and enabled libraries. Then using
grep(1) to extract the names and the same tools to extract the
dependencies. You can then use tsort(1) to topologically sort the
libraries and write the actual ${dir}_LTLIBRARIES to an included .am
file. Semantically, this is what I do. In reality, I wrote a preprocessor
for automake that does this, but I don't suggest using it, as it's beta
and very much tailored to my needs.
Hi Pippijn
Thank you for taking time and trying to help us... because we really get
stucked.
I tried to understand your answer but I honestly think I'm not good
enough with automake to be able to understand and apply your
methodology. I also doubt that the other cases given by Santhosh are the
same as our, although the symptom is.
From my understanding the previously described issues occur if the
dependences are not compiled in the right order. I think this is not our
problem because at the moment the error occurs, I know the dependence is
already compiled.
The issue occurs if the dependencies are not _installed_ in the right
order.
So, I'm sure the file $(top_builddir)/src/ctpp2/src/libctpp2.la exists.
As you noted, the dependency has been compiled and linked, but for
installation, it has to be relinked:
libtool: relink: g++ -shared -nostdlib
^^
The above occurs in make install.
So my question is why automake replaces the path with libcttp2.la in
-lctpp2, replacing a static linking with a dynamic one?
It's related to rpath linking. The resulting binaries are relinked to
include $(libdir) as rpath, so that installed binaries can be called and
libraries loaded without listing $(libdir) in LD_LIBRARY_PATH or
ld.so.conf.
It looks like you are using a recursive make, in your project. Are you
sure that libcttp2.la exists in $(libdir)? As an experiment, try make
install in libctpp2's $(builddir) and then make install in
$(top_builddir).
Regards,
--
Pippijn van Steenhoven
signature.asc
Description: Digital signature
