Re: [PATCH v4 3/3] parallel-tests: allow each test to have multiple results
* Stefano Lattarini wrote on Fri, Jun 17, 2011 at 09:33:13AM CEST: On Friday 17 June 2011, Ralf Wildenhues wrote: Actually, why not s/testcase/test/g globally in all your text. Because I'm trying to make a distinction between test scripts and test cases. With the new interface, a single test script can contain multiple testcases, and thus have multiple test results. It is a simple concept, but not immediate to get for someone who was used the older Automake tests drivers (which obeyed the principle one test script, one test result, so that you could call a test script also a testcase, or simply a test, without risk of confusion. As I said, use the name other testsuite environments use. :test-result: seems better. (EXTRA_DIST): Distribute `ostp-driver'. (test-driver-custom-multitest.log): Depend on `ostp-driver'. (test-driver-custom-multitest-recheck.log): Likewise. (test-driver-custom-multitest-recheck2.log): Likewise. (test-driver-custom-html.log): Likewise. * doc/automake.texi (API for Custom Test Drivers): Update (still in Texinfo comments only). I don't like TODO and comments stuff in finished patches documentation if we can help it (and here, we surely can). While I mostly agree, I'd like to point out that I had already rewritten and revesited the documentation several times, and after some point you risk more harm than good in tweaking and re-shaping the same text again; also, you're somewhat fed up with it, so that the risk of becoming sloppy or of producing confused text increases. I'll fix the TODOs in this patch if you insist, but honestly I'd rather to that in a follow-up patch if possible. Why not just split the whole documentation change into a followup patch then? @b{TODO!} @i{Options and flags that the driver must handle. Generation -of ``.log'' files. Console output the driver is expected to produce. -Support for colored output, XFAIL_TESTS, and DISABLE_HARD_ERRORS} +of ``.log'' files, and format they must obey. Console output the driver @file{.log} You should almost never need to use ``...'' in texinfo, at least not for any entity that has a meaning in code. Note that this hunk is just temporary, and bounded to be removed by follow-up patches that will (or would have) completed the TODOs. See above. +@cexecutes two test cases, one successful and one failing, and skip +@canother test case, the driver should end up writing the following +@cin the test log: +@c :am-testcase-result: PASS [passed testcase name or details] +@c :am-testcase-result: FAIL [failed testcase name or details] +@c :am-testcase-result: SKIP [skipped testcase name or details] is the [...] literal or metasyntactic? Metasyntactic. Please use proper notation here. What would that be? Sorry if it's a stupid question, but my grasp of Texinfo is still pretty limited. @example would seem appropriate. +@cThe above lines (each of which *must* be followed by a blank line +@cin order for the HTML output generation to work) are used only +@cwhen generating the `test-suite.log' from the individual test See above. About what exactly? @file{test-suite.log} + results2=`sed -n s/^$$rst_magic[ ]*//p $$list2`; \ tab space That's deliberate, we don't want to be too strict in parsing user's output (or in this case, third-party drivers output). Sorry. I just meant the tab should come before the space, not after. My editor highlights the space otherwise (which, granted, is a limitation in the highlighting rule, but also helps prevent other editors, human or programmatic, to mangle the code in an unwanted way). +$@ 21 | tee $tmp_out | ( + i=0 st=0 + : $tmp_res + while read line; do +case $line in + PASS:*|FAIL:*|XPASS:*|XFAIL:*|SKIP:*) +i=`expr $i + 1` +result=`LC_ALL=C expr $line : '\([A-Z]*\):.*'` This will be quite fork expensive, if done in real-world code. But this is in a script used only for testing. I don't think it's worth optimizing it. No, it's not, but your real scripts won't look all that different. Besides, why not do it right the first time? Might just want to use one single sed script replacing this while while loop. But then again, measuring is king. And BTW, premature optimization is the root of all evil ;-) True. If we'd do it right the first time then there would be no need to fix efficiency regressions afterwards. I don't actually care much about micro optimizations like the above at this point, but I do care when the whole set of code changes will introduce a factor of 2 slowdown in the test suite overhead. It looks like it may eventually, judging from your measurements done, and that's what I am trying to prevent. On w32, that would cause real pain. I agree that we should improve my code for speed later (after all, we expect it to be
Re: [PATCH] {maint} Deprecate de-ansification support, it should go away in automake 1.12
On Thursday 16 June 2011, Stefano Lattarini wrote: On Thursday 16 June 2011, Stefano Lattarini wrote: Hi Bob. On Thursday 16 June 2011, Bob Friesenhahn wrote: On Wed, 15 Jun 2011, Stefano Lattarini wrote: Problem: it seems that there are few (and luckily only few) active packages that still use the `ansi2knr' option: 1. The latest version (8c) of the libjpeg implementation from IJG (Independent JPEG Group, http://www.ijg.org/); see the `configure.ac' file from the archive http://www.ijg.org/files/jpegsrc.v8c.tar.gz I created most of the libjpeg build configury. The official maintainer (Guido Vollbeding) added the extra AM_INIT_AUTOMAKE options. So, maybe it would be worthwhile to directly inform the maintainers of the packages mentioned above that still use de-ANSI-fication feature about its oncoming deprecation and removal. WDYT? If they do not hear about it, I am sure that they will learn once the option stops working. OK, then I'll save myself the trouble. Thanks, Stefano Well, we've informed those projects anyway in the end; hope that it'll be helpful. Yes, it has been: the maintainers of all those projects have answered (thanks!) and agreed to remove the use of 'ansi2knr'. I've thus pushed the patch. Regards, Stefano
Re: [PATCH] {maint} distcheck: add support for AM_DISTCHECK_CONFIGURE_FLAGS
On Saturday 18 June 2011, Stefano Lattarini wrote: On Wednesday 15 June 2011, Eric Blake wrote: [CUT] Here is the amended patch. Note that in the end I haven't made use of the version information I've asked Eric previously, since adding it would have broken the flow of discourse IMHO. I'll push by tomorrow evening if there is no objection. [CUT] Pushed now. Regards, Stefano
Re: [PATCH] check: don't use multi-line coloring for the report
On Friday 17 June 2011, Stefano Lattarini wrote: Hi Bert, and thanks again for your patch. I have some minor observations and objections below (please do not take them as a belittling of your work; they are either constructive criticism, or requests for cosmetic changes mandated by the GNU coding standards). I hope you have time and will to address them; if not, don't worry, I can do that for you. OK, I've amended the patch on Bert's behalf, as he can't do that himself at the moment. Attached is what I've pushed (to maint). Regards, Stefano From b8c2b69913b652bcfd4665b041f11d8b5316da2b Mon Sep 17 00:00:00 2001 Message-Id: b8c2b69913b652bcfd4665b041f11d8b5316da2b.1308558605.git.stefano.lattar...@gmail.com From: Bert Wesarg bert.wes...@googlemail.com Date: Fri, 17 Jun 2011 21:59:52 +0200 Subject: [PATCH] check: don't use multi-line coloring for the report less -R can't handle multi-line coloring as it is done for the check reports of the serial and parallel testsuite, because of performance reasons. Thus, color each line of the check report by its own. * lib/am/check.am (am__text_box): Accept colors for lines, and color each line by its own. [%?PARALLEL_TESTS%] $(TEST_SUITE_LOG): Let am__text_box handle the line coloring. [!%?PARALLEL_TESTS%] $(check-TESTS): Color each report line by its own. * THANKS: Update. --- ChangeLog | 15 + THANKS |1 + lib/Automake/tests/Makefile.in | 31 +++--- lib/am/check.am| 46 +++ tests/Makefile.in | 31 +++--- 5 files changed, 79 insertions(+), 45 deletions(-) diff --git a/ChangeLog b/ChangeLog index c5652da..d79a848 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,18 @@ +2011-06-20 Bert Wesarg bert.wes...@googlemail.com (tiny change) + + check: don't use multi-line coloring for the report + less -R can't handle multi-line coloring as it is done for the + check reports of the serial and parallel testsuite, because of + performance reasons. Thus, color each line of the check report + by its own. + * lib/am/check.am (am__text_box): Accept colors for lines, and + color each line by its own. + [%?PARALLEL_TESTS%] $(TEST_SUITE_LOG): Let am__text_box handle + the line coloring. + [!%?PARALLEL_TESTS%] $(check-TESTS): Color each report line by + its own. + * THANKS: Update. + 2011-06-18 Stefano Lattarini stefano.lattar...@gmail.com docs: AM_DISTCHECK_CONFIGURE_FLAGS is for corner cases diff --git a/THANKS b/THANKS index 16a1ef8..3d71419 100644 --- a/THANKS +++ b/THANKS @@ -37,6 +37,7 @@ Benoit Sigoure ts...@lrde.epita.fr Bernard Giroud bernard.gir...@creditlyonnais.ch Bernard Urban bernard.ur...@meteo.fr Bernd Jendrissek berndfoo...@users.sourceforge.net +Bert Wesarg bert.wes...@googlemail.com Bill Currie bcur...@tssc.co.nz Bill Davidson b...@kayhay.com Bill Fenner fen...@parc.xerox.com diff --git a/lib/Automake/tests/Makefile.in b/lib/Automake/tests/Makefile.in index 7ed17d6..2f553ed 100644 --- a/lib/Automake/tests/Makefile.in +++ b/lib/Automake/tests/Makefile.in @@ -105,15 +105,21 @@ am__base_list = \ am__rst_title = sed 's/.*/ /;h;s/./=/g;p;x;p;g;p;s/.*//' am__rst_section = sed 'p;s/./=/g;p;g' # Put stdin (possibly several lines separated by . ) in a box. -am__text_box = $(AWK) '{\ - n = split($$0, lines, \\. ); max = 0; \ - for (i = 1; i = n; ++i)\ -if (max length(lines[i]))\ - max = length(lines[i]);\ - for (i = 0; i max; ++i) line = line =; \ - print line; \ - for (i = 1; i = n; ++i) if (lines[i]) print lines[i];\ - print line; \ +# Prefix each line by 'col' and terminate each with 'std', for coloring. +# Multi line coloring is problematic with less -R, so we really need +# to color each line individually. +am__text_box = $(AWK) '{ \ + n = split($$0, lines, \\. ); max = 0; \ + for (i = 1; i = n; ++i) \ +if (max length(lines[i])) \ + max = length(lines[i]); \ + for (i = 0; i max; ++i) \ +line = line =;\ + print col line std;\ + for (i = 1; i = n; ++i) \ +if (lines[i])\ + print col lines[i] std; \ + print col line std;\ }' # Solaris 10 'make', and several other traditional 'make' implementations, # pass -e to $(SHELL), and POSIX 2008 even requires this. Work around it @@ -394,12 +400,11 @@ $(TEST_SUITE_LOG): $(TEST_LOGS) test x$$VERBOSE = x || $$exit || cat $(TEST_SUITE_LOG); \ $(am__tty_colors); \ if $$exit; then \ - echo $(ECHO_N) $$grn$(ECHO_C);\ + col=$$grn; \ else\ - echo $(ECHO_N) $$red$(ECHO_C);\ + col=$$red; \ fi;\ - echo $$msg | $(am__text_box); \ - echo $(ECHO_N) $$std$(ECHO_C);\ + echo $$msg | $(am__text_box) col=$$col std=$$std; \ $$exit # Run all the tests. diff --git a/lib/am/check.am b/lib/am/check.am index 4d10ce9..0b54312 100644 --- a/lib/am/check.am
[FYI] {maint} maintcheck: avoid few spurious failures
* Makefile.am (sc_tests_plain_aclocal, sc_tests_plain_perl, sc_tests_plain_autoconf, sc_tests_plain_automake, sc_tests_plain_autoupate): Be stricter in matching an erroneous literal command, i.e., `aclocal', `automake', `perl', etc. --- ChangeLog |8 Makefile.am | 10 +- Makefile.in | 10 +- 3 files changed, 18 insertions(+), 10 deletions(-) diff --git a/ChangeLog b/ChangeLog index d79a848..82aac1e 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,11 @@ +2011-06-20 Stefano Lattarini stefano.lattar...@gmail.com + + maintcheck: avoid few spurious failures + * Makefile.am (sc_tests_plain_aclocal, sc_tests_plain_perl, + sc_tests_plain_autoconf, sc_tests_plain_automake, + sc_tests_plain_autoupate): Be stricter in matching an erroneous + literal command, i.e., `aclocal', `automake', `perl', etc. + 2011-06-20 Bert Wesarg bert.wes...@googlemail.com (tiny change) check: don't use multi-line coloring for the report diff --git a/Makefile.am b/Makefile.am index b25a30a..9277986 100644 --- a/Makefile.am +++ b/Makefile.am @@ -366,21 +366,21 @@ sc_tests_plain_make: ## Tests should never call autoconf directly. sc_tests_plain_autoconf: - @if grep -v '^#' $(srcdir)/tests/*.test | grep ':[ ]*autoconf'; then \ + @if grep -v '^#' $(srcdir)/tests/*.test | grep ':[ ]*autoconf\'; then \ echo 'Do not run autoconf in the above tests. Use $$AUTOCONF instead.' 12; \ exit 1; \ fi ## Tests should never call autoupdate directly. sc_tests_plain_autoupdate: - @if grep -v '^#' $(srcdir)/tests/*.test | grep ':[ ]*autoupdate'; then \ + @if grep -v '^#' $(srcdir)/tests/*.test | grep ':[ ]*autoupdate\'; then \ echo 'Do not run autoupdate in the above tests. Use $$AUTOUPDATE instead.' 12; \ exit 1; \ fi ## Tests should never call automake directly. sc_tests_plain_automake: - @if grep -v '^#' $(srcdir)/tests/*.test | grep -E ':[ ]*automake([^:]|$$)'; then \ + @if grep -v '^#' $(srcdir)/tests/*.test | grep -E ':[ ]*automake\([^:]|$$)'; then \ echo 'Do not run automake in the above tests. Use $$AUTOMAKE instead.' 12; \ exit 1; \ fi @@ -420,14 +420,14 @@ sc_tests_automake_fails: ## Tests should never call aclocal directly. sc_tests_plain_aclocal: - @if grep -v '^#' $(srcdir)/tests/*.test | grep ':[ ]*aclocal'; then \ + @if grep -v '^#' $(srcdir)/tests/*.test | grep ':[ ]*aclocal\'; then \ echo 'Do not run aclocal in the above tests. Use $$ACLOCAL instead.' 12; \ exit 1; \ fi ## Tests should never call perl directly. sc_tests_plain_perl: - @if grep -v '^#' $(srcdir)/tests/*.test | grep ':[ ]*perl'; then \ + @if grep -v '^#' $(srcdir)/tests/*.test | grep ':[ ]*perl\'; then \ echo 'Do not run perl in the above tests. Use $$PERL instead.' 12; \ exit 1; \ fi diff --git a/Makefile.in b/Makefile.in index 6539f3c..10d57b9 100644 --- a/Makefile.in +++ b/Makefile.in @@ -1071,19 +1071,19 @@ sc_tests_plain_make: fi sc_tests_plain_autoconf: - @if grep -v '^#' $(srcdir)/tests/*.test | grep ':[ ]*autoconf'; then \ + @if grep -v '^#' $(srcdir)/tests/*.test | grep ':[ ]*autoconf\'; then \ echo 'Do not run autoconf in the above tests. Use $$AUTOCONF instead.' 12; \ exit 1; \ fi sc_tests_plain_autoupdate: - @if grep -v '^#' $(srcdir)/tests/*.test | grep ':[ ]*autoupdate'; then \ + @if grep -v '^#' $(srcdir)/tests/*.test | grep ':[ ]*autoupdate\'; then \ echo 'Do not run autoupdate in the above tests. Use $$AUTOUPDATE instead.' 12; \ exit 1; \ fi sc_tests_plain_automake: - @if grep -v '^#' $(srcdir)/tests/*.test | grep -E ':[ ]*automake([^:]|$$)'; then \ + @if grep -v '^#' $(srcdir)/tests/*.test | grep -E ':[ ]*automake\([^:]|$$)'; then \ echo 'Do not run automake in the above tests. Use $$AUTOMAKE instead.' 12; \ exit 1; \ fi @@ -1116,13 +1116,13 @@ sc_tests_automake_fails: fi sc_tests_plain_aclocal: - @if grep -v '^#' $(srcdir)/tests/*.test | grep ':[ ]*aclocal'; then \ + @if grep -v '^#' $(srcdir)/tests/*.test | grep ':[ ]*aclocal\'; then \ echo 'Do not run aclocal in the above tests. Use $$ACLOCAL instead.' 12; \ exit 1; \ fi sc_tests_plain_perl: - @if grep -v '^#' $(srcdir)/tests/*.test | grep ':[ ]*perl'; then \ + @if grep -v '^#' $(srcdir)/tests/*.test | grep ':[ ]*perl\'; then \ echo 'Do not run perl in the above tests. Use $$PERL instead.' 12; \ exit 1; \ fi -- 1.7.2.3
[FYI] {testuite-work} maintcheck: avoid few spurious failures
* tests/depmod-data.test: Use creative quoting to avoid spuriously triggering the `sc_tests_Exit_not_exit' maintainer check. Commit `v1.11-900-g3453b8e' attempted to fix it, but succeeded only partially. * tests/cond33.test: Fix header comments, not to reference obsolescent make variable `$(mkdir_p)'. * tests/cond4.test: Rewrite $MAKE exp=... as exp=.. $MAKE -e, to please `sc_tests_overriding_macros_on_cmdline' * tests/cond19.test: Likewise. * tests/cond32.test: Likewise. * tests/add-missing.test: Use AUTOMAKE_fails instead of $AUTOMAKE ... Exit 1, to please `sc_tests_automake_fails'. --- ChangeLog | 16 tests/add-missing.test |3 +-- tests/cond19.test |8 tests/cond32.test |6 +++--- tests/cond33.test |4 ++-- tests/cond4.test |8 tests/depmod-data.test |2 +- 7 files changed, 31 insertions(+), 16 deletions(-) diff --git a/ChangeLog b/ChangeLog index 1a5c4aa..e1a774c 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,19 @@ +2011-06-20 Stefano Lattarini stefano.lattar...@gmail.com + + maintcheck: avoid few more spurious failures + * tests/depmod-data.test: Use creative quoting to avoid + spuriously triggering the `sc_tests_Exit_not_exit' maintainer + check. Commit `v1.11-900-g3453b8e' attempted to fix it, but + succeeded only partially. + * tests/cond33.test: Fix header comments, not to reference + obsolescent make variable `$(mkdir_p)'. + * tests/cond4.test: Rewrite $MAKE exp=... as exp=.. $MAKE -e, + to please `sc_tests_overriding_macros_on_cmdline' + * tests/cond19.test: Likewise. + * tests/cond32.test: Likewise. + * tests/add-missing.test: Use AUTOMAKE_fails instead of + $AUTOMAKE ... Exit 1, to please `sc_tests_automake_fails'. + 2011-06-20 Bert Wesarg bert.wes...@googlemail.com (tiny change) check: don't use multi-line coloring for the report diff --git a/tests/add-missing.test b/tests/add-missing.test index d6729f3..7e3d08c 100755 --- a/tests/add-missing.test +++ b/tests/add-missing.test @@ -141,8 +141,7 @@ check_ () # If the required auxiliary files are missing, and automake is # not told to install them, it should complain and error out, # and also give a useful suggestion. -$AUTOMAKE 2stderr { cat stderr 2; Exit 1; } -cat stderr 2 +AUTOMAKE_fails for f in $files; do grep required file ['\`]$build_aux/$f' not found stderr # Suggest the user to use `--add-missing'. diff --git a/tests/cond19.test b/tests/cond19.test index 50ae9cb..ec69bfb 100755 --- a/tests/cond19.test +++ b/tests/cond19.test @@ -60,12 +60,12 @@ $AUTOCONF $AUTOMAKE -a -i CONDITION1=true CONDITION2=true ./configure -$MAKE test exp='dlmaina1.c dlmainb2.c dlmaina1.o dlmainb2.o' +exp='dlmaina1.c dlmainb2.c dlmaina1.o dlmainb2.o' $MAKE -e test CONDITION1=true CONDITION2=false ./configure -$MAKE test exp='dlmainb1.c dlmaina2.c dlmainb1.o dlmaina2.o' +exp='dlmainb1.c dlmaina2.c dlmainb1.o dlmaina2.o' $MAKE -e test CONDITION1=false CONDITION2=true ./configure -$MAKE test exp='dlmaina1.c dlmainb2.c dlmaina1.o dlmainb2.o' +exp='dlmaina1.c dlmainb2.c dlmaina1.o dlmainb2.o' $MAKE -e test CONDITION1=false CONDITION2=false ./configure -$MAKE test exp='dlmainb1.c dlmaina2.c dlmainb1.o dlmaina2.o' +exp='dlmainb1.c dlmaina2.c dlmainb1.o dlmaina2.o' $MAKE -e test : diff --git a/tests/cond32.test b/tests/cond32.test index 30fde11..45ecf70 100755 --- a/tests/cond32.test +++ b/tests/cond32.test @@ -61,12 +61,12 @@ $AUTOCONF $AUTOMAKE ./configure -$MAKE test exp='foo.o nonsense.a' +exp='foo.o nonsense.a' $MAKE -e test ./configure two=yes three= -$MAKE test exp='bar.o' +exp='bar.o' $MAKE -e test ./configure two=yes three=yes -$MAKE test exp='baz.o' +exp='baz.o' $MAKE -e test : diff --git a/tests/cond33.test b/tests/cond33.test index 149795d..5f454db 100755 --- a/tests/cond33.test +++ b/tests/cond33.test @@ -14,8 +14,8 @@ # You should have received a copy of the GNU General Public License # along with this program. If not, see http://www.gnu.org/licenses/. -# Check that $(mkdir_p) handles well conditionally-defined install -# directories. +# Check that conditionally-defined install directories are handled +# correctly. # Report from Ralf Corsepius . ./defs || Exit 1 diff --git a/tests/cond4.test b/tests/cond4.test index 7d2aaea..3071577 100755 --- a/tests/cond4.test +++ b/tests/cond4.test @@ -57,12 +57,12 @@ grep '^@TWO_FALSE@' Makefile.in Exit 1 $AUTOCONF CONDITION1=true CONDITION2=true ./configure -$MAKE test exp='main.o one.o two.o' +exp='main.o one.o two.o' $MAKE -e test CONDITION1=true CONDITION2=false ./configure -$MAKE test exp='main.o one.o' +exp='main.o one.o' $MAKE -e test CONDITION1=false CONDITION2=true ./configure -$MAKE test exp='main.o two.o' +exp='main.o two.o' $MAKE -e test CONDITION1=false CONDITION2=false ./configure -$MAKE test exp='main.o' +exp='main.o' $MAKE
Re: [PATCH 1/2] tests: make test runner a script, not a shell function
* Stefano Lattarini wrote on Mon, Jun 20, 2011 at 10:22:28AM CEST: [Adding bug-grep, dropping bug-coreutils and automake-patches] re-adding the latter. I've noticed that grep uses a definition of TESTS_ENVIRONMENT very similar to that of coreutils (the one we've just fixed), so it will break with oncoming automake too. That may be a sign that you may not want to actually break this code with your proposed changes to Automake. Put another way: it's a good idea to estimate the level of breakage you're going to burden upon others (a couple of projects, dozens, hundreds), the amount of work needed on their side to fix it, and the amount of work (or possibility) it would take to change your code so they are not broken in the first place. Also, of course, NEWS entries (and probably automake.texi entries) for such changes are a good idea. One thing I've regularly done with new code that is not 100% backward compatible is have a new Automake option for them. That is exactly why there is a 'parallel-tests': it is not fully compatible with the simple test driver, and requires work on behalf of developers using Automake. Unfortunately, I don't have an FSF assignment for grep, so someone else should fix that issue :-( You can do minor fixes without assignments. (Of course, I don't maintain grep, so it's up to them.) Cheers, Ralf
Re: [PATCH v4 3/3] parallel-tests: allow each test to have multiple results
On Monday 20 June 2011, Ralf Wildenhues wrote: * Stefano Lattarini wrote on Fri, Jun 17, 2011 at 09:33:13AM CEST: On Friday 17 June 2011, Ralf Wildenhues wrote: Actually, why not s/testcase/test/g globally in all your text. Because I'm trying to make a distinction between test scripts and test cases. With the new interface, a single test script can contain multiple testcases, and thus have multiple test results. It is a simple concept, but not immediate to get for someone who was used the older Automake tests drivers (which obeyed the principle one test script, one test result, so that you could call a test script also a testcase, or simply a test, without risk of confusion. As I said, use the name other testsuite environments use. :test-result: seems better. That's what I had already done in my previous squash-in. In the attached squash-in, I've removed few more uses of the test case terminology, when the use the simpler term test was just as clear. I guess that a follow up will be required where we remove ambiguous or unclear uses of the test, test case and test script terms from the manual. (EXTRA_DIST): Distribute `ostp-driver'. (test-driver-custom-multitest.log): Depend on `ostp-driver'. (test-driver-custom-multitest-recheck.log): Likewise. (test-driver-custom-multitest-recheck2.log): Likewise. (test-driver-custom-html.log): Likewise. * doc/automake.texi (API for Custom Test Drivers): Update (still in Texinfo comments only). I don't like TODO and comments stuff in finished patches documentation if we can help it (and here, we surely can). While I mostly agree, I'd like to point out that I had already rewritten and revesited the documentation several times, and after some point you risk more harm than good in tweaking and re-shaping the same text again; also, you're somewhat fed up with it, so that the risk of becoming sloppy or of producing confused text increases. I'll fix the TODOs in this patch if you insist, but honestly I'd rather to that in a follow-up patch if possible. Why not just split the whole documentation change into a followup patch then? Because that would only postpone, not avoid, the continous tweaking and amending the documentation; what I'd like instead is to improve it organically and incrementally, in separate patches; i.e., commit a sketchy but correct (even if incomplete) documentation first, and then improve it with follow-ups (maybe handling one concept or one part at the time). @b{TODO!} @i{Options and flags that the driver must handle. Generation -of ``.log'' files. Console output the driver is expected to produce. -Support for colored output, XFAIL_TESTS, and DISABLE_HARD_ERRORS} +of ``.log'' files, and format they must obey. Console output the driver @file{.log} You should almost never need to use ``...'' in texinfo, at least not for any entity that has a meaning in code. Note that this hunk is just temporary, and bounded to be removed by follow-up patches that will (or would have) completed the TODOs. See above. +@cexecutes two test cases, one successful and one failing, and skip +@canother test case, the driver should end up writing the following +@cin the test log: +@c :am-testcase-result: PASS [passed testcase name or details] +@c :am-testcase-result: FAIL [failed testcase name or details] +@c :am-testcase-result: SKIP [skipped testcase name or details] is the [...] literal or metasyntactic? Metasyntactic. Please use proper notation here. What would that be? Sorry if it's a stupid question, but my grasp of Texinfo is still pretty limited. @example would seem appropriate. +@cThe above lines (each of which *must* be followed by a blank line +@cin order for the HTML output generation to work) are used only +@cwhen generating the `test-suite.log' from the individual test See above. About what exactly? @file{test-suite.log} + results2=`sed -n s/^$$rst_magic[ ]*//p $$list2`; \ tab space That's deliberate, we don't want to be too strict in parsing user's output (or in this case, third-party drivers output). Sorry. I just meant the tab should come before the space, not after. My editor highlights the space otherwise (which, granted, is a limitation in the highlighting rule, but also helps prevent other editors, human or programmatic, to mangle the code in an unwanted way). OK, fixed. +$@ 21 | tee $tmp_out | ( + i=0 st=0 + : $tmp_res + while read line; do +case $line in + PASS:*|FAIL:*|XPASS:*|XFAIL:*|SKIP:*) +i=`expr $i + 1` +result=`LC_ALL=C expr $line : '\([A-Z]*\):.*'` This will be quite fork expensive, if done in real-world code. But this is in a script used only for testing. I
Re: [PATCH] check: don't use multi-line coloring for the report
* Stefano Lattarini wrote on Mon, Jun 20, 2011 at 10:33:53AM CEST: OK, I've amended the patch on Bert's behalf, as he can't do that himself at the moment. Attached is what I've pushed (to maint). Thanks for handling this, and to Bert for the report and patch! Minor nit: --- a/lib/am/check.am +++ b/lib/am/check.am @@ -75,15 +75,21 @@ am__rst_title = sed 's/.*/ /;h;s/./=/g;p;x;p;g;p;s/.*//' am__rst_section = sed 'p;s/./=/g;p;g' # Put stdin (possibly several lines separated by . ) in a box. -am__text_box = $(AWK) '{ \ - n = split($$0, lines, \\. ); max = 0; \ - for (i = 1; i = n; ++i) \ -if (max length(lines[i])) \ - max = length(lines[i]);\ - for (i = 0; i max; ++i) line = line =; \ - print line;\ - for (i = 1; i = n; ++i) if (lines[i]) print lines[i];\ - print line;\ +# Prefix each line by 'col' and terminate each with 'std', for coloring. +# Multi line coloring is problematic with less -R, so we really need +# to color each line individually. Inconsistent comment level, when compared with below. I'd use '#' here, as this is not really interesting to anyone reading Makefile.in files. I acknowledge that they are undercommented for many users, but then again, getting them up to a comment level easily understandable for everyone would just blow their size up a real lot, so a better recommendation is to look at the .am files. And make sure they are commented well, but not too repetitious. @@ -398,14 +403,17 @@ check-TESTS: $(TESTS) fi; \ dashes=`echo $$dashes | sed s/./=/g`; \ if test $$failed -eq 0; then \ - echo $$grn$$dashes; \ + col=$$grn; \ else \ - echo $$red$$dashes; \ + col=$$red; \ fi; \ - echo $$banner; \ - test -z $$skipped || echo $$skipped; \ - test -z $$report || echo $$report; \ - echo $$dashes$$std; \ +## Multi line coloring is problematic with less -R, so we really need +## to color each line individually. + echo $${col}$$dashes$${std}; \ + echo $${col}$$banner$${std}; \ + test -z $$skipped || echo $${col}$$skipped$${std}; \ + test -z $$report || echo $${col}$$report$${std}; \ + echo $${col}$$dashes$${std}; \ test $$failed -eq 0; \ else :; fi Thanks, Ralf
Re: [PATCH 1/2] tests: make test runner a script, not a shell function
Ralf Wildenhues wrote: * Stefano Lattarini wrote on Mon, Jun 20, 2011 at 10:22:28AM CEST: [Adding bug-grep, dropping bug-coreutils and automake-patches] re-adding the latter. I've noticed that grep uses a definition of TESTS_ENVIRONMENT very similar to that of coreutils (the one we've just fixed), so it will break with oncoming automake too. It can be simplified there, since none of grep's tests start with #!/usr/bin/perl. That may be a sign that you may not want to actually break this code with your proposed changes to Automake. I suspect that I am the only culprit ;-) That usage (putting such a shell function in TESTS_ENVIRONMENT) was admittedly rather twisted. Put another way: it's a good idea to estimate the level of breakage you're going to burden upon others (a couple of projects, dozens, hundreds), the amount of work needed on their side to fix it, and the amount of work (or possibility) it would take to change your code so they are not broken in the first place. Also, of course, NEWS entries (and probably automake.texi entries) for such changes are a good idea. One thing I've regularly done with new code that is not 100% backward compatible is have a new Automake option for them. That is exactly why there is a 'parallel-tests': it is not fully compatible with the simple test driver, and requires work on behalf of developers using Automake. Your call, but don't do it for me. For things I maintain it'll be quick and easy to fix.
Re: [PATCH v4 3/3] parallel-tests: allow each test to have multiple results
Hi Stefano, * Stefano Lattarini wrote on Mon, Jun 20, 2011 at 10:26:06PM CEST: On Monday 20 June 2011, Ralf Wildenhues wrote: Why not just split the whole documentation change into a followup patch then? Because that would only postpone, not avoid, the continous tweaking and amending the documentation; what I'd like instead is to improve it organically and incrementally, in separate patches; i.e., commit a sketchy but correct (even if incomplete) documentation first, and then improve it with follow-ups (maybe handling one concept or one part at the time). Well, then please split the incomplete doc part of your patches into separate patches, so I can say no to them and yes to the other ones easily. ;-) This will be quite fork expensive, if done in real-world code. But this is in a script used only for testing. I don't think it's worth optimizing it. No, it's not, but your real scripts won't look all that different. Besides, why not do it right the first time? I still don't see the point this honestly, but I've thrown in a couple optimizations for bash, zsh and XSI shells (see attached squash-in). Is that enough? Just leave that out. You are right that doing such micro optimizing will not at all be a good strategy, if you do it unorganized and without planning. Leave it for later, but still keep half an eye on the stuff not getting like 4 times slower overhead. I don't actually care much about micro optimizations like the above at this point, but I do care when the whole set of code changes will introduce a factor of 2 slowdown in the test suite overhead. It looks like it may eventually, judging from your measurements done, and that's what I am trying to prevent. On w32, that would cause real pain. But maybe it would be worth trying to instead optimize stuff like $(am__check_pre) and $(am__vpath_adj_setup), where we could trim extra forks in the case of XSI shells or bash. Doesn't sound like it would bring your project forward at this point. I'm sorry I brought this topic up before. I shouldn't have. I.e., optimize an existing and tested implementation instead of holding back a promising design due to *possible* future performance problems. Well, I don't like this attitude. If something will have a performance problem, then maybe it was not all that promising after all. I'm not claiming your approach has, however. All I'm suggesting is that you do keep an eye on it. Also, execing the test driver in check2.am instead of spawning it could avoid an expensive fork. But we should then test at configure time that $SHELL can gracefully handle such execing w.r.t. the use of $(TESTS_ENVIRONMENT); i.e., that an usage like: 92 foo=bar exec sh -c 'echo $foo 9' does the expected thing (hint: it does with dash, bash, zsh, NetBSD /bin/sh and Debian ksh; it doesn't with Solaris /bin/ksh and /bin/sh). Such changes should only be done after demonstrating that they actually cause measureable speedups. And has no semantic changes (which I am not so sure of). For example the parallel BSD makes tend to reuse shells for running the recipe commands; I'm not so sure they like if their shells go away with exec. Cheers, Ralf
Re: [PATCH] check: don't use multi-line coloring for the report
* Ralf Wildenhues wrote on Mon, Jun 20, 2011 at 10:29:02PM CEST: * Stefano Lattarini wrote on Mon, Jun 20, 2011 at 10:33:53AM CEST: +# Prefix each line by 'col' and terminate each with 'std', for coloring. +# Multi line coloring is problematic with less -R, so we really need +# to color each line individually. Inconsistent comment level, when compared with below. I'd use '#' here, D'oh. I meant: I'd use '##' here. Sorry about that. (Also, this is so minor it can be dealt with along other changes later.) as this is not really interesting to anyone reading Makefile.in files. I acknowledge that they are undercommented for many users, but then again, getting them up to a comment level easily understandable for everyone would just blow their size up a real lot, so a better recommendation is to look at the .am files. And make sure they are commented well, but not too repetitious.
Re: [PATCH 1/2] tests: make test runner a script, not a shell function
On Monday 20 June 2011, Ralf Wildenhues wrote: * Jim Meyering wrote on Mon, Jun 20, 2011 at 10:29:50PM CEST: Ralf Wildenhues wrote: One thing I've regularly done with new code that is not 100% backward compatible is have a new Automake option for them. That is exactly why there is a 'parallel-tests': it is not fully compatible with the simple test driver, and requires work on behalf of developers using Automake. Your call, but don't do it for me. For things I maintain it'll be quick and easy to fix. OK. So, another thing to look out for, for Stefano: will the driver remain compatible to the current parallel-tests one, It should, as much as possible. It won't be truly 100% compatible, though; but the incompatibilities should be quite small; to quote from the added NEWS entry: - The parallel-tests driver is now implemented (partly at least) with the help of automake-provided auxiliary scripts (e.g., `test-driver'), instead of relying entirely on code in the generated Makefile.in. This has two noteworthy implications. The first one is that projects using the `parallel-tests' option should now either run automake with the `--add-missing' option, or manually copy the `test-driver' script into their tree. The second, and more important, implication is that now shell functions and non-environmental shell variables defined in TESTS_ENVIRONMENT are not anymore available in LOG_COMPILER and ext_LOG_COMPILER. For example, it is now no more possible to define a shell function, say `custom_test_runner', in TESTS_ENVIRONMENT, and then setting LOG_COMPILER = custom_test_runner in the Makefile.am in order to use that function to run the tests. Maybe we should also say that using TESTS_ENVIRONMENT to define a custom test runner is now not only strongly deprecated (as it already was I hope), but also unsupported and not working anymore? Also, should I look for TESTS_ENVIRONMENT usages in google code search? I was really hoping to spae myself the pain... ;-) and if not, is it possible to have a compatible one (without too much maintenance effort duplication)? No need to go the effort right away. Well, we could add a new option old-parallel-tests or something like that, that causes the old code in 'check.am' (with few tweaks in order to support the new parsing of `.log' files) to be used instead of the 'test-driver' script. By refactoring some code in handle_tests(), we could ensure not to add any real complexity to the automake script (w.r.t. to my patches at least); but the duplication between 'check.am' and 'test-driver' will unavoidable IMHO. I hope that we do not have to maintain four or more such drivers. Me too. And I also think we should start deprecating the old serial driver ASAP (i.e., in 1.12, as I think 1.11.2 is sadly tooearly); but this is for another thread, and another time. Two is already more than nice for maintenance (and has caused sync issues already). But maybe we can keep issues so small that we can deal with them in a NEWS entry after all. Cheers, Ralf Regards, Stefano
Re: [FYI] {maint} maintcheck: avoid few spurious failures
* Stefano Lattarini wrote on Mon, Jun 20, 2011 at 05:05:45PM CEST: * Makefile.am (sc_tests_plain_aclocal, sc_tests_plain_perl, sc_tests_plain_autoconf, sc_tests_plain_automake, sc_tests_plain_autoupate): Be stricter in matching an erroneous literal command, i.e., `aclocal', `automake', `perl', etc. sc_tests_plain_automake: - @if grep -v '^#' $(srcdir)/tests/*.test | grep -E ':[ ]*automake([^:]|$$)'; then \ + @if grep -v '^#' $(srcdir)/tests/*.test | grep -E ':[ ]*automake\([^:]|$$)'; then \ The RE that was there before was there specifically to emulate the nonportable '\' construct. Now, I'm not sure I should fight for using Posix compatible regular expressions in maintainer-check rules (seems I lost that battle earlier already), but if you require GNU grep, please be consistent and remove the now-unneeded stuff afterwards and the -E. sc_tests_plain_aclocal: - @if grep -v '^#' $(srcdir)/tests/*.test | grep ':[ ]*aclocal'; then \ + @if grep -v '^#' $(srcdir)/tests/*.test | grep ':[ ]*aclocal\'; then \ echo 'Do not run aclocal in the above tests. Use $$ACLOCAL instead.' 12; \ exit 1; \ fi Thanks, Ralf
Re: [FYI] {maint} maintcheck: avoid few spurious failures
On Monday 20 June 2011, Ralf Wildenhues wrote: * Stefano Lattarini wrote on Mon, Jun 20, 2011 at 05:05:45PM CEST: * Makefile.am (sc_tests_plain_aclocal, sc_tests_plain_perl, sc_tests_plain_autoconf, sc_tests_plain_automake, sc_tests_plain_autoupate): Be stricter in matching an erroneous literal command, i.e., `aclocal', `automake', `perl', etc. sc_tests_plain_automake: - @if grep -v '^#' $(srcdir)/tests/*.test | grep -E ':[ ]*automake([^:]|$$)'; then \ + @if grep -v '^#' $(srcdir)/tests/*.test | grep -E ':[ ]*automake\([^:]|$$)'; then \ The RE that was there before was there specifically to emulate the nonportable '\' construct. Now, I'm not sure I should fight for using Posix compatible regular expressions in maintainer-check rules (seems I lost that battle earlier already), Well, notice that I've just followed the existing practice in using GNU grep extensions in the maintcheck rules; for example, the use of `\b' was there before I even knew about the existence of Automake: $ cat -n Makefile.am 531 sc_tests_plain_sleep: 532 @if grep -E '\bsleep +[12345]\b' $(srcdir)/tests/*.test; then \ ... 537 ## fgrep and egrep are not required by POSIX. 538 sc_tests_plain_egrep_fgrep: 539 @if grep -E '\b[ef]grep\b' $(srcdir)/tests/*.test ; then \ ... $ git blame Makefile.am e44668c6 (Alexandre Duret-Lutz 2003-01-10 ... 532) ... f3a8b03a (Alexandre Duret-Lutz 2002-05-31 ... 539) ... but if you require GNU grep, please be consistent and remove the now-unneeded stuff afterwards and the -E. OK, I will push the attached patch if that's OK with you. And since I was at it, I noticed that the `sc_tests_plain_*' checks were incomplete, as they didn't look for autoreconf, autoheader and autom4te too. What fixing this with the second attached patch? sc_tests_plain_aclocal: - @if grep -v '^#' $(srcdir)/tests/*.test | grep ':[ ]*aclocal'; then \ + @if grep -v '^#' $(srcdir)/tests/*.test | grep ':[ ]*aclocal\'; then \ echo 'Do not run aclocal in the above tests. Use $$ACLOCAL instead.' 12; \ exit 1; \ fi Thanks, Ralf Regards, Stefano From 8b66bd1bd4addfe9327455ee98d444597975d797 Mon Sep 17 00:00:00 2001 Message-Id: 8b66bd1bd4addfe9327455ee98d444597975d797.1308607091.git.stefano.lattar...@gmail.com From: Stefano Lattarini stefano.lattar...@gmail.com Date: Mon, 20 Jun 2011 23:42:56 +0200 Subject: [PATCH 1/2] maintcheck: minor cleanup in rules * Makefile.am (sc_tests_plain_automake): Make the blacklisted regular expression simpler. Avoid now-useless use of the `-E' grep option. This also improve syncing with the other `sc_tests_plain_*' targets. * tests/location.test: Avoid spuriously triggering a failure of that check. Suggestion by Ralf Wildenhues. --- ChangeLog | 11 +++ Makefile.am |2 +- Makefile.in |2 +- tests/location.test | 38 +++--- 4 files changed, 32 insertions(+), 21 deletions(-) diff --git a/ChangeLog b/ChangeLog index 82aac1e..f171f19 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,14 @@ +2011-06-21 Stefano Lattarini stefano.lattar...@gmail.com + + maintcheck: minor cleanup in rules + * Makefile.am (sc_tests_plain_automake): Make the blacklisted + regular expression simpler. Avoid now-useless use of the + `-E' grep option. This also improve syncing with the other + `sc_tests_plain_*' targets. + * tests/location.test: Avoid spuriously triggering a failure of + that check. + Suggestion by Ralf Wildenhues. + 2011-06-20 Stefano Lattarini stefano.lattar...@gmail.com maintcheck: avoid few spurious failures diff --git a/Makefile.am b/Makefile.am index 9277986..978ec4c 100644 --- a/Makefile.am +++ b/Makefile.am @@ -380,7 +380,7 @@ sc_tests_plain_autoupdate: ## Tests should never call automake directly. sc_tests_plain_automake: - @if grep -v '^#' $(srcdir)/tests/*.test | grep -E ':[ ]*automake\([^:]|$$)'; then \ + @if grep -v '^#' $(srcdir)/tests/*.test | grep ':[ ]*automake\'; then \ echo 'Do not run automake in the above tests. Use $$AUTOMAKE instead.' 12; \ exit 1; \ fi diff --git a/Makefile.in b/Makefile.in index 10d57b9..242a073 100644 --- a/Makefile.in +++ b/Makefile.in @@ -1083,7 +1083,7 @@ sc_tests_plain_autoupdate: fi sc_tests_plain_automake: - @if grep -v '^#' $(srcdir)/tests/*.test | grep -E ':[ ]*automake\([^:]|$$)'; then \ + @if grep -v '^#' $(srcdir)/tests/*.test | grep ':[ ]*automake\'; then \ echo 'Do not run automake in the above tests. Use $$AUTOMAKE instead.' 12; \ exit 1; \ fi diff --git a/tests/location.test b/tests/location.test index 13a2183..9e602fd 100755 --- a/tests/location.test +++ b/tests/location.test @@ -49,25 +49,25 @@ AUTOMAKE_fails # Smash the useless difference of lib file locations. sed 's,^.*lib/am/\([a-z]*\.am\),\1,' stderr observed -cat expected \EOF -Makefile.am:12: VAR multiply defined in condition TRUE
Re: [PATCH] {maint} Deprecate de-ansification support, it should go away in automake 1.12
On Thursday 16 June 2011, Stefano Lattarini wrote: On Thursday 16 June 2011, Stefano Lattarini wrote: Hi Bob. On Thursday 16 June 2011, Bob Friesenhahn wrote: On Wed, 15 Jun 2011, Stefano Lattarini wrote: Problem: it seems that there are few (and luckily only few) active packages that still use the `ansi2knr' option: 1. The latest version (8c) of the libjpeg implementation from IJG (Independent JPEG Group, http://www.ijg.org/); see the `configure.ac' file from the archive http://www.ijg.org/files/jpegsrc.v8c.tar.gz I created most of the libjpeg build configury. The official maintainer (Guido Vollbeding) added the extra AM_INIT_AUTOMAKE options. So, maybe it would be worthwhile to directly inform the maintainers of the packages mentioned above that still use de-ANSI-fication feature about its oncoming deprecation and removal. WDYT? If they do not hear about it, I am sure that they will learn once the option stops working. OK, then I'll save myself the trouble. Thanks, Stefano Well, we've informed those projects anyway in the end; hope that it'll be helpful. Yes, it has been: the maintainers of all those projects have answered (thanks!) and agreed to remove the use of 'ansi2knr'. I've thus pushed the patch. Regards, Stefano
Re: libtool relink problem
On Sun, Jun 19, 2011 at 03:40:06PM +0530, Santhosh Thottingal wrote: Hi, I am facing a problem with libtool. While installing it complains a relink required and while doing so it tries to link against the shared library instead of static library. Well, the problem is better explained in this thread http://sourceware.org/ml/automake/2004-07/msg00127.html with example code. I am facing exactly the same problem. I saw a similar problem in this thread also http://www.mail-archive.com/libtool@gnu.org/msg00782.html Is there a solution available for this? Thanks Santhosh Hi Santhosh, I suggest writing your *_LTLIBRARIES with a comment in front, perhaps with # to differ between disabled and enabled libraries. Then using grep(1) to extract the names and the same tools to extract the dependencies. You can then use tsort(1) to topologically sort the libraries and write the actual ${dir}_LTLIBRARIES to an included .am file. Semantically, this is what I do. In reality, I wrote a preprocessor for automake that does this, but I don't suggest using it, as it's beta and very much tailored to my needs. Cheers, -- Pippijn van Steenhoven signature.asc Description: Digital signature