Re: automake 1.11.3 check-TESTS and command line length
On Wed, 22 Feb 2012, Stefano Lattarini wrote: Hi Bob, sorry for the delay. On 02/19/2012 07:55 PM, Bob Friesenhahn wrote: I am again bit by automake not being able run the test suite on systems with bounded command line length. Up to automake 1.11.2 I was able to apply a patch by Ralf Wildenhues. Which patch are you referring to exactly? With 1.11.3 the implementation is totally different and the patch has no hope of applying. How does this later patch of mine fares? http://debbugs.gnu.org/cgi/bugreport.cgi?bug=7868#31 Caveat: it works for GNU make only, and it's *really ugly*. So ugly that I don't feel like applying it to automake proper. I have not tried this patch, and indeed it is really ugly. Since another Automake release is forthcoming, please treat this email as a 'ping' that this problem still exists in Automake and that there is still ongoing impact. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Re: automake 1.11.3 check-TESTS and command line length
On Wed, 22 Feb 2012, Stefano Lattarini wrote: Are you referring to the GraphicsMagick testsuite? If yes, it seems to me that it could benefit greatly from the use of TAP once automake 1.12 is out; for example, all the 314 'tests/rwfile*.sh' tests could be rewritten as one single TAP test (and similarly for many other auto-generated tests), thus removing the need for a huge number of files without reducing granularity of the results. I took a look at what TAP is and see that I have already been using it already (not knowing what it was called) for Perl-related tests. The fundamental problem with TAP is that documented implementations (http://testanything.org/wiki/index.php/TAP_Consumers) require additional packages on the system such as Perl, Ruby, Javascript, Java, etc. I can not expect users who want to verify their build with 'make check' to install an additional language interpreter. It seems that (while I was sleeping) Automake 1.12 will be providing support for TAP tests (http://lists.gnu.org/archive/html/automake/2011-04/msg00069.html). Such support will not be worthwhile if it depends on Perl or some other interpreter since this might not be installed on the end-user's machine. If it is based on portable C (e.g. http://www.eyrie.org/~eagle/software/c-tap-harness/) or portable shell script then it will be sufficiently usable. So what is the story about TAP in Automake 1.12? Is it time to describe the new feature to us mere mortals (i.e. people on autom...@gnu.org) rather than Automake developers? Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Re: automake 1.11.3 check-TESTS and command line length
Bob Friesenhahn bfrie...@simple.dallas.tx.us wrote: Are you referring to the GraphicsMagick testsuite? If yes, it seems to me that it could benefit greatly from the use of TAP once automake 1.12 is out; for example, all the 314 'tests/rwfile*.sh' tests could be rewritten as one single TAP test (and similarly for many other auto-generated tests), thus removing the need for a huge number of files without reducing granularity of the results. I took a look at what TAP is and see that I have already been using it already (not knowing what it was called) for Perl-related tests. The fundamental problem with TAP is that documented implementations (http://testanything.org/wiki/index.php/TAP_Consumers) require additional packages on the system such as Perl, Ruby, Javascript, Java, etc. I can not expect users who want to verify their build with 'make check' to install an additional language interpreter. It seems that (while I was sleeping) Automake 1.12 will be providing support for TAP tests (http://lists.gnu.org/archive/html/automake/2011-04/msg00069.html). Such support will not be worthwhile if it depends on Perl or some other interpreter since this might not be installed on the end-user's machine. If it is based on portable C (e.g. http://www.eyrie.org/~eagle/software/c-tap-harness/) or portable shell script then it will be sufficiently usable. So what is the story about TAP in Automake 1.12? Is it time to describe the new feature to us mere mortals (i.e. people on autom...@gnu.org) rather than Automake developers? Development automake comes with consumers in awk and Perl. I have recently used the former (*1) with automake 1.11.1. It involved copying lib/tap-driver.sh from a development version to my project. tests/Makefile.am then just read: | TESTS_ENVIRONMENT = PERL5LIB=$(top_srcdir)/lib; export PERL5LIB; | TEST_LOG_DRIVER = env AM_TAP_AWK='$(AWK)' $(SHELL) $(top_srcdir)/build-aux/tap-driver.sh | TESTS = a.test b.test c.test I believe later automake versions prefer AM_TESTS_ENVIRONMENT for the first line, but updating would have been a major headache. Note though that IIRC if you were to combine (all) tests in one test, you'd lose the ability to run them in parallel (or you would have to implement that yourself). Tim (*1) In a project that uses Perl, using awk isn't very ra- tional, but I wanted to try it out.
Re: TAP support in automake (was: Re: automake 1.11.3 check-TESTS and command line length)
On Sat, 31 Mar 2012, Stefano Lattarini wrote: Well, automake being automake, it only depends on portable awk[1] ;-) OK, portable nawk actually (as found by the autoconf builtin macro AC_PROG_AWK), but I don't know of any non-museum machine lacking that. Good. The new feature is already documented in the automake manual on the 'master' branch; and this documentation should be release-quality already. But maybe I should make this new feature more visible before the 1.12 release? If yes, how do you suggest to do so? Certainly mentioning and discussing it again on the 'automake' list (happening now) is useful. The previous discussion I found was from almost a year ago before development of the feature was started. As for the implementation quality, there's still an annoying glitch in that you need to *manually* copy the automake-provided 'tap-driver.sh' script yourself into your package[2] -- automake --add-missing won't work -- and define some variables by hand; all of this is documented in the manual, and should hopefully be fixed for automake 1.12.1. This does not seem like a big problem. Russ Allbery's C TAP Harness looks reasonable as well. It also requires adding a single file (.c) to the source tree and a rule to build it as a test program. The philosophy of TAP is rather different than Automake's existing test suite. These differences may prove challenging when switching from Automake tests to TAP: * Every TAP test needs to somehow know its unique test number (so it can print a result ok 35). If the tests are all in the same module then they can simply increment an integer but if they are not then there is a problem (or they can always plan for one test). * TAP tests need to know if they are expected to pass or fail. A major feature of how GraphicsMagick is using the Automake tests framework is that it determines which tests are XFAIL based on what features were configured into the software. The list of tests which are expected to fail is produced based on Automake conditionals. It is useful for the user to see tests XFAIL as a reminder that the configuration may be missing something they wanted. How is this best handled for TAP tests? Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Re: automake 1.11.3 check-TESTS and command line length
On Wed, 22 Feb 2012, Stefano Lattarini wrote: Hi Bob, sorry for the delay. On 02/19/2012 07:55 PM, Bob Friesenhahn wrote: I am again bit by automake not being able run the test suite on systems with bounded command line length. Up to automake 1.11.2 I was able to apply a patch by Ralf Wildenhues. Which patch are you referring to exactly? With 1.11.3 the implementation is totally different and the patch has no hope of applying. How does this later patch of mine fares? http://debbugs.gnu.org/cgi/bugreport.cgi?bug=7868#31 Caveat: it works for GNU make only, and it's *really ugly*. So ugly that I don't feel like applying it to automake proper. I have not tried this patch, and indeed it is really ugly. Since another Automake release is forthcoming, please treat this email as a 'ping' that this problem still exists in Automake and that there is still ongoing impact. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Re: automake 1.11.3 check-TESTS and command line length
On Wed, 22 Feb 2012, Stefano Lattarini wrote: Are you referring to the GraphicsMagick testsuite? If yes, it seems to me that it could benefit greatly from the use of TAP once automake 1.12 is out; for example, all the 314 'tests/rwfile*.sh' tests could be rewritten as one single TAP test (and similarly for many other auto-generated tests), thus removing the need for a huge number of files without reducing granularity of the results. I took a look at what TAP is and see that I have already been using it already (not knowing what it was called) for Perl-related tests. The fundamental problem with TAP is that documented implementations (http://testanything.org/wiki/index.php/TAP_Consumers) require additional packages on the system such as Perl, Ruby, Javascript, Java, etc. I can not expect users who want to verify their build with 'make check' to install an additional language interpreter. It seems that (while I was sleeping) Automake 1.12 will be providing support for TAP tests (http://lists.gnu.org/archive/html/automake/2011-04/msg00069.html). Such support will not be worthwhile if it depends on Perl or some other interpreter since this might not be installed on the end-user's machine. If it is based on portable C (e.g. http://www.eyrie.org/~eagle/software/c-tap-harness/) or portable shell script then it will be sufficiently usable. So what is the story about TAP in Automake 1.12? Is it time to describe the new feature to us mere mortals (i.e. people on automake@gnu.org) rather than Automake developers? Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
TAP support in automake (was: Re: automake 1.11.3 check-TESTS and command line length)
Hi Bob. On 03/31/2012 07:15 PM, Bob Friesenhahn wrote: On Wed, 22 Feb 2012, Stefano Lattarini wrote: Are you referring to the GraphicsMagick testsuite? If yes, it seems to me that it could benefit greatly from the use of TAP once automake 1.12 is out; for example, all the 314 'tests/rwfile*.sh' tests could be rewritten as one single TAP test (and similarly for many other auto-generated tests), thus removing the need for a huge number of files without reducing granularity of the results. I took a look at what TAP is and see that I have already been using it already (not knowing what it was called) for Perl-related tests. The fundamental problem with TAP is that documented implementations (http://testanything.org/wiki/index.php/TAP_Consumers) require additional packages on the system such as Perl, Ruby, Javascript, Java, etc. I can not expect users who want to verify their build with 'make check' to install an additional language interpreter. It seems that (while I was sleeping) Automake 1.12 will be providing support for TAP tests (http://lists.gnu.org/archive/html/automake/2011-04/msg00069.html). Such support will not be worthwhile if it depends on Perl or some other interpreter since this might not be installed on the end-user's machine. Well, automake being automake, it only depends on portable awk[1] ;-) OK, portable nawk actually (as found by the autoconf builtin macro AC_PROG_AWK), but I don't know of any non-museum machine lacking that. [1] Automake also offers an implementation of its TAP consumer that is written in perl and based on the TAP::Parser module; but you are absolutely not required to use or know it -- at the moment, it is not even documented IIRC. If it is based on portable C (e.g. http://www.eyrie.org/~eagle/software/c-tap-harness/) or portable shell script then it will be sufficiently usable. So what is the story about TAP in Automake 1.12? Is it time to describe the new feature to us mere mortals (i.e. people on automake@gnu.org) rather than Automake developers? The new feature is already documented in the automake manual on the 'master' branch; and this documentation should be release-quality already. But maybe I should make this new feature more visible before the 1.12 release? If yes, how do you suggest to do so? As for the implementation quality, there's still an annoying glitch in that you need to *manually* copy the automake-provided 'tap-driver.sh' script yourself into your package[2] -- automake --add-missing won't work -- and define some variables by hand; all of this is documented in the manual, and should hopefully be fixed for automake 1.12.1. [2] The new automake option option '--print-libdir' should help mitigating this annoyance. Regards, Stefano
Re: TAP support in automake (was: Re: automake 1.11.3 check-TESTS and command line length)
On Sat, 31 Mar 2012, Stefano Lattarini wrote: Well, automake being automake, it only depends on portable awk[1] ;-) OK, portable nawk actually (as found by the autoconf builtin macro AC_PROG_AWK), but I don't know of any non-museum machine lacking that. Good. The new feature is already documented in the automake manual on the 'master' branch; and this documentation should be release-quality already. But maybe I should make this new feature more visible before the 1.12 release? If yes, how do you suggest to do so? Certainly mentioning and discussing it again on the 'automake' list (happening now) is useful. The previous discussion I found was from almost a year ago before development of the feature was started. As for the implementation quality, there's still an annoying glitch in that you need to *manually* copy the automake-provided 'tap-driver.sh' script yourself into your package[2] -- automake --add-missing won't work -- and define some variables by hand; all of this is documented in the manual, and should hopefully be fixed for automake 1.12.1. This does not seem like a big problem. Russ Allbery's C TAP Harness looks reasonable as well. It also requires adding a single file (.c) to the source tree and a rule to build it as a test program. The philosophy of TAP is rather different than Automake's existing test suite. These differences may prove challenging when switching from Automake tests to TAP: * Every TAP test needs to somehow know its unique test number (so it can print a result ok 35). If the tests are all in the same module then they can simply increment an integer but if they are not then there is a problem (or they can always plan for one test). * TAP tests need to know if they are expected to pass or fail. A major feature of how GraphicsMagick is using the Automake tests framework is that it determines which tests are XFAIL based on what features were configured into the software. The list of tests which are expected to fail is produced based on Automake conditionals. It is useful for the user to see tests XFAIL as a reminder that the configuration may be missing something they wanted. How is this best handled for TAP tests? Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Re: automake 1.11.3 check-TESTS and command line length
Bob Friesenhahn bfrie...@simple.dallas.tx.us wrote: Are you referring to the GraphicsMagick testsuite? If yes, it seems to me that it could benefit greatly from the use of TAP once automake 1.12 is out; for example, all the 314 'tests/rwfile*.sh' tests could be rewritten as one single TAP test (and similarly for many other auto-generated tests), thus removing the need for a huge number of files without reducing granularity of the results. I took a look at what TAP is and see that I have already been using it already (not knowing what it was called) for Perl-related tests. The fundamental problem with TAP is that documented implementations (http://testanything.org/wiki/index.php/TAP_Consumers) require additional packages on the system such as Perl, Ruby, Javascript, Java, etc. I can not expect users who want to verify their build with 'make check' to install an additional language interpreter. It seems that (while I was sleeping) Automake 1.12 will be providing support for TAP tests (http://lists.gnu.org/archive/html/automake/2011-04/msg00069.html). Such support will not be worthwhile if it depends on Perl or some other interpreter since this might not be installed on the end-user's machine. If it is based on portable C (e.g. http://www.eyrie.org/~eagle/software/c-tap-harness/) or portable shell script then it will be sufficiently usable. So what is the story about TAP in Automake 1.12? Is it time to describe the new feature to us mere mortals (i.e. people on automake@gnu.org) rather than Automake developers? Development automake comes with consumers in awk and Perl. I have recently used the former (*1) with automake 1.11.1. It involved copying lib/tap-driver.sh from a development version to my project. tests/Makefile.am then just read: | TESTS_ENVIRONMENT = PERL5LIB=$(top_srcdir)/lib; export PERL5LIB; | TEST_LOG_DRIVER = env AM_TAP_AWK='$(AWK)' $(SHELL) $(top_srcdir)/build-aux/tap-driver.sh | TESTS = a.test b.test c.test I believe later automake versions prefer AM_TESTS_ENVIRONMENT for the first line, but updating would have been a major headache. Note though that IIRC if you were to combine (all) tests in one test, you'd lose the ability to run them in parallel (or you would have to implement that yourself). Tim (*1) In a project that uses Perl, using awk isn't very ra- tional, but I wanted to try it out.
Re: automake 1.11.3 check-TESTS and command line length
Stefano Lattarini skrev 2012-02-22 21:54: On 02/22/2012 09:22 PM, Bob Friesenhahn wrote: On Wed, 22 Feb 2012, Stefano Lattarini wrote: I don't understand how that patch could actually work ... If there are too many tests in $(TESTS), there will be too many logs in $(TEST_LOGS), and since the recipe for $(TEST_SUITE_LOG) contents the expansion of $(TEST_LOGS), the command line length limit for /bin/sh -c will be exceeded anyway ... I don't claim to understand how the patch functions other than that it seems to delegate responsibility to a subordinate make. Basically, it was trying (and succeeding!) to pass the list of tests to the subordinate make on the stdin rather than on the command line, to avoid hitting any command-line length limit. It was a pretty simple and clever hack. The problem is that the list of tests will still *have* to be finally expanded in the recipe that creates 'test-suite.log', at which point all our attempt to avoid exceeding command line length limits in recursive make invocations becomes useless, no matter how clever of fast. A GNU make dependency would be a real problem. A dependency on a properly-working standard make would not be such a problem. Is the test suite actually failing due to exceeded command line length on any system different from Cygwin or MinGW? If not, I don't see how a GNU make dependency would be a problem (if not for the fact that it uglifies the already hideous patch even more). I have only observed it on MinGW/MSYS but that does not mean it will not appear on some Unix OS with unusually short command line length. True; but I say, let's cross that bridge when come at it :-) A GNU make dependency would be a problem if it depends on GNU make for other systems. This wouldn't be the case: the change causes the GNU make specific code to be used only if ./configure determines that the make program in use is GNU make. Otherwise, the old implementation would still been used. So, no regression for the non-GNU makes (even if the pre-existing command-line length isse wouldn't have been fixed for them). On MinGW, /bin/sh throws an error but I don't recall that it provides a useful diagnostic like exceeded command line length. It seemed to me that the argument list was truncated. I have not run into any problems when using Ralf's patch, but it apparently does not address all issues so Ralf backed it out. In fact, it fundamentally fails to address the issues -- that's why Ralf backed it out. All I know is that with the patch I did not experience a failure on any system. That's weird; clearly I'm missing something in the big picture here ... Maybe Peter can chime in with his MinGW expertise and save the day? ;-) I *think* the environment and the command line shares space (approx 64kB, I repeat *think* here, I don't know the details off the top of my head, Cygwin isn't affected since it uses the cygwin DLL to communicate this stuff between cygwin processes using normal ipc mechanisms). If I'm right, reducing the command line length for a process might sufficiently elevate the limit, even if it isn't eliminated. I'm no expert in that area... Cheers, Peter
Re: automake 1.11.3 check-TESTS and command line length
[re-adding the relevant automake bug in CC] Hi Peter, thanks for chiming in. On 02/27/2012 12:15 PM, Peter Rosin wrote: I *think* the environment and the command line shares space (approx 64kB, I repeat *think* here, I don't know the details off the top of my head, Cygwin isn't affected since it uses the cygwin DLL to communicate this stuff between cygwin processes using normal ipc mechanisms). If I'm right, reducing the command line length for a process might sufficiently elevate the limit, even if it isn't eliminated. So, basically, on MSYS, a: $ make TESTS=... invocation reduces the size available for the command lines of the make recipes, because the value of TESTS that gets exported in the environment eats away space that could be used by those command lines? Oh joy ... Anyway, if we now reintroduce Ralf's patch, we will have a good explanation of why it actually works. Thanks, Stefano
Re: automake 1.11.3 check-TESTS and command line length
On 02/27/2012 12:52 PM, Stefano Lattarini wrote: [re-adding the relevant automake bug in CC] Hi Peter, thanks for chiming in. On 02/27/2012 12:15 PM, Peter Rosin wrote: I *think* the environment and the command line shares space (approx 64kB, I can confirm this, based on my testing of Windows process spawning. So, basically, on MSYS, a: $ make TESTS=... invocation reduces the size available for the command lines of the make recipes, because the value of TESTS that gets exported in the environment eats away space that could be used by those command lines? Oh joy ... Yes. Unlike on *ix systems, where argv and environ are independent entities, Windows makes you deal with both limits at the same time, so increasing the size of environ reduces the size permitted in argv. -- Eric Blake ebl...@redhat.com+1-919-301-3266 Libvirt virtualization library http://libvirt.org signature.asc Description: OpenPGP digital signature
Re: automake 1.11.3 check-TESTS and command line length
Stefano Lattarini skrev 2012-02-22 21:54: On 02/22/2012 09:22 PM, Bob Friesenhahn wrote: On Wed, 22 Feb 2012, Stefano Lattarini wrote: I don't understand how that patch could actually work ... If there are too many tests in $(TESTS), there will be too many logs in $(TEST_LOGS), and since the recipe for $(TEST_SUITE_LOG) contents the expansion of $(TEST_LOGS), the command line length limit for /bin/sh -c will be exceeded anyway ... I don't claim to understand how the patch functions other than that it seems to delegate responsibility to a subordinate make. Basically, it was trying (and succeeding!) to pass the list of tests to the subordinate make on the stdin rather than on the command line, to avoid hitting any command-line length limit. It was a pretty simple and clever hack. The problem is that the list of tests will still *have* to be finally expanded in the recipe that creates 'test-suite.log', at which point all our attempt to avoid exceeding command line length limits in recursive make invocations becomes useless, no matter how clever of fast. A GNU make dependency would be a real problem. A dependency on a properly-working standard make would not be such a problem. Is the test suite actually failing due to exceeded command line length on any system different from Cygwin or MinGW? If not, I don't see how a GNU make dependency would be a problem (if not for the fact that it uglifies the already hideous patch even more). I have only observed it on MinGW/MSYS but that does not mean it will not appear on some Unix OS with unusually short command line length. True; but I say, let's cross that bridge when come at it :-) A GNU make dependency would be a problem if it depends on GNU make for other systems. This wouldn't be the case: the change causes the GNU make specific code to be used only if ./configure determines that the make program in use is GNU make. Otherwise, the old implementation would still been used. So, no regression for the non-GNU makes (even if the pre-existing command-line length isse wouldn't have been fixed for them). On MinGW, /bin/sh throws an error but I don't recall that it provides a useful diagnostic like exceeded command line length. It seemed to me that the argument list was truncated. I have not run into any problems when using Ralf's patch, but it apparently does not address all issues so Ralf backed it out. In fact, it fundamentally fails to address the issues -- that's why Ralf backed it out. All I know is that with the patch I did not experience a failure on any system. That's weird; clearly I'm missing something in the big picture here ... Maybe Peter can chime in with his MinGW expertise and save the day? ;-) I *think* the environment and the command line shares space (approx 64kB, I repeat *think* here, I don't know the details off the top of my head, Cygwin isn't affected since it uses the cygwin DLL to communicate this stuff between cygwin processes using normal ipc mechanisms). If I'm right, reducing the command line length for a process might sufficiently elevate the limit, even if it isn't eliminated. I'm no expert in that area... Cheers, Peter
Re: automake 1.11.3 check-TESTS and command line length
[re-adding the relevant automake bug in CC] Hi Peter, thanks for chiming in. On 02/27/2012 12:15 PM, Peter Rosin wrote: I *think* the environment and the command line shares space (approx 64kB, I repeat *think* here, I don't know the details off the top of my head, Cygwin isn't affected since it uses the cygwin DLL to communicate this stuff between cygwin processes using normal ipc mechanisms). If I'm right, reducing the command line length for a process might sufficiently elevate the limit, even if it isn't eliminated. So, basically, on MSYS, a: $ make TESTS=... invocation reduces the size available for the command lines of the make recipes, because the value of TESTS that gets exported in the environment eats away space that could be used by those command lines? Oh joy ... Anyway, if we now reintroduce Ralf's patch, we will have a good explanation of why it actually works. Thanks, Stefano
Re: automake 1.11.3 check-TESTS and command line length
On 02/27/2012 12:52 PM, Stefano Lattarini wrote: [re-adding the relevant automake bug in CC] Hi Peter, thanks for chiming in. On 02/27/2012 12:15 PM, Peter Rosin wrote: I *think* the environment and the command line shares space (approx 64kB, I can confirm this, based on my testing of Windows process spawning. So, basically, on MSYS, a: $ make TESTS=... invocation reduces the size available for the command lines of the make recipes, because the value of TESTS that gets exported in the environment eats away space that could be used by those command lines? Oh joy ... Yes. Unlike on *ix systems, where argv and environ are independent entities, Windows makes you deal with both limits at the same time, so increasing the size of environ reduces the size permitted in argv. -- Eric Blake ebl...@redhat.com+1-919-301-3266 Libvirt virtualization library http://libvirt.org signature.asc Description: OpenPGP digital signature
Re: automake 1.11.3 check-TESTS and command line length
Hi Bob, sorry for the delay. On 02/19/2012 07:55 PM, Bob Friesenhahn wrote: I am again bit by automake not being able run the test suite on systems with bounded command line length. Up to automake 1.11.2 I was able to apply a patch by Ralf Wildenhues. Which patch are you referring to exactly? With 1.11.3 the implementation is totally different and the patch has no hope of applying. How does this later patch of mine fares? http://debbugs.gnu.org/cgi/bugreport.cgi?bug=7868#31 Caveat: it works for GNU make only, and it's *really ugly*. So ugly that I don't feel like applying it to automake proper. What is the prospect that Automake will be able to run large test suites again? Again is not the correct word here -- automake, either patched or not, has *never* been able to run arbitrarily large test suites. Still, recent changes might have unintentionally reduced the maximal testsuite size automake is able to handle, and we'd want to fix that. How big is the testsuite you are referring to concretely? Anyway, for the moment, I've applied the patch (available from the link above) that introduces a testcase for this automake limitation. Better to keep the issue exposed. The exact version of the applied patch is attached, for reference. Regards, Stefano From 0f99c10d3f7087f51c84b01e33784d57194afcc4 Mon Sep 17 00:00:00 2001 Message-Id: 0f99c10d3f7087f51c84b01e33784d57194afcc4.1329903945.git.stefano.lattar...@gmail.com From: Stefano Lattarini stefano.lattar...@gmail.com Date: Wed, 28 Dec 2011 22:37:44 +0100 Subject: [PATCH] coverage: expose automake bug#7868 Expose the command-line length limit issue that can affect the Automake-generated parallel-tests harness, especially on systems where this limit is smaller (e.g., MinGW/MSYS). Suggestion by Bob Friesenhahn. * tests/parallel-tests-many.test: New test. We have verified that it actually exposes the bug#7868, as it passes when we opportunely reduce the number of test cases in $(TESTS). Checked on NetBSD 5.1 64bit, Debian unstable 32bit, Solaris 10 64bit and Cygwin 1.5 32bit. * tests/list-of-tests.mk (handwritten_TESTS): Add it. * tests/Makefile.am (XFAIL_TESTS): Likewise. --- tests/Makefile.am |1 + tests/list-of-tests.mk |1 + tests/parallel-tests-many.test | 176 3 files changed, 178 insertions(+), 0 deletions(-) create mode 100755 tests/parallel-tests-many.test diff --git a/tests/Makefile.am b/tests/Makefile.am index bb7bece..8a77681 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -41,6 +41,7 @@ gcj6.test \ override-conditional-2.test \ java-nobase.test \ objext-pr10128.test \ +parallel-tests-many.test \ pr8365-remake-timing.test \ lex-subobj-nodep.test \ remake-am-pr10111.test \ diff --git a/tests/list-of-tests.mk b/tests/list-of-tests.mk index f19c930..04ade17 100644 --- a/tests/list-of-tests.mk +++ b/tests/list-of-tests.mk @@ -726,6 +726,7 @@ parallel-tests-exit-statuses.test \ parallel-tests-console-output.test \ parallel-tests-once.test \ parallel-tests-trailing-bslash.test \ +parallel-tests-many.test \ tests-environment.test \ am-tests-environment.test \ tests-environment-backcompat.test \ diff --git a/tests/parallel-tests-many.test b/tests/parallel-tests-many.test new file mode 100755 index 000..0f2d732 --- /dev/null +++ b/tests/parallel-tests-many.test @@ -0,0 +1,176 @@ +#! /bin/sh +# Copyright (C) 2012 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see http://www.gnu.org/licenses/. + +# Check that the parallel-tests harness does not hit errors due to +# an exceeded command line length when there are many tests. +# For automake bug#7868. This test is currently expected to fail. + +am_parallel_tests=yes +. ./defs || Exit 1 + +set -e + +cat configure.in 'END' +AC_OUTPUT +END + +cat Makefile.am 'END' +# Sanity check that the $(TESTS) is going to exceed the system +# command line length. +# Extra quoting and indirections below are required to ensure the +# various make implementations (e.g, GNU make or Sun Distributed Make) +# will truly spawn a shell to execute this command, instead of relying +# on optimizations that might mask the Argument list too long error +# we expect. +this-will-fail: + @: : $(TEST_LOGS) +TEST_LOG_COMPILER = true +include list-of-tests.am +# So that we won't have to create a ton of dummy test cases. +$(TESTS): +END + +# The real
Re: automake 1.11.3 check-TESTS and command line length
On Wed, 22 Feb 2012, Stefano Lattarini wrote: Hi Bob, sorry for the delay. On 02/19/2012 07:55 PM, Bob Friesenhahn wrote: I am again bit by automake not being able run the test suite on systems with bounded command line length. Up to automake 1.11.2 I was able to apply a patch by Ralf Wildenhues. Which patch are you referring to exactly? This one: http://lists.gnu.org/archive/html/automake-patches/2010-09/msg00068.html With 1.11.3 the implementation is totally different and the patch has no hope of applying. How does this later patch of mine fares? http://debbugs.gnu.org/cgi/bugreport.cgi?bug=7868#31 I will check when I have time. Caveat: it works for GNU make only, and it's *really ugly*. So ugly that I don't feel like applying it to automake proper. A GNU make dependency would be a real problem. A dependency on a properly-working standard make would not be such a problem. Again is not the correct word here -- automake, either patched or not, has *never* been able to run arbitrarily large test suites. Still, recent changes might have unintentionally reduced the maximal testsuite size automake is able to handle, and we'd want to fix that. How big is the testsuite you are referring to concretely? I have not run into any problems when using Ralf's patch, but it apparently does not address all issues so Ralf backed it out. My test suite is pretty slim at the moment. About 610 tests (used to be about 1200) which expand to a list of tests of at least 17903 bytes when builddir == srcdir, or 36813 bytes with one of my typical build configurations with builddir != srcdir. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Re: automake 1.11.3 check-TESTS and command line length
On 02/22/2012 03:57 PM, Bob Friesenhahn wrote: On Wed, 22 Feb 2012, Stefano Lattarini wrote: Hi Bob, sorry for the delay. On 02/19/2012 07:55 PM, Bob Friesenhahn wrote: I am again bit by automake not being able run the test suite on systems with bounded command line length. Up to automake 1.11.2 I was able to apply a patch by Ralf Wildenhues. Which patch are you referring to exactly? This one: http://lists.gnu.org/archive/html/automake-patches/2010-09/msg00068.html I don't understand how that patch could actually work ... If there are too many tests in $(TESTS), there will be too many logs in $(TEST_LOGS), and since the recipe for $(TEST_SUITE_LOG) contents the expansion of $(TEST_LOGS), the command line length limit for /bin/sh -c will be exceeded anyway ... In fact, it was Ralf who explained that to me, when I tried to resurrect its proposed patch (which he had already reverted for an unrelated regression); see my proposal: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=7868#14 and Ralf's explanation for why this couldn't work: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=7868#17 With 1.11.3 the implementation is totally different and the patch has no hope of applying. I can rework it for the code in Automake 1.12 (soonish to appear) if you are still interested in it after having read the rationale above. How does this later patch of mine fares? http://debbugs.gnu.org/cgi/bugreport.cgi?bug=7868#31 I will check when I have time. Caveat: it works for GNU make only, and it's *really ugly*. So ugly that I don't feel like applying it to automake proper. A GNU make dependency would be a real problem. A dependency on a properly-working standard make would not be such a problem. Is the test suite actually failing due to exceeded command line length on any system different from Cygwin or MinGW? If not, I don't see how a GNU make dependency would be a problem (if not for the fact that it uglifies the already hideous patch even more). Again is not the correct word here -- automake, either patched or not, has *never* been able to run arbitrarily large test suites. Still, recent changes might have unintentionally reduced the maximal testsuite size automake is able to handle, and we'd want to fix that. How big is the testsuite you are referring to concretely? I have not run into any problems when using Ralf's patch, but it apparently does not address all issues so Ralf backed it out. In fact, it fundamentally fails to address the issues -- that's why Ralf backed it out. My test suite is pretty slim at the moment. About 610 tests (used to be about 1200) which expand to a list of tests of at least 17903 bytes when builddir == srcdir, or 36813 bytes with one of my typical build configurations with builddir != srcdir. Are you referring to the GraphicsMagick testsuite? If yes, it seems to me that it could benefit greatly from the use of TAP once automake 1.12 is out; for example, all the 314 'tests/rwfile*.sh' tests could be rewritten as one single TAP test (and similarly for many other auto-generated tests), thus removing the need for a huge number of files without reducing granularity of the results. HTH, Stefano
Re: automake 1.11.3 check-TESTS and command line length
Hi Bob, sorry for the delay. On 02/19/2012 07:55 PM, Bob Friesenhahn wrote: I am again bit by automake not being able run the test suite on systems with bounded command line length. Up to automake 1.11.2 I was able to apply a patch by Ralf Wildenhues. Which patch are you referring to exactly? With 1.11.3 the implementation is totally different and the patch has no hope of applying. How does this later patch of mine fares? http://debbugs.gnu.org/cgi/bugreport.cgi?bug=7868#31 Caveat: it works for GNU make only, and it's *really ugly*. So ugly that I don't feel like applying it to automake proper. What is the prospect that Automake will be able to run large test suites again? Again is not the correct word here -- automake, either patched or not, has *never* been able to run arbitrarily large test suites. Still, recent changes might have unintentionally reduced the maximal testsuite size automake is able to handle, and we'd want to fix that. How big is the testsuite you are referring to concretely? Anyway, for the moment, I've applied the patch (available from the link above) that introduces a testcase for this automake limitation. Better to keep the issue exposed. The exact version of the applied patch is attached, for reference. Regards, Stefano From 0f99c10d3f7087f51c84b01e33784d57194afcc4 Mon Sep 17 00:00:00 2001 Message-Id: 0f99c10d3f7087f51c84b01e33784d57194afcc4.1329903945.git.stefano.lattar...@gmail.com From: Stefano Lattarini stefano.lattar...@gmail.com Date: Wed, 28 Dec 2011 22:37:44 +0100 Subject: [PATCH] coverage: expose automake bug#7868 Expose the command-line length limit issue that can affect the Automake-generated parallel-tests harness, especially on systems where this limit is smaller (e.g., MinGW/MSYS). Suggestion by Bob Friesenhahn. * tests/parallel-tests-many.test: New test. We have verified that it actually exposes the bug#7868, as it passes when we opportunely reduce the number of test cases in $(TESTS). Checked on NetBSD 5.1 64bit, Debian unstable 32bit, Solaris 10 64bit and Cygwin 1.5 32bit. * tests/list-of-tests.mk (handwritten_TESTS): Add it. * tests/Makefile.am (XFAIL_TESTS): Likewise. --- tests/Makefile.am |1 + tests/list-of-tests.mk |1 + tests/parallel-tests-many.test | 176 3 files changed, 178 insertions(+), 0 deletions(-) create mode 100755 tests/parallel-tests-many.test diff --git a/tests/Makefile.am b/tests/Makefile.am index bb7bece..8a77681 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -41,6 +41,7 @@ gcj6.test \ override-conditional-2.test \ java-nobase.test \ objext-pr10128.test \ +parallel-tests-many.test \ pr8365-remake-timing.test \ lex-subobj-nodep.test \ remake-am-pr10111.test \ diff --git a/tests/list-of-tests.mk b/tests/list-of-tests.mk index f19c930..04ade17 100644 --- a/tests/list-of-tests.mk +++ b/tests/list-of-tests.mk @@ -726,6 +726,7 @@ parallel-tests-exit-statuses.test \ parallel-tests-console-output.test \ parallel-tests-once.test \ parallel-tests-trailing-bslash.test \ +parallel-tests-many.test \ tests-environment.test \ am-tests-environment.test \ tests-environment-backcompat.test \ diff --git a/tests/parallel-tests-many.test b/tests/parallel-tests-many.test new file mode 100755 index 000..0f2d732 --- /dev/null +++ b/tests/parallel-tests-many.test @@ -0,0 +1,176 @@ +#! /bin/sh +# Copyright (C) 2012 Free Software Foundation, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2, or (at your option) +# any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see http://www.gnu.org/licenses/. + +# Check that the parallel-tests harness does not hit errors due to +# an exceeded command line length when there are many tests. +# For automake bug#7868. This test is currently expected to fail. + +am_parallel_tests=yes +. ./defs || Exit 1 + +set -e + +cat configure.in 'END' +AC_OUTPUT +END + +cat Makefile.am 'END' +# Sanity check that the $(TESTS) is going to exceed the system +# command line length. +# Extra quoting and indirections below are required to ensure the +# various make implementations (e.g, GNU make or Sun Distributed Make) +# will truly spawn a shell to execute this command, instead of relying +# on optimizations that might mask the Argument list too long error +# we expect. +this-will-fail: + @: : $(TEST_LOGS) +TEST_LOG_COMPILER = true +include list-of-tests.am +# So that we won't have to create a ton of dummy test cases. +$(TESTS): +END + +# The real
Re: automake 1.11.3 check-TESTS and command line length
On Wed, 22 Feb 2012, Stefano Lattarini wrote: Hi Bob, sorry for the delay. On 02/19/2012 07:55 PM, Bob Friesenhahn wrote: I am again bit by automake not being able run the test suite on systems with bounded command line length. Up to automake 1.11.2 I was able to apply a patch by Ralf Wildenhues. Which patch are you referring to exactly? This one: http://lists.gnu.org/archive/html/automake-patches/2010-09/msg00068.html With 1.11.3 the implementation is totally different and the patch has no hope of applying. How does this later patch of mine fares? http://debbugs.gnu.org/cgi/bugreport.cgi?bug=7868#31 I will check when I have time. Caveat: it works for GNU make only, and it's *really ugly*. So ugly that I don't feel like applying it to automake proper. A GNU make dependency would be a real problem. A dependency on a properly-working standard make would not be such a problem. Again is not the correct word here -- automake, either patched or not, has *never* been able to run arbitrarily large test suites. Still, recent changes might have unintentionally reduced the maximal testsuite size automake is able to handle, and we'd want to fix that. How big is the testsuite you are referring to concretely? I have not run into any problems when using Ralf's patch, but it apparently does not address all issues so Ralf backed it out. My test suite is pretty slim at the moment. About 610 tests (used to be about 1200) which expand to a list of tests of at least 17903 bytes when builddir == srcdir, or 36813 bytes with one of my typical build configurations with builddir != srcdir. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Re: automake 1.11.3 check-TESTS and command line length
On 02/22/2012 03:57 PM, Bob Friesenhahn wrote: On Wed, 22 Feb 2012, Stefano Lattarini wrote: Hi Bob, sorry for the delay. On 02/19/2012 07:55 PM, Bob Friesenhahn wrote: I am again bit by automake not being able run the test suite on systems with bounded command line length. Up to automake 1.11.2 I was able to apply a patch by Ralf Wildenhues. Which patch are you referring to exactly? This one: http://lists.gnu.org/archive/html/automake-patches/2010-09/msg00068.html I don't understand how that patch could actually work ... If there are too many tests in $(TESTS), there will be too many logs in $(TEST_LOGS), and since the recipe for $(TEST_SUITE_LOG) contents the expansion of $(TEST_LOGS), the command line length limit for /bin/sh -c will be exceeded anyway ... In fact, it was Ralf who explained that to me, when I tried to resurrect its proposed patch (which he had already reverted for an unrelated regression); see my proposal: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=7868#14 and Ralf's explanation for why this couldn't work: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=7868#17 With 1.11.3 the implementation is totally different and the patch has no hope of applying. I can rework it for the code in Automake 1.12 (soonish to appear) if you are still interested in it after having read the rationale above. How does this later patch of mine fares? http://debbugs.gnu.org/cgi/bugreport.cgi?bug=7868#31 I will check when I have time. Caveat: it works for GNU make only, and it's *really ugly*. So ugly that I don't feel like applying it to automake proper. A GNU make dependency would be a real problem. A dependency on a properly-working standard make would not be such a problem. Is the test suite actually failing due to exceeded command line length on any system different from Cygwin or MinGW? If not, I don't see how a GNU make dependency would be a problem (if not for the fact that it uglifies the already hideous patch even more). Again is not the correct word here -- automake, either patched or not, has *never* been able to run arbitrarily large test suites. Still, recent changes might have unintentionally reduced the maximal testsuite size automake is able to handle, and we'd want to fix that. How big is the testsuite you are referring to concretely? I have not run into any problems when using Ralf's patch, but it apparently does not address all issues so Ralf backed it out. In fact, it fundamentally fails to address the issues -- that's why Ralf backed it out. My test suite is pretty slim at the moment. About 610 tests (used to be about 1200) which expand to a list of tests of at least 17903 bytes when builddir == srcdir, or 36813 bytes with one of my typical build configurations with builddir != srcdir. Are you referring to the GraphicsMagick testsuite? If yes, it seems to me that it could benefit greatly from the use of TAP once automake 1.12 is out; for example, all the 314 'tests/rwfile*.sh' tests could be rewritten as one single TAP test (and similarly for many other auto-generated tests), thus removing the need for a huge number of files without reducing granularity of the results. HTH, Stefano
Re: automake 1.11.3 check-TESTS and command line length
On Wed, 22 Feb 2012, Stefano Lattarini wrote: I don't understand how that patch could actually work ... If there are too many tests in $(TESTS), there will be too many logs in $(TEST_LOGS), and since the recipe for $(TEST_SUITE_LOG) contents the expansion of $(TEST_LOGS), the command line length limit for /bin/sh -c will be exceeded anyway ... I don't claim to understand how the patch functions other than that it seems to delegate responsibility to a subordinate make. A GNU make dependency would be a real problem. A dependency on a properly-working standard make would not be such a problem. Is the test suite actually failing due to exceeded command line length on any system different from Cygwin or MinGW? If not, I don't see how a GNU make dependency would be a problem (if not for the fact that it uglifies the already hideous patch even more). I have only observed it on MinGW/MSYS but that does not mean it will not appear on some Unix OS with unusually short command line length. A GNU make dependency would be a problem if it depends on GNU make for other systems. On MinGW, /bin/sh throws an error but I don't recall that it provides a useful diagnostic like exceeded command line length. It seemed to me that the argument list was truncated. I have not run into any problems when using Ralf's patch, but it apparently does not address all issues so Ralf backed it out. In fact, it fundamentally fails to address the issues -- that's why Ralf backed it out. All I know is that with the patch I did not experience a failure on any system. Are you referring to the GraphicsMagick testsuite? If yes, it seems to me that it could benefit greatly from the use of TAP once automake 1.12 is out; for example, all the 314 'tests/rwfile*.sh' tests could be rewritten as one single TAP test (and similarly for many other auto-generated tests), thus removing the need for a huge number of files without reducing granularity of the results. Yes, this is for the GraphicsMagick test suite. There is little doubt that the test suite is grossly inefficient. The good news is that the test suite scripts are generated by a Makefile target (manually invoked) so it should be possible to output the tests in some other format. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
Re: automake 1.11.3 check-TESTS and command line length
On 02/22/2012 09:22 PM, Bob Friesenhahn wrote: On Wed, 22 Feb 2012, Stefano Lattarini wrote: I don't understand how that patch could actually work ... If there are too many tests in $(TESTS), there will be too many logs in $(TEST_LOGS), and since the recipe for $(TEST_SUITE_LOG) contents the expansion of $(TEST_LOGS), the command line length limit for /bin/sh -c will be exceeded anyway ... I don't claim to understand how the patch functions other than that it seems to delegate responsibility to a subordinate make. Basically, it was trying (and succeeding!) to pass the list of tests to the subordinate make on the stdin rather than on the command line, to avoid hitting any command-line length limit. It was a pretty simple and clever hack. The problem is that the list of tests will still *have* to be finally expanded in the recipe that creates 'test-suite.log', at which point all our attempt to avoid exceeding command line length limits in recursive make invocations becomes useless, no matter how clever of fast. A GNU make dependency would be a real problem. A dependency on a properly-working standard make would not be such a problem. Is the test suite actually failing due to exceeded command line length on any system different from Cygwin or MinGW? If not, I don't see how a GNU make dependency would be a problem (if not for the fact that it uglifies the already hideous patch even more). I have only observed it on MinGW/MSYS but that does not mean it will not appear on some Unix OS with unusually short command line length. True; but I say, let's cross that bridge when come at it :-) A GNU make dependency would be a problem if it depends on GNU make for other systems. This wouldn't be the case: the change causes the GNU make specific code to be used only if ./configure determines that the make program in use is GNU make. Otherwise, the old implementation would still been used. So, no regression for the non-GNU makes (even if the pre-existing command-line length isse wouldn't have been fixed for them). On MinGW, /bin/sh throws an error but I don't recall that it provides a useful diagnostic like exceeded command line length. It seemed to me that the argument list was truncated. I have not run into any problems when using Ralf's patch, but it apparently does not address all issues so Ralf backed it out. In fact, it fundamentally fails to address the issues -- that's why Ralf backed it out. All I know is that with the patch I did not experience a failure on any system. That's weird; clearly I'm missing something in the big picture here ... Maybe Peter can chime in with his MinGW expertise and save the day? ;-) Are you referring to the GraphicsMagick testsuite? If yes, it seems to me that it could benefit greatly from the use of TAP once automake 1.12 is out; for example, all the 314 'tests/rwfile*.sh' tests could be rewritten as one single TAP test (and similarly for many other auto-generated tests), thus removing the need for a huge number of files without reducing granularity of the results. Yes, this is for the GraphicsMagick test suite. There is little doubt that the test suite is grossly inefficient. The good news is that the test suite scripts are generated by a Makefile target (manually invoked) so it should be possible to output the tests in some other format. Or even not generate them, but have some TAP tests doing the looping work directly; for an example of a similar move from a bunch of autogenerated tests to a single TAP test, you can take a look to this Automake commit: http://git.savannah.gnu.org/gitweb/?p=automake.git;a=commitdiff;h=8b4dfc2d Regards, Stefano
Re: automake 1.11.3 check-TESTS and command line length
On Sun, 19 Feb 2012, Bob Friesenhahn wrote: I am again bit by automake not being able run the test suite on systems with bounded command line length. Up to automake 1.11.2 I was able to apply a To clarify, it seems that this issue is only a problem with when parallel-tests is enabled. Unfortunately, this option provides the nice summary log files (more useful than the parallel execution), so it is very useful. Originally my package's test suite created its own log files but when the parallel-tests feature appeared, the test suite was converted to rely on Automake's log file mechanism and printed large amounts of test to stdout. Once use of this option was fully entrenched, then Automake added the command line length bug. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
