Re: TAP support in automake

[Stefano Lattarini]

On 03/31/2012 10:23 PM, Bob Friesenhahn wrote:
> On Sat, 31 Mar 2012, Stefano Lattarini wrote:
>>>
>> Well, automake being automake, it only depends on portable awk[1] ;-)
>> OK, portable "nawk" actually (as found by the autoconf builtin macro
>> AC_PROG_AWK), but I don't know of any non-museum machine lacking that.
> 
> Good.
> 
>> The new feature is already documented in the automake manual on the
>> 'master' branch; and this documentation should be release-quality
>> already.  But maybe I should make this new feature more visible before
>> the 1.12 release?  If yes, how do you suggest to do so?
> 
> Certainly mentioning and discussing it again on the 'automake' list 
> (happening now) is useful.
>
Let's continue the discussion then :-)

> The previous discussion I found was from almost a year ago before
> development of the feature was started.
> 
Communication failure on my part then :-(  The feature's development was
basically complete by the end of August 2011 (as mandated by the GSoC
deadlines).  And Automake has been happily using TAP in its own testsuite
since then!

>> As for the implementation quality, there's still an annoying glitch in
>> that you need to *manually* copy the automake-provided 'tap-driver.sh'
>> script yourself into your package[2] -- "automake --add-missing" won't
>> work -- and define some variables by hand; all of this is documented
>> in the manual, and should hopefully be fixed for automake 1.12.1.
> 
> This does not seem like a big problem.
> 
> Russ Allbery's C TAP Harness looks reasonable as well.  It also
> requires adding a single file (.c) to the source tree and a rule
> to build it as a test program.
>
Consider that the whole point of the TAP support in Automake is that the
testsuite harness present in Automake-generated makefiles (the one that
gets launched by "make check") will act as TAP consumer itself.  Thus
the "harness" part of Russ' package is likely of little or no use in
conjunction with the newer Automake test harness -- but of course you
can reuse the part of his project that provide a TAP *producer* for C
and/or Shell programs.

> The philosophy of TAP is rather different than Automake's existing
> test suite.
>
Yes, but not too radically IMHO; in fact, the TAP consumer implemented by
Automake reuses most of the code and interfaces of the 'parallel-tests'
test harness (that was an early design choice, and so far I have been very
happy with it).

> These differences may prove challenging when switching from Automake
> tests to TAP:
>
Note that you don't need any "flag day switch" for a conversion to TAP: "simple
tests" (whose outcome is purely based on their exit status) and TAP tests can
happily co-exists (as they currently do in the Automake's own testsuite).  This
too is an early, deliberate design choice.

>   * Every TAP test needs to somehow know its unique test number (so it
> can print a result "ok 35").
>
Actually, not really:

  $ cat foo.sh
  #!/bin/sh
  echo 1..3
  echo 'ok'
  echo 'ok # SKIP'
  echo 'not ok # TODO'

  $ prove ./foo.sh
  ./foo.sh .. ok
  All tests successful.
  Files=1, Tests=3,  1 wallclock secs ( 0.04 usr +  0.00 sys =  0.04 CPU)
  Result: PASS

But as it's reasonable, if you provide a test number, that must be correct
(this should be undeniably recognized as a feature IMO); see:

  $ cat bar.sh
  #!/bin/sh
  echo 1..3
  echo 'ok 4'
  echo 'ok # SKIP'
  echo 'not ok # TODO'

  $ prove ./bar.sh
  ./bar.sh .. All 3 subtests passed
  (less 1 skipped subtest: 2 okay)
  Test Summary Report
  ---
  ./bar.sh (Wstat: 0 Tests: 3 Failed: 0)
Parse errors: Tests out of sequence.  Found (4) but expected (1)
  Files=1, Tests=3,  0 wallclock secs ( 0.04 usr +  0.00 sys =  0.04 CPU)
  Result: FAIL

> If the tests are all in the same
> module then they can simply increment an integer but if they are
> not then there is a problem (or they can always plan for one
> test).
>
Apart from the fact that I'd advise against writing your TAP stream by
hand (you should use a small but higher-level module or library or set
of functions to do so), the situation you describe shouldn't be a problem
in any case: Automake-produced Makefiles fire off a *new* TAP-parsing
process for each TAP-based test script in $(TESTS) [1], so that you would
only need to worry about the ordering/numbering of the test cases in the
script you are modifying.

 [1] This is also extremely important in order to preserve the ability
 to run tests in parallel -- one of the major selling points of the
 Automake testsuite harness.

>   * TAP tests need to know if they are expected to pass or fail.
> 
> A major feature of how GraphicsMagick is using the Automake tests
> framework is that it determines which tests are XFAIL based on what
> features were configured into the software.  The list of tests which
> are expected to fail is produced based on Automake conditionals.
> It is useful for the user to see tests XFAIL as a remind

Re: [FYI] {master} cosmetics: fix repeated line in comment

[Peter Rosin]

On 2012-03-31 22:14, Stefano Lattarini wrote:
> * automake.in: Drop a repeated line in a comment.  Issue
> introduced in one of the commits that converted from quoting
> `like this' to quoting 'like this'.
> 
> Signed-off-by: Stefano Lattarini 
> ---
>  automake.in |1 -
>  1 files changed, 0 insertions(+), 1 deletions(-)
> 
> diff --git a/automake.in b/automake.in
> index 5357557..45f5d3d 100644
> --- a/automake.in
> +++ b/automake.in
> @@ -955,7 +955,6 @@ register_language ('name' => 'ppfc',
>  # $(CPPFLAGS)" as additional flags to the Fortran 77 compiler, since
>  # this is how GNU Make does it; see the "GNU Make Manual, Edition 0.51
>  # for `make' Version 3.76 Beta" (specifically, from info file
> -# for `make' Version 3.76 Beta" (specifically, from info file
>  # '(make)Catalogue of Rules').
>  #
>  # A better approach would be to write an Autoconf test

I can't find this one in the repo when I pull, so I assume you must
have forgotten to push it, even if it's a FYI.

Anyway, you probably don't want to keep quoting like `this'.

Cheers,
Peter

Re: TAP support in automake (was: Re: automake 1.11.3 check-TESTS and command line length)

[Bob Friesenhahn]

On Sat, 31 Mar 2012, Stefano Lattarini wrote:



Well, automake being automake, it only depends on portable awk[1] ;-)
OK, portable "nawk" actually (as found by the autoconf builtin macro
AC_PROG_AWK), but I don't know of any non-museum machine lacking that.


Good.


The new feature is already documented in the automake manual on the
'master' branch; and this documentation should be release-quality
already.  But maybe I should make this new feature more visible before
the 1.12 release?  If yes, how do you suggest to do so?


Certainly mentioning and discussing it again on the 'automake' list 
(happening now) is useful.  The previous discussion I found was from 
almost a year ago before development of the feature was started.



As for the implementation quality, there's still an annoying glitch in
that you need to *manually* copy the automake-provided 'tap-driver.sh'
script yourself into your package[2] -- "automake --add-missing" won't
work -- and define some variables by hand; all of this is documented
in the manual, and should hopefully be fixed for automake 1.12.1.


This does not seem like a big problem.

Russ Allbery's C TAP Harness looks reasonable as well.  It also 
requires adding a single file (.c) to the source tree and a rule to 
build it as a test program.


The philosophy of TAP is rather different than Automake's existing 
test suite.  These differences may prove challenging when switching 
from Automake tests to TAP:


  * Every TAP test needs to somehow know its unique test number (so it
can print a result "ok 35").  If the tests are all in the same
module then they can simply increment an integer but if they are
not then there is a problem (or they can always plan for one
test).

  * TAP tests need to know if they are expected to pass or fail.

A major feature of how GraphicsMagick is using the Automake tests 
framework is that it determines which tests are XFAIL based on what 
features were configured into the software.  The list of tests which 
are expected to fail is produced based on Automake conditionals.  It 
is useful for the user to see tests XFAIL as a reminder that the 
configuration may be missing something they wanted.  How is this best 
handled for TAP tests?


Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/

[FYI] {master} cosmetics: fix repeated line in comment

[Stefano Lattarini]

* automake.in: Drop a repeated line in a comment.  Issue
introduced in one of the commits that converted from quoting
`like this' to quoting 'like this'.

Signed-off-by: Stefano Lattarini 
---
 automake.in |1 -
 1 files changed, 0 insertions(+), 1 deletions(-)

diff --git a/automake.in b/automake.in
index 5357557..45f5d3d 100644
--- a/automake.in
+++ b/automake.in
@@ -955,7 +955,6 @@ register_language ('name' => 'ppfc',
 # $(CPPFLAGS)" as additional flags to the Fortran 77 compiler, since
 # this is how GNU Make does it; see the "GNU Make Manual, Edition 0.51
 # for `make' Version 3.76 Beta" (specifically, from info file
-# for `make' Version 3.76 Beta" (specifically, from info file
 # '(make)Catalogue of Rules').
 #
 # A better approach would be to write an Autoconf test
-- 
1.7.9

[FYI] {master} cosmetics: remove useless import in automake script

[Stefano Lattarini]

* automake.in (Automake::Version): This import is not needed;
remove it.

Signed-off-by: Stefano Lattarini 
---
 automake.in |1 -
 1 files changed, 0 insertions(+), 1 deletions(-)

diff --git a/automake.in b/automake.in
index 3e16fd8..5357557 100644
--- a/automake.in
+++ b/automake.in
@@ -150,7 +150,6 @@ use Automake::Location;
 use Automake::Condition qw/TRUE FALSE/;
 use Automake::DisjConditions;
 use Automake::Options;
-use Automake::Version;
 use Automake::Variable;
 use Automake::VarDef;
 use Automake::Rule;
-- 
1.7.9

Re: automake 1.11.3 check-TESTS and command line length

[Tim Landscheidt]

Bob Friesenhahn  wrote:

>> Are you referring to the GraphicsMagick testsuite?  If yes, it seems to me
>> that it could benefit greatly from the use of TAP once automake 1.12 is out;
>> for example, all the 314 'tests/rwfile*.sh' tests could be rewritten as one
>> single TAP test (and similarly for many other auto-generated tests), thus
>> removing the need for a huge number of files without reducing granularity
>> of the results.

> I took a look at what "TAP" is and see that I have already
> been using it already (not knowing what it was called) for
> Perl-related tests. The fundamental problem with TAP is that
> documented implementations
> (http://testanything.org/wiki/index.php/TAP_Consumers)
> require additional packages on the system such as Perl,
> Ruby, Javascript, Java, etc.  I can not expect users who
> want to verify their build with 'make check' to install an
> additional language interpreter.

> It seems that (while I was sleeping) Automake 1.12 will be
> providing support for TAP tests
> (http://lists.gnu.org/archive/html/automake/2011-04/msg00069.html). Such
> support will not be worthwhile if it depends on Perl or some
> other interpreter since this might not be installed on the
> end-user's machine.  If it is based on portable C
> (e.g. http://www.eyrie.org/~eagle/software/c-tap-harness/)
> or portable shell script then it will be sufficiently
> usable.

> So what is the story about TAP in Automake 1.12?  Is it time
> to describe the new feature to us mere mortals (i.e. people
> on autom...@gnu.org) rather than Automake developers?

Development automake comes with consumers in awk and Perl.
I have recently used the former (*1) with automake 1.11.1.
It involved copying lib/tap-driver.sh from a development
version to my project.  tests/Makefile.am then just read:

| TESTS_ENVIRONMENT = PERL5LIB=$(top_srcdir)/lib; export PERL5LIB;
| TEST_LOG_DRIVER = env AM_TAP_AWK='$(AWK)' $(SHELL) 
$(top_srcdir)/build-aux/tap-driver.sh
| TESTS = a.test b.test c.test

I believe later automake versions prefer
AM_TESTS_ENVIRONMENT for the first line, but updating would
have been a major headache.

  Note though that IIRC if you were to combine (all) tests
in one test, you'd lose the ability to run them in parallel
(or you would have to implement that yourself).

Tim

(*1)   In a project that uses Perl, using awk isn't very ra-
   tional, but I wanted to try it out.

TAP support in automake (was: Re: automake 1.11.3 check-TESTS and command line length)

[Stefano Lattarini]

Hi Bob.

On 03/31/2012 07:15 PM, Bob Friesenhahn wrote:
> On Wed, 22 Feb 2012, Stefano Lattarini wrote:
>>>
>> Are you referring to the GraphicsMagick testsuite?  If yes, it seems to me
>> that it could benefit greatly from the use of TAP once automake 1.12 is out;
>> for example, all the 314 'tests/rwfile*.sh' tests could be rewritten as one
>> single TAP test (and similarly for many other auto-generated tests), thus
>> removing the need for a huge number of files without reducing granularity
>> of the results.
> 
> I took a look at what "TAP" is and see that I have already been using it
> already (not knowing what it was called) for Perl-related tests. The
> fundamental problem with TAP is that documented implementations
> (http://testanything.org/wiki/index.php/TAP_Consumers) require additional
> packages on the system such as Perl, Ruby, Javascript, Java, etc.
> I can not expect users who want to verify their build with 'make check'
> to install an additional language interpreter.
> 
> It seems that (while I was sleeping) Automake 1.12 will be providing support
> for TAP tests 
> (http://lists.gnu.org/archive/html/automake/2011-04/msg00069.html).
> Such support will not be worthwhile if it depends on Perl or some other
> interpreter since this might not be installed on the end-user's machine.
>
Well, automake being automake, it only depends on portable awk[1] ;-)
OK, portable "nawk" actually (as found by the autoconf builtin macro
AC_PROG_AWK), but I don't know of any non-museum machine lacking that.

 [1] Automake also offers an implementation of its TAP consumer that is
 written in perl and based on the TAP::Parser module; but you are
 absolutely not required to use or know it -- at the moment, it is
 not even documented IIRC.

> If it is based on portable C
> (e.g. http://www.eyrie.org/~eagle/software/c-tap-harness/)
> or portable shell script then it will be sufficiently usable.
>
> So what is the story about TAP in Automake 1.12?  Is it time to describe
> the new feature to us mere mortals (i.e. people on autom...@gnu.org)
> rather than Automake developers?
>
The new feature is already documented in the automake manual on the
'master' branch; and this documentation should be release-quality
already.  But maybe I should make this new feature more visible before
the 1.12 release?  If yes, how do you suggest to do so?

As for the implementation quality, there's still an annoying glitch in
that you need to *manually* copy the automake-provided 'tap-driver.sh'
script yourself into your package[2] -- "automake --add-missing" won't
work -- and define some variables by hand; all of this is documented
in the manual, and should hopefully be fixed for automake 1.12.1.

 [2] The new automake option option '--print-libdir' should help
 mitigating this annoyance.

Regards,
  Stefano

Re: automake 1.11.3 check-TESTS and command line length

[Bob Friesenhahn]

On Wed, 22 Feb 2012, Stefano Lattarini wrote:



Are you referring to the GraphicsMagick testsuite?  If yes, it seems to me
that it could benefit greatly from the use of TAP once automake 1.12 is out;
for example, all the 314 'tests/rwfile*.sh' tests could be rewritten as one
single TAP test (and similarly for many other auto-generated tests), thus
removing the need for a huge number of files without reducing granularity
of the results.


I took a look at what "TAP" is and see that I have already been using 
it already (not knowing what it was called) for Perl-related tests. 
The fundamental problem with TAP is that documented implementations 
(http://testanything.org/wiki/index.php/TAP_Consumers) require 
additional packages on the system such as Perl, Ruby, Javascript, 
Java, etc.  I can not expect users who want to verify their build with 
'make check' to install an additional language interpreter.


It seems that (while I was sleeping) Automake 1.12 will be providing 
support for TAP tests 
(http://lists.gnu.org/archive/html/automake/2011-04/msg00069.html). 
Such support will not be worthwhile if it depends on Perl or some 
other interpreter since this might not be installed on the end-user's 
machine.  If it is based on portable C (e.g. 
http://www.eyrie.org/~eagle/software/c-tap-harness/) or portable shell 
script then it will be sufficiently usable.


So what is the story about TAP in Automake 1.12?  Is it time to 
describe the new feature to us mere mortals (i.e. people on 
autom...@gnu.org) rather than Automake developers?


Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/

Re: automake 1.11.3 check-TESTS and command line length

[Bob Friesenhahn]

On Wed, 22 Feb 2012, Stefano Lattarini wrote:


Hi Bob, sorry for the delay.

On 02/19/2012 07:55 PM, Bob Friesenhahn wrote:

I am again bit by automake not being able run the test suite on systems
with bounded command line length.  Up to automake 1.11.2 I was able to
apply a patch by Ralf Wildenhues.


Which patch are you referring to exactly?


With 1.11.3 the implementation is totally different and the patch has no hope
of applying.


How does this later patch of mine fares?

 

Caveat: it works for GNU make only, and it's *really ugly*.  So ugly that I 
don't
feel like applying it to automake proper.


I have not tried this patch, and indeed it is really ugly.

Since another Automake release is forthcoming, please treat this email 
as a 'ping' that this problem still exists in Automake and that there 
is still ongoing impact.


Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/