Re: TAP support in automake (was: Re: automake 1.11.3 check-TESTS and command line length)

[Bob Friesenhahn]

On Sat, 31 Mar 2012, Stefano Lattarini wrote:



Well, automake being automake, it only depends on portable awk[1] ;-)
OK, portable "nawk" actually (as found by the autoconf builtin macro
AC_PROG_AWK), but I don't know of any non-museum machine lacking that.


Good.


The new feature is already documented in the automake manual on the
'master' branch; and this documentation should be release-quality
already.  But maybe I should make this new feature more visible before
the 1.12 release?  If yes, how do you suggest to do so?


Certainly mentioning and discussing it again on the 'automake' list 
(happening now) is useful.  The previous discussion I found was from 
almost a year ago before development of the feature was started.



As for the implementation quality, there's still an annoying glitch in
that you need to *manually* copy the automake-provided 'tap-driver.sh'
script yourself into your package[2] -- "automake --add-missing" won't
work -- and define some variables by hand; all of this is documented
in the manual, and should hopefully be fixed for automake 1.12.1.


This does not seem like a big problem.

Russ Allbery's C TAP Harness looks reasonable as well.  It also 
requires adding a single file (.c) to the source tree and a rule to 
build it as a test program.


The philosophy of TAP is rather different than Automake's existing 
test suite.  These differences may prove challenging when switching 
from Automake tests to TAP:


  * Every TAP test needs to somehow know its unique test number (so it
can print a result "ok 35").  If the tests are all in the same
module then they can simply increment an integer but if they are
not then there is a problem (or they can always plan for one
test).

  * TAP tests need to know if they are expected to pass or fail.

A major feature of how GraphicsMagick is using the Automake tests 
framework is that it determines which tests are XFAIL based on what 
features were configured into the software.  The list of tests which 
are expected to fail is produced based on Automake conditionals.  It 
is useful for the user to see tests XFAIL as a reminder that the 
configuration may be missing something they wanted.  How is this best 
handled for TAP tests?


Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/

TAP support in automake (was: Re: automake 1.11.3 check-TESTS and command line length)

[Stefano Lattarini]

Hi Bob.

On 03/31/2012 07:15 PM, Bob Friesenhahn wrote:
> On Wed, 22 Feb 2012, Stefano Lattarini wrote:
>>>
>> Are you referring to the GraphicsMagick testsuite?  If yes, it seems to me
>> that it could benefit greatly from the use of TAP once automake 1.12 is out;
>> for example, all the 314 'tests/rwfile*.sh' tests could be rewritten as one
>> single TAP test (and similarly for many other auto-generated tests), thus
>> removing the need for a huge number of files without reducing granularity
>> of the results.
> 
> I took a look at what "TAP" is and see that I have already been using it
> already (not knowing what it was called) for Perl-related tests. The
> fundamental problem with TAP is that documented implementations
> (http://testanything.org/wiki/index.php/TAP_Consumers) require additional
> packages on the system such as Perl, Ruby, Javascript, Java, etc.
> I can not expect users who want to verify their build with 'make check'
> to install an additional language interpreter.
> 
> It seems that (while I was sleeping) Automake 1.12 will be providing support
> for TAP tests 
> (http://lists.gnu.org/archive/html/automake/2011-04/msg00069.html).
> Such support will not be worthwhile if it depends on Perl or some other
> interpreter since this might not be installed on the end-user's machine.
>
Well, automake being automake, it only depends on portable awk[1] ;-)
OK, portable "nawk" actually (as found by the autoconf builtin macro
AC_PROG_AWK), but I don't know of any non-museum machine lacking that.

 [1] Automake also offers an implementation of its TAP consumer that is
 written in perl and based on the TAP::Parser module; but you are
 absolutely not required to use or know it -- at the moment, it is
 not even documented IIRC.

> If it is based on portable C
> (e.g. http://www.eyrie.org/~eagle/software/c-tap-harness/)
> or portable shell script then it will be sufficiently usable.
>
> So what is the story about TAP in Automake 1.12?  Is it time to describe
> the new feature to us mere mortals (i.e. people on automake@gnu.org)
> rather than Automake developers?
>
The new feature is already documented in the automake manual on the
'master' branch; and this documentation should be release-quality
already.  But maybe I should make this new feature more visible before
the 1.12 release?  If yes, how do you suggest to do so?

As for the implementation quality, there's still an annoying glitch in
that you need to *manually* copy the automake-provided 'tap-driver.sh'
script yourself into your package[2] -- "automake --add-missing" won't
work -- and define some variables by hand; all of this is documented
in the manual, and should hopefully be fixed for automake 1.12.1.

 [2] The new automake option option '--print-libdir' should help
 mitigating this annoyance.

Regards,
  Stefano