Re: You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
On 3/8/07, Adam Kennedy <[EMAIL PROTECTED]> wrote: Andy Armstrong wrote: >> Otherwise when dealing with TAP streams that don't have a concept of >> an exit code or a seperate error channel, the most common example >> being web testing, we're left high and dry. > > In which case you'd just omit them, no? I didn't imagine they'd be > mandatory. Web testing doesn't have an error channel? HTTP has lot's of channels. They are called Status Codes. :) There are a whole slew of them for errors. Way more than you'd ever need for testing, or so I'm told. Peace, Hacker
Re: You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
On 9 Mar 2007, at 01:44, Adam Kennedy wrote: Like I said, we can't use them for anything that matters :) It's just occurred to me that it makes it easier to write tests for the harness too :) I can't quite get my head round the objection to the idea that a TAP transcript could be a complete, machine readable record of a test run. We're very nearly there anyway. -- Andy Armstrong, hexten.net
Re: You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
On 3/8/07, Andy Armstrong <[EMAIL PROTECTED]> wrote: I propose that we prefix lines from STDERR with '! ' in the same way that '# ' is used for diagnostics. wstat and exit can just be wstat 256 exit 1 How about this? wstat: 256 exit: 1 YAML, YAML, do! ;)
Re: You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
On 9 Mar 2007, at 00:28, Adam Kennedy wrote: I propose that we prefix lines from STDERR with '! ' in the same way that '# ' is used for diagnostics. wstat and exit can just be wstat 256 exit 1 The problem with STDERR and exit is that we can't actually use them for anything significant. We currently have the option of spooling parse TAP to a file during testing for later consumption. If we store STDERR + exit in that TAP it becomes possible to recreate the test run pretty accurately. Otherwise when dealing with TAP streams that don't have a concept of an exit code or a seperate error channel, the most common example being web testing, we're left high and dry. In which case you'd just omit them, no? I didn't imagine they'd be mandatory. -- Andy Armstrong, hexten.net
Re: You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
I propose that we prefix lines from STDERR with '! ' in the same way that '# ' is used for diagnostics. wstat and exit can just be wstat 256 exit 1 The problem with STDERR and exit is that we can't actually use them for anything significant. Otherwise when dealing with TAP streams that don't have a concept of an exit code or a seperate error channel, the most common example being web testing, we're left high and dry. That's always been the main reason to intentionally ignore the exit code at least. Adam K
Re: How would I use TAPx::Harness to run C++ TAP output?
Thanks for the help! For people searching the archives in the future, what I
had to do was set the HARNESS_PERL environment variable to "call" (in
windows, for an executable TAP written with libtap).
The following works as well:
===
use TAPx::Harness;
my $harness = TAPx::Harness->new( { exec => ['call']} );
$harness->runtests('SampleTest.exe');
===
Julien
On 3/8/07, Michael Peters <[EMAIL PROTECTED]> wrote:
Julien Beasley wrote:
> Just to clarify.. I'm fine with running Test::Harness instead of
> TAPx::Harness. One of the things that drew me to Test::Harness was
reading
> about TAP and how it's language agnostic. But putting in a non perl
> executable in runtests certainly doesn't work! I want to take advantage
of
> the languagage agnosticness of TAP.. whats the test harness to use that
> doesn't care if your test programs are compiled executables, and how do
you
> use it? Everything I've read in TAPX::Harness implies the use of an
> interpreter of some kind
This came up recently in a thread on Smolder where someone was trying to
use a
program that uses libtap written in C to do something similar.
http://sourceforge.net/mailarchive/forum.php?thread_id=31652415&forum_id=49450
The answer basically comes down to use "prove" which is a command line
utility
that makes it easy to use Test::Harness and is installed when
Test::Harness is
installed. You have to mess with some environment vars to make it not use
perl
on your executable though. Something like this:
HARNESS_PERL="" HARNESS_PERL_SWITCHES="" prove test.exe
I'm sure TAPx::Harness can make that much easier.
--
Michael Peters
Developer
Plus Three, LP
Re: You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
On 8 Mar 2007, at 22:47, Eric Hacker wrote: I propose that we prefix lines from STDERR with '! ' in the same way that '# ' is used for diagnostics. wstat and exit can just be wstat 256 exit 1 How about this? wstat: 256 exit: 1 YAML, YAML, do! Doesn't look like TAP though :) -- Andy Armstrong, hexten.net
Re: You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
On 8 Mar 2007, at 21:55, Michael G Schwern wrote: Michael G Schwern wrote: Stop. Stop stop stop! Stop right there. Umm, people might not realize that I tend to be a bit over dramatic and didn't actually mean to shut down the discussion. "He's not the Messiah. He's a very naughty boy". Oops, wrong language. Do we agree that the ability to capture STDERR (ordered correctly wrt STDOUT) and test script exit status are desirable? I propose that we prefix lines from STDERR with '! ' in the same way that '# ' is used for diagnostics. wstat and exit can just be wstat 256 exit 1 We also spoke about a hierarchical test numbering scheme to allow groups of tests to be nested. Test::Block and Test::Group already give a taste of how that might work. I'm +1 on that. Also tags. I haven't thought through how they might be used. Anyone got any ideas about that? -- Andy Armstrong, hexten.net
Re: You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
Michael G Schwern wrote: > Stop. Stop stop stop! Stop right there. Umm, people might not realize that I tend to be a bit over dramatic and didn't actually mean to shut down the discussion.
Re: How would I use TAPx::Harness to run C++ TAP output?
Julien Beasley wrote: > Just to clarify.. I'm fine with running Test::Harness instead of > TAPx::Harness. One of the things that drew me to Test::Harness was reading > about TAP and how it's language agnostic. But putting in a non perl > executable in runtests certainly doesn't work! I want to take advantage of > the languagage agnosticness of TAP.. whats the test harness to use that > doesn't care if your test programs are compiled executables, and how do you > use it? Everything I've read in TAPX::Harness implies the use of an > interpreter of some kind This came up recently in a thread on Smolder where someone was trying to use a program that uses libtap written in C to do something similar. http://sourceforge.net/mailarchive/forum.php?thread_id=31652415&forum_id=49450 The answer basically comes down to use "prove" which is a command line utility that makes it easy to use Test::Harness and is installed when Test::Harness is installed. You have to mess with some environment vars to make it not use perl on your executable though. Something like this: HARNESS_PERL="" HARNESS_PERL_SWITCHES="" prove test.exe I'm sure TAPx::Harness can make that much easier. -- Michael Peters Developer Plus Three, LP
Re: How would I use TAPx::Harness to run C++ TAP output?
Just to clarify.. I'm fine with running Test::Harness instead of TAPx::Harness. One of the things that drew me to Test::Harness was reading about TAP and how it's language agnostic. But putting in a non perl executable in runtests certainly doesn't work! I want to take advantage of the languagage agnosticness of TAP.. whats the test harness to use that doesn't care if your test programs are compiled executables, and how do you use it? Everything I've read in TAPX::Harness implies the use of an interpreter of some kind Julien On 3/8/07, Julien Beasley <[EMAIL PROTECTED]> wrote: I'm trying to get my project to move to TAP -- we have some perl test files and some C++ test files. Let's say I have 2 files: test.pl, and test.exe, the former is a perl program and the latter is a compiled program that outputs TAP. How do I use TAPx::Harness to run them both? I'm not really sure how to use the 'exec' switch for a program that doesn't require an interpreter :) Julien
How would I use TAPx::Harness to run C++ TAP output?
I'm trying to get my project to move to TAP -- we have some perl test files and some C++ test files. Let's say I have 2 files: test.pl, and test.exe, the former is a perl program and the latter is a compiled program that outputs TAP. How do I use TAPx::Harness to run them both? I'm not really sure how to use the 'exec' switch for a program that doesn't require an interpreter :) Julien
Re: Paying for TAP 2.0
In article <[EMAIL PROTECTED]>, Christopher H. Laco <[EMAIL PROTECTED]> wrote: > Paul Beckingham wrote: > > I'm wanting sparse output: > > > > 1..100 sparse > > 12 not ok > > 83 not ok > > > But how do you know "23 ok" if you were never told that it ran ok? For your sparse driver, it's just not emiting the "ok" messages. It's no different at the test level (or, doesn't have to be). This would be the same as filtering the current test output to get only the lines with "not" in it.
Re: Paying for TAP 2.0
--- Nicholas Clark <[EMAIL PROTECTED]> wrote: > > But how do you know "23 ok" if you were never told that it ran ok? > > Surely one can post-process a regular TAP file to "sparse" output? > And only do so if the TAP file is valid non-sparse output. > > This seems safer than generating it by default. Agreed. You can get this behavior with "runtests" by passing the '-f' switch (--failures) switch. Cheers, Ovid -- Buy the book -- http://www.oreilly.com/catalog/perlhks/ Perl and CGI -- http://users.easystreet.com/ovid/cgi_course/
Re: Paying for TAP 2.0
On 8 Mar 2007, at 16:34, Nicholas Clark wrote: But how do you know "23 ok" if you were never told that it ran ok? Surely one can post-process a regular TAP file to "sparse" output? And only do so if the TAP file is valid non-sparse output. Or post process it using gzip... -- Andy Armstrong, hexten.net
Re: Paying for TAP 2.0
On Thu, Mar 08, 2007 at 11:05:31AM -0500, Christopher H. Laco wrote: > Paul Beckingham wrote: > > Because I need to retain the output of all tests, and those files get > > large, but mostly because of the sheer redundancy. > > > > > > > > But how do you know "23 ok" if you were never told that it ran ok? Surely one can post-process a regular TAP file to "sparse" output? And only do so if the TAP file is valid non-sparse output. This seems safer than generating it by default. Nicholas Clark
Re: Paying for TAP 2.0
Paul Beckingham wrote: > I'm wanting sparse output: > > 1..100 sparse > 12 not ok > 83 not ok > > Which is three lines of output, instead of 97, but contains the same > information as: > > 1..100 > 1 ok > 2 ok > ... > 12 not ok > ... > 83 not ok > 84 ok > ... > 100 ok > > The essence of this is that with the plan of "1..100 sparse", missing > "ok N" lines are assumed to exist, if they are not specifically output. > > Because I need to retain the output of all tests, and those files get > large, but mostly because of the sheer redundancy. > > > But how do you know "23 ok" if you were never told that it ran ok? signature.asc Description: OpenPGP digital signature
Re: Paying for TAP 2.0
I'm wanting sparse output: 1..100 sparse 12 not ok 83 not ok Which is three lines of output, instead of 97, but contains the same information as: 1..100 1 ok 2 ok ... 12 not ok ... 83 not ok 84 ok ... 100 ok The essence of this is that with the plan of "1..100 sparse", missing "ok N" lines are assumed to exist, if they are not specifically output. Because I need to retain the output of all tests, and those files get large, but mostly because of the sheer redundancy.
Re: Minor Test::Class issues
--- Adrian Howard <[EMAIL PROTECTED]> wrote:
> In the mean time if you don't mind poking a private method you can
> do:
>
> INIT { Test::Class->runtests( Test::Class->_test_classes, +1 ) }
>
> If you do mind you could always copy'n'paste the code out into your
> subclass and rename appropriately. You'll just need _isa_class() and
> _test_classes().
That, by the way, has worked perfectly. Thanks!
Cheers,
Ovid
--
Buy the book -- http://www.oreilly.com/catalog/perlhks/
Perl and CGI -- http://users.easystreet.com/ovid/cgi_course/
Re: Minor Test::Class issues
On 8 Mar 2007, at 11:17, Ovid wrote:
[snip]
How do I run an individual test class which inherits from
"My::Test::Class" and have the INIT block in "My::Test::Class"
dynamically determine which test class name to put there? That's
what's stumping me.
[snip]
Damn. I knew that asymmetric treatment of runtests and expected_tests
would come and bite me in the end. I'll have to ponder a decent fix.
Added to the to do list.
In the mean time if you don't mind poking a private method you can do:
INIT { Test::Class->runtests( Test::Class->_test_classes, +1 ) }
If you do mind you could always copy'n'paste the code out into your
subclass and rename appropriately. You'll just need _isa_class() and
_test_classes().
Cheers,
Adrian
I am not on perl-qa any more
If you have anything to ask about testing stuff, or Test::Harness, or any of my Test:: modules, please mail me directly. Andy -- Andy Lester => [EMAIL PROTECTED] => www.petdance.com => AIM:petdance
Re: Customizing Your Test Modules
Ovid wrote:
> Also at http://use.perl.org/~Ovid/journal/32614
>
> I get tired of writing this all the time:
>
> #!/usr/bin/perl
>
> use Test::More tests => 23;
> use Test::Exception;
> use Test::Differences;
> use Test::NoWarnings;
>
> ...
>
> I write that *a lot*. No more. This does the same thing:
>
> #!/usr/bin/perl
>
> use My::Test::More tests => 23;
>
> And the code:
>
> package My::Test::More;
>
> use Test::Builder::Module;
> @ISA = qw(Test::Builder::Module);
> use Test::More;
> use Test::Differences;
> use Test::Exception;
>
> INIT {
> unless ( exists $INC{'Test/Class.pm'} ) {
>
> # NoWarnings doesn't play well with Test::Class
> eval "use Test::NoWarnings";
> }
> }
>
> @EXPORT = (
> @Test::More::EXPORT,
> @Test::Differences::EXPORT,
> @Test::Exception::EXPORT,
> );
>
> $ENV{RUNNING_TESTS} = 1;
>
> 1;
>
> Cheers,
> Ovid
>
> --
>
> Buy the book -- http://www.oreilly.com/catalog/perlhks/
> Perl and CGI -- http://users.easystreet.com/ovid/cgi_course/
>
>
I think it was you or someone else who got me started on this in Handel
and my other DBIC based dists where I needed both Test::More and custom
sqlite t/var initializer junk:
http://search.cpan.org/src/CLACO/DBIx-Class-InflateColumn-Currency-0.01/t/lib/DBIC/Test.pm
Yup, in fact, your still in the comments... :-)
> BEGIN {
> # little trick by Ovid to pretend to subclass+exporter Test::More
> use base qw/Test::Builder::Module Class::Accessor::Grouped/;
> use Test::More;
> use File::Spec::Functions qw/catfile catdir/;
>
> @DBIC::Test::EXPORT = @Test::More::EXPORT;
>
> __PACKAGE__->mk_group_accessors('inherited', qw/db_dir db_file/);
> };
signature.asc
Description: OpenPGP digital signature
Re: per-author META.yml
On 3/7/07, Ken Williams <[EMAIL PROTECTED]> wrote: In the specific instance we're talking about, cc_author, I wouldn't want it in any such file, whether per-author or per-distribution; I'd want it as a preference I can set within the cpan-testers system. Because if I change my mind and decide I'd rather see reports via RSS and not email, I wouldn't want to have to upload a new file somewhere. "within the cpan-testers system"? Since the CPAN Testers system is cobbled together around a mailing list, I think the only central place would be somewhere on PAUSE. David
Customizing Your Test Modules
Also at http://use.perl.org/~Ovid/journal/32614
I get tired of writing this all the time:
#!/usr/bin/perl
use Test::More tests => 23;
use Test::Exception;
use Test::Differences;
use Test::NoWarnings;
...
I write that *a lot*. No more. This does the same thing:
#!/usr/bin/perl
use My::Test::More tests => 23;
And the code:
package My::Test::More;
use Test::Builder::Module;
@ISA = qw(Test::Builder::Module);
use Test::More;
use Test::Differences;
use Test::Exception;
INIT {
unless ( exists $INC{'Test/Class.pm'} ) {
# NoWarnings doesn't play well with Test::Class
eval "use Test::NoWarnings";
}
}
@EXPORT = (
@Test::More::EXPORT,
@Test::Differences::EXPORT,
@Test::Exception::EXPORT,
);
$ENV{RUNNING_TESTS} = 1;
1;
Cheers,
Ovid
--
Buy the book -- http://www.oreilly.com/catalog/perlhks/
Perl and CGI -- http://users.easystreet.com/ovid/cgi_course/
Re: Minor Test::Class issues
--- Adrian Howard <[EMAIL PROTECTED]> wrote:
>
> On 8 Mar 2007, at 10:44, Ovid wrote:
> [snip]
> > I cannot easily use 'Test::NoWarnings' because that adds an extra
> > test,
> > thus forcing me to use 'no_plan' with one of my tests. Is there
> some
> > other way around this?
> [snip]
>
> Do something like:
>
> INIT { Test::Class->runtests( 'Foo', +1 ) }
How do I run an individual test class which inherits from
"My::Test::Class" and have the INIT block in "My::Test::Class"
dynamically determine which test class name to put there? That's
what's stumping me.
> > Also, not that when not using test names, the setup and teardown
> tests
> > are getting the names of the method they're testing, thus obscuring
>
> > tht
> > we're in setup and teardown.
>
> Good point. My logic was that the setup and teardown tests are
> logically part of the test so should have their name. I guess it
> would make sense to annotate them with the setup/teardown method
> name. Added to the todo list.
Thanks!
Cheers,
Ovid
--
Buy the book -- http://www.oreilly.com/catalog/perlhks/
Perl and CGI -- http://users.easystreet.com/ovid/cgi_course/
Re: Minor Test::Class issues
On 8 Mar 2007, at 10:44, Ovid wrote:
[snip]
I cannot easily use 'Test::NoWarnings' because that adds an extra
test,
thus forcing me to use 'no_plan' with one of my tests. Is there some
other way around this?
[snip]
Do something like:
INIT { Test::Class->runtests( 'Foo', +1 ) }
Also, not that when not using test names, the setup and teardown tests
are getting the names of the method they're testing, thus obscuring
tht
we're in setup and teardown.
Good point. My logic was that the setup and teardown tests are
logically part of the test so should have their name. I guess it
would make sense to annotate them with the setup/teardown method
name. Added to the todo list.
Cheers,
Adrian
Re: Should TAP capture exit codes
Reading through this I can't help but thing I've seen it all before. The PITA test result code conversation, if you'll recall. But when that cloned something akin to HTTP codes it was because there looked to be only 15-20 total possible results. And it's not meant to be extensible the way TAP is. And even then I'm still looking at moving to string codes. Adam K Andy Armstrong wrote: Ovid's post about TAP::Tests has reminded me: would it be useful to have a TAP statement that conveys the exit code of a test script? At the moment in a hypothetical situation where there's some distance between the harness and the test script - like perhaps they're on different machines - the test script's exit status has to be sent out-of-band. If you add exit to the grammar then the TAP transcript becomes a complete log of the test. At the moment in TAP::Harness we have the option of spooling TAP to a file - which captures everything useful /apart/ from exit status. --Andy Armstrong, hexten.net
Minor Test::Class issues
So here's a minimal test class program (v0.24):
package Foo;
use Test::More;
use Test::NoWarnings;
use base 'Test::Class';
INIT { Test::Class->runtests }
sub startup : Tests(startup => 1) { ok 1 }
sub setup : Tests(setup => 1){ ok 1 }
sub some_test : Tests(1) { ok 1 }
sub teardown : Tests(teardown => 1) { ok 1 }
sub shutdown : Tests(shutdown => 1) { ok 1 }
1;
That has a couple of issues. First, the output with 'runtests -v
foo.t':
foo..1..5
ok 1
ok 2 - some test
ok 3 - some test
ok 4 - some test
ok 5
ok 6 - no warnings
# Looks like you planned 5 tests but ran 1 extra.
Dubious, test returned 255 (wstat 65280, 0xff00)
All 5 subtests passed
Test Summary Report
---
foo.t (Wstat: 65280 Tests: 6 Failed: 1)
Failed tests: 6
Non-zero exit status: 255
Parse errors: Bad plan. You planned 5 tests but ran 6.
Files=1, Tests=6, 0 wallclock secs ( 0.10 cusr + 0.02 csys = 0.12
CPU)
I cannot easily use 'Test::NoWarnings' because that adds an extra test,
thus forcing me to use 'no_plan' with one of my tests. Is there some
other way around this?
Also, not that when not using test names, the setup and teardown tests
are getting the names of the method they're testing, thus obscuring tht
we're in setup and teardown.
And just to be fair, here's the output when run through 'prove':
$ prove -v foo.t
foo1..5
ok 1
#
# Foo->some_test
ok 2 - some test
ok 3 - some test
ok 4 - some test
ok 5
ok 6 - no warnings
# Looks like you planned 5 tests but ran 1 extra.
dubious
Test returned status 255 (wstat 65280, 0xff00)
DIED. FAILED test 6
Failed 1/5 tests, 80.00% okay
Failed Test Stat Wstat Total Fail List of Failed
---
foo.t255 65280 51 6
Failed 1/1 test scripts. -1/5 subtests failed.
Files=1, Tests=5, 1 wallclock secs ( 0.11 cusr + 0.03 csys = 0.14
CPU)
Failed 1/1 test programs. -1/5 subtests failed.
I note that '# Foo->some_test' isn't showing up with 'runtests', so
I'll have to look into that. but other than that, we have the same
issues.
Cheers,
Ovid
--
Buy the book -- http://www.oreilly.com/catalog/perlhks/
Perl and CGI -- http://users.easystreet.com/ovid/cgi_course/
