Re: nested groups in TAP

[Fergal Daly]

On 12/03/07, Michael G Schwern <[EMAIL PROTECTED]> wrote:

Fergal Daly wrote:
> Remember, the TAP producer is not the instigator of the conversation.
> The consumer is - it runs the scripts, it requests something to
> produce TAP. The consumer is the the client.

Oh but that's not necessarily true.  As pointed out there are plenty of
situations where A) the TAP producer has no idea who its producing for and
B) the resulting TAP can be read by several different parsers and C) there
may be a long gap between producing and parsing.

* Writing out to a file for later examination
* Writing out to a log file (the Test::AtRuntime concept)
* Sending via an email

Consider the case where you have a nightly smoke server.  The smoker server
dumps the TAP to a log file and archives it for later examination as part of
the history of the product.  Or maybe the results go out to a mailing list.

It is, after all, the Test Anything Protocol.


That's fine and if it produces TAP that the consumer can't read then
whoever set up this situation has work todo. I'm just trying to help
the situation where joe user is trying to get a module installed and
finds he has to update his Test::Harness when maybe he doesn't want
to.

I have no problem telling people you must keep your Test::Harness up
to date with your Test::Builder. I'm just quite surprised that so many
people are so vehemently _for_ forcing people to do this,

F

Re: nested groups in TAP

[Michael G Schwern]

Fergal Daly wrote:
> Remember, the TAP producer is not the instigator of the conversation.
> The consumer is - it runs the scripts, it requests something to
> produce TAP. The consumer is the the client.

Oh but that's not necessarily true.  As pointed out there are plenty of
situations where A) the TAP producer has no idea who its producing for and
B) the resulting TAP can be read by several different parsers and C) there
may be a long gap between producing and parsing.

* Writing out to a file for later examination
* Writing out to a log file (the Test::AtRuntime concept)
* Sending via an email

Consider the case where you have a nightly smoke server.  The smoker server
dumps the TAP to a log file and archives it for later examination as part of
the history of the product.  Or maybe the results go out to a mailing list.

It is, after all, the Test Anything Protocol.

Re: nested groups in TAP

[Fergal Daly]

On 12/03/07, Smylers <[EMAIL PROTECTED]> wrote:

Fergal Daly writes:

> On 12/03/07, chromatic <[EMAIL PROTECTED]> wrote:
>
> > On Monday 12 March 2007 10:53, A. Pagaltzis wrote:
> >
> > > Making "either the consumer talks to the producer and the
> > > producer has to implement some complex logic, or the producer
> > > sticks to the plainest TAP possible" a requirement would just
> > > lead to most ad-hoc implementations of producers being forever
> > > stuck with producing the plainest TAP possible.
> >
> > +1, has the historical support in the example of, oh, just about every
> > standard Internet protocol ever devised.
>
> What?
>
> GET / HTTP/1.1
>
> there it is right there, the consumer saying "I'd like to speak this
> version, can you handle it?"

No it isn't.  It's the instigator of the converstation saything "I am
going to speak version 1.1; deal with it".


Where "deal with it" means respond with result or indicate that you
can't deal with it (error 501).


Which seems the right thing to do with Tap as well.  Each Tap stream
merely needs to be identified by the number a version with which it is
compatible (ideally the lowest such version, but it'll still work even
if not).


Remember, the TAP producer is not the instigator of the conversation.
The consumer is - it runs the scripts, it requests something to
produce TAP. The consumer is the the client.


Then something reading that stream knows that if it supports the version
number in question it will be able to understand the stream, and that
otherwise it should give up.


If you're perfectly happy to break all previous consumers with every
increment of the version number then fine. I'm just pointing out how
that can be avoided with not much work at all,

F


Smylers

Re: nested groups in TAP

[Smylers]

Fergal Daly writes:

> On 12/03/07, chromatic <[EMAIL PROTECTED]> wrote:
> 
> > On Monday 12 March 2007 10:53, A. Pagaltzis wrote:
> > 
> > > Making "either the consumer talks to the producer and the
> > > producer has to implement some complex logic, or the producer
> > > sticks to the plainest TAP possible" a requirement would just
> > > lead to most ad-hoc implementations of producers being forever
> > > stuck with producing the plainest TAP possible.
> > 
> > +1, has the historical support in the example of, oh, just about every
> > standard Internet protocol ever devised.
> 
> What?
> 
> GET / HTTP/1.1
> 
> there it is right there, the consumer saying "I'd like to speak this
> version, can you handle it?"

No it isn't.  It's the instigator of the converstation saything "I am
going to speak version 1.1; deal with it".

Which seems the right thing to do with Tap as well.  Each Tap stream
merely needs to be identified by the number a version with which it is
compatible (ideally the lowest such version, but it'll still work even
if not).

Then something reading that stream knows that if it supports the version
number in question it will be able to understand the stream, and that
otherwise it should give up.

Smylers

Re: nested groups in TAP

[Fergal Daly]

On 12/03/07, chromatic <[EMAIL PROTECTED]> wrote:

On Monday 12 March 2007 10:53, A. Pagaltzis wrote:

> Making "either the consumer talks to the producer and the
> producer has to implement some complex logic, or the producer
> sticks to the plainest TAP possible" a requirement would just
> lead to most ad-hoc implementations of producers being forever
> stuck with producing the plainest TAP possible.

+1, has the historical support in the example of, oh, just about every
standard Internet protocol ever devised.


What?

GET / HTTP/1.1

there it is right there, the consumer saying "I'd like to speak this
version, can you handle it?"

EHLO is SMTP

Probably the 2 most important internet protocols ever (once you
actually have a network),

F

Re: nested groups in TAP

[Fergal Daly]

On 12/03/07, A. Pagaltzis <[EMAIL PROTECTED]> wrote:

Hi Fergal,

* Fergal Daly <[EMAIL PROTECTED]> [2007-03-12 18:00]:
> On 12/03/07, A. Pagaltzis <[EMAIL PROTECTED]> wrote:
> >How do you set environment variables when the producer lives
> >at the other end of an HTTP connection? How do you set
> >environment variables when you've pipe the producer's output
> >into a textfile and now you're run a consumer against that,
> >two weeks later?
>
> It's not a HTTP connection it's a HTTP request an important
> distinction because a request means that it is initiated by the
> side that will end up processing it which can therefore pass
> information as part of the request.

So I have to make up an ad-hoc way to communicate with every
possible producer?

> >Requiring bidirectional communication will greatly reduce the
> >number of scenarios in which TAP can be used easily. And if
> >you require coupling between producer and consumer to the
> >point of needing in- or inter-process communication of some
> >sort, then what is the advantage of having a protocol vs an
> >API like JUnit?
>
> Nope, without communication you simply fall back to emitting
> plain old TAP.

Why?

And why should I have to write code that tries to divine what
TAP version to produce and then conditionally produce some
version of TAP other than the highest I know, when the code
emitting TAP is machine code running on a microcontroller? Isn't
it more reasonable to expect the consumer to be complex, rather
than the producer? I'd rather reluctantly miss any useful
features from newer TAP versions and just produce plain old TAP,
regardless of what's at the the other end of the wire, than
having to go through all those gyrations.


You can make the producer as simple as you like, as long as you don't
care about both of

a features
b compatibility

you can't have both because TAP was not designed in such a way that
you cannot add a feature to it without either breaking old parser or
turning what used to be for-human fields into for-consumer fields.


> >So it makes sense to take the unproven risk of some moderate
> >future burden on consumers in exchange for retaining the
> >advantages of a protocol being a protocol and not an API.
>
> Burden on consumers? There's no burden on consumers because
> what you are saying is that TAP will forever be compatible with
> today's TAP. In that world, consumers have it easy, it's the
> producers who have to jump through hoops trying to find ways to
> add functionality without upsetting 20 year old parsers.

No, I'm saying it's reasonable to expect future consumers to
support all versions of TAP ever. In fact I'm saying it's
reasonable to offload almost all complexity to the consumer. The
only thing we need is for producers to be able to declare what
version their output conforms to. If the consumer can't cope with
that version, it can just tell the user and bail out.


Well in that case, you're saying there's no need to remain compatible.
If the consumer can't handle it we bail. The problem I have with that
is that every time we increment the version number, everyone has to
upgrade their consumer, even if they don't really need to.


Making "either the consumer talks to the producer and the
producer has to implement some complex logic, or the producer
sticks to the plainest TAP possible" a requirement would just
lead to most ad-hoc implementations of producers being forever
stuck with producing the plainest TAP possible.


There's as much or as little logic as you like. If you're a producer
and you could only be bothered supporting version 72 then you just say
that and bail or go ahead and ignore any info provided by the
consumer, output version 72 and cause the consumer to bail. What I'm
describing completely contains what you're describing as a subset but
it also offers some flexibility during transitions whereby the
producer can accommodate older consumers without explosions.



Regards,
--
Aristotle Pagaltzis // 

Re: nested groups in TAP

[chromatic]

On Monday 12 March 2007 10:53, A. Pagaltzis wrote:

> Making “either the consumer talks to the producer and the
> producer has to implement some complex logic, or the producer
> sticks to the plainest TAP possible” a requirement would just
> lead to most ad-hoc implementations of producers being forever
> stuck with producing the plainest TAP possible.

+1, has the historical support in the example of, oh, just about every 
standard Internet protocol ever devised.

-- c

Re: nested groups in TAP

[A. Pagaltzis]

Hi Fergal,

* Fergal Daly <[EMAIL PROTECTED]> [2007-03-12 18:00]:
> On 12/03/07, A. Pagaltzis <[EMAIL PROTECTED]> wrote:
> >How do you set environment variables when the producer lives
> >at the other end of an HTTP connection? How do you set
> >environment variables when you've pipe the producer's output
> >into a textfile and now you're run a consumer against that,
> >two weeks later?
> 
> It's not a HTTP connection it's a HTTP request an important
> distinction because a request means that it is initiated by the
> side that will end up processing it which can therefore pass
> information as part of the request.

So I have to make up an ad-hoc way to communicate with every
possible producer?

> >Requiring bidirectional communication will greatly reduce the
> >number of scenarios in which TAP can be used easily. And if
> >you require coupling between producer and consumer to the
> >point of needing in- or inter-process communication of some
> >sort, then what is the advantage of having a protocol vs an
> >API like JUnit?
> 
> Nope, without communication you simply fall back to emitting
> plain old TAP.

Why?

And why should I have to write code that tries to divine what
TAP version to produce and then conditionally produce some
version of TAP other than the highest I know, when the code
emitting TAP is machine code running on a microcontroller? Isn’t
it more reasonable to expect the consumer to be complex, rather
than the producer? I’d rather reluctantly miss any useful
features from newer TAP versions and just produce plain old TAP,
regardless of what’s at the the other end of the wire, than
having to go through all those gyrations.

> >So it makes sense to take the unproven risk of some moderate
> >future burden on consumers in exchange for retaining the
> >advantages of a protocol being a protocol and not an API.
> 
> Burden on consumers? There's no burden on consumers because
> what you are saying is that TAP will forever be compatible with
> today's TAP. In that world, consumers have it easy, it's the
> producers who have to jump through hoops trying to find ways to
> add functionality without upsetting 20 year old parsers.

No, I’m saying it’s reasonable to expect future consumers to
support all versions of TAP ever. In fact I’m saying it’s
reasonable to offload almost all complexity to the consumer. The
only thing we need is for producers to be able to declare what
version their output conforms to. If the consumer can’t cope with
that version, it can just tell the user and bail out.

Making “either the consumer talks to the producer and the
producer has to implement some complex logic, or the producer
sticks to the plainest TAP possible” a requirement would just
lead to most ad-hoc implementations of producers being forever
stuck with producing the plainest TAP possible.

Regards,
-- 
Aristotle Pagaltzis // 

Re: nested groups in TAP

[Fergal Daly]

On 12/03/07, A. Pagaltzis <[EMAIL PROTECTED]> wrote:

* Fergal Daly <[EMAIL PROTECTED]> [2007-03-12 00:45]:
> This brings up something else. Future harnesses should probably
> set
>
> I_UNDERSTAND_TAP_VERSIONS="1,5,8,10-22"
>
> so we know what we can output. If that's not set then we need
> to stick to plainest TAP.

No, I don't think so.

How do you set environment variables when the producer lives at
the other end of an HTTP connection? How do you set environment
variables when you've pipe the producer's output into a textfile
and now you're run a consumer against that, two weeks later?


It's not a HTTP connection it's a HTTP request an important
distinction because a request means that it is initiated by the side
that will end up processing it which can therefore pass information as
part of the request.


Requiring bidirectional communication will greatly reduce the
number of scenarios in which TAP can be used easily. And if you
require coupling between producer and consumer to the point of
needing in- or inter-process communication of some sort, then
what is the advantage of having a protocol vs an API like JUnit?


Nope, without communication you simply fall back to emitting plain old
TAP. You might lose functionality but it's functionality that was
probably only important to a developer. If you're a user installing
the module then it either passes all tests or it doesn't. If you want
to run the tests the way the developer intended (for example to
produce a helpful bug report) you can go upgrade your stuff.


For all this, I'm unconvinced of any real gains.

TAP is supposed to be so simple that you can Test Anything with
it because *anything* could be wired up to produce TAP. That
should mean TAP will not get significantly more complex than it
currently is; I'll be dismayed if I find that TAP in 2020 looks
much different from TAP today. It should also mean that even if
consumers end up having to support multiple versions of TAP, it
won't be too difficult. And lastly it should mean that at any
point there will be many fewer consumer implementations than
producer implementations.

So it makes sense to take the unproven risk of some moderate
future burden on consumers in exchange for retaining the
advantages of a protocol being a protocol and not an API.


Burden on consumers? There's no burden on consumers because what you
are saying is that TAP will forever be compatible with today's TAP. In
that world, consumers have it easy, it's the producers who have to
jump through hoops trying to find ways to add functionality without
upsetting 20 year old parsers.

F


Regards,
--
Aristotle Pagaltzis // 

Re: nested groups in TAP

[A. Pagaltzis]

* Fergal Daly <[EMAIL PROTECTED]> [2007-03-12 00:45]:
> This brings up something else. Future harnesses should probably
> set
> 
> I_UNDERSTAND_TAP_VERSIONS="1,5,8,10-22"
> 
> so we know what we can output. If that's not set then we need
> to stick to plainest TAP.

No, I don’t think so.

How do you set environment variables when the producer lives at
the other end of an HTTP connection? How do you set environment
variables when you’ve pipe the producer’s output into a textfile
and now you’re run a consumer against that, two weeks later?

Requiring bidirectional communication will greatly reduce the
number of scenarios in which TAP can be used easily. And if you
require coupling between producer and consumer to the point of
needing in- or inter-process communication of some sort, then
what is the advantage of having a protocol vs an API like JUnit?

For all this, I’m unconvinced of any real gains.

TAP is supposed to be so simple that you can Test Anything with
it because *anything* could be wired up to produce TAP. That
should mean TAP will not get significantly more complex than it
currently is; I’ll be dismayed if I find that TAP in 2020 looks
much different from TAP today. It should also mean that even if
consumers end up having to support multiple versions of TAP, it
won’t be too difficult. And lastly it should mean that at any
point there will be many fewer consumer implementations than
producer implementations.

So it makes sense to take the unproven risk of some moderate
future burden on consumers in exchange for retaining the
advantages of a protocol being a protocol and not an API.

Regards,
-- 
Aristotle Pagaltzis // 

Re: nested groups in TAP

[Mark Fowler]

On 11 Mar 2007, at 22:01, Michael G Schwern wrote:


Fergal Daly wrote:

You're suggesting that each call to extend the plan verifies that the
previous plan has been executed fully? That does not allow nesting.


Yes, nesting is what this proposal does which cannot be done now.

Unfortuantely the other thing it does well is shatter backwards
compatibility so I'd like it to do something significantly more.


If I remember correctly from our discussion at YAPC::NA::Chicago we  
suggested something like this for nesting and grouping:


1..10
1 ok - testing foo
2 ok - testing bar
..2 - grok tests
3 ok - first grok
4 ok - second grok
..end
..3 - flirble tests
5 ok - first flirble
...2 - flirble burble tests
6 ok - second flirble, first flirble burble
7 ok - third flirble, second flirble burble
...end
..end
8 ok - testing
9 ok

Please forgive me if you've discussed this on list since (I've not  
been following too closely) - if you have, this needs adding to the  
wiki.


The main reasons we liked this was that it didn't break backwards  
compatibility.  The groups would simply be ignored by existing TAP  
parsers.


Thoughts?

Mark.

Re: nested groups in TAP

[Andy Armstrong]

On 12 Mar 2007, at 13:12, Fergal Daly wrote:

I don't want every producer written from here forward to have to be
able to negotiate its way through a minefield of supported versions.


But you do want every consumer written from here forward to support
every previous version.


On the assumption that there will be fewer consumer implementations  
that producer implementations at any point in time that's where the  
burden should lie, yes. And it's OK to say "I don't know what to do  
with this" and skip the test.



It transfers a lot of burden to the producer though. It means that
every producer from now on has parse a discontinuous selection of
version numbers and then do the right thing. The negotiation stage
ends up being more onerous than the actual testing.


my $me = string_to_set($i_support);
my $he = string_to_set($harness_supports);
my $version_to_use = [sort intersect($me, $he)]->[-1];


Yes, that's fine. Then you have to actually generate TAP to whatever  
version you find in common.


[snip]

In fact, I'd even go so far as to say that an upper and lower version
are probably enough,


Can't we just keep it simple and say the harness barfs if the TAP is  
too new for it? I don't like the idea of tests automatically running  
in a degraded mode.


This isn't HTTP where the various actors need to make their best  
endeavour to communicate. We really want to be certain that nothing  
is lost in translation between the producer and the harness. If you  
have the possibility of adaptive behaviour on both sides of that  
conversation that risk goes up considerably.


--
Andy Armstrong, hexten.net

Re: nested groups in TAP

[Ovid]

--- Fergal Daly <[EMAIL PROTECTED]> wrote:

> What you're saying is we can _never_ break backwards comatibility.
> That is not the impression I got from previous discussions.

Sorry if I gave that impression.  I don't mean 'never'.  I mean we have
to have a huge amount of bang for our buck to justify this.  We also
have to have some level of consensus on how to approach something like
that.  Breaking backwards compatibility is an option, but one which
should be explored very, very carefully.
 
> What I'm saying is, this break backwards compatibility but the
> producer can fall back to plain ol TAP if it knows the consumer can't
> handle it.

I can easily envision scenarios where the producer would have no idea
who the consumer is.  I would also suggest that the producer should
never know or care who the consumer is, though I can easily understand
counter-arguments.
 
> > There are several ways we could accomplish the same thing without
> break
> > backwards compatibility.  Let's explore those.
> 
> Show me one that actually does what I'm looking for,

That could be tough, but I'm unconvinced that nothing else would work. 
  Here's one which relies on the fact that the "ok/not ok" status must
not have leading whitespace:

  1..3
  ok 1 
   2 1..2 # start a block
   2.1 1..3 # start another block
   ok 2.1.1
   ok 2.1.2
   ok 2.1.3
   ok 2.1 # inner block was all good
   ok 2.2
  ok 2 # outer block was all good
   3 1..3
   ok 3.1
   ok 3.2
  not ok 3 # planned for 3 but only ran 2 tests

Both 'prove' and 'runtests' parse this correctly.

Cheers,
Ovid

--

Buy the book -- http://www.oreilly.com/catalog/perlhks/
Perl and CGI -- http://users.easystreet.com/ovid/cgi_course/

Re: nested groups in TAP

[Fergal Daly]

On 12/03/07, Andy Armstrong <[EMAIL PROTECTED]> wrote:

On 12 Mar 2007, at 12:15, Fergal Daly wrote:
>> Highest version I'd say. I hope we're not expecting such a complex
>> melange of versions that we need to enumerate the ones we support. If
>> say we're supporting version 10 it's implicit that we also support
>> 1-9.
>
> Are we not anticipate any bad decisions either? Say version 8 was
> horribly broken and no one should ever support it. By saying you don't
> support it the producer should then fall back gracefully to something
> earlier.

If 8 is horribly broken why's the producer trying to produce it in
the first place? How likely is it that we're going to allow a
'horribly broken' TAP grammer into the wild?

> What about the future? Maybe it's 20 years later and I'm writng a TAP
> parser in C#++ and I don't want to have to cater for every version of
> TAP that's ever existed, including the broken ones. So I say
> "12,90-100" the producer can either produce up to date TAP or fall
> back to the very basics.

Madness :)

I don't want every producer written from here forward to have to be
able to negotiate its way through a minefield of supported versions.


But you do want every consumer written from here forward to support
every previous version.


 From the producer's point of view the logic should be as simple as
checking that the harness supports our version. If it doesn't emit a
diagnostic and skip.

> If you insist on full backwards compatibility all the way back to the
> start then you future TAP consumers are just going to lie. They'll say
> "100" which is supposed to mean "1-100" but they're just going to hope
> that nothing earlier than 90 ever shows up. If you're lucky they'll
> die when they see a version number they don't support but by
> communicating the reality the user could get the best available.
>
> Plus saying "1-100" is only a tiny bit more difficult than saying
> "100"

It transfers a lot of burden to the producer though. It means that
every producer from now on has parse a discontinuous selection of
version numbers and then do the right thing. The negotiation stage
ends up being more onerous than the actual testing.


my $me = string_to_set($i_support);
my $he = string_to_set($harness_supports);
my $version_to_use = [sort intersect($me, $he)]->[-1];

I don't envision a world where

$i_support = "1,7,34,36-43,78-91"
$harness_supports = "2,8,34,44-57,92-100"

I just think it should be possible in the future to say

$i_support = "78-91"
$harness_supports = "86-100"

In fact, I'd even go so far as to say that an upper and lower version
are probably enough,

F


--
Andy Armstrong, hexten.net

Re: nested groups in TAP

[Fergal Daly]

On 12/03/07, Michael G Schwern <[EMAIL PROTECTED]> wrote:

Michael G Schwern wrote:
> Subroutines?  I don't know if I follow.  Do you mean...
>
> sub foo {
>   extend(2);
>
>   pass();
>   bar();
>   pass();
> }
>
> sub bar {
>   extend(3);
>   pass();
>   pass();
>   pass();
> }
>
> I believe that can be made to work without a TAP extension.  Its the
> producer which does the stack watching and generates the errors instead of
> the TAP parser.

As a matter of fact, thinking about it more, this sort of thing has to be
able to work in the TAP producer.  How else will it know what group its
currently in to output the appropriate TAP?


Yes, the producer must track the stack but now you're also forcing it
to verify the plan which is consumer's job. I think that's acceptable
when we've fallen back to be compatible with an old consumer.

Here's another proposal that I think might work. The only plan that
really needs to be checked by the consumer is the overall plan. Within
the producer, we can consider a containing block to be the consumer
and be responsible for checking the plan without any loss of
strictness modulo bugs in the producer (not the test script but the
TAP producing library). So all that's really I need is a way to say
that some tests should not count towards the overall plan but they are
real tests that must not fail either. Going back to the original
example

1..3
1 ok
2 1..2 # start a block
2.1 1..3 # start another block
2.1.1 ok
2.1.2 ok
2.1.3 ok
2.1 ok # inner block was all good
2.2 ok
2 ok # outer block was all good
3 1..3
3.1 ok
3.2 ok
3 not ok # planned for 3 but only ran 2 tests

I care that everything is "ok" but from a plan point of view, I only
need Test::Harness to check that I gave a result for

1 ok
2 ok # outer block was all good
3 not ok # planned for 3 but only ran 2 tests

It should tell me if 2.1.2 said "not ok" but it should not count it
towards the plan.

I suppose this could be hacked up by doing the extends _after_ each
outer block finishes, although that's no good for the fork/thread use
case.

F

Re: nested groups in TAP

[Andy Armstrong]

On 12 Mar 2007, at 12:27, Andy Armstrong wrote:
I don't want every producer written from here forward to have to be  
able to negotiate its way through a minefield of supported  
versions. From the producer's point of view the logic should be as  
simple as checking that the harness supports our version. If it  
doesn't emit a diagnostic and skip.


Actually I'm not even sure if I like the idea of the harness telling  
the producer what it can support. I'd say that if you want to produce  
TAP version 53 you have to have a harness that supports it. If the  
harness sees a version later than those it knows about it should  
display a warning and skip those tests completely.


Anything else takes us down the road of having a load of hard-to- 
maintain logic floating around in both producers and harnesses to  
find the highest common version and then attempt to work with it.  
Tests that can't reliably do what they want to do should fail - not  
mess about trying to work in a crippled mode.


--
Andy Armstrong, hexten.net

Re: nested groups in TAP

[Andy Armstrong]

On 12 Mar 2007, at 12:15, Fergal Daly wrote:

Highest version I'd say. I hope we're not expecting such a complex
melange of versions that we need to enumerate the ones we support. If
say we're supporting version 10 it's implicit that we also support  
1-9.


Are we not anticipate any bad decisions either? Say version 8 was
horribly broken and no one should ever support it. By saying you don't
support it the producer should then fall back gracefully to something
earlier.


If 8 is horribly broken why's the producer trying to produce it in  
the first place? How likely is it that we're going to allow a  
'horribly broken' TAP grammer into the wild?



What about the future? Maybe it's 20 years later and I'm writng a TAP
parser in C#++ and I don't want to have to cater for every version of
TAP that's ever existed, including the broken ones. So I say
"12,90-100" the producer can either produce up to date TAP or fall
back to the very basics.


Madness :)

I don't want every producer written from here forward to have to be  
able to negotiate its way through a minefield of supported versions.  
From the producer's point of view the logic should be as simple as  
checking that the harness supports our version. If it doesn't emit a  
diagnostic and skip.



If you insist on full backwards compatibility all the way back to the
start then you future TAP consumers are just going to lie. They'll say
"100" which is supposed to mean "1-100" but they're just going to hope
that nothing earlier than 90 ever shows up. If you're lucky they'll
die when they see a version number they don't support but by
communicating the reality the user could get the best available.

Plus saying "1-100" is only a tiny bit more difficult than saying  
"100"


It transfers a lot of burden to the producer though. It means that  
every producer from now on has parse a discontinuous selection of  
version numbers and then do the right thing. The negotiation stage  
ends up being more onerous than the actual testing.


--
Andy Armstrong, hexten.net

Re: nested groups in TAP

[Fergal Daly]

On 12/03/07, Ovid <[EMAIL PROTECTED]> wrote:

--- Michael G Schwern <[EMAIL PROTECTED]> wrote:

> > How does it do this? I understand that an old harness will not be
> able
> > to parse it - it will be upset by the sub-plans and the dotted test
> > numbers)
>
> Yep.  By "backwards compatible" I mean you can feed the new TAP to an
> old
> harness and it will still be able to get the basic pass/fail
> information out
> of it.
>
> > but if we know we're running under an old harness then we can
> > just fall back to plan-at-the-end and diagnostics.
>
> Currently old harnesses do not identify what version of TAP they
> understand.
>We don't even HAVE TAP versions yet.  If we have to upgrade them
> all...
> well then they'd be new harnesses.  So I'd like to avoid breaking TAP
> compatibility without a very good reason which is why I'm giving so
> much
> pushback and searching for ways to do it with the TAP producer.

I completely agree with Schwern on this one.  We've been using TAP for
a long time and until someone comes up with an extremely compelling
reason to break all of the old TAP parsers and which we have
significant agreement on, I'm terribly reluctant to extend the
TAP::Parser to handle this.  Old TAP parsers need to read new TAP and
still generate reasonable output.


What you're saying is we can _never_ break backwards comatibility.
That is not the impression I got from previous discussions.

What I'm saying is, this break backwards compatibility but the
producer can fall back to plain ol TAP if it knows the consumer can't
handle it. You'll lose some of the benefits in terms of strict
planning and diagnosis. However they're mostly important when the
developer is running the tests, as long as a failure still shows as a
failure for the user it's all good.

In fact my hacked up version of nested groups with the current test
framework does exaclty this because there is no parser for the TAP I'd
like to output.

I also don't think there will be many times when someone has a new
whizz-bang TAP producer and a crusty old TAP consumer but I'm not
basing my case on that.


There are several ways we could accomplish the same thing without break
backwards compatibility.  Let's explore those.


Show me one that actually does what I'm looking for,

F

Re: nested groups in TAP

[Fergal Daly]

On 12/03/07, Andy Armstrong <[EMAIL PROTECTED]> wrote:

On 12 Mar 2007, at 00:00, Michael G Schwern wrote:
> That's a good idea.
>
> Whether or not you can set a list of specific versions (considering
> them as
> extensions rather than increasing versions) or just a single
> highest version
> is up for debate.

Highest version I'd say. I hope we're not expecting such a complex
melange of versions that we need to enumerate the ones we support. If
say we're supporting version 10 it's implicit that we also support 1-9.


Are we not anticipate any bad decisions either? Say version 8 was
horribly broken and no one should ever support it. By saying you don't
support it the producer should then fall back gracefully to something
earlier.

What about the future? Maybe it's 20 years later and I'm writng a TAP
parser in C#++ and I don't want to have to cater for every version of
TAP that's ever existed, including the broken ones. So I say
"12,90-100" the producer can either produce up to date TAP or fall
back to the very basics.

If you insist on full backwards compatibility all the way back to the
start then you future TAP consumers are just going to lie. They'll say
"100" which is supposed to mean "1-100" but they're just going to hope
that nothing earlier than 90 ever shows up. If you're lucky they'll
die when they see a version number they don't support but by
communicating the reality the user could get the best available.

Plus saying "1-100" is only a tiny bit more difficult than saying "100"

F

Re: nested groups in TAP

[Andy Armstrong]

On 12 Mar 2007, at 00:00, Michael G Schwern wrote:

That's a good idea.

Whether or not you can set a list of specific versions (considering  
them as
extensions rather than increasing versions) or just a single  
highest version

is up for debate.


Highest version I'd say. I hope we're not expecting such a complex  
melange of versions that we need to enumerate the ones we support. If  
say we're supporting version 10 it's implicit that we also support 1-9.


--
Andy Armstrong, hexten.net

Re: nested groups in TAP

[Ovid]

--- Michael G Schwern <[EMAIL PROTECTED]> wrote:

> > How does it do this? I understand that an old harness will not be
> able
> > to parse it - it will be upset by the sub-plans and the dotted test
> > numbers)
> 
> Yep.  By "backwards compatible" I mean you can feed the new TAP to an
> old
> harness and it will still be able to get the basic pass/fail
> information out
> of it. 
> 
> > but if we know we're running under an old harness then we can
> > just fall back to plan-at-the-end and diagnostics.
> 
> Currently old harnesses do not identify what version of TAP they
> understand.
>We don't even HAVE TAP versions yet.  If we have to upgrade them
> all...
> well then they'd be new harnesses.  So I'd like to avoid breaking TAP
> compatibility without a very good reason which is why I'm giving so
> much
> pushback and searching for ways to do it with the TAP producer.

I completely agree with Schwern on this one.  We've been using TAP for
a long time and until someone comes up with an extremely compelling
reason to break all of the old TAP parsers and which we have
significant agreement on, I'm terribly reluctant to extend the
TAP::Parser to handle this.  Old TAP parsers need to read new TAP and
still generate reasonable output.

There are several ways we could accomplish the same thing without break
backwards compatibility.  Let's explore those.

Cheers,
Ovid

--

Buy the book -- http://www.oreilly.com/catalog/perlhks/
Perl and CGI -- http://users.easystreet.com/ovid/cgi_course/

Re: nested groups in TAP

[Michael G Schwern]

Michael G Schwern wrote:
> Subroutines?  I don't know if I follow.  Do you mean...
> 
> sub foo {
>   extend(2);
> 
>   pass();
>   bar();
>   pass();
> }
> 
> sub bar {
>   extend(3);
>   pass();
>   pass();
>   pass();
> }
> 
> I believe that can be made to work without a TAP extension.  Its the
> producer which does the stack watching and generates the errors instead of
> the TAP parser.

As a matter of fact, thinking about it more, this sort of thing has to be
able to work in the TAP producer.  How else will it know what group its
currently in to output the appropriate TAP?

Re: nested groups in TAP

[Michael G Schwern]

Fergal Daly wrote:
> On 11/03/07, Michael G Schwern <[EMAIL PROTECTED]> wrote:
>> Fergal Daly wrote:
>> > You're suggesting that each call to extend the plan verifies that the
>> > previous plan has been executed fully? That does not allow nesting.
>>
>> Yes, nesting is what this proposal does which cannot be done now.
> 
> Groups without nesting gives me subroutines with a max stack of depth
> of 1. It's almost more annoying than it is useful.

Subroutines?  I don't know if I follow.  Do you mean...

sub foo {
  extend(2);

  pass();
  bar();
  pass();
}

sub bar {
  extend(3);
  pass();
  pass();
  pass();
}

I believe that can be made to work without a TAP extension.  Its the
producer which does the stack watching and generates the errors instead of
the TAP parser.


>> Unfortuantely the other thing it does well is shatter backwards
>> compatibility so I'd like it to do something significantly more.
> 
> How does it do this? I understand that an old harness will not be able
> to parse it - it will be upset by the sub-plans and the dotted test
> numbers)

Yep.  By "backwards compatible" I mean you can feed the new TAP to an old
harness and it will still be able to get the basic pass/fail information out
of it.


> but if we know we're running under an old harness then we can
> just fall back to plan-at-the-end and diagnostics.

Currently old harnesses do not identify what version of TAP they understand.
   We don't even HAVE TAP versions yet.  If we have to upgrade them all...
well then they'd be new harnesses.  So I'd like to avoid breaking TAP
compatibility without a very good reason which is why I'm giving so much
pushback and searching for ways to do it with the TAP producer.

It would also be nice if the child plans didn't have to coordinate with the
parent plan.  This would provide truly isolated nested plans.  This would be
handy if you had a test which, for example, ran other tests in wholly
separate processes.


> This brings up something else. Future harnesses should probably set
> 
> I_UNDERSTAND_TAP_VERSIONS="1,5,8,10-22"
> 
> so we know what we can output. If that's not set then we need to stick
> to plainest TAP.

That's a good idea.

Whether or not you can set a list of specific versions (considering them as
extensions rather than increasing versions) or just a single highest version
is up for debate.

Re: nested groups in TAP

[Fergal Daly]

On 11/03/07, Michael G Schwern <[EMAIL PROTECTED]> wrote:

Fergal Daly wrote:
> You're suggesting that each call to extend the plan verifies that the
> previous plan has been executed fully? That does not allow nesting.

Yes, nesting is what this proposal does which cannot be done now.


Groups without nesting gives me subroutines with a max stack of depth
of 1. It's almost more annoying than it is useful.


Unfortuantely the other thing it does well is shatter backwards
compatibility so I'd like it to do something significantly more.


How does it do this? I understand that an old harness will not be able
to parse it - it will be upset by the sub-plans and the dotted test
numbers) but if we know we're running under an old harness then we can
just fall back to plan-at-the-end and diagnostics.

This brings up something else. Future harnesses should probably set

I_UNDERSTAND_TAP_VERSIONS="1,5,8,10-22"

so we know what we can output. If that's not set then we need to stick
to plainest TAP.

Is there something in the proposal that prevents us continuing to
output TAP just as today for a test script with no blocks?

F




> There's also the issue of passing important error messages as text
> embedded in diags.

Once we have a diagnostic syntax that will not be as much of an issue.


> Finally, it doesn't help with counting. Take this example
>
> use Test::More tests => 10;
>
> my @tests = (...); #  ten things
>
> for my $t (@test) {
>  is($t, foo);
> }
>
> now if I want to change this to
>
> for my $t (@test) {
>  is($t, foo);
>  is($t, bar);
> }
>
> I must update the global plan and start multiplying things. With
> blocks I can just do
>
> for my $t (@test) {
>  block {
>plan 2;
>is($t, foo);
>is($t, bar);
>  }
> }
>
> becase the plan of each block is counted as a single test in it's
> enclosing block. With plan extensions I would either do
>
> use Test::More tests => 10;
>
> my @tests = (...); #  ten things
>
> for my $t (@test) {
>  is($t, foo);
>  extend 1;
>  is($t, bar);
> }
>
> which is clearly broken and will cause lots of confusion when you
> squirrel your tests away in subroutines. Or
>
> use Test::More tests => 0;
>
> my @tests = (...); #  ten things
>
> for my $t (@test) {
>  extend 2;
>  is($t, foo);
>  is($t, bar);
> }
>
> which will happily exit early with 0 tests run and no errors,

There seems to be the assumption that you must have an initial global count.
 Why can't there be a syntax like so?

use Test::More import => ["!plan"];
use AsYouGo;
plan "as_you_go";

for my $t (@test) {
extend 2;
is($t, "foo");
is($t, "bar");
}

Where "as_you_go" requires you to declare at least one plan extension before
exiting.  And yes, it can tell if you didn't run any tests.

The attached proof-of-concept implements it.  I had to poke at the guts of
TB to do it, there's no way to extend the plan without printing the plan, so
it would need a minor TB patch.  But its very straight forward.


package AsYouGo;

use strict;
use warnings;

use base qw(Test::Builder::Module);
our @EXPORT = qw(plan extend);

my $Extended = 0;
my $Is_As_You_Go = 0;

use CLASS;

sub plan {
my($plan) = shift;

my $tb = $CLASS->builder;

if( $plan eq "as_you_go" ) {
$Is_As_You_Go = 1;
$tb->plan("no_plan");
}
else {  # pass through to TB
$tb->plan(@_);
}
}


sub extend {
my($extra_tests) = shift;

my $tb = $CLASS->builder;

print "# Extending plan by $extra_tests\n";

$extra_tests-- unless $Extended;  # there's already one

# TB doesn't have a way to extend the plan without printing the plan.
$tb->{Expected_Tests} += $extra_tests;

$Extended = 1;

1;
}


END {
if( $Is_As_You_Go && !$Extended ) {
die "# An as_you_go plan declared but never extended!\n";
}
}

1;

__END__

Re: nested groups in TAP

[Michael G Schwern]

Michael G Schwern wrote:
> The attached proof-of-concept implements it.  I had to poke at the guts of
> TB to do it, there's no way to extend the plan without printing the plan, so
> it would need a minor TB patch.  But its very straight forward.

Figured out a way to not have to do that.  Just temporarily turn on
"no_header".  New version attached.  Feel free to take it and run.

package AsYouGo;

use strict;
use warnings;

use base qw(Test::Builder::Module);
our @EXPORT = qw(plan extend);

my $Extended = 0;
my $Is_As_You_Go = 0;

use CLASS;

sub plan {
my($plan) = shift;

my $tb = $CLASS->builder;

if( $plan eq "as_you_go" ) {
$Is_As_You_Go = 1;
$tb->plan("no_plan");
}
else {  # pass through to TB
$tb->plan(@_);
}
}


sub extend {
my($extra_tests) = shift;

my $tb = $CLASS->builder;

print "# Extending plan by $extra_tests\n";

$extra_tests-- unless $Extended;  # there's already one

# TB prints the plan when you set expected_tests() so we
# suppress that.
$tb->no_header(1);
my $expected_tests = $tb->expected_tests;
$tb->expected_tests($expected_tests + $extra_tests);
$tb->no_header(0);

$Extended = 1;

1;
}


END {
if( $Is_As_You_Go && !$Extended ) {
die "# An as_you_go plan declared but never extended!\n";
}
}

1;

__END__

Re: nested groups in TAP

[Michael G Schwern]

Fergal Daly wrote:
> You're suggesting that each call to extend the plan verifies that the
> previous plan has been executed fully? That does not allow nesting.

Yes, nesting is what this proposal does which cannot be done now.

Unfortuantely the other thing it does well is shatter backwards
compatibility so I'd like it to do something significantly more.


> There's also the issue of passing important error messages as text
> embedded in diags.

Once we have a diagnostic syntax that will not be as much of an issue.


> Finally, it doesn't help with counting. Take this example
> 
> use Test::More tests => 10;
> 
> my @tests = (...); #  ten things
> 
> for my $t (@test) {
>  is($t, foo);
> }
> 
> now if I want to change this to
> 
> for my $t (@test) {
>  is($t, foo);
>  is($t, bar);
> }
> 
> I must update the global plan and start multiplying things. With
> blocks I can just do
> 
> for my $t (@test) {
>  block {
>plan 2;
>is($t, foo);
>is($t, bar);
>  }
> }
> 
> becase the plan of each block is counted as a single test in it's
> enclosing block. With plan extensions I would either do
> 
> use Test::More tests => 10;
> 
> my @tests = (...); #  ten things
> 
> for my $t (@test) {
>  is($t, foo);
>  extend 1;
>  is($t, bar);
> }
> 
> which is clearly broken and will cause lots of confusion when you
> squirrel your tests away in subroutines. Or
> 
> use Test::More tests => 0;
> 
> my @tests = (...); #  ten things
> 
> for my $t (@test) {
>  extend 2;
>  is($t, foo);
>  is($t, bar);
> }
> 
> which will happily exit early with 0 tests run and no errors,

There seems to be the assumption that you must have an initial global count.
 Why can't there be a syntax like so?

use Test::More import => ["!plan"];
use AsYouGo;
plan "as_you_go";

for my $t (@test) {
extend 2;
is($t, "foo");
is($t, "bar");
}

Where "as_you_go" requires you to declare at least one plan extension before
exiting.  And yes, it can tell if you didn't run any tests.

The attached proof-of-concept implements it.  I had to poke at the guts of
TB to do it, there's no way to extend the plan without printing the plan, so
it would need a minor TB patch.  But its very straight forward.

package AsYouGo;

use strict;
use warnings;

use base qw(Test::Builder::Module);
our @EXPORT = qw(plan extend);

my $Extended = 0;
my $Is_As_You_Go = 0;

use CLASS;

sub plan {
my($plan) = shift;

my $tb = $CLASS->builder;

if( $plan eq "as_you_go" ) {
$Is_As_You_Go = 1;
$tb->plan("no_plan");
}
else {  # pass through to TB
$tb->plan(@_);
}
}


sub extend {
my($extra_tests) = shift;

my $tb = $CLASS->builder;

print "# Extending plan by $extra_tests\n";

$extra_tests-- unless $Extended;  # there's already one

# TB doesn't have a way to extend the plan without printing the plan.
$tb->{Expected_Tests} += $extra_tests;

$Extended = 1;

1;
}


END {
if( $Is_As_You_Go && !$Extended ) {
die "# An as_you_go plan declared but never extended!\n";
}
}

1;

__END__

Re: nested groups in TAP

[Fergal Daly]

On 11/03/07, Michael G Schwern <[EMAIL PROTECTED]> wrote:

A. Pagaltzis wrote:
> * Michael G Schwern <[EMAIL PROTECTED]> [2007-03-11 12:55]:
>> Why does this need a TAP mod? Why not let the producer handle
>> it?
>
> Because then all you can do is a global trailing plan which is
> only half a step up from no plan?

Yep.  And why is that a problem?


> And if the tests and plan disagree, how do you localise the group
> in which they diverged? It's easy as falling off a log if you
> have sub-plans.

plan add => 2;
pass;
pass;
pass;

plan add => 3;
pass;
pass;

  ok 1
  ok 2
  no ok 3
  # Planned 2 sub-tests at line 1 but ran 3!
  ok 4
  ok 5
  # Planned 3 sub-tests at line 6 but only ran 2!
  1..5

The TAP producer can do it.


You're suggesting that each call to extend the plan verifies that the
previous plan has been executed fully? That does not allow nesting.

Translating from my proposal to the above:

1 1..3
1.1 ok
1.2 1..2
1.2.1 ok
1.2.2 ok
1.2 ok
1.3 ok

becomes

1 ok # was 1.1
# Planned 3 sub-tests at line 1 but only ran 1!
2 ok # was 1.2.1
3 ok # was 1.2.2
4 ok # was 1.3
1..4

There's also the issue of passing important error messages as text
embedded in diags.

Finally, it doesn't help with counting. Take this example

use Test::More tests => 10;

my @tests = (...); #  ten things

for my $t (@test) {
 is($t, foo);
}

now if I want to change this to

for my $t (@test) {
 is($t, foo);
 is($t, bar);
}

I must update the global plan and start multiplying things. With
blocks I can just do

for my $t (@test) {
 block {
   plan 2;
   is($t, foo);
   is($t, bar);
 }
}

becase the plan of each block is counted as a single test in it's
enclosing block. With plan extensions I would either do

use Test::More tests => 10;

my @tests = (...); #  ten things

for my $t (@test) {
 is($t, foo);
 extend 1;
 is($t, bar);
}

which is clearly broken and will cause lots of confusion when you
squirrel your tests away in subroutines. Or

use Test::More tests => 0;

my @tests = (...); #  ten things

for my $t (@test) {
 extend 2;
 is($t, foo);
 is($t, bar);
}

which will happily exit early with 0 tests run and no errors,

F


> I thought the benefits of local plans are dead obvious.

When you're talking about extending a protocol that everyone has to obey
forever and ever, you try not to leave things to "obvious".

Re: nested groups in TAP

[Fergal Daly]

On 11/03/07, Michael G Schwern <[EMAIL PROTECTED]> wrote:

Fergal Daly wrote:
> Did group 1 produce 2 tests and group 2 produce 3 or was it the other
> way around?

Why is that important to know?


Because the first case conforms to the plan, the second doesn't.


> Worse,
>
> ok 1
> ok 2
> ok 3
> ...
> ok 1001
> 1..1000
>
> where did my extra test come from?

That's something the TAP producer can tell us, right now.  No TAP extension
necessary.  Its even better than relying on TAP groups because it will work
in any test, it doesn't rely on your having small groups.


I'll address this point replying to your other mail as it includes an example.

F

TB patch welcome.


> Finally, this allows you to fork a process to run a block with no
> prior knowledge of how many tests are going to happen inside the
> forked process, not a killer app but it might be useful for people who
> have tests involving lots of network latency, they could run in
> parallel.

I see, with the groups being independent of each other in ordering now.
That is handy.

Re: nested groups in TAP

[Michael G Schwern]

A. Pagaltzis wrote:
> * Michael G Schwern <[EMAIL PROTECTED]> [2007-03-11 12:55]:
>> Why does this need a TAP mod? Why not let the producer handle
>> it?
> 
> Because then all you can do is a global trailing plan which is
> only half a step up from no plan?

Yep.  And why is that a problem?


> And if the tests and plan disagree, how do you localise the group
> in which they diverged? It’s easy as falling off a log if you
> have sub-plans.

plan add => 2;
pass;
pass;
pass;

plan add => 3;
pass;
pass;

  ok 1
  ok 2
  no ok 3
  # Planned 2 sub-tests at line 1 but ran 3!
  ok 4
  ok 5
  # Planned 3 sub-tests at line 6 but only ran 2!
  1..5

The TAP producer can do it.


> I thought the benefits of local plans are dead obvious.

When you're talking about extending a protocol that everyone has to obey
forever and ever, you try not to leave things to "obvious".

Re: nested groups in TAP

[Michael G Schwern]

Fergal Daly wrote:
> Did group 1 produce 2 tests and group 2 produce 3 or was it the other
> way around?

Why is that important to know?


> Worse,
> 
> ok 1
> ok 2
> ok 3
> ...
> ok 1001
> 1..1000
> 
> where did my extra test come from?

That's something the TAP producer can tell us, right now.  No TAP extension
necessary.  Its even better than relying on TAP groups because it will work
in any test, it doesn't rely on your having small groups.

TB patch welcome.


> Finally, this allows you to fork a process to run a block with no
> prior knowledge of how many tests are going to happen inside the
> forked process, not a killer app but it might be useful for people who
> have tests involving lots of network latency, they could run in
> parallel.

I see, with the groups being independent of each other in ordering now.
That is handy.

Re: nested groups in TAP

[Fergal Daly]

On 11/03/07, Michael G Schwern <[EMAIL PROTECTED]> wrote:

Fergal Daly wrote:
> http://perl-qa.yi.org/index.php/Test_Groups

Rationale?

Why does this need a TAP mod?  Why not let the producer handle it?

  plan "as_you_go";

  plan add => 2;
  pass;
  pass;

  plan add => 3;
  pass;
  pass;
  pass;

Would output:

  ok 1
  ok 2
  ok 3
  ok 4
  ok 5
  1..5

which is perfectly legal TAP.


Did group 1 produce 2 tests and group 2 produce 3 or was it the other
way around? Worse,

ok 1
ok 2
ok 3
...
ok 1001
1..1000

where did my extra test come from? I know, that's what we have right
now but that's no reason to not make it better.

Finally, this allows you to fork a process to run a block with no
prior knowledge of how many tests are going to happen inside the
forked process, not a killer app but it might be useful for people who
have tests involving lots of network latency, they could run in
parallel.

Updated the wiki.


And what's with the ALL CAPS OK?


It's me misremembering what TAP looks like.

Also updated.

F

Re: nested groups in TAP

[A. Pagaltzis]

* Michael G Schwern <[EMAIL PROTECTED]> [2007-03-11 12:55]:
> Why does this need a TAP mod? Why not let the producer handle
> it?

Because then all you can do is a global trailing plan which is
only half a step up from no plan?

And if the tests and plan disagree, how do you localise the group
in which they diverged? It’s easy as falling off a log if you
have sub-plans.

I thought the benefits of local plans are dead obvious.

Regards,
-- 
Aristotle Pagaltzis // 

Re: nested groups in TAP

[Michael G Schwern]

Fergal Daly wrote:
> http://perl-qa.yi.org/index.php/Test_Groups

Rationale?

Why does this need a TAP mod?  Why not let the producer handle it?

  plan "as_you_go";

  plan add => 2;
  pass;
  pass;

  plan add => 3;
  pass;
  pass;
  pass;

Would output:

  ok 1
  ok 2
  ok 3
  ok 4
  ok 5
  1..5

which is perfectly legal TAP.

And what's with the ALL CAPS OK?