Re: done_testing() and test counts

[Michael G Schwern]

Andy Lester wrote:
> 
> On Jun 9, 2009, at 2:57 PM, Michael G Schwern wrote:
> 
>>> use Test::More tests => 14;
>>> ok( 1 );
>>> done_testing();
>>
>> You would try that. :P
> 
> 
> It's not that I tried it, it's that I didn't know what would happen if I
> did both together, because it wasn't clear to me up front if
> done_testing() replaced no_plan or worked with it.

Its shown in the plan documentation working with no_plan and stated at the end
of the done_testing() docs: This is safer than and replaces the "no_plan" plan.

Oddly enough, done_testing() and no_plan do not work together.  They probably
should.

  use Test::More "no_plan";
  pass();
  done_testing()'

  ok 1
  1..1
  The plan was already output at
/usr/local/perl/5.10.0/lib/5.10.0/Test/Builder.pm line 2341.
  1..1

Anyhow, I was just poking fun because you're Mr. Test Count.


-- 
91. I am not authorized to initiate Jihad.
-- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army
   http://skippyslist.com/list/

Re: done_testing() and test counts

[Andy Lester]

On Jun 9, 2009, at 2:57 PM, Michael G Schwern wrote:


use Test::More tests => 14;
ok( 1 );
done_testing();


You would try that. :P



It's not that I tried it, it's that I didn't know what would happen if  
I did both together, because it wasn't clear to me up front if  
done_testing() replaced no_plan or worked with it.


xoa

--
Andy Lester => a...@petdance.com => www.theworkinggeek.com => AIM:petdance

Re: done_testing() and test counts

[Michael G Schwern]

Andy Lester wrote:
> I'm so glad for done_testing().  I don't like no_plan, but
> done_testing() makes it better.
> 
> I was surprised/confused to see this behavior:
> 
> $ cat foo.t
> use Test::More tests => 14;
> ok( 1 );
> done_testing();

You would try that. :P

I guess its a belt and suspenders approach.


> $ prove -v foo.t
> [13:55:39] foo.t ..
> 1..14
> ok 1
> not ok 2 - planned to run 14 but done_testing() expects 1
> 
> #   Failed test 'planned to run 14 but done_testing() expects 1'
> #   at /usr/lib/perl5/5.8.8/Test/More.pm line 220.
> # Looks like you planned 14 tests but ran 2.
> # Looks like you failed 1 test of 2 run.
> Dubious, test returned 1 (wstat 256, 0x100)
> Failed 13/14 subtests
> [13:55:39]
> 
> Test Summary Report
> ---
> foo.t (Wstat: 256 Tests: 2 Failed: 1)
>   Failed test:  2
>   Non-zero exit status: 1
>   Parse errors: Bad plan.  You planned 14 tests but ran 2.
> Files=1, Tests=2,  0 wallclock secs ( 0.03 usr  0.01 sys +  0.02 cusr 
> 0.01 csys =  0.07 CPU)
> Result: FAIL
> 
> 
> "Looks like you failed 1 test of 2 run".  I guess that it's counting
> done_testing() as a test in itself, but that doesn't seem to be right. 
> Is that intentional?

The extra test is intentional.  Its recording the plan failure as TAP.  This
may be unnecessary, it really should be letting the plan do that.  It could
get away with just a diagnostic.

What it is preventing is this:

  use Test::More tests => 2;
  pass("legit pass");
  done_testing();
  pass("I thought we were done?");

That should be a failure, and it is because done_testing() is a failing test.

Unfortunately, I just discovered that this is not a failure:

  use Test::More tests => 2;
  pass("legit pass");
  done_testing(2);
  pass("I thought we were done?");

And that's a bug.  Normally done_testing() would output "1..2" itself, so
you'd have:

  ok 1 - legit pass
  1..2
  ok 2 - I thought we were done?

And that's a plan failure, which is good.  But because the plan has already
been output it doesn't do it twice and we get:

  1..2
  ok 1 - legit pass
  ok 2 - I thought we were done?

The problem is done_testing() enforces "everything after this is a failure" by
outputting a plan in the wrong spot.  But if the plan has already been output,
what can it do?  Best I can think of is turn it into a failing test.  I agree
its a little weird to have a magic meta-test.  I don't want done_testing() to
simply die, that's not encodable in TAP and you don't see the rest of your
test results.  I also don't want the following tests to fail, that's a lie
they passed, its the plan that failed.

It could output a second plan if the tests don't match, that would cause a
plan failure.

  1..2
  ok 1 - legit pass
  # done_testing() expected 2 but only got 1
  1..2
  ok 2 - I thought we were done?

What is a mistake is that done_testing() with no argument shouldn't be
expecting anything.  The diagnostic should be the normal "Looks like you
planned 14 tests but ran 1."


-- 
Reality is that which, when you stop believing in it, doesn't go away.
-- Phillip K. Dick

done_testing() and test counts

[Andy Lester]

I'm so glad for done_testing().  I don't like no_plan, but  
done_testing() makes it better.


I was surprised/confused to see this behavior:

$ cat foo.t
use Test::More tests => 14;
ok( 1 );
done_testing();


$ prove -v foo.t
[13:55:39] foo.t ..
1..14
ok 1
not ok 2 - planned to run 14 but done_testing() expects 1

#   Failed test 'planned to run 14 but done_testing() expects 1'
#   at /usr/lib/perl5/5.8.8/Test/More.pm line 220.
# Looks like you planned 14 tests but ran 2.
# Looks like you failed 1 test of 2 run.
Dubious, test returned 1 (wstat 256, 0x100)
Failed 13/14 subtests
[13:55:39]

Test Summary Report
---
foo.t (Wstat: 256 Tests: 2 Failed: 1)
  Failed test:  2
  Non-zero exit status: 1
  Parse errors: Bad plan.  You planned 14 tests but ran 2.
Files=1, Tests=2,  0 wallclock secs ( 0.03 usr  0.01 sys +  0.02 cusr   
0.01 csys =  0.07 CPU)

Result: FAIL


"Looks like you failed 1 test of 2 run".  I guess that it's counting  
done_testing() as a test in itself, but that doesn't seem to be  
right.  Is that intentional?


xoxo,
Andy


--
Andy Lester => a...@petdance.com => www.theworkinggeek.com => AIM:petdance

Re: [ANNOUNCE] Test::More 0.87_01 (with done_testing())

[Michael G Schwern]

Evgeny wrote:
> Super! Well done!
> 
> Makes me feel all warm inside.
> 
> Would kiss you if you were near here somewhere! :)

I'm kind of glad I'm not, but thank you all the same.


-- 
24. Must not tell any officer that I am smarter than they are, especially
if it's true.
-- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army
   http://skippyslist.com/list/

Re: [ANNOUNCE] Test::More 0.87_01 (with done_testing())

[Evgeny]

Super! Well done!

Makes me feel all warm inside.

Would kiss you if you were near here somewhere! :)

-
evgeny

On Sun, Mar 29, 2009 at 12:52 PM, Michael G Schwern  wrote:
> Just pushed out a new alpha of Test::More.  There's a bunch of new features
> having to do with the plan.  In general it's less restrictive, you can now run
> tests without first having declared a plan.
>
> This enables the long called for new feature of done_testing().  It's a safer
> replacement for no_plan.  Looks like this:
>
>    use Test::More;  # look ma, no plan!
>
>    pass("Some sort of test");
>    pass("Some other test")
>
>    done_testing();
>
> If your test does not reach done_testing(), no plan will be output and your
> test will be a failure.  This is safer than "no_plan" because if your test
> exits in the middle it will fail.
>
> You can also give done_testing() a number of tests.  In that case it works
> just like "plan tests => #" except instead of having to put it at the
> beginning of the test it goes at the end.  This gives you the opportunity to
> calculate the number of tests as things go.
>
> plan( add => # ) didn't make the cut.  There were too many edge cases still to
> be worked out and I don't want to delay done_testing() because it fixes 90% of
> the problems with plans.  Here at the hackathon, Ovid is working on a full-on
> sub-plan TAP syntax which may eliminate the need for it entirely.  Those
> wanting a look at that can see http://github.com/ovid/test-more/master
>
> Finally, as a little aid to testing, Test::Builder->output() and friends now
> take scalar references.  This makes it much easier to trap test output without
> having to try and write backwards compatible tied capturing filehandles (5.6
> will not do scalar ref filehandles).  I had to suck in a copy of IO::Scalar to
> do it (one more argument in favor of just dropping 5.6) but it made the tests
> so much simpler.  There's Test::Builder::NoOutput in t/lib which is a
> convenience subclass to make testing even simpler.  I'm considering rolling
> the functionality into Test::Builder proper because I'm sure others will find
> it very useful.
>
>
> 0.87_01  Sun Mar 29 09:56:52 BST 2009
>    New Features
>    * done_testing() allows you to declare that you have finished
>      running tests and how many you ran.  It is a safer no_plan and
>      effectively replaces it.
>    * output() now supports scalar references.
>
>    Feature Changes
>    * You can now run a test without first declaring a plan.  This allows
>      done_testing() to work.
>    * You can now call current_test() without first declaring a plan.
>
>    Bug Fixes
>    * skip_all() with no reason would output "1..0" which is invalid TAP.
>      It will now always include the SKIP directive.
>
>    Other
>    * Repository moved to github.
>
> 0.86  Sun Nov  9 01:09:05 PST 2008
>    Same as 0.85_01
>
>
>
> --
> You are wicked and wrong to have broken inside and peeked at the
> implementation and then relied upon it.
>        -- tchrist in <31832.969261...@chthon>
>
>

[ANNOUNCE] Test::More 0.87_01 (with done_testing())

[Michael G Schwern]

Just pushed out a new alpha of Test::More.  There's a bunch of new features
having to do with the plan.  In general it's less restrictive, you can now run
tests without first having declared a plan.

This enables the long called for new feature of done_testing().  It's a safer
replacement for no_plan.  Looks like this:

use Test::More;  # look ma, no plan!

pass("Some sort of test");
pass("Some other test")

done_testing();

If your test does not reach done_testing(), no plan will be output and your
test will be a failure.  This is safer than "no_plan" because if your test
exits in the middle it will fail.

You can also give done_testing() a number of tests.  In that case it works
just like "plan tests => #" except instead of having to put it at the
beginning of the test it goes at the end.  This gives you the opportunity to
calculate the number of tests as things go.

plan( add => # ) didn't make the cut.  There were too many edge cases still to
be worked out and I don't want to delay done_testing() because it fixes 90% of
the problems with plans.  Here at the hackathon, Ovid is working on a full-on
sub-plan TAP syntax which may eliminate the need for it entirely.  Those
wanting a look at that can see http://github.com/ovid/test-more/master

Finally, as a little aid to testing, Test::Builder->output() and friends now
take scalar references.  This makes it much easier to trap test output without
having to try and write backwards compatible tied capturing filehandles (5.6
will not do scalar ref filehandles).  I had to suck in a copy of IO::Scalar to
do it (one more argument in favor of just dropping 5.6) but it made the tests
so much simpler.  There's Test::Builder::NoOutput in t/lib which is a
convenience subclass to make testing even simpler.  I'm considering rolling
the functionality into Test::Builder proper because I'm sure others will find
it very useful.


0.87_01  Sun Mar 29 09:56:52 BST 2009
New Features
* done_testing() allows you to declare that you have finished
  running tests and how many you ran.  It is a safer no_plan and
  effectively replaces it.
* output() now supports scalar references.

Feature Changes
    * You can now run a test without first declaring a plan.  This allows
  done_testing() to work.
* You can now call current_test() without first declaring a plan.

Bug Fixes
* skip_all() with no reason would output "1..0" which is invalid TAP.
  It will now always include the SKIP directive.

Other
* Repository moved to github.

0.86  Sun Nov  9 01:09:05 PST 2008
Same as 0.85_01



-- 
You are wicked and wrong to have broken inside and peeked at the
implementation and then relied upon it.
-- tchrist in <31832.969261...@chthon>

Re: done_testing()

[David E. Wheeler]

On Feb 23, 2009, at 1:39 AM, Aristotle Pagaltzis wrote:


Or differently, to RSS?

It may well be that we’ll need to break backcompat over this
issue, and if so, OK, but “I don’t care about backcompat” is
no way to go about designing a format that succeeds widely.


Hence my suggestion for a deprecation policy. And Schwern was already  
on it.


Best,

David

Re: done_testing()

[Aristotle Pagaltzis]

* David E. Wheeler  [2009-02-22 07:20]:
> I care less and less about backwards compatibility every day.

This is a format spec, not code.

Guess what happened to XML 1.1? To XHTML2?

Or differently, to RSS?

It may well be that we’ll need to break backcompat over this
issue, and if so, OK, but “I don’t care about backcompat” is
no way to go about designing a format that succeeds widely.

Regards,
-- 
Aristotle Pagaltzis // 

Re: done_testing()

[David E. Wheeler]

On Feb 22, 2009, at 3:45 PM, Michael G Schwern wrote:


I think that second one has value.


Agreed, but that's what a deprecation cycle is for.


TAP currently has no deprecation cycle.  That's what I introduced  
earlier.

http://www.ietf.org/mail-archive/web/tap/current/msg00411.html


Good!

D

Re: done_testing()

[Michael G Schwern]

David E. Wheeler wrote:
> On Feb 21, 2009, at 10:54 PM, Michael G Schwern wrote:
> 
>> There is Perl 5 style backwards compatibility where you never, ever break
>> anything for years and years and years and even for code that you're
>> not sure
>> even exists.  That's what chromatic is on about.
>>
>> And then there's backwards compatibility where you try not to clearly
>> break
>> everyone's well-written code!
>>
>> I think that second one has value.
> 
> Agreed, but that's what a deprecation cycle is for.

TAP currently has no deprecation cycle.  That's what I introduced earlier.
http://www.ietf.org/mail-archive/web/tap/current/msg00411.html


-- 
87. If the thought of something makes me giggle for longer than 15
seconds, I am to assume that I am not allowed to do it.
-- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army
   http://skippyslist.com/list/

Re: done_testing()

[David E. Wheeler]

On Feb 21, 2009, at 10:54 PM, Michael G Schwern wrote:

There is Perl 5 style backwards compatibility where you never, ever  
break
anything for years and years and years and even for code that you're  
not sure

even exists.  That's what chromatic is on about.

And then there's backwards compatibility where you try not to  
clearly break

everyone's well-written code!

I think that second one has value.


Agreed, but that's what a deprecation cycle is for.

That said, just a couple days ago I sent along a proposal to the TAP  
list to
change the way the TAP version works to allow incompatible changes  
to the
protocol.  Then we can work on sub-plans without having to bend over  
backwards

to compatibility.


Right!


PS  The signature choice was entirely random.  Eerie.

--
The past has a vote, but not a veto.
   -- Mordecai M. Kaplan


Amen to that!

D

Re: done_testing()

[Michael G Schwern]

David E. Wheeler wrote:
> On Feb 21, 2009, at 5:44 PM, Michael G Schwern wrote:
> 
>>> Yeah, I'm suggesting this more for a new version of TAP.
>>
>> It won't work because it's not backwards compatible.
> 
> I care less and less about backwards compatibility every day. Also,
> chromatic.

There is Perl 5 style backwards compatibility where you never, ever break
anything for years and years and years and even for code that you're not sure
even exists.  That's what chromatic is on about.

And then there's backwards compatibility where you try not to clearly break
everyone's well-written code!

I think that second one has value.

That said, just a couple days ago I sent along a proposal to the TAP list to
change the way the TAP version works to allow incompatible changes to the
protocol.  Then we can work on sub-plans without having to bend over backwards
to compatibility.


PS  The signature choice was entirely random.  Eerie.

-- 
The past has a vote, but not a veto.
-- Mordecai M. Kaplan

Re: done_testing()

[David E. Wheeler]

On Feb 21, 2009, at 5:44 PM, Michael G Schwern wrote:


Yeah, I'm suggesting this more for a new version of TAP.


It won't work because it's not backwards compatible.


I care less and less about backwards compatibility every day. Also,  
chromatic.


Technically "ok 1.1" should be read as an unnumbered test with the  
description
of "1.1".  TAP::Parser reads "ok 1.1" as test #1 with the  
description of ".1"

because it doesn't require spaces.

In Oslo we tried to make this one work and be backwards compatible  
but never

quite got it to work.
http://testanything.org/wiki/index.php/Test_Blocks

I don't feel like dragging out the details at the moment.


I don't think you can do it without breaking compatibility.

Best,

David

Re: done_testing()

[Michael G Schwern]

David E. Wheeler wrote:
> On Feb 21, 2009, at 11:35 AM, Michael G Schwern wrote:
> 
>> David E. Wheeler wrote:
>>> Dot notation?
>>>
>>> ok 1.1
>>> ok 1.2
>>> ok 2.1
>>> 1..2
>>
>> If you don't want any existing TAP parser to be able to read it and delay
>> release until they do, sure!
>>
>> I am totally not waiting for TAP to work out sub-plan syntax.
> 
> Yeah, I'm suggesting this more for a new version of TAP.

It won't work because it's not backwards compatible.

Technically "ok 1.1" should be read as an unnumbered test with the description
of "1.1".  TAP::Parser reads "ok 1.1" as test #1 with the description of ".1"
because it doesn't require spaces.

In Oslo we tried to make this one work and be backwards compatible but never
quite got it to work.
http://testanything.org/wiki/index.php/Test_Blocks

I don't feel like dragging out the details at the moment.


-- 
s7ank: i want to be one of those guys that types "s/j&jd//.^$ueu*///djsls/sm."
   and it's a perl script that turns dog crap into gold.

Re: done_testing()

[David E. Wheeler]

On Feb 21, 2009, at 11:35 AM, Michael G Schwern wrote:


David E. Wheeler wrote:

Dot notation?

ok 1.1
ok 1.2
ok 2.1
1..2


If you don't want any existing TAP parser to be able to read it and  
delay

release until they do, sure!

I am totally not waiting for TAP to work out sub-plan syntax.


Yeah, I'm suggesting this more for a new version of TAP.

David

Re: done_testing()

[Michael G Schwern]

David E. Wheeler wrote:
> Dot notation?
> 
>  ok 1.1
>  ok 1.2
>  ok 2.1
>  1..2

If you don't want any existing TAP parser to be able to read it and delay
release until they do, sure!

I am totally not waiting for TAP to work out sub-plan syntax.


-- 
185. My name is not a killing word.
-- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army
   http://skippyslist.com/list/

Re: done_testing()

[Michael G Schwern]

Aristotle Pagaltzis wrote:
> * Michael G Schwern  [2009-02-20 23:35]:
>> And we come back to the beginning: it's all going to be ad hoc
>> anyway until TAP formalizes it. Fine for eyeballing. If someone
>> wants to scrape the information out they can do it from the
>> description (with the usual caveats about scraping).
> 
> What I am saying is that just because they are… *wrinkles nose*
> …scraping [yuck, I feel dirty] doesn’t mean we shouldn’t try to
> make it as easy as possible.
> 
> Something about cow paths… so do we want those established by the
> scraping plebes or not?

No, their stuff will just break when the formatting changes.  Such is the
death of all screen scraping.


>> I plan on doing this:
> 
> So you see no way of doing it as a test numbering scheme? I would
> still prefer that…

Let's say we do.  How does this let you programmatically detect sub-plans?

What you write:

use Test::More;

subplan(2, "First thing");
pass();
pass();

subplan(1, "Second thing");
pass();

What it might output:

# subplan "First thing"
ok 1
ok 2
# subplan "Second thing"
ok 3
1..3

There's even LESS formal information there than if the subplan() calls are
tests themselves.  Worse than scraping test descriptions, you're scraping
comments.


-- 
Being faith-based doesn't trump reality.
-- Bruce Sterling

Re: done_testing()

[Aristotle Pagaltzis]

* Michael G Schwern  [2009-02-20 23:35]:
> And we come back to the beginning: it's all going to be ad hoc
> anyway until TAP formalizes it. Fine for eyeballing. If someone
> wants to scrape the information out they can do it from the
> description (with the usual caveats about scraping).

What I am saying is that just because they are… *wrinkles nose*
…scraping [yuck, I feel dirty] doesn’t mean we shouldn’t try to
make it as easy as possible.

Something about cow paths… so do we want those established by the
scraping plebes or not?

> I plan on doing this:

So you see no way of doing it as a test numbering scheme? I would
still prefer that…

Although, that solution is acceptable as a distant second choice.

Regards,
-- 
Aristotle Pagaltzis // 

Re: done_testing()

[Michael G Schwern]

Aristotle Pagaltzis wrote:
> * Michael G Schwern  [2009-02-19 21:15]:
>> As TAP has no formal means to express that, and I'm not waiting
>> for a TAP extension, any TAP reader will need extra logic to
>> figure that out. So worrying about that seems moot.
> 
> If it takes a lot longer and TB offers subplans in the meantime,
> people will want to be able to get the information back out of
> their TAP streams before formal subplans become part of the
> syntax. So worrying about how they will process the output of a
> stream containing implicit subplans seems entirely appropriate.

And we come back to the beginning: it's all going to be ad hoc anyway until
TAP formalizes it.  Fine for eyeballing.  If someone wants to scrape the
information out they can do it from the description (with the usual caveats
about scraping).

I have a feeling we're arguing about two different things.  Or we're arguing
about a plan that was already rejected.  Or it's gotten very meta and we're
arguing about concepts.

I plan on doing this:

use Test::More;

plan( add => 1 );
pass("First test");

plan( add => 2 );
pass("Second");
pass("Third");

done_testing(5);

to produce this:

ok 1 - First test
ok 2 - sub-plan at foo.t line 3
ok 3 - Second
ok 4 - Third
ok 5 - sub-plan at foo.t line 6
1..5

Hmm, maybe it should be subplan() instead of plan().  That lines up with the
output, and it allows a subplan description.

subplan(2, "frobnitzing the widget");
# ok 4 - subplan: frobnitzing the widget


-- 
164. There is no such thing as a were-virgin.
-- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army
   http://skippyslist.com/list/

Re: done_testing()

[Aristotle Pagaltzis]

* Michael G Schwern  [2009-02-19 21:15]:
> As TAP has no formal means to express that, and I'm not waiting
> for a TAP extension, any TAP reader will need extra logic to
> figure that out. So worrying about that seems moot.

If it takes a lot longer and TB offers subplans in the meantime,
people will want to be able to get the information back out of
their TAP streams before formal subplans become part of the
syntax. So worrying about how they will process the output of a
stream containing implicit subplans seems entirely appropriate.

Regards,
-- 
Aristotle Pagaltzis // 

Re: done_testing()

[Michael G Schwern]

David E. Wheeler wrote:
> On Feb 19, 2009, at 12:12 PM, Michael G Schwern wrote:
> 
>>> I’m not talking about pass/fail, I’m talking about finding out
>>> about subplans, from the consumer end.
>>
>> As TAP has no formal means to express that, and I'm not waiting for a TAP
>> extension, any TAP reader will need extra logic to figure that out.  So
>> worrying about that seems moot.
> 
> Should TAP be extended for this?

Yes.  Sub-plans have been considered, but we never quite got something to work
that was backwards compatible.  Anyhow, I'm not waiting for it.


-- 
emacs -- THAT'S NO EDITOR... IT'S AN OPERATING SYSTEM!

Re: done_testing()

[David E. Wheeler]

On Feb 19, 2009, at 12:12 PM, Michael G Schwern wrote:


I’m not talking about pass/fail, I’m talking about finding out
about subplans, from the consumer end.


As TAP has no formal means to express that, and I'm not waiting for  
a TAP
extension, any TAP reader will need extra logic to figure that out.   
So

worrying about that seems moot.


Should TAP be extended for this?

Best,

David

Re: done_testing()

[Michael G Schwern]

Aristotle Pagaltzis wrote:
> * Michael G Schwern  [2009-02-19 08:40]:
>> it doesn't require any extra TAP reader logic to determine pass
>> or fail.
> 
> I’m not talking about pass/fail, I’m talking about finding out
> about subplans, from the consumer end.

As TAP has no formal means to express that, and I'm not waiting for a TAP
extension, any TAP reader will need extra logic to figure that out.  So
worrying about that seems moot.


-- 
87. If the thought of something makes me giggle for longer than 15
seconds, I am to assume that I am not allowed to do it.
-- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army
   http://skippyslist.com/list/

Re: done_testing()

[Aristotle Pagaltzis]

* Michael G Schwern  [2009-02-19 08:40]:
> it doesn't require any extra TAP reader logic to determine pass
> or fail.

I’m not talking about pass/fail, I’m talking about finding out
about subplans, from the consumer end.

Regards,
-- 
Aristotle Pagaltzis // 

Re: done_testing()

[Michael G Schwern]

Aristotle Pagaltzis wrote:
>> If you're referring to having two tests with the same number,
>> that's perfectly valid TAP which will cause the suite to fail.
>> Test::Builder should support it whether we use it for sub-plans
>> or not.
> 
> I agree with that. Let me try again:
> 
> I’m saying that outputting extra tests to force a failure means
> that suddenly a test result does not just represent the result of
> a test, but can also be a signal that the incremental plan failed
> to pan out. You’ve added a new meaning to the syntax for test
> results.

What do test results mean but that an assertion failed?  A sub-plan is an
assertion.


> Worse, though, if you output these only sometimes, a TAP consumer
> which wants to extract this information will need funky logic to
> deal with the conditional presence of these tests.

I admit that having tests which only show up on failure is weird, which is why
if it went that way the test result would always be there, but it doesn't
require any extra TAP reader logic to determine pass or fail.


>> So something like this? […] Not such a bad idea, and pretty
>> straight forward if each plan( add => # ) is considered a test
>> itself.
> 
> Yes, exactly like that. I still don’t quite like it. I can see,
> though, that using the numbering to communicate this while
> keeping TB-dependent code working could be a problem, whereas
> this version would be feasible – in which case at least there
> is a certain logic to it, and the subplan stages could be
> interpreted by TAP consumers with much less complex condition
> checks than the output you proposed.


-- 
Defender of Lexical Encapsulation

Re: done_testing()

[Aristotle Pagaltzis]

* Michael G Schwern  [2009-02-19 06:35]:
> Aristotle Pagaltzis wrote:
> > But injecting artificial test results seems like a fairly big
> > modification to the format’s semantics to me, and I’m not
> > comfortable with the idea of doing that for no greater reason
> > than that the existing codebase has inadequacies in exposing
> > the full existing semantics.
>
> Sorry, you've lost me. What?
>
> If you're referring to having two tests with the same number,
> that's perfectly valid TAP which will cause the suite to fail.
> Test::Builder should support it whether we use it for sub-plans
> or not.

I agree with that. Let me try again:

I’m saying that outputting extra tests to force a failure means
that suddenly a test result does not just represent the result of
a test, but can also be a signal that the incremental plan failed
to pan out. You’ve added a new meaning to the syntax for test
results.

Worse, though, if you output these only sometimes, a TAP consumer
which wants to extract this information will need funky logic to
deal with the conditional presence of these tests.

Using the numbering scheme to indicate these failures, however,
would be fine.

That’s what I was trying to say. Sorry it was less than clear.

> So something like this? […] Not such a bad idea, and pretty
> straight forward if each plan( add => # ) is considered a test
> itself.

Yes, exactly like that. I still don’t quite like it. I can see,
though, that using the numbering to communicate this while
keeping TB-dependent code working could be a problem, whereas
this version would be feasible – in which case at least there
is a certain logic to it, and the subplan stages could be
interpreted by TAP consumers with much less complex condition
checks than the output you proposed.

Regards,
-- 
Aristotle Pagaltzis // 

Re: done_testing()

[Michael G Schwern]

Aristotle Pagaltzis wrote:
> * Michael G Schwern  [2009-02-18 22:45]:
>> Aristotle Pagaltzis wrote:
>>> * Michael G Schwern  [2009-02-18 21:55]:
>>>> One of the issues with that approach is Test::Builder's
>>>> history can't store test #2 twice. So history is lost.
>>> Shouldn’t this be fixed?
>> Sure, but how?
> 
> I don’t know.
> 
> But injecting artificial test results seems like a fairly big
> modification to the format’s semantics to me, and I’m not
> comfortable with the idea of doing that for no greater reason
> than that the existing codebase has inadequacies in exposing the
> full existing semantics.

Sorry, you've lost me.  What?

If you're referring to having two tests with the same number, that's perfectly
valid TAP which will cause the suite to fail.  Test::Builder should support it
whether we use it for sub-plans or not.


> New tools will forever have to make special accomodations to
> interpret this crud correctly just because the old ones were
> inadequate at some point in time. I understand that the TB API
> can’t just be changed to accommodate this either… it’s a
> difficult problem. But I’d prefer if some more thought went into
> how to proceed, rather than just going forward with whatever’s
> expedient.

This would be why I posted the question.  I was in the middle of writing up
some plan add tests and still don't know how best to handle a sub-plan overrun.


> I might be more comfortable if using `plan add` counted as an
> assertion itself, so it would *always* emit a test result, and
> you’d get N extra tests (properly accounted for in the plan, of
> course) if you called `plan add` N times. I’m stil unhappy with
> the idea, but at least that leads to less variability in the
> output, reducing the amount of cleverness required to correctly
> interpret the resulting streams.

So something like this?

  use Test::More;

  plan add => 2;
  pass("first");

  plan add => 2;
  pass("second");
  pass("third");

  # Four tests, plus the two sub-plans.
  done_testing(6);

Would produce:

  ok 1 - first
  not ok 2 - sub-plan at line 3
  # 2 planned, but only 1 ran
  ok 3 - second
  ok 4 - third
  ok 5 - sub-plan at line 7
  1..6

Not such a bad idea, and pretty straight forward if each plan( add => # ) is
considered a test itself.


-- 
You know what the chain of command is? It's the chain I go get and beat you
with 'til you understand who's in ruttin' command here.
-- Jayne Cobb, "Firefly"

Re: done_testing()

[David E. Wheeler]

On Feb 18, 2009, at 3:57 PM, Aristotle Pagaltzis wrote:


Shouldn’t this be fixed?


Sure, but how?


I don’t know.

But injecting artificial test results seems like a fairly big
modification to the format’s semantics to me, and I’m not
comfortable with the idea of doing that for no greater reason
than that the existing codebase has inadequacies in exposing the
full existing semantics.


+1


New tools will forever have to make special accomodations to
interpret this crud correctly just because the old ones were
inadequate at some point in time. I understand that the TB API
can’t just be changed to accommodate this either… it’s a
difficult problem. But I’d prefer if some more thought went into
how to proceed, rather than just going forward with whatever’s
expedient.

I might be more comfortable if using `plan add` counted as an
assertion itself, so it would *always* emit a test result, and
you’d get N extra tests (properly accounted for in the plan, of
course) if you called `plan add` N times. I’m stil unhappy with
the idea, but at least that leads to less variability in the
output, reducing the amount of cleverness required to correctly
interpret the resulting streams.


That would have to go for `use Test::More tests => 4;`, too. That  
would mean 5 tests. Personally I'm not too keen on this idea, but  
agree that at least then it would be consistent with Schwern's  
proposal for `plan add`.


Best,

David

Re: done_testing()

[Aristotle Pagaltzis]

* Michael G Schwern  [2009-02-18 22:45]:
> Aristotle Pagaltzis wrote:
> > * Michael G Schwern  [2009-02-18 21:55]:
> >> One of the issues with that approach is Test::Builder's
> >> history can't store test #2 twice. So history is lost.
> >
> > Shouldn’t this be fixed?
>
> Sure, but how?

I don’t know.

But injecting artificial test results seems like a fairly big
modification to the format’s semantics to me, and I’m not
comfortable with the idea of doing that for no greater reason
than that the existing codebase has inadequacies in exposing the
full existing semantics.

New tools will forever have to make special accomodations to
interpret this crud correctly just because the old ones were
inadequate at some point in time. I understand that the TB API
can’t just be changed to accommodate this either… it’s a
difficult problem. But I’d prefer if some more thought went into
how to proceed, rather than just going forward with whatever’s
expedient.

I might be more comfortable if using `plan add` counted as an
assertion itself, so it would *always* emit a test result, and
you’d get N extra tests (properly accounted for in the plan, of
course) if you called `plan add` N times. I’m stil unhappy with
the idea, but at least that leads to less variability in the
output, reducing the amount of cleverness required to correctly
interpret the resulting streams.

Regards,
-- 
Aristotle Pagaltzis // 

Re: done_testing()

[Michael G Schwern]

Aristotle Pagaltzis wrote:
> * Michael G Schwern  [2009-02-18 21:55]:
>> One of the issues with that approach is Test::Builder's history
>> can't store test #2 twice. So history is lost.
> 
> Shouldn’t this be fixed?

Sure, but how?

Internally, fine, it can be stored using a list of lists.  But the interfaces
to the history, Test::Builder->summary() and Test::Builder->details(), both
return flat lists where $test[$num] is test $num + 1.  And I really hate
mixing refs and non-refs in a list.  The user has to do shenanigans like this:

for my $test (map { ref $_ ? @$_ : $_ } $tb->details) {
...
}

Which they'll forget to do.

For TB2 I'll probably remove the assumption that element N is test N+1 to
retain the flat list rather than make all the elements be refs.


-- 
"Clutter and overload are not an attribute of information,
 they are failures of design"
-- Edward Tufte

Re: done_testing()

[Aristotle Pagaltzis]

* Michael G Schwern  [2009-02-18 21:55]:
> One of the issues with that approach is Test::Builder's history
> can't store test #2 twice. So history is lost.

Shouldn’t this be fixed?

Regards,
-- 
Aristotle Pagaltzis // 

Re: done_testing()

[Michael G Schwern]

David E. Wheeler wrote:
> 1..4? God no. I think this:
> 
>  # Planning 2 more tests at foo.t line 3.
>  ok 1 - First test
>  # Looks like you planned 2 tests, but only 1 was run
>  # at foo.t line 6.
>  ok 3 - Second test
>  ok 4 - Third test
>  1..3
> 
> That is, all tests should pass, but the test suite itself should fail,
> just as happens now if you have the wrong number of tests. Not sure how
> you'd show that, though.

Ok, makes sense.  Now riddle me this, Batman:

use Test::More;

plan add => 1;
pass("first");
pass("second");
plan add => 2;
pass("third");
done_testing(3);

Is that...

ok 1 - first
ok 2 - second
# Looks like you planned 1 tests, but 2 were run
#   at foo.t line 3.
ok 2 - third
# Looks like you planned 2 tests, but only 1 was run
#   at foo.t line 6.
1..3

One of the issues with that approach is Test::Builder's history can't store
test #2 twice.  So history is lost.


-- 
There will be snacks.

Re: done_testing()

[Aristotle Pagaltzis]

* David E. Wheeler  [2009-02-06 04:05]:
> On Feb 5, 2009, at 6:23 PM, Michael G Schwern wrote:
>> Or would that be 1..4?  Having an overall plan violation on
>> top of the plan addition failure seems a bit much, because the
>> number of tests run was correct (until we added one of our
>> own).
>
> 1..4? God no. I think this:
>
>  # Planning 2 more tests at foo.t line 3.
>  ok 1 - First test
>  # Looks like you planned 2 tests, but only 1 was run
>  # at foo.t line 6.
>  ok 3 - Second test
>  ok 4 - Third test
>  1..3
>
> That is, all tests should pass, but the test suite itself
> should fail,  just as happens now if you have the wrong number
> of tests. Not sure how you'd show that, though.

++

I didn’t think of it, but now that I see it, that is exactly what
should happen. Only three tests were planned, but the first leg
of the test program did not complete (test 2 is missing) and the
second leg overshot (there’s an extra test 4). That TAP stream
expresses these things precisely with no artificial extra test
results injected.

Regards,
-- 
Aristotle Pagaltzis // 

Re: done_testing()

[Gabor Szabo]

On Mon, Feb 9, 2009 at 10:48 AM, Philippe Bruhat (BooK)
 wrote:
> On Tue, Feb 03, 2009 at 08:21:33PM -0800, Michael G Schwern wrote:
>>
>> Finally, this makes it now possible to build up the test plan as you go.  I'd
>> like to put first order support into Test::Builder and Test::More for it, but
>> for the moment this will work:
>>
>>   use Test::More;
>>
>>   my $tests = 2;
>>   pass;
>>   pass;
>>
>>   $tests += 1;
>>   pass;
>>
>>   done_testing($tests);
>
> Just a side note: this has always been possible, as I've seen people do the 
> following:
>
>my $tests;
>use Test::More tests => $tests;
>
>BEGIN { $tests += 2 };
>ok( ... );
>ok( ... );
>BEGIN { $tests += 1 };
>ok( ... );
>
> I like the plan add => $n interface a lot, especially with DrHyde's
> suggestion to use it for optional tests. That may look better than
> skipped tests, but I guess it's mostly a difference in the message one
> wants to send: skipped tests include a reason why tests were skipped.
>
> It may also make the plan computation much easier for complicated test
> suites where one tests a list of cases with a varying number of tests
> for each case, and doesn't want to put the hairy computation in a map {}
> at the plan() stage. Now that I think about it, this latter case is
> probably a better use case for plan add.
>


I have been using the BEGIN {} code for some time since I learned it on
the perl-qa list but the "plan add" is much better IMHO especially if
it can also report on incorrect number of tests in each section.

Gabor

Re: done_testing()

[Philippe Bruhat (BooK)]

On Tue, Feb 03, 2009 at 08:21:33PM -0800, Michael G Schwern wrote:
> 
> Finally, this makes it now possible to build up the test plan as you go.  I'd
> like to put first order support into Test::Builder and Test::More for it, but
> for the moment this will work:
> 
>   use Test::More;
> 
>   my $tests = 2;
>   pass;
>   pass;
> 
>   $tests += 1;
>   pass;
> 
>   done_testing($tests);

Just a side note: this has always been possible, as I've seen people do the 
following:

my $tests;
use Test::More tests => $tests;

BEGIN { $tests += 2 };
ok( ... );
ok( ... );
BEGIN { $tests += 1 };
ok( ... );

I like the plan add => $n interface a lot, especially with DrHyde's
suggestion to use it for optional tests. That may look better than
skipped tests, but I guess it's mostly a difference in the message one
wants to send: skipped tests include a reason why tests were skipped.

It may also make the plan computation much easier for complicated test
suites where one tests a list of cases with a varying number of tests
for each case, and doesn't want to put the hairy computation in a map {}
at the plan() stage. Now that I think about it, this latter case is
probably a better use case for plan add.

-- 
 Philippe Bruhat (BooK)

 When you open a new door, the bad comes in with the good.
   (Moral from Groo The Wanderer #102 (Epic))

Re: done_testing()

[David E. Wheeler]

On Feb 5, 2009, at 6:23 PM, Michael G Schwern wrote:


 # Planning 2 more tests at foo.t line 3.
 ok 1 - First test
 not ok 2 - 2 tests were planned but only 1 was run
 # Failed test "2 tests were planned but only 1 was run"
 # at foo.t line 6.
 # Planning 1 more test at foo.t line 6.
 ok 3 - Second test
 ok 4 - Third test
 1..3

Or would that be 1..4?  Having an overall plan violation on top of  
the plan
addition failure seems a bit much, because the number of tests run  
was correct

(until we added one of our own).


1..4? God no. I think this:

 # Planning 2 more tests at foo.t line 3.
 ok 1 - First test
 # Looks like you planned 2 tests, but only 1 was run
 # at foo.t line 6.
 ok 3 - Second test
 ok 4 - Third test
 1..3

That is, all tests should pass, but the test suite itself should fail,  
just as happens now if you have the wrong number of tests. Not sure  
how you'd show that, though.


Best,

David

Re: done_testing()

[Michael G Schwern]

David E. Wheeler wrote:
> On Feb 5, 2009, at 12:34 PM, Michael G Schwern wrote:
> 
>> Though we don't have incremental TAP plans, Test::Builder can check that
>> you've run all the tests you said you'd run before you add more.  Thus...
>>
>>  use Test::More;
>>
>>  plan add => 2;
>>  pass;
>>
>>  plan add => 1;
>>  pass;   # failure
>>  pass;
>>
>> Thoughts?
> 
> All makes pretty good sense to me.

How should that plan violation be expressed?  Making a passing test fail is
wrong, because the test didn't fail, it's a plan violation.  I can make the
plan addition itself a test, but only if it fails.

So this...

  use Test::More;

  plan add => 2;
  pass("First test");

  plan add => 1;
  pass("Second test");
  pass("Third test");

Might produce:

  # Planning 2 more tests at foo.t line 3.
  ok 1 - First test
  not ok 2 - 2 tests were planned but only 1 was run
  # Failed test "2 tests were planned but only 1 was run"
  # at foo.t line 6.
  # Planning 1 more test at foo.t line 6.
  ok 3 - Second test
  ok 4 - Third test
  1..3

Or would that be 1..4?  Having an overall plan violation on top of the plan
addition failure seems a bit much, because the number of tests run was correct
(until we added one of our own).


-- 
170. Not allowed to "defect" to OPFOR during training missions.
-- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army
   http://skippyslist.com/list/

Re: done_testing()

[David E. Wheeler]

On Feb 5, 2009, at 12:34 PM, Michael G Schwern wrote:

Though we don't have incremental TAP plans, Test::Builder can check  
that
you've run all the tests you said you'd run before you add more.   
Thus...


 use Test::More;

 plan add => 2;
 pass;

 plan add => 1;
 pass;   # failure
 pass;

Thoughts?


All makes pretty good sense to me.

Best,

David

Re: done_testing()

[Michael G Schwern]

David Cantrell wrote:
> On Thu, Feb 05, 2009 at 10:00:38AM +0200, Gabor Szabo wrote:
> 
>> You might want to report error if both plan(2) and add_plan(2) was
>> called in the same file.
> 
> Or you might not.
> 
> plan(2);
> # generic tests
> ok(...);
> ok(...);
> if($ENV{RUN_REALLY_SLOW_TESTS}) {
>   add_plan(2);
>   ok(...);
>   ok(...);
> }
> done_testing();

Test::NoWarnings, which has to add a test without you knowing, would find that
advantageous.

I also allowed this:

  use Test::More 'no_plan';
  pass();
  pass();
  done_testing(2);

I'm not entirely sure why you'd do it, maybe if a human writes "no_plan" but a
module issues the done_testing(), but I can't think of a strong reason to make
it fail.

Separating adding to the plan from declaring a plan is also a good idea.
Rather than make a new function, plan() already takes key/values so
plan(add => $num_tests) would seem to be the way to do it.

Also, Test::More must know that your plan is additive so it knows to issue the
TAP plan at the end.  So this won't work:

  use Test::More tests => 1;  # at this point, "1..1" is printed
  pass;
  plan add => 1;  # what now?
  pass;

So you'd do this:

  use Test::More;

  plan add => 1;
  pass;

  plan add => 1;
  pass;

Though we don't have incremental TAP plans, Test::Builder can check that
you've run all the tests you said you'd run before you add more.  Thus...

  use Test::More;

  plan add => 2;
  pass;

  plan add => 1;
  pass;   # failure
  pass;

Thoughts?


-- 
You know what the chain of command is? It's the chain I go get and beat you
with 'til you understand who's in ruttin' command here.
-- Jayne Cobb, "Firefly"

Re: done_testing()

[Michael G Schwern]

Gabor Szabo wrote:
> BTW what about
> 
> use Test::More;
> pass;
> pass;
> # done_testing() # not being called at all.
> 
> Is this a failure now as there was no plan, not even a no_plan ?

It's a failure by virtue of there being no TAP plan.  I didn't feel it was
necessary to output an explicit failing test as the lack of a plan will fail
just fine and reflects what failed.

ok 1
ok 2
# Tests were run but no plan was declared and done_testing() was not seen.



-- 
emacs -- THAT'S NO EDITOR... IT'S AN OPERATING SYSTEM!

Re: done_testing()

[David Cantrell]

On Thu, Feb 05, 2009 at 10:00:38AM +0200, Gabor Szabo wrote:

> You might want to report error if both plan(2) and add_plan(2) was
> called in the same file.

Or you might not.

plan(2);
# generic tests
ok(...);
ok(...);
if($ENV{RUN_REALLY_SLOW_TESTS}) {
  add_plan(2);
  ok(...);
  ok(...);
}
done_testing();

-- 
David Cantrell | top google result for "internet beard fetish club"

You don't need to spam good porn

Re: [test-more-users] Re: done_testing()

[Ovid]

- Original Message 

> From: Gaurav Vaidya 

> I just made a branch of Test::Builder which does this. You can see the 
> expected 
> behaviour in the tests at:
> 
> http://github.com/gaurav/test-more/tree/incremental-planning/t/incremental_planning/
> (or http://tinyurl.com/ad8mb4)
> 
> I've modified Test::Builder to emit incremental plans ("1..2 [individual test 
> results] 3..4 [..] 5..7") since this was easier to throw together in an 
> evening, 
> but if you avoid Centirely and put a Cat the end of 
> the test, you'll get one, single plan combining all the tests run until then 
> (see the end of t/incremental_planning/simple.t to see what I mean). This 
> should 
> work in current TAP parsers.
> 
> This code's quick, dirty and messy, but I thought having a prototype 
> incremental 
> T::B to play around with might help us think out the best way to go forward 
> with 
> incremental planning. Git's flexibility means that moving code back into the 
> Test::Builder mainline should be straightforward, too. If you don't want to 
> muck 
> about with Git, you can get a tarball of my branch at:
> http://github.com/gaurav/test-more/tarball/incremental-planning-try1

Yay for git :)

 
Cheers,
Ovid
--
Buy the book - http://www.oreilly.com/catalog/perlhks/
Tech blog- http://use.perl.org/~Ovid/journal/
Twitter  - http://twitter.com/OvidPerl
Official Perl 6 Wiki - http://www.perlfoundation.org/perl6

Re: done_testing()

[Gabor Szabo]

On Wed, Feb 4, 2009 at 4:35 PM, Ovid  wrote:
> - Original Message 
>
>> From: Michael G Schwern 
>
> First of all, thank you!  This is fantastic work and I'm sure it will make a 
> lot of people happy.

++


> Thoughts on first order support:
>
> use Test::More;
>
> plan(3);
> pass;
> pass;
> pass;
> plan(2)
> pass;
> pass;
> done_testing() # optional
>
> Then, you can incrementally build a plan for those who want it and it seems 
> backwards almost compatible (since done_testing() wouldn't be required).
>
> The problem is that internally, TB will see that plan() has been called and 
> will die if a plan has been called twice.

why not call it something else then?
plan_more(2)   or  add_plan(2)

You might want to report error if both plan(2) and add_plan(2) was
called in the same file.


BTW what about

use Test::More;
pass;
pass;
# done_testing() # not being called at all.

Is this a failure now as there was no plan, not even a no_plan ?

Gabor

Re: done_testing()

[David E. Wheeler]

On Feb 4, 2009, at 3:15 PM, Ovid wrote:

Only one.  The BBC frowns on it if you come back *completely*  
wasted.  I'd probably get a stern talking to if I did.


Ooh, scary.

D

Re: done_testing()

[Ovid]

- Original Message 

> From: David E. Wheeler 

> > (The first idea is bugging me because I swear I had thought of a 
> > show-stopper 
> over lunch, but for the life of me, I can't recall what it was).
> 
> Must've been a damn good lunch. How many pints did you kill?

Only one.  The BBC frowns on it if you come back *completely* wasted.  I'd 
probably get a stern talking to if I did.

  
Cheers,
Ovid
--
Buy the book - http://www.oreilly.com/catalog/perlhks/
Tech blog- http://use.perl.org/~Ovid/journal/
Twitter  - http://twitter.com/OvidPerl
Official Perl 6 Wiki - http://www.perlfoundation.org/perl6

Re: [test-more-users] Re: done_testing()

[Gaurav Vaidya]

Hello,

On Feb 4, 2009, at 10:35 PM, Ovid wrote:

Thoughts on first order support:

use Test::More;

plan(3);
pass;
pass;
pass;
plan(2)
pass;
pass;
done_testing() # optional
I just made a branch of Test::Builder which does this. You can see the  
expected behaviour in the tests at:


http://github.com/gaurav/test-more/tree/incremental-planning/t/incremental_planning/
(or http://tinyurl.com/ad8mb4)

I've modified Test::Builder to emit incremental plans ("1..2  
[individual test results] 3..4 [..] 5..7") since this was easier to  
throw together in an evening, but if you avoid C entirely and  
put a C at the end of the test, you'll get one, single  
plan combining all the tests run until then (see the end of t/ 
incremental_planning/simple.t to see what I mean). This should work in  
current TAP parsers.


This code's quick, dirty and messy, but I thought having a prototype  
incremental T::B to play around with might help us think out the best  
way to go forward with incremental planning. Git's flexibility means  
that moving code back into the Test::Builder mainline should be  
straightforward, too. If you don't want to muck about with Git, you  
can get a tarball of my branch at:

http://github.com/gaurav/test-more/tarball/incremental-planning-try1

cheers,
Gaurav

Re: done_testing()

[David E. Wheeler]

On Feb 4, 2009, at 6:35 AM, Ovid wrote:


Thoughts?

(The first idea is bugging me because I swear I had thought of a  
show-stopper over lunch, but for the life of me, I can't recall what  
it was).


Must've been a damn good lunch. How many pints did you kill?

D

Re: done_testing()

[Ovid]

- Original Message 

> From: Michael G Schwern 

First of all, thank you!  This is fantastic work and I'm sure it will make a 
lot of people happy.

> Finally, this makes it now possible to build up the test plan as you go.  I'd
> like to put first order support into Test::Builder and Test::More for it, but
> for the moment this will work:
> 
>   use Test::More;
> 
>   my $tests = 2;
>   pass;
>   pass;
> 
>   $tests += 1;
>   pass;
> 
>   done_testing($tests);

Thoughts on first order support:

use Test::More;

plan(3);
pass;
pass;
pass;
plan(2) 
pass;
pass;
done_testing() # optional

Then, you can incrementally build a plan for those who want it and it seems 
backwards almost compatible (since done_testing() wouldn't be required).

The problem is that internally, TB will see that plan() has been called and 
will die if a plan has been called twice.

Alternatively:

use Test::More;

plan(3);
pass;
pass;
pass;
add_plan(2) 
pass;
pass;
done_testing() # NOT optional

Or:

use Test::More 'deferred';

plan(3);
pass;
pass;
pass;
plan(2) 
pass;
pass;
done_testing() # NOT optional

Thoughts?

(The first idea is bugging me because I swear I had thought of a show-stopper 
over lunch, but for the life of me, I can't recall what it was).

Cheers,
Ovid

done_testing()

[Michael G Schwern]

Well it's about damn time.

  use Test::More;

  pass;
  done_testing;

This now works and effectively replaces no_plan.

This works, too:

  use Test::More;

  pass;
  done_testing(1);

I was expecting it to be hard and require a lot of reworking the tests.  It
turns out it was really easy, so I feel I probably missed some edge cases.
This might also mean the coverage isn't so good.  Here's the patch for review.
http://github.com/schwern/test-more/commit/5cd0ad7c09187fd9244a0153e1fe0545acfe454e

Pay particular attention to the way done_testing() interacts with existing
plans.  For example, this is legal:

  use Test::More tests => 2;

  pass;
  pass;
  done_testing(2);

This is legal, but it's a failure:

  use Test::More tests => 2;

  pass;
  pass;
  done_testing(3);

And it produces this:

1..2
ok 1
ok 2
not ok 3 - planned to run 2 but done_testing() expects 3
#   Failed test 'planned to run 2 but done_testing() expects 3'
#   at lib/Test/More.pm line 221.
# Looks like you planned 2 tests but ran 3.
# Looks like you failed 1 test of 3 run.

An anomaly in done_testing() is expressed as an additional failed test.  For
example.

  use Test::More;

  pass;
  pass;
  done_testing();
  done_testing();

ok 1
ok 2
1..2
not ok 3 - done_testing() was already called at line 221
#   Failed test 'done_testing() was already called at line 221'
#   at lib/Test/More.pm line 221.
# Looks like you planned 2 tests but ran 3.
# Looks like you failed 1 test of 3 run.

Which is a little weird.  I'm not entirely happy with it, but it will
certainly result in a failure and its the most reliable way to produce one.

Finally, this makes it now possible to build up the test plan as you go.  I'd
like to put first order support into Test::Builder and Test::More for it, but
for the moment this will work:

  use Test::More;

  my $tests = 2;
  pass;
  pass;

  $tests += 1;
  pass;

  done_testing($tests);


-- 
40. I do not have super-powers.
-- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army
   http://skippyslist.com/list/