Re: shouldn't "UNEXPECTEDLY SUCCEEDED" mean failure?

[Michael G Schwern]

I'm going to sum up this reply, because it got long but kept on the same themes.

*  TODO tests provide you with information about what tests the author decided
to ignore.
**  Commented out tests provide you with NO information.
**  Most TODO tests would have otherwise been commented out.

*  How you interpret that information is up to you.
**  Most folks don't care, so the default is to be quiet.

*  The decision for what is success and what is failure lies with the author
**  There's nothing we can do to stop that.
**  But TODO tests allow you to reinterpret the author's desires.

*  TAP::Harness (aka Test::Harness 3) has fairly easy ways to control how
   TODO tests are interpreted.
**  It could be made easier, especially WRT controlling "make test"
**  CPAN::Reporter could be made aware of TODO passes.


Fergal Daly wrote:
> On 05/12/2007, Michael G Schwern <[EMAIL PROTECTED]> wrote:
>> This this whole discussion has unhinged a bit from reality, maybe you can 
>> give
>> some concrete examples of the problems you're talking about?  You obviously
>> have some specific breakdowns in mind.
> 
> I don't, I'm arguing against what has been put forward as good
> practice when there are other better practices that are approximately
> as easy and don't have the same downsides.
> 
> In fairness though these bad practices were far more strongly
> advocated in the previous thread on this topic than in this one.

I don't know what thread that was, or if I was involved, so maybe I'm not the
best person to be arguing with.


>> The final choice, incrementing the dependency version to one that does not 
>> yet
>> exist, boils down to "it won't work".  It's also ill advised to anticipate
>> that version X+1 will fix a given bug as on more than one occasion an
>> anticipated bug has not been fixed in the next version.
> 
> As I said earlier though, in Module::Build you have the option of
> saying version < X and then when it's finally fixed, you can say !X
> (and !X+1 if that didn't fix it).

Yep, rich dependencies are helpful.


>> There is also the "I don't think feature X works in Y environment" problem.
>> For example, say you have something that depends on symlinks.  You could hard
>> code in your test to skip if on Windows or some such, but that's often too
>> broad.  Maybe they'll add them in a later version, or with a different
>> filesystem (it's happened on VMS) or with some fancy 3rd party hack.  It's
>> nice to get that information back.
> 
> How do you get this information back? Unexpected passes are not
> reported to you. If you want to be informed about things like this a
> TODO is not a very good way to do it.

The TODO test is precisely the way to do it, it provides all the information
needed.  We just don't have the infrastructure to report it back.

As discussed before, what's needed is a higher resolution then just "pass" and
"fail" for the complete test run.  That's the "Result: PASS/TODO" discussed
earlier.  Things like CPAN::Reporter could then send that information back to
the author.  It's a fairly trivial change for Test::Harness.

The important thing is that "report back" is no longer locked to "fail".


>>> I'm talking about people converting tests that were working just fine
>>> to be TODO tests because the latest version of Foo (an external
>>> module) has a new bug. While Foo is broken, they don't want lots of
>>> bug reports from CPAN testers that they can't do anything about.
>>>
>>> This use of TODO allows you to silence the alarm and also gives you a
>>> way to spot when the alarm condition has passed. It's convenient for
>>> developers but it's 2 fingers to users who can now get false passes
>>> from the test suites,
>> It still boils down to what known bugs the author is willing to release with.
>>  Once the author has decided they don't want to hear about a broken
>> dependency,  and that the breakage isn't important, the damage is done.  The
>> TODO test is orthogonal.
>>
>> Again, consider the alternative which is to comment the test out.  Then you
>> have NO information.
> 
> Who's "you"?

You == user.


> If you==user then a failing TODO test and commented out test are
> indistinguishable unless you go digging in the code or TAP stream.

As they say, "works as designed".  The author decided the failures aren't
important.  Don't like it?  Take it up with the author.  Most folks don't care
about that information, they just want the thing installed.

You (meaning Fergal Daly) can dig them out with some Test::Harness hackery,
and maybe that should be easier if you really care about it.  The important
thing is the information is there, encoded in the tests, and you can get at it
programatically.

The alternative is to comment the failing test out in which case you have *no*
information and those who are interested cannot get it out.


> A passing TODO is just confusing.

That's a function of how it's displayed.  "UNEXPECTEDLY SUCCEEDED", I agree,
was confusing.  No question.  TH 3's 

Re: Why not run a test without a plan?

[Michael G Schwern]

Eric Wilhelm wrote:
>> A. Pagaltzis wrote:
>>> ...
>>> which would still be an error. That way a mistake in a test
>>> script won’t lead to Test::More silently converting an up-front
>>> plan declarations into trailing ones.
>> Which brings us back to the original question:  why should that be an
>> error?
> 
> It's a matter of stricture?  If the developer intended the plan to be 
> before-hand, they'll be expecting an error to enforce that practice.

Why do they care if the plan is output at the beginning or end?  How does this
stricture improve the quality of the test?  What mistake does it prevent?  If
we were to propose the "no test without a plan" stricture today, what would
the arguments in favor be?

I'm not worried about the shock of violating existing programmer's calcified
minor expectations.  They'll live.

About the only thing I can think of is consistency.  skip_all must still come
first, so that this sort of thing will sometimes work, sometimes not,
depending on $^O.

use Test::More;

if( $^O eq 'BrokenOS' ) {
plan skip_all => 'Your shit is broke';
}
else {
plan tests => 42;
}

BEGIN { use_ok 'Some::Module' }

This might be solved by the oft-requested "skip_rest".

use Test::More tests => 42;

skip_rest("Your shit is broke") if $^O eq 'BrokenOS';

BEGIN { use_ok 'Some::Module' }

Hmm, it's also shorter.  This even allows something like this:

use Test::More tests => 42;

BEGIN {
use_ok 'Optional::Module' ||
skip_rest('Optional::Module not available');
}


> The current planning functions impose strictures.  (Yes, it happens to 
> be due to an old implementation detail which no longer governs -- but 
> that doesn't change the fact that behavior expectations have already 
> been set.)  Taking away the error essentially means that you've changed 
> the API.  Imagine if strict.pm suddenly stopped being strict about 
> symbolic refs.
>
> That is, users should somehow be able to rely on the same strictures 
> that they've had in the past (hopefully by-default.)  So, either this 
> new non-strict plan scheme should be declared in the import() params or 
> be not named plan().

I see where you're going, but I think this is going too far wrt backwards
compatibility.

The strict analogy is spurious because that would be changing a fundamental
part of strict.  Whereas this is an incidental part of Test::More.  Scale does
matter.

Furthermore, it's not going to cause any passing tests to fail, or any
legitimately failing tests (ie. due to a real bug, not Test::More stricture)
to pass.

The only breakage I can think of are all highly convoluted and improbable,
where you've somehow written a test that checks that this specific feature
works.  But the only one who should be doing that is Test::More's own tests.
Or some highly paranoid dependent on that specific feature, in which case
congratulations!  Your test did its job!

I'm not worried.


> I'm still wishing for the "plan to make it to a given statement" model 
> (e.g. done().)

Honestly all that's really holding that up is a good name for the plan style
and "I'm done testing" terminator.  Nothing has really lept out at me yet.
Maybe something as straight forward as...

plan 'until_done';



done_testing;


-- 
Ahh email, my old friend.  Do you know that revenge is a dish that is best
served cold?  And it is very cold on the Internet!

Re: Why not run a test without a plan?

[Chris Dolan]

On Dec 4, 2007, at 9:06 AM, David Golden wrote:


On Dec 4, 2007 12:40 AM, Chris Dolan <[EMAIL PROTECTED]> wrote:

Or maybe not reinvent the wheel and get tons of extra functionality
for free!

use base 'Test::Class';
sub block : Test(2) {

   ok(1, "wibble");
   ok(1, "wobble");
}


But if it goes into Test::More, it eventually goes into core.  :-)

David


Are you proposing that Test::Class go in core?  I second the  
motion!  :-)


Chris

Re: shouldn't "UNEXPECTEDLY SUCCEEDED" mean failure?

[Fergal Daly]

On 05/12/2007, Michael G Schwern <[EMAIL PROTECTED]> wrote:
> This this whole discussion has unhinged a bit from reality, maybe you can give
> some concrete examples of the problems you're talking about?  You obviously
> have some specific breakdowns in mind.

I don't, I'm arguing against what has been put forward as good
practice when there are other better practices that are approximately
as easy and don't have the same downsides.

In fairness though these bad practices were far more strongly
advocated in the previous thread on this topic than in this one.

> Fergal Daly wrote:
> >> Modules do not have a binary state of working or not working.  They're
> >> composed of piles of (often too many) features.  Code can be shippable 
> >> without
> >> every single thing working.
> >
> > You're right, I was being binary, but you were being unary. There are 3 
> > cases,
> >
> > 1 the breakage was not so important, so you don't bail no matter what
> > version you find.
> > 2 it's fuzzy, maybe it's OK to use Foo version X but once Foo version
> > X+1 has been released you want to force people to use it
> > 3 the breakage is serious, you always want to bail if you find Foo
> > version X (and so you definitely don't switch the tests to TODO).
> >
> > You claimed 2 is always the case.  I claimed that 1 and 3 occur.
>
> If I did, that wasn't my intent.  I only talked about #2 because it's the only
> one that results in the user seeing passing TODO tests, which is what we were
> talking about.
>
>
> > I'm
> > happy to say admit that 2 can also occur. The point remains, you would
> > not necessarily change your modules requirements as a reaction to X+1
> > being released. You might, or you might change it beforehand if it
> > really matters or you might not change it at all.
>
> And I might dip my head in whipped cream and go give a random stranger a foot
> bath.  You seem to have covered all possibilities, good and bad.  I'm not sure
> to what end.
>
> The final choice, incrementing the dependency version to one that does not yet
> exist, boils down to "it won't work".  It's also ill advised to anticipate
> that version X+1 will fix a given bug as on more than one occasion an
> anticipated bug has not been fixed in the next version.

As I said earlier though, in Module::Build you have the option of
saying version < X and then when it's finally fixed, you can say !X
(and !X+1 if that didn't fix it).

> Anyhow, to get back to the point, it boils down to an author's decision how to
> deal with a known bug.  TODO tests are orthogonal.
>
>
> >> Maybe we're arguing two different situations.  Yours seems to be when 
> >> there is
> >> a broken version of a dependency, but a known working version exists.  In 
> >> this
> >> case, you're right, it's better resolved with a rich dependency system.
> >
> > I think maybe we are.
> >
> > You're talking about where someone writes a TODO for a feature that
> > has never worked. That's legit, although I still think there's
> > something odd about it as you personally have nothing "to do". I agree
> > it's not dangerous.
>
> Sure you do, you have to watch for when the dependency fixes its bug.  But
> that's boring and rote, what computers are for!  So you write a TODO test to
> automate the process.  [1]

That's back on the other case, I'm only talking about taking an
existing, previously passing test and marking it TODO.

> In a large project, sometimes things get implemented when you implement other
> things.  This is generally more applicable to bugs, but sometimes to minor
> features.
>
> Then there are folks who embrace the whole test first thing and write out lots
> and lots of tests beforehand.  Maybe you decide not to implement them all
> before shipping.  Rather than delete or comment out those tests, just wrap
> them in TODO blocks.  Then you don't have to do any fiddling with the tests
> before and after release, something which leads to an annoying shear between
> the code the author uses and the code users use.

That's all fine.

> There is also the "I don't think feature X works in Y environment" problem.
> For example, say you have something that depends on symlinks.  You could hard
> code in your test to skip if on Windows or some such, but that's often too
> broad.  Maybe they'll add them in a later version, or with a different
> filesystem (it's happened on VMS) or with some fancy 3rd party hack.  It's
> nice to get that information back.

How do you get this information back? Unexpected passes are not
reported to you. If you want to be informed about things like this a
TODO is not a very good way to do it.

I would say you should test if the feature is there. If it is, run the
tests and enable the feature, if not don't run the tests and disable
the feature.

I think conditional enabling of not-so-important features depending on
test results is actually a far better way to do this, although we have
no infrastructure for that at the moment.

> > I'm talking about peopl

Re: Why not run a test without a plan?

[Michael G Schwern]

Geoffrey Young wrote:
> I guess what I thought you were getting at was a natural decoupling of
> comparison functions with the planning without all the hackery involved
> to get that sepraration working now.  so I was suggesting that the
> decoupling go further than just no_plan, and that yeah, rock on, great
> idea.  'tis all :)

I see what you're getting at.  I don't think I'm going that far, though I'm
willing to help somehow with the "I want to paste a bunch of subtest processes
together" problem.

One of the original issues Test::More was designed to deal with was the
problem of running individual tests without having to parse the test output
(by eye or by computer) to get an accurate result.

That's why it changes the exit code on failure and why it has the ending
diagnostic message if there's a failure.  While prove and TAP::Parser help, I
still like running tests by hand to get complete control.


-- 
If at first you don't succeed--you fail.
-- "Portal" demo

Re: Why not run a test without a plan?

[Eric Wilhelm]

# from Michael G Schwern
# on Tuesday 04 December 2007 15:24:

>A. Pagaltzis wrote:
>>...
>> which would still be an error. That way a mistake in a test
>> script won’t lead to Test::More silently converting an up-front
>> plan declarations into trailing ones.
>
>Which brings us back to the original question:  why should that be an
> error?

It's a matter of stricture?  If the developer intended the plan to be 
before-hand, they'll be expecting an error to enforce that practice.

The current planning functions impose strictures.  (Yes, it happens to 
be due to an old implementation detail which no longer governs -- but 
that doesn't change the fact that behavior expectations have already 
been set.)  Taking away the error essentially means that you've changed 
the API.  Imagine if strict.pm suddenly stopped being strict about 
symbolic refs.

That is, users should somehow be able to rely on the same strictures 
that they've had in the past (hopefully by-default.)  So, either this 
new non-strict plan scheme should be declared in the import() params or 
be not named plan().

E.g. add_to_plan() for the plan-as-you-go usage and plan_was() for 
planning at the end or something.

I'm still wishing for the "plan to make it to a given statement" model 
(e.g. done().)

--Eric
-- 
"It is impossible to make anything foolproof because fools are so 
ingenious."
--Murphy's Second Corollary
---
http://scratchcomputing.com
---

Re: shouldn't "UNEXPECTEDLY SUCCEEDED" mean failure?

[Michael G Schwern]

This this whole discussion has unhinged a bit from reality, maybe you can give
some concrete examples of the problems you're talking about?  You obviously
have some specific breakdowns in mind.


Fergal Daly wrote:
>> Modules do not have a binary state of working or not working.  They're
>> composed of piles of (often too many) features.  Code can be shippable 
>> without
>> every single thing working.
> 
> You're right, I was being binary, but you were being unary. There are 3 cases,
> 
> 1 the breakage was not so important, so you don't bail no matter what
> version you find.
> 2 it's fuzzy, maybe it's OK to use Foo version X but once Foo version
> X+1 has been released you want to force people to use it
> 3 the breakage is serious, you always want to bail if you find Foo
> version X (and so you definitely don't switch the tests to TODO).
>
> You claimed 2 is always the case.  I claimed that 1 and 3 occur.

If I did, that wasn't my intent.  I only talked about #2 because it's the only
one that results in the user seeing passing TODO tests, which is what we were
talking about.


> I'm
> happy to say admit that 2 can also occur. The point remains, you would
> not necessarily change your modules requirements as a reaction to X+1
> being released. You might, or you might change it beforehand if it
> really matters or you might not change it at all.

And I might dip my head in whipped cream and go give a random stranger a foot
bath.  You seem to have covered all possibilities, good and bad.  I'm not sure
to what end.

The final choice, incrementing the dependency version to one that does not yet
exist, boils down to "it won't work".  It's also ill advised to anticipate
that version X+1 will fix a given bug as on more than one occasion an
anticipated bug has not been fixed in the next version.

Anyhow, to get back to the point, it boils down to an author's decision how to
deal with a known bug.  TODO tests are orthogonal.


>> Maybe we're arguing two different situations.  Yours seems to be when there 
>> is
>> a broken version of a dependency, but a known working version exists.  In 
>> this
>> case, you're right, it's better resolved with a rich dependency system.
> 
> I think maybe we are.
> 
> You're talking about where someone writes a TODO for a feature that
> has never worked. That's legit, although I still think there's
> something odd about it as you personally have nothing "to do". I agree
> it's not dangerous.

Sure you do, you have to watch for when the dependency fixes its bug.  But
that's boring and rote, what computers are for!  So you write a TODO test to
automate the process.  [1]

In a large project, sometimes things get implemented when you implement other
things.  This is generally more applicable to bugs, but sometimes to minor
features.

Then there are folks who embrace the whole test first thing and write out lots
and lots of tests beforehand.  Maybe you decide not to implement them all
before shipping.  Rather than delete or comment out those tests, just wrap
them in TODO blocks.  Then you don't have to do any fiddling with the tests
before and after release, something which leads to an annoying shear between
the code the author uses and the code users use.

There is also the "I don't think feature X works in Y environment" problem.
For example, say you have something that depends on symlinks.  You could hard
code in your test to skip if on Windows or some such, but that's often too
broad.  Maybe they'll add them in a later version, or with a different
filesystem (it's happened on VMS) or with some fancy 3rd party hack.  It's
nice to get that information back.


> I'm talking about people converting tests that were working just fine
> to be TODO tests because the latest version of Foo (an external
> module) has a new bug. While Foo is broken, they don't want lots of
> bug reports from CPAN testers that they can't do anything about.
> 
> This use of TODO allows you to silence the alarm and also gives you a
> way to spot when the alarm condition has passed. It's convenient for
> developers but it's 2 fingers to users who can now get false passes
> from the test suites,

It still boils down to what known bugs the author is willing to release with.
 Once the author has decided they don't want to hear about a broken
dependency,  and that the breakage isn't important, the damage is done.  The
TODO test is orthogonal.

Again, consider the alternative which is to comment the test out.  Then you
have NO information.

So I think the problem you're concerned with is poor release decisions.  TODO
tests are just a tool being employed therein.


[1] Don't get to hung up on names, things only get one even though they can do
lots of things.  I'm sure you've written lots of perl programs that didn't do
much extracting or reporting.


-- 
Reality is that which, when you stop believing in it, doesn't go away.
-- Phillip K. Dick

Re: shouldn't "UNEXPECTEDLY SUCCEEDED" mean failure?

[Eric Wilhelm]

# from Fergal Daly
# on Tuesday 04 December 2007 15:12:

>I'm talking about people converting tests that were working just fine
>to be TODO tests because the latest version of Foo (an external
>module) has a new bug. While Foo is broken, they don't want lots of
>bug reports from CPAN testers that they can't do anything about.

To me, the fact that we're discussing TODO tests in this context is an 
indicator that the testchain badly needs richer reporting and/or 
there's something missing in the CPAN/CPAN testers system/workflow.

--Eric
-- 
Anyone who has the power to make you believe absurdities has the power
to make you commit injustices.
--Voltaire
---
http://scratchcomputing.com
---

Re: Why not run a test without a plan?

[Geoffrey Young]

Michael G Schwern wrote:
> Geoffrey Young wrote:
>> Andy Armstrong wrote:
>>> On 4 Dec 2007, at 15:22, Geoffrey Young wrote:
 it would be nice if this were enforced on the TAP-digestion side and not
 from the TAP-emitter side - the coupling of TAP rules within the
 TAP-emitter is what lead to my trouble in the first place.
>>> A valid plan - at the beginning or the end - is required by Test::Harness.
>> yup, I get that.  but that has nothing to do with the Test::More errors
>> that started the thread - I ought to be able to use is() functionality
>> to emit into whatever stream I want and not have it complain about
>> missing plans, especially when Test::Harness will catch malformed TAP
>> and complain anyway... if I decide to send it to Test::Harness, which I
>> may not.
> 
> You can turn off all the ending checks with Test::Builder->no_ending(1) and
> the header being printed with no_header(1).  That's what we came up with back
> then.
> http://www.nntp.perl.org/group/perl.qa/2006/07/msg6212.html

yup, I've been doing that.

I guess what I thought you were getting at was a natural decoupling of
comparison functions with the planning without all the hackery involved
to get that sepraration working now.  so I was suggesting that the
decoupling go further than just no_plan, and that yeah, rock on, great
idea.  'tis all :)

--Geoff

Re: UNKNOWN despite only failing tests -- how come?

[Michael G Schwern]

Andreas J. Koenig wrote:
> Bug in CPAN::Reporter and/or Test::Harness and/or CPAN.pm?
> 
>   http://www.nntp.perl.org/group/perl.cpan.testers/796974
>   http://www.nntp.perl.org/group/perl.cpan.testers/825449
> 
> All tests fail but Test::Harness reports NOTESTS and CPAN::Reporter
> concludes UNKNOWN and CPAN.pm then installs it.

Test::Harness bug where it concludes "NOTESTS" if it sees no test output, as
is the case when every test dies.  I'll see about fixing it.


-- 
On error resume stupid

Re: Why not run a test without a plan?

[Michael G Schwern]

Geoffrey Young wrote:
> 
> Andy Armstrong wrote:
>> On 4 Dec 2007, at 15:22, Geoffrey Young wrote:
>>> it would be nice if this were enforced on the TAP-digestion side and not
>>> from the TAP-emitter side - the coupling of TAP rules within the
>>> TAP-emitter is what lead to my trouble in the first place.
>>
>> A valid plan - at the beginning or the end - is required by Test::Harness.
> 
> yup, I get that.  but that has nothing to do with the Test::More errors
> that started the thread - I ought to be able to use is() functionality
> to emit into whatever stream I want and not have it complain about
> missing plans, especially when Test::Harness will catch malformed TAP
> and complain anyway... if I decide to send it to Test::Harness, which I
> may not.

You can turn off all the ending checks with Test::Builder->no_ending(1) and
the header being printed with no_header(1).  That's what we came up with back
then.
http://www.nntp.perl.org/group/perl.qa/2006/07/msg6212.html


-- 
Hating the web since 1994.

Re: Why not run a test without a plan?

[Michael G Schwern]

A. Pagaltzis wrote:
> That would work. Of course once you have that, you don’t need to
> allow assertions to run without a plan, since one can always say
> 
> use Test::More tests => variable => 0;
> pass();
> plan add_tests => 2;
> pass();
> 
> instead of
> 
> use Test::More;
> pass();
> plan tests => 2;
> pass();
> 
> which would still be an error. That way a mistake in a test
> script won’t lead to Test::More silently converting an up-front
> plan declarations into trailing ones.

Which brings us back to the original question:  why should that be an error?


-- 
There will be snacks.

Re: shouldn't "UNEXPECTEDLY SUCCEEDED" mean failure?

[Fergal Daly]

On 02/12/2007, Michael G Schwern <[EMAIL PROTECTED]> wrote:
> Fergal Daly wrote:
> >> As long as you're releasing a new version, why would you not upgrade your
> >> module's dependency to use the version that works?
> >
> > Your module either is or isn't usable with version X of Foo.
> >
> > If it is usable then you would not change your dependency before or
> > after the bug in version X is fixed (maybe I have a good reason not to
> > upgrade Foo and you wouldn't want your module to refuse to install if
> > it is actually usable).
> >
> > If it isn't usable then marking your tests as TODO was the wrong thing
> > to do in the first place, you should have bailed out due to
> > incompatibility with version X and not bothered to run any tests at
> > all. I think Extutils::MM does not have any way to specify complex
> > version dependencies but with Module::Build you could say
>
> ETOOBINARY
>
> Modules do not have a binary state of working or not working.  They're
> composed of piles of (often too many) features.  Code can be shippable without
> every single thing working.

You're right, I was being binary, but you were being unary. There are 3 cases,

1 the breakage was not so important, so you don't bail no matter what
version you find.
2 it's fuzzy, maybe it's OK to use Foo version X but once Foo version
X+1 has been released you want to force people to use it
3 the breakage is serious, you always want to bail if you find Foo
version X (and so you definitely don't switch the tests to TODO).

You claimed 2 is always the case. I claimed that 1 and 3 occur. I'm
happy to say admit that 2 can also occur. The point remains, you would
not necessarily change your modules requirements as a reaction to X+1
being released. You might, or you might change it beforehand if it
really matters or you might not change it at all.

> The TODO test is useful when the working version *does not yet* exist.  If
> it's a minor feature or bug then rather than hold up the whole release waiting
> for someone else to fix their shit, you can mark it TODO and release.  This is
> the author's decision to go ahead and release with a known bug.  We do it all
> the time, just not necessarily with a formal TODO test.
>
> > I am basically against the practice of using TODO to cope with
> > external breakage. Not taking unexpected passes seriously encourages
> > this practice. Apart from there being other ways to handle external
> > breakage that seem easier, using TODO is actually dangerous as it can
> > cause false passes in 2 ways. Says version X of Foo has a non-serious
> > bug so you release version Y of Bar with some tests marked TODO. The
> > we risk
>
> Maybe we're arguing two different situations.  Yours seems to be when there is
> a broken version of a dependency, but a known working version exists.  In this
> case, you're right, it's better resolved with a rich dependency system.

I think maybe we are.

You're talking about where someone writes a TODO for a feature that
has never worked. That's legit, although I still think there's
something odd about it as you personally have nothing "to do". I agree
it's not dangerous.

I'm talking about people converting tests that were working just fine
to be TODO tests because the latest version of Foo (an external
module) has a new bug. While Foo is broken, they don't want lots of
bug reports from CPAN testers that they can't do anything about.

This use of TODO allows you to silence the alarm and also gives you a
way to spot when the alarm condition has passed. It's convenient for
developers but it's 2 fingers to users who can now get false passes
from the test suites,

F

> My case is when a working version of the dependency does not exist, or the
> last working version is so old it's more trouble than it's worth.  In this
> case the author decides the bug is not critical, can't be worked around and
> doesn't want to wait for fix in the dependency.  The decision is whether or
> not to release with a known bug.  After that, wrapping it in a TODO test is
> just an alternative to commenting it out.
>
> Compare with the more common alternative for shipping with a known bug which
> is to simply not have a test at all.
>
>
> > 1 Version X+1 of Foo is even worse and will cause Bar to eat your dog.
> > Sadly for your dog, the test that might have warned him has been
> > marked TODO.
>
> If they release Bar with a known bug against Foo X where your dog's fur is
> merely a bit ruffled, then that's ok.  If version X+1 of Foo causes Bar to eat
> your dog then why didn't their tests catch that?  Was there not a "dog not
> eaten" test?  If not then that's just an incomplete test, the TODO test has
> nothing to do with that.
>
> The "dog not eaten" test wouldn't have been part of the TODO test, that part
> worked fine when the author released and they'd have gotten the "todo passed"
> message and known to move it out of the TODO block.
>
> Or maybe they're just a cat person.
>
> Point is, there's multiple 

UNKNOWN despite only failing tests -- how come?

[Andreas J. Koenig]

Bug in CPAN::Reporter and/or Test::Harness and/or CPAN.pm?

  http://www.nntp.perl.org/group/perl.cpan.testers/796974
  http://www.nntp.perl.org/group/perl.cpan.testers/825449

All tests fail but Test::Harness reports NOTESTS and CPAN::Reporter
concludes UNKNOWN and CPAN.pm then installs it.

Your opinions welcome,
-- 
andreas

Re: UNKNOWN despite only failing tests -- how come?

[David Golden]

CPAN::Reporter says explicitly that UNKNOWN will return success to
CPAN.pmand not prevent installation.  I think this is a Test::Harness
bug.

David

On Dec 4, 2007 3:24 PM, Andreas J. Koenig <
[EMAIL PROTECTED]> wrote:

> Bug in CPAN::Reporter and/or Test::Harness and/or CPAN.pm?
>
>  http://www.nntp.perl.org/group/perl.cpan.testers/796974
>  http://www.nntp.perl.org/group/perl.cpan.testers/825449
>
> All tests fail but Test::Harness reports NOTESTS and CPAN::Reporter
> concludes UNKNOWN and CPAN.pm then installs it.
>
> Your opinions welcome,
> --
> andreas
>

Re: Why not run a test without a plan?

[Geoffrey Young]

Andy Armstrong wrote:
> On 4 Dec 2007, at 15:22, Geoffrey Young wrote:
>> it would be nice if this were enforced on the TAP-digestion side and not
>> from the TAP-emitter side - the coupling of TAP rules within the
>> TAP-emitter is what lead to my trouble in the first place.
> 
> 
> A valid plan - at the beginning or the end - is required by Test::Harness.

yup, I get that.  but that has nothing to do with the Test::More errors
that started the thread - I ought to be able to use is() functionality
to emit into whatever stream I want and not have it complain about
missing plans, especially when Test::Harness will catch malformed TAP
and complain anyway... if I decide to send it to Test::Harness, which I
may not.

--Geoff

Re: Why not run a test without a plan?

[Andy Armstrong]

On 4 Dec 2007, at 15:22, Geoffrey Young wrote:
it would be nice if this were enforced on the TAP-digestion side and  
not

from the TAP-emitter side - the coupling of TAP rules within the
TAP-emitter is what lead to my trouble in the first place.



A valid plan - at the beginning or the end - is required by  
Test::Harness.


--
Andy Armstrong, Hexten

Re: Why not run a test without a plan?

[Geoffrey Young]

> Since the technical restriction is gone, and I see no particular benefit to it
> being there, and it eliminates some tricky plan counting situations, I don't
> see why it shouldn't be removed.

this would be great, and help with situations like this:

  http://www.nntp.perl.org/group/perl.qa/2006/02/msg5445.html

> 
> 
> PS  To be clear, a plan is still eventually needed before the test exits.

it would be nice if this were enforced on the TAP-digestion side and not
from the TAP-emitter side - the coupling of TAP rules within the
TAP-emitter is what lead to my trouble in the first place.

--Geoff

Re: Why not run a test without a plan?

[David Golden]

On Dec 4, 2007 12:40 AM, Chris Dolan <[EMAIL PROTECTED]> wrote:
> Or maybe not reinvent the wheel and get tons of extra functionality
> for free!
>
> use base 'Test::Class';
> sub block : Test(2) {
>
>ok(1, "wibble");
>ok(1, "wobble");
> }

But if it goes into Test::More, it eventually goes into core.  :-)

David

Re: Why not run a test without a plan?

[A. Pagaltzis]

* Michael G Schwern <[EMAIL PROTECTED]> [2007-12-04 11:05]:
> A. Pagaltzis wrote:
> >* Michael G Schwern <[EMAIL PROTECTED]> [2007-12-04 03:35]:
> >> It also makes it technically possible to allow the test to
> >> change it's plan mid-stream
> > 
> > Without some hypothetical future version of TAP this is only
> > possible if you have run tests before declaring a plan at
> > all, because otherwise the plan will already have been output
> > as the first line of the TAP stream.
> 
> Just needs a way to declare that you're going to add to the
> plan up front.

That would work. Of course once you have that, you don’t need to
allow assertions to run without a plan, since one can always say

use Test::More tests => variable => 0;
pass();
plan add_tests => 2;
pass();

instead of

use Test::More;
pass();
plan tests => 2;
pass();

which would still be an error. That way a mistake in a test
script won’t lead to Test::More silently converting an up-front
plan declarations into trailing ones.


* Michael G Schwern <[EMAIL PROTECTED]> [2007-12-04 11:10]:
> Smylers wrote:
> > Wasn't there general agreement only a week or so ago to now
> > allow plans to be specified at the end rather than the start?
> > I was presuming that Schwern's suggestions were in the light
> > of this other change.
> 
> No, that was a much more involved thing which involves nested
> plans and multiple plans and such.

Actually, what I remember was about Ovid asking about adding a
way to say “I don’t want to decide on a plan yet but I’ll supply
one later.” AndyA brought up the nested test stuff in the course
of that thread but that issue was not central to Ovid’s RFC.

Essentially it was the same discussion as this one.


Regards,
-- 
Aristotle Pagaltzis // 

Re: Why not run a test without a plan?

[Chris Dolan]

On Dec 3, 2007, at 10:37 PM, Eric Wilhelm wrote:


# from David Golden
# on Monday 03 December 2007 19:55:

With some sugar, that could actually be quite handy for something  
like

test blocks.  E.g.:

{
  plan add => 2;
  ok( 1, "wibble" );
  ok(1, "wobble" );
}


or maybe make the block a sub

block {
  subplan 2;
  ok(1, "wibble");
  ok(1, "wobble");
};


Or maybe not reinvent the wheel and get tons of extra functionality  
for free!


use base 'Test::Class';
sub block : Test(2) {
  ok(1, "wibble");
  ok(1, "wobble");
}

Chris

RE: Why not run a test without a plan?

[Eric Harrison]

 
I would find something along those lines incredibly helpful.  Most of our tests 
have a variable number of tests based on a number of variables and so I use 
no_plan and every single test.  Being able to add to the plan within a block of 
code would be massivley helpful and would allow us to be able to use a plan.


Eric Harrison 
QA Developer

[EMAIL PROTECTED]
w: www.rea-group.com


REA Group refers to realestate.com.au Ltd (ASX:REA)

This email transmission may contain confidential information. If you have 
received this transmission in error, please notify us immediately on (61 3) 
9897 1121 or reply by email to the sender. You must destroy the email 
immediately and not use, copy, distribute or disclose the contents. Thank you.  


-Original Message-
From: Eric Wilhelm [mailto:[EMAIL PROTECTED] 
Sent: Tuesday, 4 December 2007 3:37 PM
To: perl-qa@perl.org
Subject: Re: Why not run a test without a plan?

# from David Golden
# on Monday 03 December 2007 19:55:

>With some sugar, that could actually be quite handy for something like 
>test blocks.  E.g.:
>
>{
>  plan add => 2;
>  ok( 1, "wibble" );
>  ok(1, "wobble" );
>}

or maybe make the block a sub

block {
  subplan 2;
  ok(1, "wibble");
  ok(1, "wobble");
};

The bit where prototypes "must have the & first" hurts, but then there's always 
λ{}. :-D

--Eric
--
The reasonable man adapts himself to the world; the unreasonable man persists 
in trying to adapt the world to himself. Therefore all progress depends on the 
unreasonable man.
--George Bernard Shaw
---
http://scratchcomputing.com
---

Re: Why not run a test without a plan?

[Michael G Schwern]

Smylers wrote:
>>> It also makes it technically possible to allow the test to change
>>> it's plan mid-stream
>> Without some hypothetical future version of TAP this is only possible
>> if you have run tests before declaring a plan at all, because
>> otherwise the plan will already have been output as the first line of
>> the TAP stream.
> 
> Wasn't there general agreement only a week or so ago to now allow plans
> to be specified at the end rather than the start?  I was presuming that
> Schwern's suggestions were in the light of this other change.

No, that was a much more involved thing which involves nested plans and
multiple plans and such.  This simply takes advantage of the existing ability
to put the plan at the end instead of at the front.

1..2
ok 1
ok 2

and

ok 1
ok 2
1..2

are equivalent test output.  This is how no_plan works and it's been around
since 2001.

Nothing new happened to allow this change, I just never really gave it thought
before.


-- 
I have a date with some giant cartoon robots and booze.

Re: Why not run a test without a plan?

[Michael G Schwern]

A. Pagaltzis wrote:
> Yes, so this should be allowed:
> 
> pass();
> plan 'no_plan';
> pass();
> 
> Whereas this should not:
> 
> pass();
> plan tests => 2;
> pass();

Umm, why not?  That's exactly what I was proposing and it would result in...

ok 1
ok 2
1..2


> Consider also:
> 
> pass();
> plan skip_all => 'Surprise!';
> pass();

Good point.  That wouldn't work, there's no way to express "skip_all" once a
test has been issued.  There are ways Test::More could cheat to make it work,
but that goes against it's intent to be as explicit as possible.  Running a
test and then stating that you're going to skip all tests is ambiguous.

It does splash some cold water on eliminating the common mistake of running a
use_ok() before deciding if you can or cannot run the tests.


>> It also makes it technically possible to allow the test to
>> change it's plan mid-stream
> 
> Without some hypothetical future version of TAP this is only
> possible if you have run tests before declaring a plan at all,
> because otherwise the plan will already have been output as the
> first line of the TAP stream.

Just needs a way to declare that you're going to add to the plan up front.


>> Since the technical restriction is gone, and I see no
>> particular benefit to it being there, and it eliminates some
>> tricky plan counting situations, I don't see why it shouldn't
>> be removed.
> 
> Because declaring a plan after running tests is effectively a
> no_plan and the programmer should be aware that that’s what they
> did. It’s fine if that’s their conscious choice; just make sure
> it was.

No, it's critically different from a no_plan in that the number of tests to be
run is still fixed by the programmer.  For example...

pass();
plan tests => 3;
pass();

Would produce...

ok 1
ok 2
1..3

Which would be a failure, just as if the plan was at the top.


-- 
I do have a cause though. It's obscenity. I'm for it.
- Tom Lehrer

Re: Why not run a test without a plan?

[Smylers]

A. Pagaltzis writes:

> Hi Michael,
> 
> * Michael G Schwern <[EMAIL PROTECTED]> [2007-12-04 03:35]:
> 
> >   use Test::More;
> >   pass();
> >   plan tests => 2;
> >   pass();
> > 
> > Why shouldn't this work?
> 
> ... this should be allowed:
> 
> pass();
> plan 'no_plan';
> pass();
> 
> Whereas this should not:
> 
> pass();
> plan tests => 2;
> pass();

Why not?  That's exactly what Schwern was suggesting should work.

> > It also makes it technically possible to allow the test to change
> > it's plan mid-stream
> 
> Without some hypothetical future version of TAP this is only possible
> if you have run tests before declaring a plan at all, because
> otherwise the plan will already have been output as the first line of
> the TAP stream.

Wasn't there general agreement only a week or so ago to now allow plans
to be specified at the end rather than the start?  I was presuming that
Schwern's suggestions were in the light of this other change.

Smylers

Re: Why not run a test without a plan?

[A. Pagaltzis]

Hi Michael,

* Michael G Schwern <[EMAIL PROTECTED]> [2007-12-04 03:35]:
>   use Test::More;
>   pass();
>   plan tests => 2;
>   pass();
> 
> Why shouldn't this work? Currently you get a "You tried to run
> a test without a plan" error, but what is it really protecting
> the test author from?
> 
> Historically, there was a clear technical reason. It used to be
> that the plan had to come first in the TAP output, so a plan
> had to come before any tests were run. Simple.
> 
> But that technical restriction no longer holds true. The plan
> can come at the end, primarily used for "no_plan". If a test is
> run before the plan is declared, simply delay the plan output
> until the end.

Yes, so this should be allowed:

pass();
plan 'no_plan';
pass();

Whereas this should not:

pass();
plan tests => 2;
pass();

So the error would be deferred from the first assertion run to
the `plan` invocation.

Consider also:

pass();
plan skip_all => 'Surprise!';
pass();

> It also makes it technically possible to allow the test to
> change it's plan mid-stream

Without some hypothetical future version of TAP this is only
possible if you have run tests before declaring a plan at all,
because otherwise the plan will already have been output as the
first line of the TAP stream.

> Since the technical restriction is gone, and I see no
> particular benefit to it being there, and it eliminates some
> tricky plan counting situations, I don't see why it shouldn't
> be removed.

Because declaring a plan after running tests is effectively a
no_plan and the programmer should be aware that that’s what they
did. It’s fine if that’s their conscious choice; just make sure
it was.

Regards,
-- 
Aristotle Pagaltzis //