Re: shouldn't "UNEXPECTEDLY SUCCEEDED" mean failure?

[A. Pagaltzis]

* Michael G Schwern <[EMAIL PROTECTED]> [2007-12-04 03:00]:
> So I read two primary statements here.
> 
> 1) Anything unexpected is suspicious. This includes unexpected
>success.
> 
> 2) Anything unexpected should be reported back to the author.
> 
> The first is controversial, and leads to the conclusion that
> TODO passes should fail.

The first doesn’t seem controversial to me; everyone agrees, I
think, that passing TODOs merit investigation in some sense or
other.

> The second is not controversial, but it erroneously leads to
> the conclusion that TODO passes should fail.

The second one wasn’t primarily proposed that I could see at all.
Some people brought it up, but it was not a particularly central
part of the discussion.

Eric mentioned asking the author, as part of investigating TODO
passes (which as I mentioned I think we all agree about). That
seems like a reasonable position to me; how it implies that TODO
passes should count as failures, I don’t understand.

> So what we need is a "pass with caveats" or, as Eric pointed
> out, some way for the harness to communicate it's results in
> a machine parsable way.

I thought that was already the overall thrust of the discussion.

I agree completely, in any case.

> Maybe "Result: TODO_PASS" is enough.

WFM.

Regards,
-- 
Aristotle Pagaltzis // 

Re: Why not run a test without a plan?

[Michael G Schwern]

Eric Wilhelm wrote:
> # from David Golden
> # on Monday 03 December 2007 19:55:
> 
>> With some sugar, that could actually be quite handy for something like
>> test blocks.  E.g.:
>>
>> {
>>   plan add => 2;
>>   ok( 1, "wibble" );
>>   ok(1, "wobble" );
>> }
> 
> or maybe make the block a sub
> 
> block {
>   subplan 2;
>   ok(1, "wibble");
>   ok(1, "wobble");
> };

I guess the unspoken benefit is block() can check the sub-plan when the
subroutine ref is done?

I'm always wary of using subs-as-blocks for general testing as they
transparently mess up the call stack which will effect testing anything that
plays with caller() (such as Carp).  Then you have to introduce more
complexity, like Sub::Uplevel, to mask that.


-- 
'All anyone gets in a mirror is themselves,' she said. 'But what you
gets in a good gumbo is everything.'
-- "Witches Abroad" by Terry Prachett

Re: Why not run a test without a plan?

[Michael G Schwern]

David Golden wrote:
> Michael G Schwern <[EMAIL PROTECTED]> wrote:
>> It also makes it technically possible to allow the test to change it's plan
>> mid-stream, though the consequences and interface for that do require some
>> thought.
> 
> With some sugar, that could actually be quite handy for something like
> test blocks.  E.g.:
> 
> {
>   plan add => 2;
>   ok( 1, "wibble" );
>   ok(1, "wobble" );
> }

Yep, something like that.  There will likely be a "change the plan" method in
Test::Builder at minimum.  Right now changing the expected number of tests is
tied to printing out the header.


-- 
 What we learned was if you get confused, grab someone and swing
  them around a few times
-- Life's lessons from square dancing

Re: Why not run a test without a plan?

[Eric Wilhelm]

# from David Golden
# on Monday 03 December 2007 19:55:

>With some sugar, that could actually be quite handy for something like
>test blocks.  E.g.:
>
>{
>  plan add => 2;
>  ok( 1, "wibble" );
>  ok(1, "wobble" );
>}

or maybe make the block a sub

block {
  subplan 2;
  ok(1, "wibble");
  ok(1, "wobble");
};

The bit where prototypes "must have the & first" hurts, but then there's 
always λ{}. :-D

--Eric
-- 
The reasonable man adapts himself to the world; the unreasonable man
persists in trying to adapt the world to himself. Therefore all progress
depends on the unreasonable man.
--George Bernard Shaw
---
http://scratchcomputing.com
---

Re: Why not run a test without a plan?

[David Golden]

Michael G Schwern <[EMAIL PROTECTED]> wrote:
> It also makes it technically possible to allow the test to change it's plan
> mid-stream, though the consequences and interface for that do require some
> thought.

With some sugar, that could actually be quite handy for something like
test blocks.  E.g.:

{
  plan add => 2;
  ok( 1, "wibble" );
  ok(1, "wobble" );
}

David

Why not run a test without a plan?

[Michael G Schwern]

use Test::More;
pass();
plan tests => 2;
pass();

Why shouldn't this work?  Currently you get a "You tried to run a test without
a plan" error, but what is it really protecting the test author from?

Historically, there was a clear technical reason.  It used to be that the plan
had to come first in the TAP output, so a plan had to come before any tests
were run.  Simple.

But that technical restriction no longer holds true.  The plan can come at the
end, primarily used for "no_plan".  If a test is run before the plan is
declared, simply delay the plan output until the end.

Removing this restriction eliminates some unnecessarily difficult planing
problems, especially of the annoying "the plan is calculated but I have to run
some tests at BEGIN time".  Like this common mistake:

use Test::More;

if( $something ) {
plan tests => 1;
}
else {
plan skip_all => "Because";
}

BEGIN { use_ok "Some::Module" }

It also allows you to run some tests before you determine the plan, though I
can't think of a particular use for this.

It also makes it technically possible to allow the test to change it's plan
mid-stream, though the consequences and interface for that do require some
thought.

Since the technical restriction is gone, and I see no particular benefit to it
being there, and it eliminates some tricky plan counting situations, I don't
see why it shouldn't be removed.


PS  To be clear, a plan is still eventually needed before the test exits.


-- 
The interface should be as clean as newly fallen snow and its behavior
as explicit as Japanese eel porn.

Re: shouldn't "UNEXPECTEDLY SUCCEEDED" mean failure?

[Michael G Schwern]

So I read two primary statements here.

1)  Anything unexpected is suspicious.  This includes unexpected success.

2)  Anything unexpected should be reported back to the author.

The first is controversial, and leads to the conclusion that TODO passes
should fail.

The second is not controversial, but it erroneously leads to the conclusion
that TODO passes should fail.  That's the only mechanism we currently have for
telling the user "hey, something weird happened.  Pay attention!"  It's also
how we normally report stuff back to the author.  Also there's only two easily
identifiable states for a test: Pass and fail.

So what we need is a "pass with caveats" or, as Eric pointed out, some way for
the harness to communicate it's results in a machine parsable way.  The very
beginnings of such a hack was put in for CPAN::Reporter in the "Result" line
that is output at the end of the test.  Ideally you'd have the harness
spitting out its full conclusions... somehow... without cluttering up the
human readable output.  But maybe "Result: TODO_PASS" is enough.


-- 
Stabbing you in the face for your own good.

Re: shouldn't "UNEXPECTEDLY SUCCEEDED" mean "unexpectedly succeeded"?

[Eric Wilhelm]

# from nadim khemir
# on Monday 03 December 2007 15:11:

>There is not simple solution to this problem.

There is, but the data needs to be more complicated than a boolean.

Please.  Calling it "failure" just gets everyone hung-up on semantics 
(and rightly so, because computers are awfully picky about semantics.)

>but I would like to be 
> able to check my modules more thorowly and I would like to be able to
> setup cpan so I can say "unexpectedly succeeded" == failed

I would be all-for harness enhancements where "unexpectedly succeeded" 
== "unexpectedly succeeded".  You could then treat it as "disallow 
unexpectedly succeeding results".

Adding machine-readable information to the output (or hooks to the API) 
and using it to apply policy is *much* different than optionally and 
arbitrarily defining $foo as failure (and, IMO, will cause fewer 
plagues of locusts.)

Now, as far as whether this means a non-zero return value from prove, or 
some bit of YAML at the end of the output or some-other-thing is quite 
a different question.

Also, how is this hooked into CPAN.pm?  (I suspect the linkage is 
limited to a boolean, which might mean you'll break CPAN::Reporter or 
something if you're not careful.)  Anybody more-in-the-know care to 
comment on how this works now?

The internal API of TAP::Harness already has a pretty rich system for 
determining what failed or passed or skipped or todo_passed -- why not 
use that to drive some kind of result_policy system?

--Eric
-- 
Those who cannot remember the past are condemned to repeat it.
--George Santayana
---
http://scratchcomputing.com
---

Re: shouldn't "UNEXPECTEDLY SUCCEEDED" mean failure?

[nadim khemir]

On Sunday 02 December 2007 18:48, Chris Dolan wrote:
> ...
>
> In this fashion, we use TODO tests to track when bugs in our
> dependent packages get fixed, and then when we can remove workarounds
> therefore.
>
> Chris


This discussion is interresting and it's always educating to understand why 
other do thing in what might first appear to be a strange way.

One of the things I wrote was that I wanted _my_ tests to fail on a passed 
TODO. This would let other use the process they see fit.

I have no doubt that any of us can deal with the "passed TODO" but we may not 
be the most representative developers. I like the principle of least surprise 
and passing todo are very very surprising, so surprising that (most) people 
writing Todos don't expect them to pass at all (I like to believe) and most 
people using modules might be surprised by that too;

How we deal with passed todos in _our_ modules is easy. The question is how do 
you deal with _other's modules_ passing todo? 

I mail the author, look in the code, mail here and discuss.

Most I believe, and it's regreatable, don't even bother looking. they scan 
for 'OK' and are happy with that. (I used to; now I like to see xx/yyy tests, 
that gives me a , false, sense of security) 

There is a third kind of module users. Entreprise users, people who follow a 
process, people investing money in development and that don't like to take 
risks. For those people, "Unexpectedly passed" screams "BROKEN".

There is not simple solution to this problem. but I would like to be able to 
check my modules more thorowly and I would like to be able to setup cpan so I 
can say "unexpectedly succeeded" == failed so I can avoid some less competent 
manager come and put a list of 'weird modules used in my project' under my 
nose.

Cheers, Nadim.

PS: some of you always send two mails, one to the original mail author and one 
to the list. it's OK but getting the information just once is, IMO, better.

Re: shouldn't "UNEXPECTEDLY SUCCEEDED" mean failure?

[Adrian Howard]

On 3 Dec 2007, at 04:34, Michael G Schwern wrote:
[snip]

This doesn't mean that people don't use dies_ok() when they should use
throws_ok().  Every tool is open to abuse.  The solution is not to  
remove the
tool, but figure out why it's being abused.  Maybe the answer is as  
simple as

putting throws_ok() first in the Test::Exception documentation?

[snip]

Which it has in the subversion repo for some time now. I just have to  
get my lazy arse in gear to release it. Which I will do once I've  
fixed the extant Win bug. Tonight probably.


Adrian 'in the TODO passes are not failures camp' Howard

Re: shouldn't "UNEXPECTEDLY SUCCEEDED" mean failure?

[Adrian Howard]

On 3 Dec 2007, at 10:26, A. Pagaltzis wrote:


* Ovid <[EMAIL PROTECTED]> [2007-12-02 16:50]:

Breaking the toolchain is bad.


You can almost imagine Curtis murmuring those words even in
his sleep…


I have lost count of the number of times Andy/Ovid said this over the  
LPW weekend :-)


Adrian

Re: shouldn't "UNEXPECTEDLY SUCCEEDED" mean failure?

[A. Pagaltzis]

* Ovid <[EMAIL PROTECTED]> [2007-12-02 16:50]:
> Breaking the toolchain is bad.

You can almost imagine Curtis murmuring those words even in
his sleep…

Regards,
-- 
Aristotle Pagaltzis // 

Re: skip vs todo vs $other_thing

[Eric Wilhelm]

# from Chris Dolan
# on Sunday 02 December 2007 17:35:

>The problem with skipped tests is that they're easier for developers  
>to ignore than TODO tests.

This is true, but if the test can correctly detect whether it should 
skip, that's probably a better way to go.  An option for the 
*developer* to force skipped tests to run would probably be a nice way 
to handle that -- particularly when you're running against an 
unreleased version of a dependency.

Maybe they should be DEPENDS tests.  (pun intended ;-)

--Eric
-- 
You can't whack a chisel without a big wooden mallet.
---
http://scratchcomputing.com
---