Re: What's the common denominator in these NYTProf failures?

[David Golden]

On Wed, Jan 28, 2009 at 8:50 AM, Tim Bunce  wrote:

> I'm strugling to find a common denominator in these test results:
>
>
> http://bbbike.radzeit.de/~slaven/cpantestersmatrix.cgi?dist=Devel-NYTProf+2.07_94
>
>
The common denominator appears to be that they're all from BinGOs.  Which
means they're all CPANPLUS-based smoke tests running on virtual machines.
I'm not sure what impact that has.

He's pretty responsive so you can email him at his cpan.org address.
(bin...@...)

For future reference, these kinds of questions are better directed to the
cpan-testers-discuss mailing list, which is regularly monitored by most of
the high-volume testers.

-- David

Re: TAP and the exit code

[Michael G Schwern]

Eric Wilhelm wrote:
> # from Michael G Schwern
> # on Tuesday 27 January 2009 18:16:
> 
>> For example, if a test program knows that it is going to
>> exit abnormally, even after it has finished emitting TAP according to
>> its plan, it should emit one final "not ok" line before exiting.
> 
> So,
> 
> 1.  TAP::Harness should store that when archiving?

Yes, it's just a test like any other.

1..3
ok 1
ok 2
ok 3
not ok 4 - exited with status 1

That will cause both a test failure and a plan failure.  Alternatively, you
can plan to issue a final exit status test, it's really stylistic.


> 2.  Is it possible to have something more explicit than that?  (or does 
> the exit code go in the test name or what?

At the moment, nothing more than the description.

In the future, yes.  This use case came up at the Oslo hackathon.  First TAP
extension allows structured diagnostics associated with each test, so you can
put the information in explicitly.

1..3
ok 1
ok 2
ok 3
not ok 4 - exited with status 1
  ---
  exit: 1
  ...

(I keep forgetting that the structured diagnostics are indented)

Beyond that, meta data for the whole test suite was proposed to record things
which otherwise would not be recorded such as time, versions, system
information, etc... as well as things like exit codes.
http://testanything.org/wiki/index.php/TAP_meta_information

It might look like this:

TAP version 15
  ---
  file:  foo.t
  date:  2009-01-28 16:26:07
  uname: Darwin 9.6.0
  ...
1..3
ok 1
ok 2
ok 3
  ---
  exit: 1
  ...

Or consider if you're reading the TAP stream from a web server and archiving it:

TAP version 15
  ---
  url:http://example.com/test.tap
  date:   2009-01-28 16:26:07
  server: Apache/2.2.9 (Unix)
  ...
1..2
ok 1
ok 2
  ---
  http status: 200
  ...

As you can see, it's ambiguous whether that final bit of information is
associated with the last test or is trailing suite meta data.  I don't recall
if we had a solution to that.


-- 
185. My name is not a killing word.
-- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army
   http://skippyslist.com/list/

RE: What's the common denominator in these NYTProf failures?

[Jan Dubois]

On Wed, 28 Jan 2009, David Cantrell wrote:
> On Wed, Jan 28, 2009 at 03:56:28PM +, Martin Evans wrote:
> > Not intel 64 bit int and multi-threaded perl then? Looked common to
> > all the failures and unless I missed one looked missing from the
> > successes.
>
> It passed on what I thought were my 64 bit threaded AMD builds, but
> from looking carefully at my -V it seems that they might not be 64 bit
> after all - despite building with '-de -Duse64bitall -Dusethreads', I
> have intsize=4 (but longsize=8 and ivtype=long), so I'm not sure if my
> perl is truly 64 bit or not.
> http://www.nntp.perl.org/group/perl.cpan.testers/2009/01/msg3116778.html

That is a 64-bit Perl. The important %Config value to look at is
ptrsize. If it is set to 8, then you have a full 64-bit Perl. If the
ivtype is set to a type that is 8 bytes, but ptrsize is 4, then you have
a 32-bit Perl with 64-bit integers (which from the OS point of view is a
32-bit process).

Cheers,
-Jan

Re: TAP and the exit code

[Eric Wilhelm]

# from Ovid
# on Wednesday 28 January 2009 07:57:

>> How would such a UI handle situations where all the tests pass, but
>> the plan fails (too many tests, too few tests or a missing plan)? If
>> those become flashing/stripped lines as well, then the current spec
>> works as written. If not, we should fix this up.
>
>It's all up to whomever implements it.  TAP should provide
> information, not dictate its usage.

Right.  And anticipating usage is a good way to decide how to 
distinguish information, but never a good excuse for not 
distinguishing.

--Eric
-- 
"I've often gotten the feeling that the only people who have learned
from computer assisted instruction are the authors."
--Ben Schneiderman
---
http://scratchcomputing.com
---

Re: What's the common denominator in these NYTProf failures?

[David Cantrell]

On Wed, Jan 28, 2009 at 03:56:28PM +, Martin Evans wrote:
> Tim Bunce wrote:
> >I'm strugling to find a common denominator in these test results:
> >
> >http://bbbike.radzeit.de/~slaven/cpantestersmatrix.cgi?dist=Devel-NYTProf+2.07_94
> Not intel 64 bit int and multi-threaded perl then? Looked common to all 
> the failures and unless I missed one looked missing from the successes.

It passed on what I thought were my 64 bit threaded AMD builds, but from
looking carefully at my -V it seems that they might not be 64 bit after
all - despite building with '-de -Duse64bitall -Dusethreads', I have
intsize=4 (but longsize=8 and ivtype=long), so I'm not sure if my perl
is truly 64 bit or not.
  http://www.nntp.perl.org/group/perl.cpan.testers/2009/01/msg3116778.html

-- 
header   FROM_DAVID_CANTRELLFrom =~ /david.cantrell/i
describe FROM_DAVID_CANTRELLMessage is from David Cantrell
scoreFROM_DAVID_CANTRELL15.72 # This figure from experimentation

Re: What's the common denominator in these NYTProf failures?

[David Cantrell]

On Wed, Jan 28, 2009 at 01:50:05PM +, Tim Bunce wrote:

> I'm strugling to find a common denominator in these test results:
> http://bbbike.radzeit.de/~slaven/cpantestersmatrix.cgi?dist=Devel-NYTProf+2.07_94

Try http://www.cpantesters.org/show/Devel-NYTProf.html (requires
Javascript), then in the drop-downs on the right, choose:

dist maturity: all releases

then select 2.07_94 in the "version summary" box below that.  You can
then use the dropdowns to switch back and forth between PASS and FAIL
reports to look for patterns.

I see that all the failures are on architecture *-thread-multi-64int.
There is also one pass with that, but that's also the only result for
perl 5.8.7.

This tool will convert from a report id to the tester's email address,
hopefully one of them will be able to help you:
  http://stats.cpantesters.org/cpanmail.html

-- 
David Cantrell | Enforcer, South London Linguistic Massive

"Every normal man must be tempted at times to spit on his hands,
 hoist the black flag, and begin slitting throats." -- H. L. Mencken

Re: TAP and the exit code

[Ovid]

- Original Message 

> From: Gaurav Vaidya 

> As I understand it, the idea behind the final 'not ok' isn't as an 
> additionally 
> failed test, but as a way of forcing the plan to fail. That'll work as long 
> as 
> you don't have 'N' tests, and exactly 'N-1' tests finish before the suite 
> exists 
> prematurely - in which case we *still* tell the user something went wrong, 
> but 
> now you can't tell it apart from a single, failed test.

Good point!
 
> > As an example, someone writing a test runner GUI could have standard 
> > green/red 
> pass/fail lines, but maybe have flashing or striped red lines for premature 
> exits, letting the programmer prioritize what he/she needs to deal with.
> How would such a UI handle situations where all the tests pass, but the plan 
> fails (too many tests, too few tests or a missing plan)? If those become 
> flashing/stripped lines as well, then the current spec works as written. If 
> not, 
> we should fix this up.

It's all up to whomever implements it.  TAP should provide information, not 
dictate its usage.

 
Cheers,
Ovid
--
Buy the book - http://www.oreilly.com/catalog/perlhks/
Tech blog- http://use.perl.org/~Ovid/journal/
Twitter  - http://twitter.com/OvidPerl
Official Perl 6 Wiki - http://www.perlfoundation.org/perl6

Re: What's the common denominator in these NYTProf failures?

[Martin Evans]

Tim Bunce wrote:

I'm strugling to find a common denominator in these test results:

http://bbbike.radzeit.de/~slaven/cpantestersmatrix.cgi?dist=Devel-NYTProf+2.07_94

It would be wonderful if there was some tool that would analyse the
perl -V output and help identify the combinations of settings associated
with failures - assuming there is a pattern.

Meanwhile, can anyone reproduce the failure? (I've not been able to.)

Tim.




Not intel 64 bit int and multi-threaded perl then? Looked common to all 
the failures and unless I missed one looked missing from the successes.


Martin
--
Martin J. Evans
Easysoft Limited
http://www.easysoft.com

Re: TAP and the exit code

[Gaurav Vaidya]

Hey,

On Jan 28, 2009, at 5:19 PM, Ovid wrote:
That being said, 'not ok - test exited prematurely' is not terribly  
structured and doesn't necessarily let an advanced TAP consumer  
present this information any better.
As I understand it, the idea behind the final 'not ok' isn't as an  
additionally failed test, but as a way of forcing the plan to fail.  
That'll work as long as you don't have 'N' tests, and exactly 'N-1'  
tests finish before the suite exists prematurely - in which case we  
*still* tell the user something went wrong, but now you can't tell it  
apart from a single, failed test.


As an example, someone writing a test runner GUI could have standard  
green/red pass/fail lines, but maybe have flashing or striped red  
lines for premature exits, letting the programmer prioritize what he/ 
she needs to deal with.
How would such a UI handle situations where all the tests pass, but  
the plan fails (too many tests, too few tests or a missing plan)? If  
those become flashing/stripped lines as well, then the current spec  
works as written. If not, we should fix this up.


cheers,
Gaurav

What's the common denominator in these NYTProf failures?

[Tim Bunce]

I'm strugling to find a common denominator in these test results:

http://bbbike.radzeit.de/~slaven/cpantestersmatrix.cgi?dist=Devel-NYTProf+2.07_94

It would be wonderful if there was some tool that would analyse the
perl -V output and help identify the combinations of settings associated
with failures - assuming there is a pattern.

Meanwhile, can anyone reproduce the failure? (I've not been able to.)

Tim.

Re: TAP and the exit code

[Michael Peters]

Eric Wilhelm wrote:


1.  TAP::Harness should store that when archiving?


It would be nice if TAP::Harness::Archive noted the exit code in the meta.yml file included in the 
archive.


--
Michael Peters
Plus Three, LP

Re: TAP and the exit code

[Ovid]

- Original Message 

> From: Eric Wilhelm 

> >For example, if a test program knows that it is going to
> >exit abnormally, even after it has finished emitting TAP according to
> > its plan, it should emit one final "not ok" line before exiting.
> 
> So,
> 
> 1.  TAP::Harness should store that when archiving?
> 2.  Is it possible to have something more explicit than that?  (or does 
> the exit code go in the test name or what?

The current discussion has generally been along the lines of "Core TAP should 
be all that is needed to know if tests have passed or failed".  Thus, Schwern's 
'not ok' final line meets this criteria.

That being said, 'not ok - test exited prematurely' is not terribly structured 
and doesn't necessarily let an advanced TAP consumer present this information 
any better.  Structured TAP diagnostics (preferably in JSON and not feasible 
with the current Test::Builder)  would be an optional feature which would allow 
additional information to be passed.  It shouldn't affect pass/fail status, but 
instead augment that status to allow the TAP consumer to respond appropriately, 
if needed.

As an example, someone writing a test runner GUI could have standard green/red 
pass/fail lines, but maybe have flashing or striped red lines for premature 
exits, letting the programmer prioritize what he/she needs to deal with.

The key here is to make all of this optional (as much as is feasible), but 
possible.

 
Cheers,
Ovid
--
Buy the book - http://www.oreilly.com/catalog/perlhks/
Tech blog- http://use.perl.org/~Ovid/journal/
Twitter  - http://twitter.com/OvidPerl
Official Perl 6 Wiki - http://www.perlfoundation.org/perl6