Re: Non-Perl TAP implementations (and diag() problems)

[Adrian Howard]

On 20 Apr 2006, at 16:55, Michael Peters wrote:
[snip]

I'm not sure I agree that there is a difference between them. They are
both comments output by the tests. Just because one comes from the
testing routine used by the test and the other from the test itself
doesn't mean they aren't both just human readable comments on the  
test run.

[snip]

It's useful to distinguish between them for things like home-brew  
test runners - so I can accurately determine which diagnostics are  
associated with a particular test failure, and which ones are just  
informative.


Adrian

Re: Non-Perl TAP implementations (and diag() problems)

[Ovid]

(Oops.  Accidentally sent this to Michael directly rather than to the
list.)

--- Michael Peters <[EMAIL PROTECTED]> wrote:
> I'm not sure I agree that there is a difference between them. They
> are
> both comments output by the tests. Just because one comes from the
> testing routine used by the test and the other from the test itself
> doesn't mean they aren't both just human readable comments on the
> test run.

  is 3, 2, 'foo';
  diag 'bar';
  is 4, 5, 'baz';

If you run those tests, you know that (assuming we can get around the
buffering problem) the lines beginning with "#" after the "not ok" line
are the failure messages associated with a given test.  Unfortunately,
where "diag 'bar'" belong?  Can we associate it with the first test or
the second?

There's no way of knowing what the programmer and you *can't* handle
diag correctly if you can't disambiguate it.  This is another reason
why writing automated tools for TAP is problematic.  There's more stuff
folks would like to do and we need these issues resolved to make those
things happen.

Cheers,
Ovid

-- 
If this message is a response to a question on a mailing list, please send 
follow up questions to the list.

Web Programming with Perl -- http://users.easystreet.com/ovid/cgi_course/

Re: Non-Perl TAP implementations (and diag() problems)

[Michael Peters]

Ovid wrote:
> --- Adrian Howard <[EMAIL PROTECTED]> wrote:
>> I've thought in the past about about using /^## / for non-test  
>> related diagnostics
>>
>> ## Start the fribble tests
>> ok 1 - fribble foo
>> not ok 2 - fribble bar
>> #   Failed test 'fribble bar'
>> #   in untitled text 2 at line 5.
>> #  got: 'baz'
>> # expected: 'bar'
>> ## Start the blart tests
>> # ok 1 - blart foo
>> ... etc ...
>>
>> Reads reasonably to me and has the advantage of being backward  
>> compatible.
> 
> Hmm, I was thinking something more along the lines of ">" or "*" to
> make it more visually distinctive, but backward compatibility is a
> glorious thing.  I don't mind "##".

I'm not sure I agree that there is a difference between them. They are
both comments output by the tests. Just because one comes from the
testing routine used by the test and the other from the test itself
doesn't mean they aren't both just human readable comments on the test run.

-- 
Michael Peters
Developer
Plus Three, LP

Re: Non-Perl TAP implementations (and diag() problems)

[Ovid]

--- Adrian Howard <[EMAIL PROTECTED]> wrote:
> I've thought in the past about about using /^## / for non-test  
> related diagnostics
> 
> ## Start the fribble tests
> ok 1 - fribble foo
> not ok 2 - fribble bar
> #   Failed test 'fribble bar'
> #   in untitled text 2 at line 5.
> #  got: 'baz'
> # expected: 'bar'
> ## Start the blart tests
> # ok 1 - blart foo
> ... etc ...
> 
> Reads reasonably to me and has the advantage of being backward  
> compatible.

Hmm, I was thinking something more along the lines of ">" or "*" to
make it more visually distinctive, but backward compatibility is a
glorious thing.  I don't mind "##".

Anyone else?

Cheers,
Ovid

-- 
If this message is a response to a question on a mailing list, please send 
follow up questions to the list.

Web Programming with Perl -- http://users.easystreet.com/ovid/cgi_course/

Re: Non-Perl TAP implementations (and diag() problems)

[Adrian Howard]

On 19 Apr 2006, at 09:02, Ovid wrote:
[snip]

From a parser standpoint, there's no clean way of distinguishing that
from what the test functions are going to output.  As a result, I
really think that "diag" and normal test failure information should be
marked differently (instead of the /^# / that we see).

[snip]

I've thought in the past about about using /^## / for non-test  
related diagnostics


## Start the fribble tests
ok 1 - fribble foo
not ok 2 - fribble bar
#   Failed test 'fribble bar'
#   in untitled text 2 at line 5.
#  got: 'baz'
# expected: 'bar'
## Start the blart tests
# ok 1 - blart foo
... etc ...

Reads reasonably to me and has the advantage of being backward  
compatible.


?

Adrian

Re: Non-Perl TAP implementations

[Adam Kennedy]

I can't remember the exact list off the top of my head.

At the time, I did the cascade tracing by hand using the CPANTS webpage, 
it took for-bloody-ever. The only reason I kept doing as deep as I did 
was that I kept being astonished at what was getting caught in the web.


There's been something of a shuffle in there since, a lot of people 
dropped Test::Warn (which was a big branch) and Test::Builder::Tester 
merged in.


I'm hoping, if we can get this CDBI-SQLite stuff working, to have 
similar things running automatically in future, so I could actually 
answer your question.


Adam K

Fergal Daly wrote:

On 4/18/06, Adam Kennedy <[EMAIL PROTECTED]> wrote:

The aformentioned change to Test::Builder broke 3 different Test-Testing
modules that relied on it.


3? I only know of 2 - Test::Builder::Tester (which scrapes and broke)
and Test::Tester (which doesn't scrape and didn't break). Is there
another Test-Testing module?

F


That broke 28 Test modules which used them.

That broke 115 various CPAN modules.

That broken 880 other CPAN modules.

And so on and so forth, until the end number ended up somewhere between
2000 and 3000 distributions.

So 30% of all CPAN modules broke.

This included things like almost ALL of WWW::, because Mechanize got
sucked into it. God knows how many darkpan Mech modules got hurt as well.

Do we blame ALL of those 100s of developers?

Does blame even matter at that point?

This is what I mean by becoming an API even though you didn't want to be.

Adam K

Re: Non-Perl TAP implementations (and diag() problems)

[Ricardo SIGNES]

* Fergal Daly <[EMAIL PROTECTED]> [2006-04-19T15:24:51]
> On 4/19/06, Ricardo SIGNES <[EMAIL PROTECTED]> wrote:
> >
> > There are other things that test test output, like Test::Tester.  Will they
> > break?  To find out, I downloaded a pristene copy of the latest Test-Simple
> > and Test-Tester and changed Test::Builder to use STDOUT for failure_output.
> > The only test that failed in the whole set was one that checked whether
> > failure_output defaulted to STDERR.
> ...
> I'm not sure what test you are referring to, as far as I can tell,
> Test-Tester doesn't check anything about failure_output. What .t file
> is it in?

When I said "the whole set" I meant "the whole set [of tests from Test-Simple
and Test-Tester."  That particular test was part of Test-Simple, t/reset.t,
which checks that on reset failure_output is again TESTERR.

-- 
rjbs


signature.asc
Description: Digital signature

Re: Non-Perl TAP implementations

[Fergal Daly]

On 4/19/06, David Wheeler <[EMAIL PROTECTED]> wrote:
> On Apr 19, 2006, at 12:14, Fergal Daly wrote:
>
> > One other reason (that I didn't see mentioned) is that objects imply
> > that the harness and tests are in the same process which means that
> > the tests can corrupt the harness and that the harness can fail to
> > report if the test process dies,
>
> Well, the harness can be corrupted by bad output, too (do something
> like this in a test to see what I mean:
>
>print STDOUT "ok - Ha ha ha!\n";
>
> ). But in the JavaScript port, tests run in a browser, and there's no
> such thing a separate processes, so I had no choice there. So I
> decided to do both things: Test.Harness uses the objects it collects
> from Test.Builder to summarize test pasess, failures, what to output
> to the browser, and what not. But Test.Builder also sends all output
> to a series of appropriate function calls (which in the browser all
> go to the same place), so the test can run without the harness and
> display results, and so that some other harness could potentially
> scrape the output to summarize the results.

Yes, in the context of javascript testing, there's no choice but, for
example, I have used DUnit (Delphi) and it has a nice GUI test runner
but it's very annoying to have your test launching and result display
program wedge, crash or silently disappear because of a problem with
the module you're testing,

F

Re: Non-Perl TAP implementations

[David Wheeler]

On Apr 19, 2006, at 12:14, Fergal Daly wrote:


One other reason (that I didn't see mentioned) is that objects imply
that the harness and tests are in the same process which means that
the tests can corrupt the harness and that the harness can fail to
report if the test process dies,


Well, the harness can be corrupted by bad output, too (do something  
like this in a test to see what I mean:


  print STDOUT "ok - Ha ha ha!\n";

). But in the JavaScript port, tests run in a browser, and there's no  
such thing a separate processes, so I had no choice there. So I  
decided to do both things: Test.Harness uses the objects it collects  
from Test.Builder to summarize test pasess, failures, what to output  
to the browser, and what not. But Test.Builder also sends all output  
to a series of appropriate function calls (which in the browser all  
go to the same place), so the test can run without the harness and  
display results, and so that some other harness could potentially  
scrape the output to summarize the results.


Best,

David

Re: Non-Perl TAP implementations (and diag() problems)

[Fergal Daly]

On 4/19/06, Ricardo SIGNES <[EMAIL PROTECTED]> wrote:
> * Ovid <[EMAIL PROTECTED]> [2006-04-19T04:02:31]
> > From a parser standpoint, there's no clean way of distinguishing that
> > from what the test functions are going to output.  As a result, I
> > really think that "diag" and normal test failure information should be
> > marked differently (instead of the /^# / that we see).
>
> I strongly agree.  This came up when TAP was being named, and also during a
> previous Harness upgrade.  Unfortunately, Harness and Straps discard comments
> and, as mentioned before, the STDOUT/STDERR makes it hard to associate
> diagonstic output with tests.
>
> How many things rely on which stream the various output methods in
> Test::Builder use?  I know there was trouble last time that things changed, 
> but
> wasn't that entirely because of the disconnect between Test::Builder and
> Test::Builder::Tester?  Since they're now together, isn't that particular 
> issue
> solved?
>
> There are other things that test test output, like Test::Tester.  Will they
> break?  To find out, I downloaded a pristene copy of the latest Test-Simple 
> and
> Test-Tester and changed Test::Builder to use STDOUT for failure_output.  The
> only test that failed in the whole set was one that checked whether
> failure_output defaulted to STDERR.

Test-Tester (and modules using it) should not be impacted by this as
it is based on the Test::Builer object API. It collects the test
results while they are still Perl data, rather than scraping them out
of the test stream.

I'm not sure what test you are referring to, as far as I can tell,
Test-Tester doesn't check anything about failure_output. What .t file
is it in?

F

>
> I think that a real investigation into the impact of using one stream by
> default is in order.
>
> --
> rjbs
>
>
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.2 (GNU/Linux)
>
> iD8DBQFERjFI5IEwYcR13KMRAhL+AKCJWpGtsfclNyyQtfKm61W6F6bVgQCfaNbc
> V2TENE9WeZ2Fm+ZMTRjPi2U=
> =kG3g
> -END PGP SIGNATURE-
>
>
>

Re: Non-Perl TAP implementations

[Fergal Daly]

On 4/18/06, Ovid <[EMAIL PROTECTED]> wrote:
> --- David Wheeler <[EMAIL PROTECTED]> wrote:
> > Test.Simple—JavaScript. It looks and acts just like tap, although in
> > reality it's tracking test results in an object rather than scraping
> > them from a print buffer.
> >
> >http://openjsan.org/doc/t/th/theory/Test/Simple/
>
> Tracking the results in an object is a better choice than scraping from
> a print buffer.  One of the frustrating issues with Perl's testing
> tools is the limited flexibility we have due to reading the output from
> STDOUT.

One other reason (that I didn't see mentioned) is that objects imply
that the harness and tests are in the same process which means that
the tests can corrupt the harness and that the harness can fail to
report if the test process dies,

F

>
> The TAP output should really just be for humans.  It should also be
> reconfigurable, but obviously we can't do that because Test::Harness
> would choke.
>
> Since it looks like we're going to stick with reading information from
> a print buffer, we should at least publish an EBNF grammar for the
> output.  (Interestingly, if we did that, we could potentially
> incorporate that into Test::Harness and allow folks to provide their
> own grammars and thus structure the output to better suit their needs.
> Of course, I would like a Ponie with that, too).
>
> Cheers,
> Ovid
>
> --
> If this message is a response to a question on a mailing list, please send 
> follow up questions to the list.
>
> Web Programming with Perl -- http://users.easystreet.com/ovid/cgi_course/
>

Re: Non-Perl TAP implementations

[Fergal Daly]

On 4/18/06, Adam Kennedy <[EMAIL PROTECTED]> wrote:
> The aformentioned change to Test::Builder broke 3 different Test-Testing
> modules that relied on it.

3? I only know of 2 - Test::Builder::Tester (which scrapes and broke)
and Test::Tester (which doesn't scrape and didn't break). Is there
another Test-Testing module?

F

>
> That broke 28 Test modules which used them.
>
> That broke 115 various CPAN modules.
>
> That broken 880 other CPAN modules.
>
> And so on and so forth, until the end number ended up somewhere between
> 2000 and 3000 distributions.
>
> So 30% of all CPAN modules broke.
>
> This included things like almost ALL of WWW::, because Mechanize got
> sucked into it. God knows how many darkpan Mech modules got hurt as well.
>
> Do we blame ALL of those 100s of developers?
>
> Does blame even matter at that point?
>
> This is what I mean by becoming an API even though you didn't want to be.
>
> Adam K
>

Re: Non-Perl TAP implementations (and diag() problems)

[A. Pagaltzis]

* A. Pagaltzis <[EMAIL PROTECTED]> [2006-04-19 19:10]:
> Use JSON. It’s cross-language, can fit data structures on a
> single line, is reasonably widely supported, and can even be
> read by a YAML parser.

Oh, and I forgot: in contrast to YAML, the spec is so simple
that it’s trivial to cook an implementation in an evening if
a language lacks libraries for it.

Regards,
-- 
Aristotle Pagaltzis // 

Re: Non-Perl TAP implementations (and diag() problems)

[A. Pagaltzis]

* Ovid <[EMAIL PROTECTED]> [2006-04-19 10:05]:
> --- demerphq <[EMAIL PROTECTED]> wrote:
> > I should think that youd have a much easier time using
> > Data::Dumper and its ->Pad and ->Indent(0) method than you
> > ever would with YAML.
> 
> That fails when we have PHP, C, or Java producing the
> information. They don't have Data::Dumper, but they do have
> YAML.

Use JSON. It’s cross-language, can fit data structures on a
single line, is reasonably widely supported, and can even be
read by a YAML parser.

Regards,
-- 
Aristotle Pagaltzis // 

Re: Non-Perl TAP implementations (and diag() problems)

[Ricardo SIGNES]

* Ovid <[EMAIL PROTECTED]> [2006-04-19T04:02:31]
> From a parser standpoint, there's no clean way of distinguishing that
> from what the test functions are going to output.  As a result, I
> really think that "diag" and normal test failure information should be
> marked differently (instead of the /^# / that we see).

I strongly agree.  This came up when TAP was being named, and also during a
previous Harness upgrade.  Unfortunately, Harness and Straps discard comments
and, as mentioned before, the STDOUT/STDERR makes it hard to associate
diagonstic output with tests.

How many things rely on which stream the various output methods in
Test::Builder use?  I know there was trouble last time that things changed, but
wasn't that entirely because of the disconnect between Test::Builder and
Test::Builder::Tester?  Since they're now together, isn't that particular issue
solved?

There are other things that test test output, like Test::Tester.  Will they
break?  To find out, I downloaded a pristene copy of the latest Test-Simple and
Test-Tester and changed Test::Builder to use STDOUT for failure_output.  The
only test that failed in the whole set was one that checked whether
failure_output defaulted to STDERR.

I think that a real investigation into the impact of using one stream by
default is in order.

-- 
rjbs


signature.asc
Description: Digital signature

Re: Non-Perl TAP implementations (and diag() problems)

[Ovid]

--- Ovid <[EMAIL PROTECTED]> wrote:
> > If you mean you want pluck YAML test results from a noisy input
> > stream I'd say youd probably be wrong.

Naturally, I forgot to include the most compelling argument.  The
"noisy input stream" is noisy only because we've never tamed it.  If it
becomes well-defined, a grammar can be applied.  If we accept the
status quo, we accept suboptimal tools.

Cheers,
Ovid

-- 
If this message is a response to a question on a mailing list, please send 
follow up questions to the list.

Web Programming with Perl -- http://users.easystreet.com/ovid/cgi_course/

Re: Non-Perl TAP implementations (and diag() problems)

[Ovid]

--- demerphq <[EMAIL PROTECTED]> wrote:

> On 4/19/06, Ovid <[EMAIL PROTECTED]> wrote:
> > to handle got/expected failure information in Java or C?  There are
> > pretty rich data structures we could put out there and YAML might
> help.
> >  That would also likely simplify a parser.
> 
> If you mean you want pluck YAML test results from a noisy input
> stream I'd say youd probably be wrong.

I might agree with you, but only because what we currently have may not
be sufficient.  TAP essentially provides three types of information. 
The first is whether or not an individual test has passed.  The second
is the reason why a test has failed.  Those reasons are included in
lines beginning with a hash mark:

  ovid $ perl -MTest::More=tests,1 -e 'is 2,3, "fail"'
  1..1
  not ok 1 - fail
  #   Failed test 'fail'
  #   in -e at line 1.
  #  got: '2'
  # expected: '3'
  # Looks like you failed 1 test of 1.

(The third type of information is human-readable "diag()" information
and I'll address those issues below)

There is a lot of potential here, but also problems.  First, "Failed
test 'fail' does not really indicate which tests fail (duplicate tests
names cause problems), so it's a human-readable bit of information, but
it's not useful for a parser.  It *is* useful for a parser if we
guarantee the error information is always after the failed test.

> I should think that youd have a much easier time using Data::Dumper
> and its ->Pad and ->Indent(0) method than you ever would with YAML.

That fails when we have PHP, C, or Java producing the information. 
They don't have Data::Dumper, but they do have YAML.

Another annoying issue is "diag()":

  ok foo(), 'foo() works';
  diag $some_message;
  ok bar(), 'bar() works';

For the above snippet, is diag() associated with the first test or the
second?

In reality, there's no way to guarantee that it's associated with
either, so what would you do with that diag() info when it pops up in a
JUnit style GUI test runner?

Further, we'll get real unhappy the first time when someone
drool-n-pastes the error output to diag statements:

  diag "   got: '2'";

>From a parser standpoint, there's no clean way of distinguishing that
from what the test functions are going to output.  As a result, I
really think that "diag" and normal test failure information should be
marked differently (instead of the /^# / that we see).

Thoughts?

Cheers,
Ovid

-- 
If this message is a response to a question on a mailing list, please send 
follow up questions to the list.

Web Programming with Perl -- http://users.easystreet.com/ovid/cgi_course/

Re: Non-Perl TAP implementations

[demerphq]

On 4/19/06, Ovid <[EMAIL PROTECTED]> wrote:
> to handle got/expected failure information in Java or C?  There are
> pretty rich data structures we could put out there and YAML might help.
>  That would also likely simplify a parser.

If you mean you want pluck YAML test results from a noisy input stream
Id say youd probably be wrong. YAML is about as inappropriate a tool
for that purpose as I can imagine. Id say that using a white-space
sensitive data representation in such a circumstance would  be an
abject nightmare.

I should think that youd have a much easier time using Data::Dumper
and its ->Pad and ->Indent(0) method than you ever would with YAML.

DD output is more regular than YAML, and can be output so the full
dump is on a single line meaning that buffering issues shouldnt be as
difficult to resolve. Using ->Pad you could easily add a "magic
marker" to identify the beginning of a dump stream (regardless as to
its size). Heck with a DD structure you could use a regex to match the
entire dump as DD's output is that predictable.

Seriously Id be surprised to hear a single thing that makes YAML
preferable to something like DD for the purposes I understand we are
discussing. And i can think of a lot of reasons YAML would be
suboptimal.

cheers,
Yves

--
perl -Mre=debug -e "/just|another|perl|hacker/"

Re: Non-Perl TAP implementations

[A. Pagaltzis]

* Ovid <[EMAIL PROTECTED]> [2006-04-19 07:20]:
> What if someone wants more diagnostic information or more
> complete failure information?  We might find things useful
> enough that that we feel compelled to update TAP.  If so, TAP
> needs to be versioned and we need to figure out how to
> accomodate that.

Hmm, has anyone else ever designed an extensible, versioned
plaintext protocol before, maybe?

:-)

Just think of TAP as a wire protocol. Then most questions just
disappear. There is also have a very large body of prior art to
consult.

Regards,
-- 
Aristotle Pagaltzis // 

Re: Non-Perl TAP implementations

[Ovid]

--- Adam Kennedy <[EMAIL PROTECTED]> wrote:

> > While I agree that this caused problems, those modules were relying
> on a
> > format that was not spec'ed out or documented.
> 
> That is irrelevant. You put something into CPAN, get massive numbers
> of 
> people using it, and leave it alone and have it remain stable for 4 
> years, it becomes an API whether you wanted it to be or not :)

I'm sure most folks here are familiar with the aforementioned buffering
issues, but on the off chance you don't know the sorts of difficulties
that writing to separate output streams causes, read Dominus'
"Suffering from Buffering" (http://perl.plover.com/FAQs/Buffering.html)

What's frustrating about this is that I completely agree with Adam.  We
cannot simply have error and diagnostic output shunted to STDOUT.  What
about having the error and diagnostic information sent to both STDERR
*and* STDOUT?  That could solve the buffering problem, but we'd likely
have duplicate information being output.  Is there an easy solution?

If we can solve this problem, then I can more reliably write a grammar
that can be applied to a single stream of data.  I've toyed with the
idea of TAP::Parser, but we're not there yet.  That raises the next
issue.  What if someone wants more diagnostic information or more
complete failure information?  We might find things useful enough that
that we feel compelled to update TAP.  If so, TAP needs to be versioned
and we need to figure out how to accomodate that.  Also, how do we want
to handle got/expected failure information in Java or C?  There are
pretty rich data structures we could put out there and YAML might help.
 That would also likely simplify a parser.

Cheers,
Ovid

-- 
If this message is a response to a question on a mailing list, please send 
follow up questions to the list.

Web Programming with Perl -- http://users.easystreet.com/ovid/cgi_course/

Re: Non-Perl TAP implementations

[Adam Kennedy]

Geoffrey Young wrote:


Adam Kennedy wrote:

Schwern made one small change in the STDERR format, and the recursive
cascade of failing test-testing modules hit something like 3000 CPAN
distributions.


While I agree that this caused problems, those modules were relying on a
format that was not spec'ed out or documented.


That is irrelevant. You put something into CPAN, get massive numbers of
people using it, and leave it alone and have it remain stable for 4
years, it becomes an API whether you wanted it to be or not :)


really?  so I use an age old (but undocumented) feature of Config, then
Config changes and it's _not_ my fault?[1]  of course it is - perl is great
because you can go mucking around with stuff you shouldn't, but everyone
knows it's your own fault if it bites you later on.


At one level of abstraction, I think the blame is absolutely clear.

The trouble is when you introduce recursion.

The aformentioned change to Test::Builder broke 3 different Test-Testing 
modules that relied on it.


That broke 28 Test modules which used them.

That broke 115 various CPAN modules.

That broken 880 other CPAN modules.

And so on and so forth, until the end number ended up somewhere between 
2000 and 3000 distributions.


So 30% of all CPAN modules broke.

This included things like almost ALL of WWW::, because Mechanize got 
sucked into it. God knows how many darkpan Mech modules got hurt as well.


Do we blame ALL of those 100s of developers?

Does blame even matter at that point?

This is what I mean by becoming an API even though you didn't want to be.

Adam K

Re: Non-Perl TAP implementations

[Smylers]

Geoffrey Young writes:

> Adam Kennedy wrote:
> 
> > That is irrelevant. You put something into CPAN, get massive numbers
> > of people using it, and leave it alone and have it remain stable for
> > 4 years, it becomes an API whether you wanted it to be or not :)
> 
> really?

I'm with Adam on this one.

> so I use an age old (but undocumented) feature of Config, then Config
> changes and it's _not_ my fault?[1] 
> 
> [1] http://dev.perl.org/perl5/list-summaries/2005/20051003.html

But there Schwern invoked Tom Christian:

  You are wicked and wrong to have broken inside and peeked at the
  implementation and then relied upon it.
-- tchrist in <[EMAIL PROTECTED]>

Note that's specifically to do with peeking inside.

If you don't peek inside some code but merely observe its output, and
its output always behaves in a particular way, then relying on that
output is not the same as relying on implementation details.

Smylers

Re: Non-Perl TAP implementations

[Geoffrey Young]

Adam Kennedy wrote:
> 
>>> Schwern made one small change in the STDERR format, and the recursive
>>> cascade of failing test-testing modules hit something like 3000 CPAN
>>> distributions.
>>
>>
>> While I agree that this caused problems, those modules were relying on a
>> format that was not spec'ed out or documented.
> 
> 
> That is irrelevant. You put something into CPAN, get massive numbers of
> people using it, and leave it alone and have it remain stable for 4
> years, it becomes an API whether you wanted it to be or not :)

really?  so I use an age old (but undocumented) feature of Config, then
Config changes and it's _not_ my fault?[1]  of course it is - perl is great
because you can go mucking around with stuff you shouldn't, but everyone
knows it's your own fault if it bites you later on.

--Geoff

[1] http://dev.perl.org/perl5/list-summaries/2005/20051003.html

Re: Non-Perl TAP implementations

[Geoffrey Young]

chromatic wrote:
> On Monday 17 April 2006 18:50, Ovid wrote:
> 
> 
>>The only problem I see with that is the occasional buffering errors I
>>see on my Mac where the STDERR and STDOUT don't line up.
> 
> 
> Agreed.  Is it too late to send everything to STDOUT where it belongs?

just for everyone's knowledge, for much of Apache-Test STDERR is the apache
error_log, so we override things so errors are sent to STDOUT where they're
immediately visible as output from the http client.

so, as long as there remains an official way to redirect the two streams as
I deem fit then it doesn't matter to me.  but I thought folks might like to
be reminded that there's no guarantee that STDERR and STDOUT are the same
"screen" so mixing up where test output and/or diagnostics ends up may or
may not be a good thing, depending on your pov.

--Geoff

Re: Non-Perl TAP implementations

[Adam Kennedy]

Schwern made one small change in the STDERR format, and the recursive
cascade of failing test-testing modules hit something like 3000 CPAN
distributions.


While I agree that this caused problems, those modules were relying on a
format that was not spec'ed out or documented.


That is irrelevant. You put something into CPAN, get massive numbers of 
people using it, and leave it alone and have it remain stable for 4 
years, it becomes an API whether you wanted it to be or not :)


Adam K

Re: Non-Perl TAP implementations

[Michael Peters]

Adam Kennedy wrote:
> chromatic wrote:
>> On Monday 17 April 2006 18:50, Ovid wrote:
>>
>>> The only problem I see with that is the occasional buffering errors I
>>> see on my Mac where the STDERR and STDOUT don't line up.
>>
>> Agreed.  Is it too late to send everything to STDOUT where it belongs?
> 
> Unfortunately, it probably is too late.
> 
> Schwern made one small change in the STDERR format, and the recursive
> cascade of failing test-testing modules hit something like 3000 CPAN
> distributions.

While I agree that this caused problems, those modules were relying on a
format that was not spec'ed out or documented.

In fact, the TAP spec explicitly says that diagnostics can/should be
mixed with test output. At least to me, that means they're on the same
output stream.

http://search.cpan.org/~petdance/Test-Harness-2.56/lib/Test/Harness/TAP.pod#Diagnostics

> Changing TAP without extreme caution, planning and co-ordination could
> get be very very nasty, especially out there in the darkpan.

I do agree that caution and coordination is necessary. It's the test
modules themselves (Test::More, etc) that print to STDOUT/STDERR, right?
 And there sure are a lot of them. If I had any TAP producing test
modules on CPAN I'd volunteer to change them :)

-- 
Michael Peters
Developer
Plus Three, LP

Re: Non-Perl TAP implementations

[A. Pagaltzis]

* Adam Kennedy <[EMAIL PROTECTED]> [2006-04-18 09:30]:
> especially out there in the darkpan.

What about the deadpan?

SCNR,
-- 
#Aristotle
*AUTOLOAD=*_;sub _{s/(.*)::(.*)/print$2,(",$\/"," ")[defined wantarray]/e;$1};
&Just->another->Perl->hacker;

Re: Non-Perl TAP implementations

[chromatic]

On Tuesday 18 April 2006 00:20, Ovid wrote:

> As a language-independent tool, an API is silly and
> I'm a fool for shooting my keyboard off for thinking only about the
> implementation problems as opposed to its benefits.

I'm sure you're not the only person who has thought "Hey, screen scraping the 
results is tedious work."  I've certainly wished occasionally for an API 
before reading reality's greeting card full of counterarguments.

-- c

Re: Non-Perl TAP implementations

[Adam Kennedy]

chromatic wrote:

On Monday 17 April 2006 18:50, Ovid wrote:


The only problem I see with that is the occasional buffering errors I
see on my Mac where the STDERR and STDOUT don't line up.


Agreed.  Is it too late to send everything to STDOUT where it belongs?


Unfortunately, it probably is too late.

Schwern made one small change in the STDERR format, and the recursive 
cascade of failing test-testing modules hit something like 3000 CPAN 
distributions.


Changing TAP without extreme caution, planning and co-ordination could 
get be very very nasty, especially out there in the darkpan.


Adam K

Re: Non-Perl TAP implementations

[Ovid]

--- "A. Pagaltzis" <[EMAIL PROTECTED]> wrote:
> If that were universally true, we wouldn’t have plaintext network
> protocols, would we? And I think making TAP a protocol instead of
> an API was the right choice.

As already noted, I freely admit I was wrong.  If TAP was a "Perl only"
thing, I might (might) be inclined to argue the point, but even then
it's questionable.  As a language-independent tool, an API is silly and
I'm a fool for shooting my keyboard off for thinking only about the
implementation problems as opposed to its benefits.

Cheers,
Ovid

-- 
If this message is a response to a question on a mailing list, please send 
follow up questions to the list.

Web Programming with Perl -- http://users.easystreet.com/ovid/cgi_course/

Re: Non-Perl TAP implementations

[Adam Kennedy]

Ovid wrote:

--- David Wheeler <[EMAIL PROTECTED]> wrote:
Test.Simple—JavaScript. It looks and acts just like tap, although in 
reality it's tracking test results in an object rather than scraping 
them from a print buffer.


   http://openjsan.org/doc/t/th/theory/Test/Simple/


Tracking the results in an object is a better choice than scraping from
a print buffer.  One of the frustrating issues with Perl's testing
tools is the limited flexibility we have due to reading the output from
STDOUT.

The TAP output should really just be for humans.  It should also be
reconfigurable, but obviously we can't do that because Test::Harness
would choke.


I disagree.

I think one of the key strengths of TAP, as opposed to most other 
testing systems out there, is that the test results are language-agnostic.


It means we can do things like using a Perl Harness to hit web pages 
built by PHP-generated TAP output.


Or mix tests for multiple languages in a single environment/system/dist.

Certainly a more robust grammar could be useful, and an EBNF grammar 
sounds wonderful.


Adam K


Since it looks like we're going to stick with reading information from
a print buffer, we should at least publish an EBNF grammar for the
output.  (Interestingly, if we did that, we could potentially
incorporate that into Test::Harness and allow folks to provide their
own grammars and thus structure the output to better suit their needs. 
Of course, I would like a Ponie with that, too).


Cheers,
Ovid

Re: Non-Perl TAP implementations

[A. Pagaltzis]

* Ovid <[EMAIL PROTECTED]> [2006-04-18 03:15]:
> Tracking the results in an object is a better choice than
> scraping from a print buffer.

If that were universally true, we wouldn’t have plaintext network
protocols, would we? And I think making TAP a protocol instead of
an API was the right choice.

Regards,
-- 
Aristotle Pagaltzis // 

Re: Non-Perl TAP implementations

[chromatic]

On Monday 17 April 2006 18:50, Ovid wrote:

> The only problem I see with that is the occasional buffering errors I
> see on my Mac where the STDERR and STDOUT don't line up.

Agreed.  Is it too late to send everything to STDOUT where it belongs?

-- c

Re: Non-Perl TAP implementations

[Ovid]

--- chromatic <[EMAIL PROTECTED]> wrote:
> There are a lot of reasons why having separate testing and analysis
> processes are very good and there are a lot of reasons why having
> file-based communication is very, very good.

OK, you raise some good points.  Then by relying on the output and
scraping that while the tests are running (instead of afterwards), we
could still get many of the benefits folks want (such as GUI test
runners, colored tests, etc.) and they should be easy to implement.

The only problem I see with that is the occasional buffering errors I
see on my Mac where the STDERR and STDOUT don't line up.

Cheers,
Ovid

-- 
If this message is a response to a question on a mailing list, please send 
follow up questions to the list.

Web Programming with Perl -- http://users.easystreet.com/ovid/cgi_course/

Re: Non-Perl TAP implementations

[chromatic]

On Monday 17 April 2006 18:09, Ovid wrote:

> Tracking the results in an object is a better choice than scraping from
> a print buffer.  One of the frustrating issues with Perl's testing
> tools is the limited flexibility we have due to reading the output from
> STDOUT.

... an object of which TAP-supporting language?
... from STDOUT on which computer?
... is it really STDOUT?
... can you actually run code on the remote machine?
... can you replay the object?
... can you serialize the object for later?
... can you provide a default language-independent object that supports all of 
the TAP features you want to exercise with the implementation of a 
programming language you've never seen before?
... or one that no one has written yet?

There are a lot of reasons why having separate testing and analysis processes 
are very good and there are a lot of reasons why having file-based 
communication is very, very good.

-- c

Re: Non-Perl TAP implementations

[Tyler MacDonald]

Ovid <[EMAIL PROTECTED]> wrote:
> Tracking the results in an object is a better choice than scraping from
> a print buffer.  One of the frustrating issues with Perl's testing
> tools is the limited flexibility we have due to reading the output from
> STDOUT.

I like that aspect about TAP... it's simple, humans and machines can
read it easily, and there's no reason why it has to be langauge-specific. I
think the only thing that could make it better would be if TAP output was
parseable by YAML. :-) 

> we should at least publish an EBNF grammar for the output.

Definately!  

- Tyler

Re: Non-Perl TAP implementations

[Ovid]

--- Andy Lester <[EMAIL PROTECTED]> wrote:

> > Since it looks like we're going to stick with reading information
> from
> > a print buffer, we should at least publish an EBNF grammar for the
> > output.
> 
> Patches welcome!

OK, I've a bunch of TPF stuff I'm wrapping up, but if I have some tuits
soon, I'll see what I can put together.
 
> > (Interestingly, if we did that, we could potentially
> > incorporate that into Test::Harness and allow folks to provide
> their
> > own grammars and thus structure the output to better suit their
> needs.
> 
> Patches less welcome! :-)

You're hurtin' me, Andy, you're hurtin' me ;)

Cheers,
Ovid

-- 
If this message is a response to a question on a mailing list, please send 
follow up questions to the list.

Web Programming with Perl -- http://users.easystreet.com/ovid/cgi_course/

Re: Non-Perl TAP implementations

[Andy Lester]

Since it looks like we're going to stick with reading information from
a print buffer, we should at least publish an EBNF grammar for the
output.


Patches welcome!


(Interestingly, if we did that, we could potentially
incorporate that into Test::Harness and allow folks to provide their
own grammars and thus structure the output to better suit their needs.


Patches less welcome! :-)

--
Andy Lester => [EMAIL PROTECTED] => www.petdance.com => AIM:petdance

Re: Non-Perl TAP implementations

[Ovid]

--- David Wheeler <[EMAIL PROTECTED]> wrote:
> Test.Simple—JavaScript. It looks and acts just like tap, although in 
> reality it's tracking test results in an object rather than scraping 
> them from a print buffer.
> 
>http://openjsan.org/doc/t/th/theory/Test/Simple/

Tracking the results in an object is a better choice than scraping from
a print buffer.  One of the frustrating issues with Perl's testing
tools is the limited flexibility we have due to reading the output from
STDOUT.

The TAP output should really just be for humans.  It should also be
reconfigurable, but obviously we can't do that because Test::Harness
would choke.

Since it looks like we're going to stick with reading information from
a print buffer, we should at least publish an EBNF grammar for the
output.  (Interestingly, if we did that, we could potentially
incorporate that into Test::Harness and allow folks to provide their
own grammars and thus structure the output to better suit their needs. 
Of course, I would like a Ponie with that, too).

Cheers,
Ovid

-- 
If this message is a response to a question on a mailing list, please send 
follow up questions to the list.

Web Programming with Perl -- http://users.easystreet.com/ovid/cgi_course/

Re: Non-Perl TAP implementations

[David Wheeler]

On Apr 17, 2006, at 06:03, Andy Lester wrote:

Can you please give me a short couple of sentences on it for  
someone who has no idea how/why you'd use TAP outside of Perl?


It's a direct port of Test::Builder, ::Simple, and ::More, along with  
a harness for showing test results in a browser. What am I missing in  
terms of explanation?


D

Re: Non-Perl TAP implementations

[Geoffrey Young]

Andy Lester wrote:
> 
> I'm adding a section to Test::Harness::TAP on non-Perl TAP.
> 
> http://svn.perl.org/modules/Test-Harness/trunk/lib/Test/Harness/TAP.pod
> 
> If you know of one, please send me some text to add.

all the big PHP players now produce TAP

  o phpt (outputs TAP by default as of the yet-to-be-released PEAR 1.5.0)

  http://pear.php.net/PEAR


  o PHPUnit has a TAP logger (since 2.3.4)

  http://www.phpunit.de/wiki/Main_Page


  o there's a third-party TAP reporting extension for SimpleTest

http://www.digitalsandwich.com/archives/51-Updated-Simpletest+Apache-Test.html


  o Apache-Test's PHP writes TAP by default and includes the standalone
test-more.php

  http://search.cpan.org/dist/Apache-Test/


there's also libtap (written in C)

  http://jc.ngo.org.uk/trac-bin/trac.cgi/wiki/LibTap

HTH

--Geoff

Re: Non-Perl TAP implementations

[Andy Lester]

Test.Simple—JavaScript. It looks and acts just like tap, although  
in reality it's tracking test results in an object rather than  
scraping them from a print buffer.


  http://openjsan.org/doc/t/th/theory/Test/Simple/


Can you please give me a short couple of sentences on it for someone  
who has no idea how/why you'd use TAP outside of Perl?


--
Andy Lester => [EMAIL PROTECTED] => www.petdance.com => AIM:petdance

Re: Non-Perl TAP implementations

[Andy Lester]

How non-Perl do you want?  Does the Perl 6 version of Test.pm or
Test::Builder/Test::More count?  How about the Parrot versions?


Sure, lemme have 'em.

--
Andy Lester => [EMAIL PROTECTED] => www.petdance.com => AIM:petdance

Re: Non-Perl TAP implementations

[chromatic]

On Sunday 16 April 2006 20:08, Andy Lester wrote:

> I'm adding a section to Test::Harness::TAP on non-Perl TAP.
>
> http://svn.perl.org/modules/Test-Harness/trunk/lib/Test/Harness/TAP.pod
>
> If you know of one, please send me some text to add.

How non-Perl do you want?  Does the Perl 6 version of Test.pm or 
Test::Builder/Test::More count?  How about the Parrot versions?

-- c

Re: Non-Perl TAP implementations

[David Wheeler]

On Apr 16, 2006, at 20:08, Andy Lester wrote:


I'm adding a section to Test::Harness::TAP on non-Perl TAP.

http://svn.perl.org/modules/Test-Harness/trunk/lib/Test/Harness/ 
TAP.pod


If you know of one, please send me some text to add.


Test.Simple—JavaScript. It looks and acts just like tap, although in  
reality it's tracking test results in an object rather than scraping  
them from a print buffer.


  http://openjsan.org/doc/t/th/theory/Test/Simple/

Best,

David

Re: Non-Perl TAP implementations

[Ricardo SIGNES]

* Ricardo SIGNES <[EMAIL PROTECTED]> [2006-04-16T23:33:19]
> It's not really ready to be publicized, and I haven't touched it in a little
> while, but I'll mention PyTap: http://svn.codesimply.com/projects/pytap

I got a request, off-list, for more info, so here is some:

PyTap will, when it's done, provide a simple, assertive
(Test::Builder/More-like) interface for writing tests in Python.  It will
output TAP and will include the functionality found in Test::Builder and
Test::More.  It will try to make it easy to add more test code (so you can
write your own TAP.StringDiff, for example.

Right now, it's got a fair bit of the basics needed to emulate Test::More, and
I think it's easy to add more stuff -- just like Test::Builder, there's a
singleton that you can get at easily.

I need to better identify and finish implementing the most basic tests.  I am
not a Python guru, I just use it from time to time, so my aim may not be true.
I need to write tests for it, which means either relying on Perl for the tester
tester, or writing one in Python.

Here's a sample test, as found in my Subversion:

  from TAP.Simple import *

  plan(15)

  ok(1)
  ok(1, "everything is OK!")
  ok(0, "always fails")

  is_ok(10, 10, "is ten ten?")
  is_ok(ok, ok, "even ok is ok!")
  ok(id(ok),"ok is not the null pointer")
  ok(True,  "the Truth will set you ok")
  ok(not False, "and nothing but the truth")
  ok(False, "and we'll know if you lie to us")

  isa_ok(10, int, "10")
  isa_ok('ok', str, "some string")

  ok(0,"zero is true", todo="be more like Ruby!")
  ok(None, "none is true", skip="not possible in this universe")

  eq_ok("not", "equal", "two strings are not equal");

-- 
rjbs


signature.asc
Description: Digital signature

Re: Non-Perl TAP implementations

[Ricardo SIGNES]

* Andy Lester <[EMAIL PROTECTED]> [2006-04-16T23:08:26]
> I'm adding a section to Test::Harness::TAP on non-Perl TAP.
> 
> http://svn.perl.org/modules/Test-Harness/trunk/lib/Test/Harness/TAP.pod
> 
> If you know of one, please send me some text to add.

It's not really ready to be publicized, and I haven't touched it in a little
while, but I'll mention PyTap: http://svn.codesimply.com/projects/pytap

I'll get it in a useful state soon.  Comments from more regular producers of
Python code are welcome.

-- 
rjbs


signature.asc
Description: Digital signature

Non-Perl TAP implementations

[Andy Lester]

I'm adding a section to Test::Harness::TAP on non-Perl TAP.

http://svn.perl.org/modules/Test-Harness/trunk/lib/Test/Harness/TAP.pod

If you know of one, please send me some text to add.

Thanks,
xoxo,
Andy

--
Andy Lester => [EMAIL PROTECTED] => www.petdance.com => AIM:petdance