Re: Eliminating STDERR without any disruption.
Michael G Schwern wrote: >> How about >> "diag Failure\n". Or even levels of keywords debug/info/notice/warning/ >> err/crit/alert/emerg (stolen from syslog.h). > > That's an interesting idea. My worry is making it human readable. > > not ok 2 > err Test failed in foo.t line 2 > err got: foo > err expected: bar > ok 3 > > It doesn't leap out at you the way something like this does. > > not ok 2 > !!! Test failed in foo.t line 2 > !!! got: foo > !!! expected: bar > ok 3 What if the number of bangs indicated the severity level? We already do this thing things like: !! REALLY IMPORTANT MESSAGE !!! I don't know if we need all 8 levels used in syslog. I'm not sure where the distinction comes between "Emergency", "Alert", "Critical" and "Error" when it comes to testing. But its a good start. Some undefined levels we can define later seems like a good idea, leaves some wiggle room in the protocol. http://tools.ietf.org/html/rfc3164 Numerical Severity Code 0 Emergency: system is unusable 1 Alert: action must be taken immediately 2 Critical: critical conditions 3 Error: error conditions 4 Warning: warning conditions 5 Notice: normal but significant condition 6 Informational: informational messages 7 Debug: debug-level messages We would, of course, invert the numerics. Granted, syslog is pretty old technology. Anyone familiar with the current thinking on this sort of stuff?
Re: Eliminating STDERR without any disruption.
Yitzchak Scott-Thoennes wrote: > Michael G Schwern wrote: >> print "TAP version 15\n"; >> print "1..1\n"; >> print "# Information\n"; >> print "not ok 1\n"; >> print "! Failure\n"; > > I'd really not like to see meaningful punctuation. >> I'm going to say its if a line starts with a ! just for discussion >> purposes. ->>> "Just for discussion purposes" <<<- I haven't given any thought to the real syntax yet. > How about > "diag Failure\n". Or even levels of keywords debug/info/notice/warning/ > err/crit/alert/emerg (stolen from syslog.h). That's an interesting idea. My worry is making it human readable. not ok 2 err Test failed in foo.t line 2 err got: foo err expected: bar ok 3 It doesn't leap out at you the way something like this does. not ok 2 !!! Test failed in foo.t line 2 !!! got: foo !!! expected: bar ok 3 > Oh, and "yaml" if that idea lives. The TAP diagnostic syntax is largely orthogonal to this although it makes most uses of displayed diagnostics obsolete.
Re: Eliminating STDERR without any disruption.
Michael G Schwern wrote: > print "TAP version 15\n"; > print "1..1\n"; > print "# Information\n"; > print "not ok 1\n"; > print "! Failure\n"; I'd really not like to see meaningful punctuation. How about "diag Failure\n". Or even levels of keywords debug/info/notice/warning/ err/crit/alert/emerg (stolen from syslog.h). Oh, and "yaml" if that idea lives.
Re: [tapx-dev] TAP::Parser, structured diagnostics
Andy Armstrong wrote: > On 16 Mar 2007, at 02:52, Michael G Schwern wrote: >> The reason being that the parser should know if there's diagnostics >> forthcoming for the current test. This allows a TAP displayer to >> display the >> complete information for a test without having to wait, possibly a >> long time, >> for the next test line. > > It only has to wait for end-of-YAML - which it'd have to wait for > anyway before it could display anything. So as soon as it sees the > '...' it outputs the decoded YAML. The problem is not knowing when the YAML ends, its knowing if there's any forthcoming at all. Consider if we go with your plan. print "not ok 1\n"; print <<'END'; --- you're fucked: so fucking fucked ... END print "not ok 2\n"; sleep 1000; # simulating a long test print "ok 3\n"; Consider that we're a GUI TAP displayer. When a test fails we want to pop up a little balloon saying what failed and why, so we want to know what failed and why before we do that. We read the first line and see that test 1 failed, maybe there's some diagnostics forthcoming? We read the next line and see the ---. Ah ha, diagnostics. Ok, we read until the ... and then display our popup with all the failure information. We read the next line and see test 2 failed. Maybe there's some diagnostics forthcoming so we go to read the next line... and block for 1000 seconds. Finally we see test 3 passed. Now we know there's no diagnostics forthcoming so we can display that test 2 failed, but 1000 seconds delayed. That's the situation I'm trying to avoid, I'd like TAP to stream without needing any sort of look-ahead. So the marker indicating "there diagnostics forthcoming about this test" goes on the same line as the test itself. The parser can know just by reading one line if there's anything forthcoming or if it has all the information its going to get. I chose "#---" since most existing TAP parsers should ignore it, backwards compatible. The particulars of the syntax are debatable but it has to stream.
Eliminating STDERR without any disruption.
I believe I now know how to move towards no longer using STDERR for failure information display AND keep compatibility with existing test scripts, even those not written using Test::Builder or Test.pm AND not require Test::Builder, Test.pm and TH not be upgraded in lock step AND not introduce ambiguity to TAP AND (most importantly) not effect what diagnostics people installing tests with "make test" see. That last one is the most critical use case, people who are just running existing tests (whether they wrote them or they're someone else's) and upgrade their modules from CPAN and don't really think about it. The information they receive to their eyeballs and brain should remain the same. It goes a little like this. Let's assume a simple test like this: print "1..1\n"; print "# Information\n"; print "not ok 1\n"; print STDERR "# Failure\n"; With the existing system a user sees: Oh god test one failed! # Failure "Oh god test one faled!" comes from the harness. "# Failure" is on STDERR. What's important is that this is what the user sees. Ok, now let's say we change the harness to display diagnostics. The user sees: Oh god test one failed! # Information # Failure A change in what is displayed to the user. So no good. Let's say we add in a new syntax which means "display this verbatim". I'm going to say its if a line starts with a ! just for discussion purposes. We change our test: print "TAP version 15\n"; print "1..1\n"; print "# Information\n"; print "not ok 1\n"; print "! Failure\n"; And we upgrade the harness to recognize this new syntax and display it. Oh god test one failed! ! Failure We're no longer using STDERR yet we retain the ability to differentiate between a straight comment and information for the user. It would be implemented in Test::Builder by making diag() go to STDOUT and having it prefix messages with a ! instead of a #. That's the new producer/new parser case and it works fine. What about the old producer/new parser case? Well we have an old style script that looks like this: print "1..1\n"; print "# Information\n"; print "not ok 1\n"; print STDERR "# Failure\n"; And the upgraded harness, which still ignores STDERR, displays it like this: Oh god test one failed! # Failure So no change, an old style producer still works with a new style parser. This is good, it means the parser can be upgraded without having to change everyone's existing tests. We have a much greater control over the parser because currently there is only one in wide-spread use, Test::Harness. Now what about a new style producer with an old style parser? print "TAP version 15\n"; print "1..1\n"; print "# Information\n"; print "not ok 1\n"; print "! Failure\n"; And the old harness displays... Oh god test one failed! Uh oh, loss of information. The old parser simply ignores the "! Failure", thinks its junk. That's not good. One possible work around is to have the new producer print information to STDERR as well as STDOUT but that means we're all noisy and gross and it something I'd like to avoid. A better work around is to simply avoid this case. As mentioned before we have much more control over the parser then the producer, since there's only one parser (Test::Harness) and lots of producers including ad-hoc ones. We upgrade Test::Harness first, we can do that because old producer / new harness works fine. Then we upgrade Test::Builder and Test.pm to be new-style producers and have them depend on the new version of Test::Harness. Voila! STDERR is eliminated from new style producers. There's no heuristics or ambiguity necessary to determine what should and should not be displayed. Existing tests continue to display just as they did. We retain the ability to put undisplayed comments in the TAP stream. Comments, issues, problems?
Re: The price of synching STDOUT and STDERR
chromatic wrote: > Yeah, much more than a sentence and people start to take me seriously. > > I think diagnostics have to go into the TAP stream at some point. > > I think expecting a harness to merge STDOUT and STDERR when it runs a test > file is prone to errors. > > I think there are plenty of uses for harnesses where they don't have access > to > STDERR, whether that's because they have access only to the TAP stream or for > some other good reason. (I rely on some of them, actually.) > > I think it's a great idea to put some sort of structure in the diagnostics to > give failure information. > > I don't believe this is all possible without upgrading TAP producers and > consumers. > > I do believe that this is possible in stages without breaking code or tests > and with minor disruption to old TAP consumers. > > I'm happy to patch both of the Test::Builder ports I maintain to switch > diag() > over to STDOUT at any point, and I'm confident that it won't break anything. We're in agreement here except on the last three. I believe its possible without requiring a simultaneous upgrade of both TAP producers and consumers. That is, all combinations (new producer / new consumer, new producer / old consumer, old producer / new consumer and old producer / old consumer) can be made to work. I believe its possible to do it without any disruption to old TAP consumers. I believe switching diag() over to STDOUT is a good idea BUT it needs to be done in concert with changes to TAP. Switching diag() over to STDOUT now without any other changes won't break the parsing of tests but it will cause users to stop seeing failure diagnostics. So you put it in with a dependency on an upgraded Test::Harness which displays diagnostics. But then diagnostics on STDOUT which previously were not displayed (TODO tests are one example but people print all sorts of stuff to STDOUT) get displayed. This is the situation I'm not happy about. It makes existing passing tests noisy, even those using Test::Builder. So you put in a heuristic to Test::Harness so that any diagnostics after a failing TODO test are not displayed. But then you introduce uncertainty into what is and is not displayed. Diagnostics coming after a test are not necessarily associated with that test. I'm not happy about introducing ambiguity. It also leaves us with no mechanism to say "put this comment in the TAP stream but don't display it" which is what printing a diagnostic to STDOUT currently does. Not keen on losing that functionality. So my problems with the "make diag() go to STDOUT and make TH display diagnostics" plan is that it makes what you should and should not display ambiguous and it loses functionality. I have a plan to avoid this which is more like "make diag() go to STDOUT and add a TAP syntax to indicate 'display this'". Next post.
Re: [tapx-dev] TAP::Parser, structured diagnostics
On 16 Mar 2007, at 02:52, Michael G Schwern wrote: The reason being that the parser should know if there's diagnostics forthcoming for the current test. This allows a TAP displayer to display the complete information for a test without having to wait, possibly a long time, for the next test line. It only has to wait for end-of-YAML - which it'd have to wait for anyway before it could display anything. So as soon as it sees the '...' it outputs the decoded YAML. -- Andy Armstrong, hexten.net
Re: [tapx-dev] TAP::Parser, structured diagnostics
Andy Armstrong wrote: > I've added to TAP::Parser an experimental YAML syntax for embedding > machine readable diagnostic information in TAP streams. The syntax looks > like this: > > 1..5 > ok 1 > ok 2 > --- > - >fnurk: skib >ponk: gleeb > - >bar: krup >foo: plink > ... That should be this: 1..5 ok 1 ok 2 #--- - fnurk: skib ponk: gleeb - bar: krup foo: plink ... ok 3 The reason being that the parser should know if there's diagnostics forthcoming for the current test. This allows a TAP displayer to display the complete information for a test without having to wait, possibly a long time, for the next test line.
Re: [tapx-dev] TAP::Parser, structured diagnostics
On 15 Mar 2007, at 23:49, Andy Armstrong wrote: The YAML parser we're currently using needs to be fed an entire YAML document in a string so the parser has to recognise the boundaries of ^ TAP the YAML and only then pass it to the parser. This means that if the ^ YAML -- Andy Armstrong, hexten.net
Test::More is broken before 5.8, but its not as bad as it looks
The last two releases of Test-Simple (0.68 and 0.69) break on perls before 5.8. The reason is the, seemingly innocent, change to Test::Builder->is_fh() to use ->isa() instead of UNIVERSAL::isa(). This means things like UNIVERSAL::isa($non_object, "GLOB") don't work any more. Its not as bad as it looks in the tests, the situation which breaks is passing a reference to a tied filehandle to the TB->output methods. Pretty rare but it does make the tests puke their guts out. [1] And I now realize that UNIVERSAL::isa($non_object, "GLOB") is pretty stupid and I can just check with ref. [2] Ok, new release coming. [1] http://youtube.com/watch?v=_9UlxZfDmoA [2] If anyone complains about the flaws in ref I'm going to make them drink Ipecac.
TAP::Parser, structured diagnostics
I've added to TAP::Parser an experimental YAML syntax for embedding machine readable diagnostic information in TAP streams. The syntax looks like this: 1..5 ok 1 ok 2 --- - fnurk: skib ponk: gleeb - bar: krup foo: plink ... ok 3 ok 4 --- expected: - 1 - 2 - 4 got: - 1 - pong - 4 ... ok 5 I'm not actually doing anything with the YAML blocks yet but the parser handles them correctly. I'm open to suggestions for what we do with the data now we have it :) The YAML parser we're currently using needs to be fed an entire YAML document in a string so the parser has to recognise the boundaries of the YAML and only then pass it to the parser. This means that if the trailing '...' is missing the whole of the remainder of the TAP stream will be consumed. I'll probably modify the YAML parser so that it reads directly from the TAP stream. If anyone's interested you can get it from the SVN:. http://svn.hexten.net/tapx -- Andy Armstrong, hexten.net
Does anyone have 5.8.5 installed on Windows?
This is a bit of a long shot. We have a test failure for TAP::Parser 0.51 on Windows: http://www.nntp.perl.org/group/perl.cpan.testers/2007/03/msg437065.html It's Perl 5.8.5 and to this (untutored) eye it looks as if it's not ActiveState's build. I've tried to build 5.8.5 here but can't get it to play nice with VC++ 8.0. If anyone might happen to have 5.8.5 installed on a Windows box and could make test against both T::P 0.51 and a snapshot from the SVN[1] and report back I'd be most grateful. FWIW the tests all pass on Windows/5.8.8. [1] http://svn.hexten.net/tapx -- Andy Armstrong, hexten.net
Re: No, sending diag() to STDOUT won't work. Yet again.
chromatic wrote: > One issue that hasn't come up much is that you can't always rely on STDERR > being available when a human looks at the test results. > > Think of testing long-running programs in process, testing in the browser > (whether via JavaScript Test.Builder or Apache::Test), and automated smoke > tests. > > Sure, dumping diagnostics to STDERR works fairly well when a human enters > the > command to run the tests and sits in front of the controlling terminal and > can scroll back to see the STDERR output, but that's not the only case of > running tests at all (and getting diagnostic output is much, much more > useful > when you don't have access to the particular computer where the tests > fail). Another way of saying that is that TAP is at heart a file/stream format, and where it tries to be something else, it loses benefits that come with that (merging STDOUT/STDERR and the bailout function being two instances that spring to mind).
Re: No, sending diag() to STDOUT won't work. Yet again.
On Thursday 15 March 2007 09:49, brian d foy wrote: > I'm not advocating any change here because I'm perfectly happy with > what we have now, but isn't the problem there that # means too many > things? > > If a comment started with a # (because this is perl), and other things > had some other sigil, would anyone be arguing about which filehandle > they are on? Maybe. The nice thing about # being overloaded is that really old harnesses *should* recognize diagnostics that even implement the future extensions and ignore or display them as they see fit. They won't get any extra meaning, but they won't crash. One issue that hasn't come up much is that you can't always rely on STDERR being available when a human looks at the test results. Think of testing long-running programs in process, testing in the browser (whether via JavaScript Test.Builder or Apache::Test), and automated smoke tests. Sure, dumping diagnostics to STDERR works fairly well when a human enters the command to run the tests and sits in front of the controlling terminal and can scroll back to see the STDERR output, but that's not the only case of running tests at all (and getting diagnostic output is much, much more useful when you don't have access to the particular computer where the tests fail). -- c
Re: The price of synching STDOUT and STDERR
David Cantrell wrote:
> Yitzchak Scott-Thoennes wrote:
>> David Cantrell wrote:
>>> Any test suite that blithely ignores warnings is BROKEN.
>>> The second type of warning is the one that tells you when you the
>>> author
>>> have fucked up, like, when you say "my $variable" twice, or saying
>>> "$variable = 'one thing'" and "$varable = 'something else'". I deal
>>> with those by having $SIG{__WARN__} turn them into die()s in the tests.
>>> If you don't deal with them, you're saying that you don't care about
>>> tripping yourself up in the future.
>> I have to disagree here. You are saying that it is the test's
>> responsibility to fail if it issues any warnings. I believe
>> an author may choose to scan the test output and rely on seeing
>> messages to go to stderr.
>
> Please, tell me how I can scan the test output on - say - Windows. Bear
> in mind that I do not own a Windows licence, nor do I have any hardware
> that will run Windows without lots of annoying faffing around.
You don't. Or you take the extra step to have your test detect and fail
on warnings. But there are downsides, see below.
> When a user installs my module on Windows, he will probably type
> something like:
>
> C:\> perl -MCPAN -e 'install Daves::Module'
>
> which will handle any dependencies for him. If my module - or one of
> the dependencies - spits out a warning, the user is likely to miss it
> when it scrolls past too fast to read. Even worse, he might see it, and
> ignore it because of the bit at the bottom that says "all tests passed".
>THAT is why I believe warnings in tests should be fatal by default.
I support your right to make your warnings fatal, but I'd not like
to see that become the default. New versions of perl can enable new
warnings. If you want to ensure that you are an early tester of new
perl versions against your modules, you can get by with having to
release new module versions as needed. But for the greater number
of perfectly decent modules out there that just sit unchanged for
years, it's too harsh to have them start failing tests out of the
blue.
> It means I am more likely to get useful bug reports. It also means I'll
> get "pro-active" bug reports from the CPAN testers.
>
> > Obviously it's better to do as you
>> propose, but that takes extra work in every test to implement.
>
> $SIG{__WARN__} = sub { die(@_) };
>
> so much extra work! Or if you're less of a depdencies-fascist than I
> am, use Test::NoWarnings for even less typing.
>
>> Authors should not be counted on to do so
>
> Authors are giving their work away for free. They shouldn't be counted
> on to do *anything*.
>
> >and when they don't
>> go that extra mile, their warnings should not just get lost.
>
> Of course. It would be nice if they weren't *ignored* though.
That's exactly what I'm saying; that changing from displaying STDERR
to silently suppressing it is a BAD THING(tm). I thought you were
arguing the opposite; that it's ok to do so because tests that don't
catch warnings are already broken.
> Hrrm, I'm going to go away now and think about how I can inject that
> __WARN__ handler into everyone elses tests when I run them :-)
sitecustomize.pl and $0 =~ /(?:\.t|test\.pl)\z/ ?
Re: No, sending diag() to STDOUT won't work. Yet again.
In article <[EMAIL PROTECTED]>, Michael G Schwern <[EMAIL PROTECTED]> wrote: > Piping all diagnostics to STDOUT solves nothing except maybe allowing runtests > to display warnings again. You still can't tell the difference between a > comment (what currently is "# foo" printed to STDOUT) and a failure diagnostic > (what currently is "# foo" printed to STDERR) and diagnostics associated with > a TODO test (which is "# foo" printed to STDOUT). I'm not advocating any change here because I'm perfectly happy with what we have now, but isn't the problem there that # means too many things? If a comment started with a # (because this is perl), and other things had some other sigil, would anyone be arguing about which filehandle they are on? Again, I'm not saying that anyone should do any work to change this, but after reading the links Schwern provided and checking the wiki, it seems that people talk about the format of the diagnostic but not the thing to signal it.
Fwd: CPAN Upload: P/PE/PETDANCE/Test-Harness-2.65_01.tar.gz
This version of T::H makes prove automatically run under -w, and also adds a -X flag for turning off warnings. Begin forwarded message: From: PAUSE <[EMAIL PROTECTED]> Date: March 15, 2007 11:46:59 AM CDT To: "Andy Lester" <[EMAIL PROTECTED]> Subject: CPAN Upload: P/PE/PETDANCE/Test-Harness-2.65_01.tar.gz Reply-To: cpan-testers@perl.org The uploaded file Test-Harness-2.65_01.tar.gz has entered CPAN as file: $CPAN/authors/id/P/PE/PETDANCE/Test-Harness-2.65_01.tar.gz size: 70579 bytes md5: e8d71a1d8f874765ed48bdfe77acedd9 No action is required on your part Request entered by: PETDANCE (Andy Lester) Request entered on: Thu, 15 Mar 2007 16:45:40 GMT Request completed: Thu, 15 Mar 2007 16:46:59 GMT Thanks, -- paused, v845 -- Andy Lester => [EMAIL PROTECTED] => www.petdance.com => AIM:petdance
Re: The price of synching STDOUT and STDERR
Yitzchak Scott-Thoennes wrote:
David Cantrell wrote:
Any test suite that blithely ignores warnings is BROKEN.
The second type of warning is the one that tells you when you the author
have fucked up, like, when you say "my $variable" twice, or saying
"$variable = 'one thing'" and "$varable = 'something else'". I deal
with those by having $SIG{__WARN__} turn them into die()s in the tests.
If you don't deal with them, you're saying that you don't care about
tripping yourself up in the future.
I have to disagree here. You are saying that it is the test's
responsibility to fail if it issues any warnings. I believe
an author may choose to scan the test output and rely on seeing
messages to go to stderr.
Please, tell me how I can scan the test output on - say - Windows. Bear
in mind that I do not own a Windows licence, nor do I have any hardware
that will run Windows without lots of annoying faffing around.
When a user installs my module on Windows, he will probably type
something like:
C:\> perl -MCPAN -e 'install Daves::Module'
which will handle any dependencies for him. If my module - or one of
the dependencies - spits out a warning, the user is likely to miss it
when it scrolls past too fast to read. Even worse, he might see it, and
ignore it because of the bit at the bottom that says "all tests passed".
THAT is why I believe warnings in tests should be fatal by default.
It means I am more likely to get useful bug reports. It also means I'll
get "pro-active" bug reports from the CPAN testers.
> Obviously it's better to do as you
propose, but that takes extra work in every test to implement.
$SIG{__WARN__} = sub { die(@_) };
so much extra work! Or if you're less of a depdencies-fascist than I
am, use Test::NoWarnings for even less typing.
Authors should not be counted on to do so
Authors are giving their work away for free. They shouldn't be counted
on to do *anything*.
>and when they don't
go that extra mile, their warnings should not just get lost.
Of course. It would be nice if they weren't *ignored* though.
Hrrm, I'm going to go away now and think about how I can inject that
__WARN__ handler into everyone elses tests when I run them :-)
--
David Cantrell
Re: utApia
On 15 Mar 2007, at 09:20, Eric Wilhelm wrote: At the moment, what I'm seeing is differences in priorities placed on wants #1 and #2 and/or how much of "which want" you're willing to give up for the other. Right. Agreed. So let's nail this down to specific actions. My plan for TAP::Parser - assuming Ovid et al agree is: 1) Disable OUT/ERR merging by default. Enabled by a --merge flag to runtests. Documentation says: "using this flag destroys the important distinction between output and errors. Errors may be parsed as TAP. Bad things may happen as a result". We already have the merging implementation and some people are finding it valuable. So we don't destroy it but we make it non-default. 2) Implement TAP grammar for structured diagnostic information. This is patently a good idea. The only friction is that Test::Builder et al don't yet have support. We have to start somewhere. Unless anyone can demonstrate that either of these actions actually move us further away from our goals I'm going to get started on them, probably tomorrow. OK? -- Andy Armstrong, hexten.net
utApia
The diag() debate raged on in pdx tonight. Of course, the sides are roughly in agreement about most things, but with differing priorities and ideas about particulars of the implementation. Perhaps it's time to collect the issues and do some thinking. Fundamentals: 1. Anything on STDERR cannot be meaningful to a tap consumer. Facts: 1. STDERR has historically been used for (non-parsable) failure messages. 2. Completely synchronizing two filehandles is not possible at the harness level (unless the harness is the kernel, then *maybe*.) 3. Merging STDOUT and STDERR breaks starting with `warn "ok";` Wants: 1. Identifiable (tagged) error messages as a formal part of TAP. ~1.5 Display errors synchronously in relation to failing tests. 2. Backwards parser/producer compatibility (mostly with expectations created by fact 1.) Please correct and clarify as needed. I've tried to cook this down into the essentials. If there is a significant fact or want which is not a subset of the above, then I have completely misunderstood. If I've got the fundamentals wrong, then I'm done -- stick a fork in me. If I've got the facts wrong, correct me; otherwise, they're uh... facts. At the moment, what I'm seeing is differences in priorities placed on wants #1 and #2 and/or how much of "which want" you're willing to give up for the other. Forget for a moment about 'Undefined variable' and such warnings (except to remember that they exist and can ruin your day if the protocol gets in their way (I think that's the "cannot be meaningful" part of the fundamental.)) --Eric -- "Beware of bugs in the above code; I have only proved it correct, not tried it." --Donald Knuth --- http://scratchcomputing.com ---
