Re: Redirecting STDERR/STDOUT on Win32 (was Re: Terrible diagnostic failure)
On 9/23/06, Adam Kennedy <[EMAIL PROTECTED]> wrote: > Yeah, but thats a can of worms in of itself. Using backticks is > simple, and requires no special stuff. If you dont mind blocking until > the other process completes, I see no reason to use another more > complex approach. I seem to recall Randal talking about exploding buffers or something, Forgive me, but i get this ridiculous mental image of bits flying everywhere. :-) Yves -- perl -Mre=debug -e "/just|another|perl|hacker/"
Re: Redirecting STDERR/STDOUT on Win32 (was Re: Terrible diagnostic failure)
Ovid wrote: > - Original Message > From: Michael G Schwern <[EMAIL PROTECTED]> > >> Something to consider in all this is that TAPx::Parser >> needs to know to display what goes to STDERR >> and suppress what goes to STDOUT. This means >> you simply can't mash the two streams together, >> you have to read them separately and get the order right. > > Agreed that the order has to be correct. Are you saying that by telling > Test::Builder > to send everything to the same place that *it* can't get the order right? > That seems wrong. > All I need is an environment variable I can pass it and this problem seems > solved for the time > being. What is about Test::Builder which prevents this? Well, for one, not everything uses Test::Builder. Nor do they use the latest version of Test::Builder. Test.pm, for example, also spits to STDERR. So will most home-rolled libraries. That's because its the only way to get a test to display diagnostic information under Test::Harness. Adding a feature to Test::Builder will not solve the problem for you. It won't work with Test::Builder anyway because of the way TODO tests work. When a TODO test runs and inevitably fails you don't want the diagnostics to be normally displayed to the user but you would like it to show up when run verbose or when the raw output is examined. To accomplish this TODO test diagnostics are sent to STDOUT rather than STDERR. Test::Harness then eats the comments. If everything was mashed into one stream the TAP parser wouldn't know what to display and what to suppress. In order to be compatible with the way tests are currently written you're going to have to capture both streams separately and read them in sync. That's why I'm edging you towards not worrying about this right now, its hard.
Re: Redirecting STDERR/STDOUT on Win32 (was Re: Terrible diagnostic failure)
- Original Message From: Michael G Schwern <[EMAIL PROTECTED]> > Something to consider in all this is that TAPx::Parser > needs to know to display what goes to STDERR > and suppress what goes to STDOUT. This means > you simply can't mash the two streams together, > you have to read them separately and get the order right. Agreed that the order has to be correct. Are you saying that by telling Test::Builder to send everything to the same place that *it* can't get the order right? That seems wrong. All I need is an environment variable I can pass it and this problem seems solved for the time being. What is about Test::Builder which prevents this? > Once again, you will probably be best served by just > leaving STDERR alone and working on a real TAP > diagnostic syntax. Sure, but the in the meantime, I'd like TAPx::Parser to work. Cheers, Ovid
Re: Redirecting STDERR/STDOUT on Win32 (was Re: Terrible diagnostic failure)
I had lots of trouble getting IPC::Run to work well on Win32; if it worked well there, I'd recommend it for this application. If anyone knows how, in Perl, to open up to 3 pipes and then use WaitForMultipleObjects() on the ends the parent process reads & writes, please let me know. That's the most likely way to get anything like this working on Win32. - Barrie Michael G Schwern wrote: demerphq wrote: I found that the suggested code for saving and restoring STDOUT and STDERR given in "perldoc -f open" seems to work OK. This is essentially what IPC::Run3 is doing -- capturing to an external file and then reading it back in and making it available. Yeah, but thats a can of worms in of itself. Using backticks is simple, and requires no special stuff. If you dont mind blocking until the other process completes, I see no reason to use another more complex approach. We do mind blocking until the other process (ie. the test) completes. Otherwise you don't get that nice "30/309 tests" progress counter. I'm surprised simply tying STDOUT and STDERR doesn't work... Well, its testable. Talk with the CPANPLUS guys. They had to solve this problem I believe. Some combination of IPC::Run, IPC::Run3 and IPC::Open3 should do it. Something to consider in all this is that TAPx::Parser needs to know to display what goes to STDERR and suppress what goes to STDOUT. This means you simply can't mash the two streams together, you have to read them separately and get the order right. This is Hard and one of the reasons Test::Harness just leaves STDERR alone. Once again, you will probably be best served by just leaving STDERR alone and working on a real TAP diagnostic syntax.
Re: Redirecting STDERR/STDOUT on Win32 (was Re: Terrible diagnostic failure)
demerphq wrote: >> I found that the suggested code for saving and restoring STDOUT and >> STDERR given in "perldoc -f open" seems to work OK. This is essentially >> what IPC::Run3 is doing -- capturing to an external file and then >> reading it back in and making it available. > > Yeah, but thats a can of worms in of itself. Using backticks is > simple, and requires no special stuff. If you dont mind blocking until > the other process completes, I see no reason to use another more > complex approach. We do mind blocking until the other process (ie. the test) completes. Otherwise you don't get that nice "30/309 tests" progress counter. I'm surprised simply tying STDOUT and STDERR doesn't work... Well, its testable. Talk with the CPANPLUS guys. They had to solve this problem I believe. Some combination of IPC::Run, IPC::Run3 and IPC::Open3 should do it. Something to consider in all this is that TAPx::Parser needs to know to display what goes to STDERR and suppress what goes to STDOUT. This means you simply can't mash the two streams together, you have to read them separately and get the order right. This is Hard and one of the reasons Test::Harness just leaves STDERR alone. Once again, you will probably be best served by just leaving STDERR alone and working on a real TAP diagnostic syntax.
Re: Redirecting STDERR/STDOUT on Win32 (was Re: Terrible diagnostic failure)
On 9/18/06, David Golden <[EMAIL PROTECTED]> wrote: demerphq wrote: > On 9/18/06, Ovid <[EMAIL PROTECTED]> wrote: >> I've gotten a report that the open command fails on Windows. Not a >> surprise, now that I think about it. However, I don't know of any >> portable way of forcing STDERR to STDOUT (and I don't have a Windows >> box handy). This means that my 2000+ TAPx::Parser tests are in >> trouble. If Test::Builder accepted an environment variable which >> allowed me to override this, I might have a way out. So far removing >> the 2>&1 seems to make my tests pass on a Linux box, but that strikes >> me as bizarre as I thought STDERR wouldn't get read that way. What >> the heck am I misunderstanding? > > The easiest way I know of to execute a process in win32 and get both > the stderr and stdout back is to use backticks. > > my $res=`$cmd 2>&1`; I found that the suggested code for saving and restoring STDOUT and STDERR given in "perldoc -f open" seems to work OK. This is essentially what IPC::Run3 is doing -- capturing to an external file and then reading it back in and making it available. Yeah, but thats a can of worms in of itself. Using backticks is simple, and requires no special stuff. If you dont mind blocking until the other process completes, I see no reason to use another more complex approach. yves -- perl -Mre=debug -e "/just|another|perl|hacker/"
Redirecting STDERR/STDOUT on Win32 (was Re: Terrible diagnostic failure)
demerphq wrote:
On 9/18/06, Ovid <[EMAIL PROTECTED]> wrote:
I've gotten a report that the open command fails on Windows. Not a
surprise, now that I think about it. However, I don't know of any
portable way of forcing STDERR to STDOUT (and I don't have a Windows
box handy). This means that my 2000+ TAPx::Parser tests are in
trouble. If Test::Builder accepted an environment variable which
allowed me to override this, I might have a way out. So far removing
the 2>&1 seems to make my tests pass on a Linux box, but that strikes
me as bizarre as I thought STDERR wouldn't get read that way. What
the heck am I misunderstanding?
The easiest way I know of to execute a process in win32 and get both
the stderr and stdout back is to use backticks.
my $res=`$cmd 2>&1`;
I found that the suggested code for saving and restoring STDOUT and
STDERR given in "perldoc -f open" seems to work OK. This is essentially
what IPC::Run3 is doing -- capturing to an external file and then
reading it back in and making it available.
I use it in CPAN::Reporter -- see t\03_test_report.t:
local *OLDOUT;
open( OLDOUT, ">&STDOUT" )
or die "Couldn't save STDOUT before testing";
open( STDOUT, ">$temp_stdout" )
or die "Couldn't redirect STDOUT before testing";
$|++;
my $makefile_rc = ! system("$perl Makefile.PL");
my $test_make_rc = CPAN::Reporter::test( $dist, "$make test" );
system("$make realclean");
close(STDOUT); open(STDOUT, ">&OLDOUT");
Tests pass on Linux, Darwin, Win32, Netbsd and Freebsd accorting to
CPAN::Testers. While that's only STDOUT, the recipe in perldoc seems to
imply you could reopen STDERR to a duplicate of STDOUT (after
redirecting it) to capture both to the same place.
As a side note, it would be nice if IO::Capture or Test::Output used
that method instead of ties so they could be used with external
processes, too.
In the meantime, I'd suggest trying it with IPC::Run3, which should do
the right thing.
Regards,
David Golden
Re: Terrible diagnostic failure
On 9/18/06, Ovid <[EMAIL PROTECTED]> wrote:
- Original Message
From: Michael G Schwern <[EMAIL PROTECTED]>
> > What about an optional environment variable
> > which forcess *all* output to STDOUT or STDERR
> > but, if not present, leaves things as is?
>
> Did anyone think to try it?
>
> $ cat ~/tmp/stdout.t
> #!/usr/bin/perl -w
>
> use Test::More tests => 1;
>
> my $tb = Test::More->builder;
>
> $tb->failure_output( $tb->output );
>
> is 23, 42;
>
>
> $ perl -MTest::Harness -wle 'runtests @ARGV' ~/tmp/stdout.t
> /Users/schwern/tmp/stdoutdubious
>Test returned status 1 (wstat 256, 0x100)
> DIED. FAILED test 1
>Failed 1/1 tests, 0.00% okay
> Failed Test Stat Wstat Total Fail Failed List of Failed
>
---
> /Users/schwern/tmp/stdout.t1 256 11 100.00% 1
> Failed 1/1 test scripts, 0.00% okay. 1/1 subtests failed, 0.00% okay.
>
> Test::Harness throws out all non-TAP stuff going to STDOUT.
> This includes comments. So if Test::Builder started sending
> its diagnostics to STDOUT they'd disappear into the ether.
I have a bit of a problem, I think. It could simply be a matter of
misunderstanding how things work, but I have the following bit of code in
TAPx::Parser::Source::Perl:
my $sym = gensym;
if ( open $sym, "$command 2>&1 |" ) {
return TAPx::Parser::Iterator->new($sym);
}
else {
$self->exit($? >> 8);
$self->error("Could not execute ($command): $!");
warn $self->error;
return;
}
I've gotten a report that the open command fails on Windows. Not a surprise, now that
I think about it. However, I don't know of any portable way of forcing STDERR to
STDOUT (and I don't have a Windows box handy). This means that my 2000+ TAPx::Parser
tests are in trouble. If Test::Builder accepted an environment variable which allowed
me to override this, I might have a way out. So far removing the 2>&1 seems to
make my tests pass on a Linux box, but that strikes me as bizarre as I thought STDERR
wouldn't get read that way. What the heck am I misunderstanding?
The easiest way I know of to execute a process in win32 and get both
the stderr and stdout back is to use backticks.
my $res=`$cmd 2>&1`;
I guess you would have to wrap the result in something so that you get
an iterator over it, but it does work as you can see below.
D:\dev\cpan\re-0.0601>perl -e"my $r=`cl`; print qq(\n); print $r"
Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 13.00.9466 for 80x86
Copyright (C) Microsoft Corporation 1984-2001. All rights reserved.
usage: cl [ option... ] filename... [ /link linkoption... ]
D:\dev\cpan\re-0.0601>perl -e"my $r=`cl 2>&1`; print qq(\n); print $r"
Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 13.00.9466 for 80x86
Copyright (C) Microsoft Corporation 1984-2001. All rights reserved.
usage: cl [ option... ] filename... [ /link linkoption... ]
D:\dev\cpan\re-0.0601>
--
perl -Mre=debug -e "/just|another|perl|hacker/"
Re: Terrible diagnostic failure
* Ovid <[EMAIL PROTECTED]> [2006-09-18T13:18:19] > Anyone have a Windows box and is willing to test this out for me? I will gladly run tests under WinXP + Strawberry if you tell me where to get it and what you want to know. Drop me an IM or email. -- rjbs
Re: Terrible diagnostic failure
- Original Message From: Chris Dolan <[EMAIL PROTECTED]> > Try IPC::Open3, it's in the Perl core. > http://search.cpan.org/perldoc?IPC::Open3 > > IPC::Run3 is supposed to be good on Windows, but I haven't tried it > enough. > http://search.cpan.org/perldoc?IPC::Run3 Anyone have a Windows box and is willing to test this out for me? > Finally, you can try duplicating the STDOUT filehandle and saving it > in STDERR. Does this work as expected in Windows? > > open STDERR, '&STDOUT'; >From perldoc perlfaq8: Note that you cannot simply open STDERR to be a dup of STDOUT in your Perl program and avoid calling the shell to do the redirection. This doesn’t work: open(STDERR, ">&STDOUT"); $alloutput = ‘cmd args‘; # stderr still escapes Cheers, Ovid -- Buy the book -- http://www.oreilly.com/catalog/perlhks/ Perl and CGI -- http://users.easystreet.com/ovid/cgi_course/
Re: Terrible diagnostic failure
- Original Message From: Ovid <[EMAIL PROTECTED]> > If Test::Builder accepted an environment variable which allowed > me to override this, I might have a way out. So far removing the > 2>&1 seems to make my tests pass on a Linux box, but that > strikes me as bizarre as I thought STDERR wouldn't get read > that way. What the heck am I misunderstanding? OK, now I know what the problem is. I wasn't thinking clearly :( Basically, in reading perlopentut, I don't see any way of portably capturing both STDOUT and STDERR, much less keeping them in synch. The reason my tests worked is that the regression tests put everything in STDOUT and my other tests didn't cover the case of STDERR and STDOUT being separate streams. Thus, TAPx::Parser seems to be almost dead in the water for much of what I was hoping to accomplish. Is there some other option? Schwern, I know you initially showed that redirecting STDERR to STDOUT causes no diagnostic information to appear, but that's only in the case of Test::Harness, yes? Would you accept a patch which allowed this redirection so that other TAP parsers have a chance of overcoming this limitation? Again, am I missing something fundamental here? Cheers, Ovid -- Buy the book -- http://www.oreilly.com/catalog/perlhks/ Perl and CGI -- http://users.easystreet.com/ovid/cgi_course/
Re: Terrible diagnostic failure
On Sep 18, 2006, at 11:55 AM, Ovid wrote:
I have a bit of a problem, I think. It could simply be a matter of
misunderstanding how things work, but I have the following bit of
code in TAPx::Parser::Source::Perl:
my $sym = gensym;
if ( open $sym, "$command 2>&1 |" ) {
return TAPx::Parser::Iterator->new($sym);
}
else {
$self->exit($? >> 8);
$self->error("Could not execute ($command): $!");
warn $self->error;
return;
}
I've gotten a report that the open command fails on Windows. Not a
surprise, now that I think about it. However, I don't know of any
portable way of forcing STDERR to STDOUT (and I don't have a
Windows box handy). This means that my 2000+ TAPx::Parser tests
are in trouble. If Test::Builder accepted an environment variable
which allowed me to override this, I might have a way out. So far
removing the 2>&1 seems to make my tests pass on a Linux box, but
that strikes me as bizarre as I thought STDERR wouldn't get read
that way. What the heck am I misunderstanding?
Cheers,
Ovid
Try IPC::Open3, it's in the Perl core.
http://search.cpan.org/perldoc?IPC::Open3
IPC::Run3 is supposed to be good on Windows, but I haven't tried it
enough.
http://search.cpan.org/perldoc?IPC::Run3
Finally, you can try duplicating the STDOUT filehandle and saving it
in STDERR. Does this work as expected in Windows?
open STDERR, '&STDOUT';
Chris
--
Chris Dolan, Software Developer, Clotho Advanced Media Inc.
608-294-7900, fax 294-7025, 1435 E Main St, Madison WI 53703
vCard: http://www.chrisdolan.net/ChrisDolan.vcf
Clotho Advanced Media, Inc. - Creators of MediaLandscape Software
(http://www.media-landscape.com/) and partners in the revolutionary
Croquet project (http://www.opencroquet.org/)
Re: Terrible diagnostic failure
- Original Message
From: Michael G Schwern <[EMAIL PROTECTED]>
> > What about an optional environment variable
> > which forcess *all* output to STDOUT or STDERR
> > but, if not present, leaves things as is?
>
> Did anyone think to try it?
>
> $ cat ~/tmp/stdout.t
> #!/usr/bin/perl -w
>
> use Test::More tests => 1;
>
> my $tb = Test::More->builder;
>
> $tb->failure_output( $tb->output );
>
> is 23, 42;
>
>
> $ perl -MTest::Harness -wle 'runtests @ARGV' ~/tmp/stdout.t
> /Users/schwern/tmp/stdoutdubious
>Test returned status 1 (wstat 256, 0x100)
> DIED. FAILED test 1
>Failed 1/1 tests, 0.00% okay
> Failed Test Stat Wstat Total Fail Failed List of Failed
> ---
> /Users/schwern/tmp/stdout.t1 256 11 100.00% 1
> Failed 1/1 test scripts, 0.00% okay. 1/1 subtests failed, 0.00% okay.
>
> Test::Harness throws out all non-TAP stuff going to STDOUT.
> This includes comments. So if Test::Builder started sending
> its diagnostics to STDOUT they'd disappear into the ether.
I have a bit of a problem, I think. It could simply be a matter of
misunderstanding how things work, but I have the following bit of code in
TAPx::Parser::Source::Perl:
my $sym = gensym;
if ( open $sym, "$command 2>&1 |" ) {
return TAPx::Parser::Iterator->new($sym);
}
else {
$self->exit($? >> 8);
$self->error("Could not execute ($command): $!");
warn $self->error;
return;
}
I've gotten a report that the open command fails on Windows. Not a surprise,
now that I think about it. However, I don't know of any portable way of
forcing STDERR to STDOUT (and I don't have a Windows box handy). This means
that my 2000+ TAPx::Parser tests are in trouble. If Test::Builder accepted an
environment variable which allowed me to override this, I might have a way out.
So far removing the 2>&1 seems to make my tests pass on a Linux box, but that
strikes me as bizarre as I thought STDERR wouldn't get read that way. What the
heck am I misunderstanding?
Cheers,
Ovid
Re: Terrible diagnostic failure
Geoffrey Young wrote: and seems a little broken so I doubt anyone's using it, I use it every day, and it's not broken for me... By broken I mean I was surprised to find the behavior of Apache::Test changed by changing the backend. I would have expected the interface to remain the same. Looking at it again it is clearly documented otherwise (I guess its not a backend after all) but nevertheless surprising to me. $ perl -Mblib -wle 'use Apache::Test qw(-withtestmore); plan tests => 1; ok 1, 2' 1..1 # Using Apache/Test.pm version 1.28 # You named your test '2'. You shouldn't use numbers for your test names. # Very confusing. ok 1 - 2 your turn for coffee :) $ perl -e 'use Test::More; plan tests => 1; ok 1,2' $ perl -e 'use Apache::Test qw(-withtestmore); plan tests => 1; is 1,2' --Geoff
Re: Terrible diagnostic failure
> Its been doing that for the last 10 years or so. Try an espresso. yeah, ok. > Apache::Test, by default, sends diagnostics to STDERR. This is because > by default it uses Test.pm which sends its errors to STDERR. right. I haven't actually used the Test.pm interface in ages. but most other people do I guess. seems like we should change the Test.pm behavior though - when running tests on the server we probably want STDERR to appear on the command line as well, not be lost in the error_log. > Only when you explicitly say -withtestmore does it go to STDOUT. right. > -withtestmore seems like it changes the behavior of Apache::Test yes. A-T's dependence on Test.pm predates my involvement, and there has been significant pushback to make it rely solely on T::B and friends. so, T::B support is a side venture for people like me who care. > and > seems a little broken so I doubt anyone's using it, I use it every day, and it's not broken for me... > the tests don't, > which is why you haven't noticed. yeah, very few tests for it, unfortunately. > > $ perl -Mblib -wle 'use Apache::Test qw(-withtestmore); plan tests => > 1; ok 1, 2' > 1..1 > # Using Apache/Test.pm version 1.28 > > # You named your test '2'. You shouldn't use numbers for your test > names. > # Very confusing. > ok 1 - 2 your turn for coffee :) $ perl -e 'use Test::More; plan tests => 1; ok 1,2' $ perl -e 'use Apache::Test qw(-withtestmore); plan tests => 1; is 1,2' --Geoff
Re: Terrible diagnostic failure
Geoffrey Young wrote: $ perl -MTest::Harness -wle 'runtests @ARGV' ~/tmp/stdout.t /Users/schwern/tmp/stdoutdubious Test returned status 1 (wstat 256, 0x100) DIED. FAILED test 1 Failed 1/1 tests, 0.00% okay Failed Test Stat Wstat Total Fail Failed List of Failed --- /Users/schwern/tmp/stdout.t1 256 11 100.00% 1 Failed 1/1 test scripts, 0.00% okay. 1/1 subtests failed, 0.00% okay. Oh look, no diagnostics at all. um, ok, so I haven't quite finished my coffee this morning, but isn't that what we would expect without $Test::Harness::verbose being set? Have more coffee. It has never worked that way. $Test::Harness::verbose is about showing the raw output of the .t file. This means all the "ok" and "not ok" lines which it normally obscures. It only effects stuff going to STDOUT. Things going to STDERR have always been displayed. Here's a simple example which you can try while brewing up a pot. $ cat fail.t #!/usr/bin/perl -w use Test::More tests => 1; is 23, 42; $ perl -MTest::Harness -wle 'runtests @ARGV' fail.t fail failNOK 1# Failed test in fail.t at line 5. # got: '23' # expected: '42' # Looks like you failed 1 test of 1. faildubious Test returned status 1 (wstat 256, 0x100) DIED. FAILED test 1 Failed 1/1 tests, 0.00% okay Failed Test Stat Wstat Total Fail List of Failed --- fail.t 1 256 11 1 Failed 1/1 test scripts. 1/1 subtests failed. Files=1, Tests=1, 0 wallclock secs ( 0.02 cusr + 0.01 csys = 0.03 CPU) Failed 1/1 test programs. 1/1 subtests failed. I certainly don't want diagnostics clogging up my output unless I ask for them. Its been doing that for the last 10 years or so. Try an espresso. Yes, you do want failure diagnostics. Otherwise you're left not knowing why things failed and we might as well go back to ed-style error reporting (ie. just print a "?"). You get into fun situations where users report test failures with no diagnostic info and you have to go tell them to rerun the tests with the verbose flag on. Or you run the tests and get a failure and then you have to rerun them again with "verbose" on to find out why (and its a heisenbug so it doesn't appear that time). $ perl -MTest::Harness -wle '$Test::Harness::verbose = 1; runtests @ARGV' /tmp/stdout.t Test::Harness throws out all non-TAP stuff going to STDOUT. This includes comments. So if Test::Builder started sending its diagnostics to STDOUT they'd disappear into the ether. I haven't found that to be true I don't think - because STDERR is the error_log file, Apache-Test sends everything to STDOUT and it works just fine for us. or so it has seemed for a while now... Apache::Test, by default, sends diagnostics to STDERR. This is because by default it uses Test.pm which sends its errors to STDERR. $ perl -Mblib -wle 'use Apache::Test; plan tests => 1; ok 1, 2' > /dev/null # Test 1 got: "1" (-e at line 1) # Expected: "2" Only when you explicitly say -withtestmore does it go to STDOUT. $ perl -Mblib -wle 'use Apache::Test qw(-withtestmore); plan tests => 1; is 1, 2' > /dev/null $ -withtestmore seems like it changes the behavior of Apache::Test and seems a little broken so I doubt anyone's using it, the tests don't, which is why you haven't noticed. $ perl -Mblib -wle 'use Apache::Test qw(-withtestmore); plan tests => 1; ok 1, 2' 1..1 # Using Apache/Test.pm version 1.28 # You named your test '2'. You shouldn't use numbers for your test names. # Very confusing. ok 1 - 2
Re: Terrible diagnostic failure
> $ perl -MTest::Harness -wle 'runtests @ARGV' ~/tmp/stdout.t > /Users/schwern/tmp/stdoutdubious >Test returned status 1 (wstat 256, 0x100) > DIED. FAILED test 1 >Failed 1/1 tests, 0.00% okay > Failed Test Stat Wstat Total Fail Failed List of Failed > --- > > /Users/schwern/tmp/stdout.t1 256 11 100.00% 1 > Failed 1/1 test scripts, 0.00% okay. 1/1 subtests failed, 0.00% okay. > > > Oh look, no diagnostics at all. um, ok, so I haven't quite finished my coffee this morning, but isn't that what we would expect without $Test::Harness::verbose being set? I certainly don't want diagnostics clogging up my output unless I ask for them. $ perl -MTest::Harness -wle '$Test::Harness::verbose = 1; runtests @ARGV' /tmp/stdout.t > > Test::Harness throws out all non-TAP stuff going to STDOUT. This > includes comments. So if Test::Builder started sending its diagnostics > to STDOUT they'd disappear into the ether. I haven't found that to be true I don't think - because STDERR is the error_log file, Apache-Test sends everything to STDOUT and it works just fine for us. or so it has seemed for a while now... --Geoff
Re: Terrible diagnostic failure
Ovid wrote: Are there any Test:: modules which test the STDERR output directly or something like that? * Test::Output * IO::Capture There are tests that specifically check what's being sent to STDERR (as well as STDOUT). Of course, these modules intercept STDOUT and STDERR to keep them away from Test::Harness so as to do that safely. Depending on where STDOUT and STDERR are merged, this may or may not be an issue. If Test::Harness merges them before calling the test files, then there will be a problem. If Test::Builder merges them before issuing TAP output and diagnostics, then it won't interfere. David
Re: Terrible diagnostic failure
On Tue, Sep 05, 2006 at 12:01:47 +0300, Yuval Kogman wrote: > FWIW, pugs' test suite outputs diags to STDOUT. Things like > Test::TAP::Builder are much more useful this way, too. Err, i mean Test::TAP::Model. -- Yuval Kogman <[EMAIL PROTECTED]> http://nothingmuch.woobling.org 0xEBD27418 pgpQjk6E0l3W7.pgp Description: PGP signature
Re: Terrible diagnostic failure
On Tue, Sep 05, 2006 at 01:08:45 -0700, Ovid wrote: > So who wants to bet against Adam and me? I'm willing to bet that it'll take you at least a week to win, thus making it seem like you actually lost ;-) FWIW, pugs' test suite outputs diags to STDOUT. Things like Test::TAP::Builder are much more useful this way, too. -- Yuval Kogman <[EMAIL PROTECTED]> http://nothingmuch.woobling.org 0xEBD27418 pgpn0AMgdGKGV.pgp Description: PGP signature
Re: Terrible diagnostic failure
- Original Message From: Adam Kennedy <[EMAIL PROTECTED]> > Without knowing anything that will specifically break, I think I'll take > that bet. There's just too much code scattered around the place not to > have SOMETHING break... > > Loser buys the winner a restaurant meal at the next YAPC? The only problem being that I'm sure something is going to break, too. We can't both bet on the same side :) So who wants to bet against Adam and me? Cheers, Ovid -- Buy the book -- http://www.oreilly.com/catalog/perlhks/ Perl and CGI -- http://users.easystreet.com/ovid/cgi_course/
Re: Terrible diagnostic failure
Ovid wrote: - Original Message From: ivorw <[EMAIL PROTECTED]> I've had similar issues with test output out of sequence, especially when I pipe the output into more or tee (sometimes 2>&1 helps, but not always). What about an optional environment variable which forcess *all* output to STDOUT or STDERR but, if not present, leaves things as is? Seems like that would work and would also be backwards compatible. Did anyone think to try it? $ cat ~/tmp/stdout.t #!/usr/bin/perl -w use Test::More tests => 1; my $tb = Test::More->builder; $tb->failure_output( $tb->output ); is 23, 42; $ perl -MTest::Harness -wle 'runtests @ARGV' ~/tmp/stdout.t /Users/schwern/tmp/stdoutdubious Test returned status 1 (wstat 256, 0x100) DIED. FAILED test 1 Failed 1/1 tests, 0.00% okay Failed Test Stat Wstat Total Fail Failed List of Failed --- /Users/schwern/tmp/stdout.t1 256 11 100.00% 1 Failed 1/1 test scripts, 0.00% okay. 1/1 subtests failed, 0.00% okay. Oh look, no diagnostics at all. Test::Harness throws out all non-TAP stuff going to STDOUT. This includes comments. So if Test::Builder started sending its diagnostics to STDOUT they'd disappear into the ether. Now we'd have to change the behavior of Test::Harness AND Test::Builder at the same time. Test::Builder (and by extension most of the Test modules on CPAN) would now depend on a very, very recent version of Test::Harness. I'm not ready to be that cavalier with critical core toolchain dependencies. This is not worth that kind of hell (for me). And no, I'm not going to have Test::Builder trying to guess at what version of Test::Harness its being run under. Even so its still not a good idea because when a TODO test fails the way it hides its failure diagnostics is by sending them to STDOUT. Now they'd start showing up and Test::Harness would need a way to tell the difference between diagnostics associated with a real failure and ones associated with a TODO failure. People, this ain't gonna fly. Here's how you really fix it. 1) Give up trying to fix the current diagnostics. They're unformatted comments. 2) Hammer out a real diagnostic TAP syntax such as this http://perl-qa.yi.org/index.php/TAP_diagnostic_syntax
Re: Terrible diagnostic failure
Ovid wrote: - Original Message From: Andy Lester <[EMAIL PROTECTED]> I think I might just JFDI a "T::H goes to STDOUT" release. You can call it the "Ovid wants to have Petdance's children" release. Anyone care to place bets as to whether or not anything depends on this behavior? Without knowing anything that will specifically break, I think I'll take that bet. There's just too much code scattered around the place not to have SOMETHING break... Loser buys the winner a restaurant meal at the next YAPC? Adam K
Re: Terrible diagnostic failure
Rafael Garcia-Suarez wrote: Ovid wrote in perl.qa : I've had similar issues with test output out of sequence, especially when I pipe the output into more or tee (sometimes 2>&1 helps, but not always). What about an optional environment variable which forcess *all* output to STDOUT or STDERR but, if not present, leaves things as is? Seems like that would work and would also be backwards compatible. What's exactly the problem with sending everything to stdout by default? T::H knows how to cope with that. I don't like the environment variable idea. It's easy to forget your environment and be surprised when switching to anthoer session or machine. Wouldn't the main problem here be that the code being tested could be randomly spewing stuff to STDERR on error, and we can't be certain that isn't going to clash with the TAP output? Adam K
Re: Terrible diagnostic failure
On Sep 4, 2006, at 4:15 PM, Ovid wrote: You can call it the "Ovid wants to have Petdance's children" release. Anyone care to place bets as to whether or not anything depends on this behavior? PLease note as David pointed out that it's really Test::Builder that's the issue here. -- Andy Lester => [EMAIL PROTECTED] => www.petdance.com => AIM:petdance
Re: Terrible diagnostic failure
- Original Message From: Andy Lester <[EMAIL PROTECTED]> > I think I might just JFDI a "T::H goes to STDOUT" release. You can call it the "Ovid wants to have Petdance's children" release. Anyone care to place bets as to whether or not anything depends on this behavior? Cheers, Ovid -- Buy the book -- http://www.oreilly.com/catalog/perlhks/ Perl and CGI -- http://users.easystreet.com/ovid/cgi_course/
Re: Terrible diagnostic failure
On Sep 4, 2006, at 10:58, Andy Lester wrote: I think I might just JFDI a "T::H goes to STDOUT" release. That won't preveing T::B from sending output to two different file handles… Best, David
Re: Terrible diagnostic failure
On Sep 4, 2006, at 07:19, Ovid wrote: Actually, I'm not sure how this would break anything and *I* don't see a problem with it, but since the current behavior is to use separate file handles, I'm not sure if there's anyone relying on the current behavior. However, given that it's so widespread, I'm sure something does. I agree. Test.Builder (JavaScript) uses one file handle (function) for all output by default, but there are hooks for users to send there various types of outputs to different places, if they like. This makes it easy to, say, make failures red, for example. Best, David
Re: Terrible diagnostic failure
On Sep 4, 2006, at 12:44 PM, chromatic wrote: T::H didn't always know how to cope with that. I patched it within the past couple of years. I think it's a long-time backwards compatibility issue, but I don't feel strongly about it. I think I might just JFDI a "T::H goes to STDOUT" release. -- Andy Lester => [EMAIL PROTECTED] => www.petdance.com => AIM:petdance
Re: Terrible diagnostic failure
On Monday 04 September 2006 07:12, Rafael Garcia-Suarez wrote: > What's exactly the problem with sending everything to stdout by default? > T::H knows how to cope with that. T::H didn't always know how to cope with that. I patched it within the past couple of years. I think it's a long-time backwards compatibility issue, but I don't feel strongly about it. -- c
Re: Terrible diagnostic failure
- Original Message From: Rafael Garcia-Suarez <[EMAIL PROTECTED]> > What's exactly the problem with sending everything to stdout by default? > T::H knows how to cope with that. I don't like the environment variable > idea. It's easy to forget your environment and be surprised when > switching to anthoer session or machine. Actually, I'm not sure how this would break anything and *I* don't see a problem with it, but since the current behavior is to use separate file handles, I'm not sure if there's anyone relying on the current behavior. However, given that it's so widespread, I'm sure something does. Are there any Test:: modules which test the STDERR output directly or something like that? Cheers, Ovid -- Buy the book -- http://www.oreilly.com/catalog/perlhks/ Perl and CGI -- http://users.easystreet.com/ovid/cgi_course/
Re: Terrible diagnostic failure
Ovid wrote in perl.qa : >> I've had similar issues with test output out of sequence, especially when >> I pipe the output into more or tee (sometimes 2>&1 helps, but not >> always). > > What about an optional environment variable which forcess *all* output > to STDOUT or STDERR but, if not present, leaves things as is? Seems > like that would work and would also be backwards compatible. What's exactly the problem with sending everything to stdout by default? T::H knows how to cope with that. I don't like the environment variable idea. It's easy to forget your environment and be surprised when switching to anthoer session or machine. -- * What system had proved more effective? * Indirect suggestion implicating selfinterest. -- Ulysses
Re: Terrible diagnostic failure
- Original Message From: ivorw <[EMAIL PROTECTED]> > I've had similar issues with test output out of sequence, especially when > I pipe the output into more or tee (sometimes 2>&1 helps, but not > always). What about an optional environment variable which forcess *all* output to STDOUT or STDERR but, if not present, leaves things as is? Seems like that would work and would also be backwards compatible. Cheers, Ovid
RE: Terrible diagnostic failure
> -Original Message- > From: demerphq [mailto:[EMAIL PROTECTED] > Sent: 04 September 2006 13:13 > To: Ovid > Cc: perl-qa@perl.org > Subject: Re: Terrible diagnostic failure > > > On 9/4/06, Ovid <[EMAIL PROTECTED]> wrote: > > Once again we have an example of how using different output > streams for regular and diagnostic information causes serious > problems. This is the worst I've seen. In one method I'm > testing I embedded the following: > > > > use Data::Dumper; > > ::diag(Data::Dumper->Dump( > > [$stripped, [EMAIL PROTECTED], > > [qw<$stripped *records> ] > > )); > > > > And here's the output in the test: > > > > ok 14 - ... and splitting the sql into individual > statements should succeed > > # $stripped = 'alter table test_colors add column foo > varchar(200)'; > > ok 15 - ... and splitting the sql should succeed > > # @records = ( > > ok 16 - ... but the original SQL should remain unchanged > > # 'alter table test_colors add column foo > varchar(200)' > > #); > > not ok 17 - We should be able to "split" a single statement > > > > Ouch. This is what I 'normally' see when I run this: > > > > ok 14 - ... and splitting the sql into individual > statements should succeed > > ok 15 - ... and splitting the sql should succeed > > ok 16 - ... but the original SQL should remain unchanged > > # $stripped = 'alter table test_colors add column foo > varchar(200)'; > > # @records = ( > > # 'alter table test_colors add column foo > varchar(200)' > > #); > > This looks like buffering issues to me. I see stuff like this all the > time when i run code through an editor. Perl tests to see if stdout > (and maybe stderr) are terminals and then automatically turns on > auto-flush when they are. But when i launch the script via my editor > apparently perl thinks its not dealing with a terminal and doesnt > buffer the filehandles, resulting in output much like you have above. > So i wonder if you manually turn on autoflushing of all the output > handles if that helps. > I've had similar issues with test output out of sequence, especially when I pipe the output into more or tee (sometimes 2>&1 helps, but not always). I've not had consistent behaviour between operating systems and between perl versions, but I've not had time to nail the circumstances. I've also had flushing problems with the debugger, with tests that fork a child process (especially with open2 and open3). I find that I lose the results from the child, but if I run outside the debugger I am OK. Ivor.
Re: Terrible diagnostic failure
On 9/4/06, Ovid <[EMAIL PROTECTED]> wrote: Once again we have an example of how using different output streams for regular and diagnostic information causes serious problems. This is the worst I've seen. In one method I'm testing I embedded the following: use Data::Dumper; ::diag(Data::Dumper->Dump( [$stripped, [EMAIL PROTECTED], [qw<$stripped *records> ] )); And here's the output in the test: ok 14 - ... and splitting the sql into individual statements should succeed # $stripped = 'alter table test_colors add column foo varchar(200)'; ok 15 - ... and splitting the sql should succeed # @records = ( ok 16 - ... but the original SQL should remain unchanged # 'alter table test_colors add column foo varchar(200)' #); not ok 17 - We should be able to "split" a single statement Ouch. This is what I 'normally' see when I run this: ok 14 - ... and splitting the sql into individual statements should succeed ok 15 - ... and splitting the sql should succeed ok 16 - ... but the original SQL should remain unchanged # $stripped = 'alter table test_colors add column foo varchar(200)'; # @records = ( # 'alter table test_colors add column foo varchar(200)' #); This looks like buffering issues to me. I see stuff like this all the time when i run code through an editor. Perl tests to see if stdout (and maybe stderr) are terminals and then automatically turns on auto-flush when they are. But when i launch the script via my editor apparently perl thinks its not dealing with a terminal and doesnt buffer the filehandles, resulting in output much like you have above. So i wonder if you manually turn on autoflushing of all the output handles if that helps. Yves -- perl -Mre=debug -e "/just|another|perl|hacker/"
Terrible diagnostic failure
Once again we have an example of how using different output streams for regular and diagnostic information causes serious problems. This is the worst I've seen. In one method I'm testing I embedded the following: use Data::Dumper; ::diag(Data::Dumper->Dump( [$stripped, [EMAIL PROTECTED], [qw<$stripped *records> ] )); And here's the output in the test: ok 14 - ... and splitting the sql into individual statements should succeed # $stripped = 'alter table test_colors add column foo varchar(200)'; ok 15 - ... and splitting the sql should succeed # @records = ( ok 16 - ... but the original SQL should remain unchanged # 'alter table test_colors add column foo varchar(200)' #); not ok 17 - We should be able to "split" a single statement Ouch. This is what I 'normally' see when I run this: ok 14 - ... and splitting the sql into individual statements should succeed ok 15 - ... and splitting the sql should succeed ok 16 - ... but the original SQL should remain unchanged # $stripped = 'alter table test_colors add column foo varchar(200)'; # @records = ( # 'alter table test_colors add column foo varchar(200)' #); Cheers, Ovid -- Buy the book -- http://www.oreilly.com/catalog/perlhks/ Perl and CGI -- http://users.easystreet.com/ovid/cgi_course/
