On Wednesday 14 March 2007 17:02, Michael G Schwern wrote: > I forgot, I'm on a mailing list. Thou shalt not let any inaccuracy, no > matter how minor or totally inconsequential to the point being made, go > uncorrected.
I generally consider it good debating technique to characterize the opposing positions fairly and accurately. "No this technique will not work for <insert completely innacurate reason here>" does not apply. > 2) Not everyone uses Test.pm and Test::Builder. Really. TH is one of the > most widely used modules in Perldom and most folks have no idea they're > using it. The alternate TAP producers continue printing to STDERR and things continue unbroken. > 3) TAP comments on STDOUT have never been displayed, it will break that > assumption which has been around for about a decade. Lots of folks take > advantage of this by printing to either STDOUT or STDERR for "don't > display" or "display". You'll start displaying all sorts of comments that > never were before. That's an untrue assumption--but these are *diagnostics*. They don't affect whether tests pass or fail. I don't understand how changing diag() to print to STDOUT somehow causes prints to STDERR to fail or cause strange test failures somehow. > 4) It requires a simultaneous Test::Harness, Test and Test::More upgrade. > I don't want to even think of the dependency mobius strip that will be. So does any change to TAP--but these are *diagnostics*. They don't affect whether tests pass or fail. > 5) It couples the harness and the producer violating one of the central > principles of TAP, what sets it apart from XUnit. If Test::Harness::Straps has parsed diagnostics for almost two years (and it has), I don't see how this is possibly true. > I covered this in the original post. The heuristics used to determine what > diagnostics are associated with what test are just that, heuristics. > There's nothing which guarantees that 1) diagnostics must follow a test and > 2) that all the diagnostics between two tests belong to the proceeding > test. Its just a convention. You can easily lose important failure > diagnostics this way. There are ways of making better diagnostics and improving the heuristics, but not unless there are easy ways of allowing TAP consumers to get at the diagnostics! Horse, then cart. > That might not have been clear, it was intended to show the effect of what > you're proposing on existing tests. I used runtests because it does > display all diagnostics (except for its TODO heuristic). It illustrates > that the STDOUT/STDERR split is an important one in determining what should > be displayed. If you start displaying all diagnostics from STDOUT you're > going to display things which were not intended to be and passing tests > will be noisy. This goes against the principle that passing tests should > be quiet. So, don't display any diagnostics until after a test fails and then only until the next test passes. If all tests pass, there are no diagnostics. Everything's quiet. No problem. If a test fails, there are diagnostics. There may be extra diagnostics until there's a bulletproof way to detect failure diagnostics from extra information, but that doesn't violate the principle that passing tests should be quiet. > > Currently: > > - the test harness displays all free-form information to the user by > > virtue of not touching it at all > > "Currently" as in what Test::Harness 2 currently does or in what > Test::Harness 3 might do as a stop gap? I'm going to assume the former. > > That's not entirely true. "free-form information" (aka TAP diagnostics) > are not displayed by the harness. The only reason diagnostics going to > STDERR are displayed is because the harness does nothing to STDERR, it has > no knowledge. Yes, and the non-TB TAP producers can happily continue to do what they do here without any modification! > > In the future: > > - TAP producers that continue to send diagnostics to STDERR are no worse > > off! - TAP consumers that use Test::Harness::Straps can handle diagnostic > > information > > THS has no future, its going away when TAP::Parser takes over as the guts > of TH 3. It was an experiment and has served its purpose. I was not aware that TAP::Parser can't also support diagnostic information, nor that Andy would reject the five lines it takes to make T::H to take advantage of my patch he applied to T::H::S almost two years ago. > > Diagnostics > > > > Additional information may be put into the testing output on > > separate lines. Diagnostic lines should begin with a "#", which the > > harness must ignore, at least as far as analyzing the test results. The > > har‐ ness is free, however, to display the diagnostics. > > Test::Harness::TAP is not a bible but a draft. That the harness is free to > display diagnostics does not mean that its a good idea to do so by default. Who said by default? Certainly not me. > > Perl warnings? The ones that go to STDERR by default? How in the world > > do they go through diag()? > > That's the point, they don't. I'm not talking about merging the streams. I said that was a silly idea. I'm well familiar with the buffering and ordering problems. That's why I said it was a silly idea! That's precisely why I said that changing diag() to write to the single TAP stream solves so many problems! > Furthermore, its ok for a TAP producer (such as Test::Builder) to require a > certain of Test::Harness (which is really saying "I require a parser which > can handle at least X version of TAP" but we have no direct way of a Perl > module saying that) but the other way around is a biiiig no no. The > harness should have no knowledge of the internals of a test script. I fail to see how changing diag() to write to the single TAP stream forces the harness to have knowledge of the internals of a test script. Either the TAP consumers can handle the Diagnostics section of the TAP "guidelines" or they can't. I know that Test::Harness can because *I wrote that code almost two years ago*. I suspect that TAP::Parser can because it only takes a couple of lines of code to support and it's been in the "guidelines" for ages. > I'm not sure yet what the solution is, but I know that merging the streams > and shunting diag() to STDOUT is not it. I've long since rejected them and > said why many times. The more time we spend repeating the same arguments > about rejected solutions the less we have to think about how to really fix > the problem. > One thing's for sure, I want to eliminate the TAP producer printing > anything to STDERR on purpose. One stream. These two paragraphs have absolutely baffled me. How about creating some TAP streams--with diagnostics--that you think will break a TAP consumer and we let running code decide? TAP::Parser, is that acceptable? -- c
