chromatic wrote: > Yeah, much more than a sentence and people start to take me seriously. > > I think diagnostics have to go into the TAP stream at some point. > > I think expecting a harness to merge STDOUT and STDERR when it runs a test > file is prone to errors. > > I think there are plenty of uses for harnesses where they don't have access > to > STDERR, whether that's because they have access only to the TAP stream or for > some other good reason. (I rely on some of them, actually.) > > I think it's a great idea to put some sort of structure in the diagnostics to > give failure information. > > I don't believe this is all possible without upgrading TAP producers and > consumers. > > I do believe that this is possible in stages without breaking code or tests > and with minor disruption to old TAP consumers. > > I'm happy to patch both of the Test::Builder ports I maintain to switch > diag() > over to STDOUT at any point, and I'm confident that it won't break anything.
We're in agreement here except on the last three. I believe its possible without requiring a simultaneous upgrade of both TAP producers and consumers. That is, all combinations (new producer / new consumer, new producer / old consumer, old producer / new consumer and old producer / old consumer) can be made to work. I believe its possible to do it without any disruption to old TAP consumers. I believe switching diag() over to STDOUT is a good idea BUT it needs to be done in concert with changes to TAP. Switching diag() over to STDOUT now without any other changes won't break the parsing of tests but it will cause users to stop seeing failure diagnostics. So you put it in with a dependency on an upgraded Test::Harness which displays diagnostics. But then diagnostics on STDOUT which previously were not displayed (TODO tests are one example but people print all sorts of stuff to STDOUT) get displayed. This is the situation I'm not happy about. It makes existing passing tests noisy, even those using Test::Builder. So you put in a heuristic to Test::Harness so that any diagnostics after a failing TODO test are not displayed. But then you introduce uncertainty into what is and is not displayed. Diagnostics coming after a test are not necessarily associated with that test. I'm not happy about introducing ambiguity. It also leaves us with no mechanism to say "put this comment in the TAP stream but don't display it" which is what printing a diagnostic to STDOUT currently does. Not keen on losing that functionality. So my problems with the "make diag() go to STDOUT and make TH display diagnostics" plan is that it makes what you should and should not display ambiguous and it loses functionality. I have a plan to avoid this which is more like "make diag() go to STDOUT and add a TAP syntax to indicate 'display this'". Next post.
