Wheeling back over to the extra diagnostic output that Schwern originally proposed, I agree with Adam in that any additions we make to TAP must be completely backward-compatible.
I hereby recant my burblings. After reading Adam's replies, I think I might have pushed the thread in the wrong direction. Let me try another angle... Say we want better debugging support in an editor. A programmer runs the test suite, a few tests fail. She selects the first failure and is whisked to a highlighted line in a file. Some other part of the UI points out what output the test expected and what it actually received. What information does the editor need to make this happen? Schwern has already answered this: the file, line, and description, plus the actual and expected output. How do we get this information to the editor? More information would need to be included in the TAP output. How do we put more information in the TAP output? That's the tougher question -- we either mark up directives or add new kinds of output lines. The difference in syntax between these two options appears to be an octothorpe, but I'm not sure about compatibility: "Any output line that is not a plan, a test line or a diagnostic is incorrect. How a harness handles the incorrect line is undefined. Test::Harness silently ignores incorrect lines, but will become more stringent in the future." -- http://xrl.us/o3tq (You guys were right. I now think YAML, or any kind of general-purpose logical markup, is overkill; our requirements are very succinct.) How do we implement this? If the editor application developers didn't care, they could run the tests with custom Test::Builder and parser replacement, or they could follow standards that we decide. The former (proprietary replacement modules) wouldn't be a Good Thing. Sure, it would be the most backward compatible, as they would "only" be used by the editor. However, if we decide how it's done now, multiple editors can share modules for producing TAP and parsing the results. A final random thought: Not all TAP output contains merely one-line actual/expected value diagnostics. Consider Test::Differences, which outputs a side-by-side, linewise comparison of the values given to it. What then? On 7/11/06, Adam Kennedy <[EMAIL PROTECTED]> wrote:
Geoffrey Young wrote: >> However, most perl tests don't care about TAP, they use Test::More and >> Test::Harness and happen to exchange data via TAP. If Test::More and >> Test::Harness decied to use "YAP" (YAML Anything Protocol? :), then most >> applications would probably never notice. > > most _perl_ applications would never notice. > > IMHO however this overhaul ends up playing out, TAPs current marvel is > that it's wonderfully simple and very forgiving - if the new version > isn't completely back compatible y'all should call it something else, > lest you risk alienating all the non-perl folks who embraced the > protocol because of it's simplicity. > > in other words, call it TAP but break current non-perl TAP > implementations and I think you'll wipe out the userbase that some of us > spent a lot of effort trying to win over (and succeeding :) Indeed, I think we were pretty much clear that under no circumstances whatsoever would we break full back-compatibility. The idea of the extra mime-like things was to be able to provide some relatively simple and cross-language enhancements, so that, for example, if running tests with a higher verbosity, there could be some feedback to, say, an editor or a tool about the location of the errors. This is part of why I dislike YAML for this. Regardless of what the marketing says, it isn't cross-language for anything more than trivial use cases, for example objects or !!perl-specific things. As others have mentioned, the beauty of TAP is it's simplicity and accessibility. It Just Works across languages with almost no effort at all, can be done in pure-$your_language easily, and is generally quite forgiving. In introducing some form of semi-standard enhancements (such as the failure diagnostics) I really don't want to lose this. In using YAML, we're already adding a lot of extra weight, adding dependencies on C libraries, and tempting people into using the more ehnanced, non-platform-neutral features. Lets at least let the YAML spec stop morphing and the language implementations become stable before we start thinking of making TAP inherit any baggage from other protocols by adding them as dependencies. Fair enough a "Layer 1" TAP parser might not care, but why not make it as equally easy to implement a "Layer 2" parser as well. Adam K
-- Ian Langworth
