Re: You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
chromatic wrote: > On Friday 09 March 2007 14:50, Michael G Schwern wrote: > >> We can leverage any existing status system we want. HTTP status. Exit >> status. Throw them all in! Don't find TAP's existing statuses rich >> enough? Add your own extension keys! A particular status code not make >> sense for your application? Leave it out! > > Frequently Unasked Question: > > But Schwern, if we run all of our tests through Test::Harness, not only > will > it not report that extra diagnostic information but it doesn't run non-Perl > tests very easily! Therefore we can't use TAP! > > Frequently Unanswered Answer: > > You can *run* the tests and *analyze* the results in separate steps. > That's > why they're separate steps. > > I think people get hung up on the idea that there is or has to be only one > way > to run-and-analyze tests in one step. That's great, put it on the wiki please.
Re: You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
On Friday 09 March 2007 14:50, Michael G Schwern wrote: > We can leverage any existing status system we want. HTTP status. Exit > status. Throw them all in! Don't find TAP's existing statuses rich > enough? Add your own extension keys! A particular status code not make > sense for your application? Leave it out! Frequently Unasked Question: But Schwern, if we run all of our tests through Test::Harness, not only will it not report that extra diagnostic information but it doesn't run non-Perl tests very easily! Therefore we can't use TAP! Frequently Unanswered Answer: You can *run* the tests and *analyze* the results in separate steps. That's why they're separate steps. I think people get hung up on the idea that there is or has to be only one way to run-and-analyze tests in one step. -- c
Re: You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
Adam Kennedy wrote: >> I propose that we prefix lines from STDERR with '! ' in the same way >> that '# ' is used for diagnostics. wstat and exit can just be >> >> wstat 256 >> exit 1 > > The problem with STDERR and exit is that we can't actually use them for > anything significant. > > Otherwise when dealing with TAP streams that don't have a concept of an > exit code or a seperate error channel, the most common example being web > testing, we're left high and dry. > > That's always been the main reason to intentionally ignore the exit code > at least. Am I TALKING TO MYSELF?! We do *not* have to decide on a fixed set of diagnostics you attach at the end of a test! We can allow the wait status, the exit status, the HTTP status... whatever you like. Put them in as you need. http://www.nntp.perl.org/group/perl.qa/2007/03/msg8155.html Extensibility is the key. This is why I like a YAML or JSON format so much. We can always add to it. http://perl-qa.yi.org/index.php/TAP_diagnostic_syntax 1..2 ok 1 ok 2 done (or whatever we wind up using for "end of tests") --- exit status: 23 wait status: 0 http status: 404 human readable status: Yer screwed. psychological status: I'm feeling fine, thanks ... We can leverage any existing status system we want. HTTP status. Exit status. Throw them all in! Don't find TAP's existing statuses rich enough? Add your own extension keys! A particular status code not make sense for your application? Leave it out!
Re: You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
Andy Armstrong wrote: > On 8 Mar 2007, at 22:47, Eric Hacker wrote: >>> I propose that we prefix lines from STDERR with '! ' in the same way >>> that '# ' is used for diagnostics. wstat and exit can just be >>> >>> wstat 256 >>> exit 1 >> >> How about this? >> >> wstat: 256 >> exit: 1 >> >> YAML, YAML, do! > > Doesn't look like TAP though :) I'm starting to get the feeling that every time I mention this: http://perl-qa.yi.org/index.php/TAP_diagnostic_syntax everyone sort of stares off into space for a few seconds with their mouths gaping open and then blinks twice and continues on with whatever they were doing without ever having seen the URL. It never gets any comment. Why is that? Is it the fnords? Are there fnords in that URL? Its ok, they won't eat you. I'll protect you. Everyone keeps reinventing the same wheel and coming around to the same conclusion. We have a wiki to avoid exactly this sort of thing. Either comment on what's wrong with it or code that proposal! If I see another "let's hack up the Test::More diagnostics" thread I'm going to puke.
Re: You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
On 3/8/07, Adam Kennedy <[EMAIL PROTECTED]> wrote: Andy Armstrong wrote: >> Otherwise when dealing with TAP streams that don't have a concept of >> an exit code or a seperate error channel, the most common example >> being web testing, we're left high and dry. > > In which case you'd just omit them, no? I didn't imagine they'd be > mandatory. Web testing doesn't have an error channel? HTTP has lot's of channels. They are called Status Codes. :) There are a whole slew of them for errors. Way more than you'd ever need for testing, or so I'm told. Peace, Hacker
Re: You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
On 9 Mar 2007, at 01:44, Adam Kennedy wrote: Like I said, we can't use them for anything that matters :) It's just occurred to me that it makes it easier to write tests for the harness too :) I can't quite get my head round the objection to the idea that a TAP transcript could be a complete, machine readable record of a test run. We're very nearly there anyway. -- Andy Armstrong, hexten.net
Re: You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
On 3/8/07, Andy Armstrong <[EMAIL PROTECTED]> wrote: I propose that we prefix lines from STDERR with '! ' in the same way that '# ' is used for diagnostics. wstat and exit can just be wstat 256 exit 1 How about this? wstat: 256 exit: 1 YAML, YAML, do! ;)
Re: You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
On 9 Mar 2007, at 00:28, Adam Kennedy wrote: I propose that we prefix lines from STDERR with '! ' in the same way that '# ' is used for diagnostics. wstat and exit can just be wstat 256 exit 1 The problem with STDERR and exit is that we can't actually use them for anything significant. We currently have the option of spooling parse TAP to a file during testing for later consumption. If we store STDERR + exit in that TAP it becomes possible to recreate the test run pretty accurately. Otherwise when dealing with TAP streams that don't have a concept of an exit code or a seperate error channel, the most common example being web testing, we're left high and dry. In which case you'd just omit them, no? I didn't imagine they'd be mandatory. -- Andy Armstrong, hexten.net
Re: You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
I propose that we prefix lines from STDERR with '! ' in the same way that '# ' is used for diagnostics. wstat and exit can just be wstat 256 exit 1 The problem with STDERR and exit is that we can't actually use them for anything significant. Otherwise when dealing with TAP streams that don't have a concept of an exit code or a seperate error channel, the most common example being web testing, we're left high and dry. That's always been the main reason to intentionally ignore the exit code at least. Adam K
Re: You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
On 8 Mar 2007, at 22:47, Eric Hacker wrote: I propose that we prefix lines from STDERR with '! ' in the same way that '# ' is used for diagnostics. wstat and exit can just be wstat 256 exit 1 How about this? wstat: 256 exit: 1 YAML, YAML, do! Doesn't look like TAP though :) -- Andy Armstrong, hexten.net
Re: You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
On 8 Mar 2007, at 21:55, Michael G Schwern wrote: Michael G Schwern wrote: Stop. Stop stop stop! Stop right there. Umm, people might not realize that I tend to be a bit over dramatic and didn't actually mean to shut down the discussion. "He's not the Messiah. He's a very naughty boy". Oops, wrong language. Do we agree that the ability to capture STDERR (ordered correctly wrt STDOUT) and test script exit status are desirable? I propose that we prefix lines from STDERR with '! ' in the same way that '# ' is used for diagnostics. wstat and exit can just be wstat 256 exit 1 We also spoke about a hierarchical test numbering scheme to allow groups of tests to be nested. Test::Block and Test::Group already give a taste of how that might work. I'm +1 on that. Also tags. I haven't thought through how they might be used. Anyone got any ideas about that? -- Andy Armstrong, hexten.net
Re: You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
Michael G Schwern wrote: > Stop. Stop stop stop! Stop right there. Umm, people might not realize that I tend to be a bit over dramatic and didn't actually mean to shut down the discussion.
Re: You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
On Mar 7, 2007, at 6:40 PM, Michael G Schwern wrote: Repeat after me, "We cannot predict how TAP will be used". That's why it's the "Test Anything Protocol". The A is Anything! -- Andy Lester => [EMAIL PROTECTED] => www.petdance.com => AIM:petdance
