Re: You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
chromatic wrote: > On Friday 09 March 2007 14:50, Michael G Schwern wrote: > >> We can leverage any existing status system we want. HTTP status. Exit >> status. Throw them all in! Don't find TAP's existing statuses rich >> enough? Add your own extension keys! A particular status code not make >> sense for your application? Leave it out! > > Frequently Unasked Question: > > But Schwern, if we run all of our tests through Test::Harness, not only > will > it not report that extra diagnostic information but it doesn't run non-Perl > tests very easily! Therefore we can't use TAP! > > Frequently Unanswered Answer: > > You can *run* the tests and *analyze* the results in separate steps. > That's > why they're separate steps. > > I think people get hung up on the idea that there is or has to be only one > way > to run-and-analyze tests in one step. That's great, put it on the wiki please.
Re: You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
On Friday 09 March 2007 14:50, Michael G Schwern wrote: > We can leverage any existing status system we want. HTTP status. Exit > status. Throw them all in! Don't find TAP's existing statuses rich > enough? Add your own extension keys! A particular status code not make > sense for your application? Leave it out! Frequently Unasked Question: But Schwern, if we run all of our tests through Test::Harness, not only will it not report that extra diagnostic information but it doesn't run non-Perl tests very easily! Therefore we can't use TAP! Frequently Unanswered Answer: You can *run* the tests and *analyze* the results in separate steps. That's why they're separate steps. I think people get hung up on the idea that there is or has to be only one way to run-and-analyze tests in one step. -- c
Re: You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
Adam Kennedy wrote: >> I propose that we prefix lines from STDERR with '! ' in the same way >> that '# ' is used for diagnostics. wstat and exit can just be >> >> wstat 256 >> exit 1 > > The problem with STDERR and exit is that we can't actually use them for > anything significant. > > Otherwise when dealing with TAP streams that don't have a concept of an > exit code or a seperate error channel, the most common example being web > testing, we're left high and dry. > > That's always been the main reason to intentionally ignore the exit code > at least. Am I TALKING TO MYSELF?! We do *not* have to decide on a fixed set of diagnostics you attach at the end of a test! We can allow the wait status, the exit status, the HTTP status... whatever you like. Put them in as you need. http://www.nntp.perl.org/group/perl.qa/2007/03/msg8155.html Extensibility is the key. This is why I like a YAML or JSON format so much. We can always add to it. http://perl-qa.yi.org/index.php/TAP_diagnostic_syntax 1..2 ok 1 ok 2 done (or whatever we wind up using for "end of tests") --- exit status: 23 wait status: 0 http status: 404 human readable status: Yer screwed. psychological status: I'm feeling fine, thanks ... We can leverage any existing status system we want. HTTP status. Exit status. Throw them all in! Don't find TAP's existing statuses rich enough? Add your own extension keys! A particular status code not make sense for your application? Leave it out!
Re: You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
Andy Armstrong wrote: > On 8 Mar 2007, at 22:47, Eric Hacker wrote: >>> I propose that we prefix lines from STDERR with '! ' in the same way >>> that '# ' is used for diagnostics. wstat and exit can just be >>> >>> wstat 256 >>> exit 1 >> >> How about this? >> >> wstat: 256 >> exit: 1 >> >> YAML, YAML, do! > > Doesn't look like TAP though :) I'm starting to get the feeling that every time I mention this: http://perl-qa.yi.org/index.php/TAP_diagnostic_syntax everyone sort of stares off into space for a few seconds with their mouths gaping open and then blinks twice and continues on with whatever they were doing without ever having seen the URL. It never gets any comment. Why is that? Is it the fnords? Are there fnords in that URL? Its ok, they won't eat you. I'll protect you. Everyone keeps reinventing the same wheel and coming around to the same conclusion. We have a wiki to avoid exactly this sort of thing. Either comment on what's wrong with it or code that proposal! If I see another "let's hack up the Test::More diagnostics" thread I'm going to puke.
Re: You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
On 3/8/07, Adam Kennedy <[EMAIL PROTECTED]> wrote: Andy Armstrong wrote: >> Otherwise when dealing with TAP streams that don't have a concept of >> an exit code or a seperate error channel, the most common example >> being web testing, we're left high and dry. > > In which case you'd just omit them, no? I didn't imagine they'd be > mandatory. Web testing doesn't have an error channel? HTTP has lot's of channels. They are called Status Codes. :) There are a whole slew of them for errors. Way more than you'd ever need for testing, or so I'm told. Peace, Hacker
Re: You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
On 9 Mar 2007, at 01:44, Adam Kennedy wrote: Like I said, we can't use them for anything that matters :) It's just occurred to me that it makes it easier to write tests for the harness too :) I can't quite get my head round the objection to the idea that a TAP transcript could be a complete, machine readable record of a test run. We're very nearly there anyway. -- Andy Armstrong, hexten.net
Re: You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
On 3/8/07, Andy Armstrong <[EMAIL PROTECTED]> wrote: I propose that we prefix lines from STDERR with '! ' in the same way that '# ' is used for diagnostics. wstat and exit can just be wstat 256 exit 1 How about this? wstat: 256 exit: 1 YAML, YAML, do! ;)
Re: You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
On 9 Mar 2007, at 00:28, Adam Kennedy wrote: I propose that we prefix lines from STDERR with '! ' in the same way that '# ' is used for diagnostics. wstat and exit can just be wstat 256 exit 1 The problem with STDERR and exit is that we can't actually use them for anything significant. We currently have the option of spooling parse TAP to a file during testing for later consumption. If we store STDERR + exit in that TAP it becomes possible to recreate the test run pretty accurately. Otherwise when dealing with TAP streams that don't have a concept of an exit code or a seperate error channel, the most common example being web testing, we're left high and dry. In which case you'd just omit them, no? I didn't imagine they'd be mandatory. -- Andy Armstrong, hexten.net
Re: You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
I propose that we prefix lines from STDERR with '! ' in the same way that '# ' is used for diagnostics. wstat and exit can just be wstat 256 exit 1 The problem with STDERR and exit is that we can't actually use them for anything significant. Otherwise when dealing with TAP streams that don't have a concept of an exit code or a seperate error channel, the most common example being web testing, we're left high and dry. That's always been the main reason to intentionally ignore the exit code at least. Adam K
Re: You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
On 8 Mar 2007, at 22:47, Eric Hacker wrote: I propose that we prefix lines from STDERR with '! ' in the same way that '# ' is used for diagnostics. wstat and exit can just be wstat 256 exit 1 How about this? wstat: 256 exit: 1 YAML, YAML, do! Doesn't look like TAP though :) -- Andy Armstrong, hexten.net
Re: You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
On 8 Mar 2007, at 21:55, Michael G Schwern wrote: Michael G Schwern wrote: Stop. Stop stop stop! Stop right there. Umm, people might not realize that I tend to be a bit over dramatic and didn't actually mean to shut down the discussion. "He's not the Messiah. He's a very naughty boy". Oops, wrong language. Do we agree that the ability to capture STDERR (ordered correctly wrt STDOUT) and test script exit status are desirable? I propose that we prefix lines from STDERR with '! ' in the same way that '# ' is used for diagnostics. wstat and exit can just be wstat 256 exit 1 We also spoke about a hierarchical test numbering scheme to allow groups of tests to be nested. Test::Block and Test::Group already give a taste of how that might work. I'm +1 on that. Also tags. I haven't thought through how they might be used. Anyone got any ideas about that? -- Andy Armstrong, hexten.net
Re: You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
Michael G Schwern wrote: > Stop. Stop stop stop! Stop right there. Umm, people might not realize that I tend to be a bit over dramatic and didn't actually mean to shut down the discussion.
Re: Should TAP capture exit codes
Reading through this I can't help but thing I've seen it all before. The PITA test result code conversation, if you'll recall. But when that cloned something akin to HTTP codes it was because there looked to be only 15-20 total possible results. And it's not meant to be extensible the way TAP is. And even then I'm still looking at moving to string codes. Adam K Andy Armstrong wrote: Ovid's post about TAP::Tests has reminded me: would it be useful to have a TAP statement that conveys the exit code of a test script? At the moment in a hypothetical situation where there's some distance between the harness and the test script - like perhaps they're on different machines - the test script's exit status has to be sent out-of-band. If you add exit to the grammar then the TAP transcript becomes a complete log of the test. At the moment in TAP::Harness we have the option of spooling TAP to a file - which captures everything useful /apart/ from exit status. --Andy Armstrong, hexten.net
Re: You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
On Mar 7, 2007, at 6:40 PM, Michael G Schwern wrote: Repeat after me, "We cannot predict how TAP will be used". That's why it's the "Test Anything Protocol". The A is Anything! -- Andy Lester => [EMAIL PROTECTED] => www.petdance.com => AIM:petdance
You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
Eric Hacker wrote: I think this is a situation where you want a code rather than just using text. The conciseness eases automatic interpretation and assures clarity of what was said. If so, then how many codes? Probably not as many as hundreds. Best to group codes so that similar codes are easily identified. Best not to do this using some cool binary or hex code which pushes the calculation to people rather than the big calculator/space heater. So lets say that the codes will be 3 digits, so we'll have enough room to grow for unforseen needs. Stop. Stop stop stop! Stop right there. Repeat after me, "We cannot predict how TAP will be used". We can't. Anything we design now will seem silly and narrow 5, 10, 15 years from now. TAP, after all, is 20 years old. 20 years ago would be have considered HTTP status codes? No, they didn't exist. Would we even have cared about other operating systems? No, it would be Unix exit codes... just like it is now. If we reduce exit status down to a series of hard coded 3 digit codes we're erecting an iron crib around the cradle. So limiting. (999 status codes|64K) will be enough for anyone, right? Extensibility is the key. This is why I like a YAML or JSON format so much. We can always add to it. http://perl-qa.yi.org/index.php/TAP_diagnostic_syntax 1..2 ok 1 ok 2 done (or whatever we wind up using for "end of tests") --- exit status: 23 wait status: 0 http status: 404 human readable status: Yer screwed. psychological status: I'm feeling fine, thanks ... We can leverage any existing status system we want. HTTP status. Exit status. Throw them all in! Don't find TAP's existing statuses rich enough? Add your own extension keys! A particular status code not make sense for your application? Leave it out! And uses the same mechanism as the TAP diagnostics proposal so little extra work on the part of the TAP parser is necessary. And we don't have to plan everything out now and be so very, very, very wrong.
Re: Should TAP capture exit codes
Gary Hawkins wrote: 2. Option to inject a clearcut delimiter between tests Distinct from, say, outputting a diagnostic between groups of tests? By 'tests' I'm thinking 'file' with its subtests, so-to-speak, so yes, anything that is clearly delineatable (LOL) programmatically where one 'file' output stops and another starts is fine. Different in what way from the current behaviour of Test::Harness? Each test script gets its own line in the output currently. Not if verbose output. Surely you jest. It is not reliably parsable, there can be garbage text on the end of those summary lines at times. There's already a proposal to handle per-test diagnostics. http://perl-qa.yi.org/index.php/TAP_diagnostic_syntax
Re: Should TAP capture exit codes
demerphq wrote:
If you want to say "Temporary Redirect" don't say "307" say "Temporary
Redirect"! If you want to put lots of information into one value, like
categorization, use a hash! { type => "Redirect", permanent => 0 }
Numeric response codes have the advantage that they are language agnostic.
So are strings. Wow, I'm having deja vu.
I should have been more clear, you wouldn't use a literal hash. You'd use
YAML or JSON or something.
error:
type: Redirect
permanent: 0
Re: Should TAP capture exit codes
On Wednesday 07 March 2007 11:39, Eric Hacker wrote: > Now if all TAP can handle is Perl, or even just code output, then > those of us pushing the envelope will eventually migrate to some other > test output format. TAP can certainly handle more than just Perl and code output. In my mind, the questions are: 1) Do existing TAP features already provide this feature? 2) Do existing TAP features almost already provide this feature, with a little bit of work? 3) Is this a feature that TAP should support somehow? 4) Is this generalizable to different domains? 5) Is there a clean way to add this to TAP while retaining backwards compatibility? I'm still not sure what you're doing that requires anything other than a machine-parsable diagnostic, but I have a failure of imagination sometimes. -- c
Re: Should TAP capture exit codes
On 3/7/07, Andy Armstrong <[EMAIL PROTECTED]> wrote:
It sounds as if you're doing monitoring rather than testing though.
Although they're related the requirements are quite different.
Poor explaining on my part then. Monitoring has similar needs, but us
usually much more shallow. Consider a web app test (simplified):
# Set tail before running test
$database->ok("tail test add $data");
$client->ok("submit $data to url http://www.example.com/form.cgi";);
Monitoring would check that the http server is up, and check that the
database server is up, but this is functional testing. Does the
application work, end to end.
Now stick some new web app security thingy in from of the web server,
and a database security tool in between the web app server and the
database. We need to run a regression test on the changes, but how.
This is not monitoring. This is scripted testing.
You
need real time capture of as much information as you can get; the
tests we're talking about aren't time dependent and it's quite
acceptable to run any test multiple times until the source of the
problem is found. The ability to capture more nuanced information
about the outcome of a particular test would burden the tests with
producing that information.
That's what Test::More does. Otherwise, everything would just be Ok.
In summary I think you're saying "don't just tell me it failed, tell
me /why/ it failed dammit" and we're saying pretty much the opposite:
"just tell me if it worked or not and leave it to me to find out why".
is_deeply and cmp_ok start to say why.
Now why is a matter of degree, and if a test could tell that the
developer meant && instead of || or that the new HTTP firewall was
dropping all POSTS with "hacker" in the data, then we'd be out of
work. I thought the spirit was to present as much info as practical.
> Functional testing is years behind, especially testing distributed
> applications like network security. I'm trying to leverage what I can
> to get something useful working. Perl and Test::Harness seem to be
> good tools to try.
I'd say not actually. Perl's certainly an excellent tool for your
application but I don't think Perl's testing framework is a good
match for what you're doing. Use the testing framework to prove that
your code works of course but don't try to make it do something it
wasn't intended to do and then blame it when it fights back :)
I really don't care too much, because I am already making what's there
work for me now. But if you are asking what to improve... :)
I took a look at http://www.testobsessed.com the other day. Check out
the functional test tools next generation dream. That makes status
codes look like Lego Duplos.
Re: Should TAP capture exit codes
On 3/7/07, Michael G Schwern <[EMAIL PROTECTED]> wrote:
Any time you start writing a system that involves representing states as
numbers and doing bitmasks and math to add extra meaning, step back and
remind yourself that its 2007 and this is not C and you're not writing a
network protocol. You shouldn't have to memorize a table or do math in your
head to figure out the basics of what a message means.
I agree, except we are writing a network protocol. Once the test
results are being shipped around to different systems, a network
protocol is in the works.
And god forbid we had more than 100 failure types!
If you want to say "Temporary Redirect" don't say "307" say "Temporary
Redirect"!
How about "temporär adressieren Sie um" or "provisoire réorientez" or
"de tijdelijke werkkracht richt opnieuw". :)
A decent coding system simplifies the translation between machines and
humans and human languages. It is a means of data sanitizing. It
establishes a scheme of the important details within an area that are
agreed upon by the participants. This provides guidance to new
participants, who can see from the coding system what is relevant.
If an existing coding system that many of the user population is
already familiar with can be leveraged, all the better.
If you want to put lots of information into one value, like
categorization, use a hash! { type => "Redirect", permanent => 0 }
Ok. YAML or XML for serialization? :)
Re: Should TAP capture exit codes
How gross would it be just to have a logical channel in TAP that could represent output to STDERR? That plus the exit status of the test script is pretty much all you have at the moment. Would that be so bad? Perhaps, for non-Unix testing, perhaps not. Here is what I have. There are bots that are running on various systems. One asks the bot to do something, and it responds. My bots are being set up to do testing things. Can you ping the server, can you open a TCP connection to port 80, can you get a URL over HTTP, can you run this script, etc. The idea is that without a traditional debugger, there should be a command line user interface that does these things as well as an RPC method for automated tests. When I send an RPC response, it has the human readable reply and the response code. Sure, I could have written a regex to figure out if the response is correct of failed, and I allow that. It is easier to process the codes though, and that suffices for most tests. Sure all of this could have been done in SSH, if everyone had account on and the machines and etc. There are other ways, this is what I'm trying to do and I may fail. Then you all can learn from my 409, 416 or 417. Now, I know you are thinking about exit status on test scripts and I'm thinking individual tests (of which running another test script might be an instance), but in the distributed functional testing space, one really can't rely on independent test scripts to do much. I need to set up a tail with regex on a log file on one bot, hit it from another bot, and know that both were successful real time. Then keep going. Now if all TAP can handle is Perl, or even just code output, then those of us pushing the envelope will eventually migrate to some other test output format. Functional testing is years behind, especially testing distributed applications like network security. I'm trying to leverage what I can to get something useful working. Perl and Test::Harness seem to be good tools to try.
Re: Should TAP capture exit codes
On 7 Mar 2007, at 21:24, Gary Hawkins wrote: Not if verbose output. Surely you jest. It is not reliably parsable, there can be garbage text on the end of those summary lines at times. [snip] If you want /parsed/ test results you'll be able to use TAPx::Parser and/or TAPx::Harness to get at them directly. No need to parse the output from the harness. -- Andy Armstrong, hexten.net
Re: Should TAP capture exit codes
On 7 Mar 2007, at 21:15, Eric Hacker wrote: Monitoring would check that the http server is up, and check that the database server is up, but this is functional testing. Does the application work, end to end. OK, /now/ I understand. That's what Test::More does. Otherwise, everything would just be Ok. Sure - but that's human readable information. As far as the harness is concerned it's succeed or fail plus a bunch of text it passes through verbatim. In summary I think you're saying "don't just tell me it failed, tell me /why/ it failed dammit" and we're saying pretty much the opposite: "just tell me if it worked or not and leave it to me to find out why". is_deeply and cmp_ok start to say why. Now why is a matter of degree, and if a test could tell that the developer meant && instead of || or that the new HTTP firewall was dropping all POSTS with "hacker" in the data, then we'd be out of work. I thought the spirit was to present as much info as practical. Sure, absolutely. Correct me if I'm wrong (again) but you were advocating returning an HTTP-like status code per individual test, yes? Given that there innumerable ways for a particular test to fail I can't quite see how you can distil that down to a status code. I really don't care too much, because I am already making what's there work for me now. But if you are asking what to improve... :) Thanks for taking the time. I appreciate that even if I can't actually work out what you're advocating :) I took a look at http://www.testobsessed.com the other day. Check out the functional test tools next generation dream. That makes status codes look like Lego Duplos. I'm clearly having a stupid day. Is this what I'm supposed to be looking at: What Problem Would Next-Generation Functional Testing Tools Solve? http://lyxus.net/gdc I can't see anything there that's much of a reach for us. -- Andy Armstrong, hexten.net
RE: Should TAP capture exit codes
2. Option to inject a clearcut delimiter between tests >>> >>> Distinct from, say, outputting a diagnostic between groups of tests? >> >> By 'tests' I'm thinking 'file' with its subtests, so-to-speak, so yes, >> anything that is clearly delineatable (LOL) programmatically where one >> 'file' output stops and another starts is fine. > >Different in what way from the current behaviour of Test::Harness? >Each test script gets its own line in the output currently. Not if verbose output. Surely you jest. It is not reliably parsable, there can be garbage text on the end of those summary lines at times. If the harness was run from one level above a 't' directory, you'll have t/ at the beginning of the line, but what if not? No way to predict the relative path someone will use, how many '.', or whether the last character on the line will be a digit, ok, NOK, dubious, or whatever else, including just a dot. Add to that failed or died diagnostics. Parse this to find the results per test to write them to db. (This does not take verbose into account where the challenge is more desperate). somegreattestfoobars_exceptions..ok xwhateverr_exceptionsok 35/71 # Failed test (xwhateverr_exceptions.t at line 1011) xwhateverr_exceptionsok 71/71# Looks like you failed 1 test of 71. xwhateverr_exceptionsdubious Test returned status 1 (wstat 256, 0x100) DIED. FAILED test 36 Failed 1/71 tests, 98.59% okay foobar_limit.ok 3/16 # Failed test (foobar_limit.t at line 321) foobar_limit.ok 16/16# Looks like you failed 1 test of 16. foobar_limit.dubious Test returned status 1 (wstat 256, 0x100) DIED. FAILED test 4 Failed 1/16 tests, 93.75% okay It would be healthy, smart, and advantagous for the good of TAP to add an option for delimiter. _..._ ~_~ _..._ <- kilroy? somegreattestfoobars_exceptions..ok _..._ ~_~ _..._ xwhateverr_exceptionsok 35/71 # Failed test (xwhateverr_exceptions.t at line 1011) xwhateverr_exceptionsok 71/71# Looks like you failed 1 test of 71. xwhateverr_exceptionsdubious Test returned status 1 (wstat 256, 0x100) DIED. FAILED test 36 Failed 1/71 tests, 98.59% okay _..._ ~_~ _..._ foobar_limit.ok 3/16 # Failed test (foobar_limit.t at line 321) foobar_limit.ok 16/16# Looks like you failed 1 test of 16. foobar_limit.dubious Test returned status 1 (wstat 256, 0x100) DIED. FAILED test 4 Failed 1/16 tests, 93.75% okay _..._ ~_~ _..._ That's all I have to say about that. -Forest Gump The information contained in this message may be privileged, confidential, and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the message and deleting it from your computer.
Re: Should TAP capture exit codes
On 7 Mar 2007, at 20:35, Gary Hawkins wrote: 2. Option to inject a clearcut delimiter between tests Distinct from, say, outputting a diagnostic between groups of tests? By 'tests' I'm thinking 'file' with its subtests, so-to-speak, so yes, anything that is clearly delineatable (LOL) programmatically where one 'file' output stops and another starts is fine. Different in what way from the current behaviour of Test::Harness? Each test script gets its own line in the output currently. -- Andy Armstrong, hexten.net
RE: Should TAP capture exit codes
> > 2. Option to inject a clearcut delimiter between tests > Distinct from, say, outputting a diagnostic between groups of tests? By 'tests' I'm thinking 'file' with its subtests, so-to-speak, so yes, anything that is clearly delineatable (LOL) programmatically where one 'file' output stops and another starts is fine. Thanks, Gary The information contained in this message may be privileged, confidential, and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the message and deleting it from your computer.
Re: Should TAP capture exit codes
On 7 Mar 2007, at 19:50, Gary Hawkins wrote:
1. Access to both STDERR and STDOUT, in proper order ('prove'
jumbles them), capturable into a variable (T::B snatches away STDERR)
TAP's a line oriented protocol so I imagine the best we can do is to
keep /lines/ from STDERR and STDOUT in the right order. I imagine we
can just dump lines of STDERR prefixed by '!' or whatever into the
TAP output - much the same way we handle diagnostic output currently.
2. Option to inject a clearcut delimiter between tests
Distinct from, say, outputting a diagnostic between groups of tests?
--
Andy Armstrong, hexten.net
Re: Should TAP capture exit codes
On 3/7/07, Andy Armstrong <[EMAIL PROTECTED]> wrote: On 7 Mar 2007, at 13:48, Eric Hacker wrote: > I think it was Ovid who recently called it the Test Anything Protocol. > If really what is desired, then some additional complexity is > required. Sure - I'm completely in favour of being able to test anything and capture everything that might be useful. But if we want more nuanced exit information (did the test even run? did a connection fail?) we could still capture that without having to massage everything into HTTP style status codes. Not massage, leverage. :) I think this is a situation where you want a code rather than just using text. The conciseness eases automatic interpretation and assures clarity of what was said. If so, then how many codes? Probably not as many as hundreds. Best to group codes so that similar codes are easily identified. Best not to do this using some cool binary or hex code which pushes the calculation to people rather than the big calculator/space heater. So lets say that the codes will be 3 digits, so we'll have enough room to grow for unforseen needs. The first digit can be a grouping by success/failure. They'll be another status group for situations where the test didn't run at all, because of an operation issue. Might want a status group for keepalive info and whatnot on long tests. Sure the Harness might not ever get them, but why have some entirely different way of communication this type of status. I'm not /strongly/ opposed to your proposal - just trying to understand what it might do that we can't do more simply by other means. So then if I'm not too far off base with the above, then to use something different than HTTP::Status type codes would be reinventing. 1xx Info 2xx Success 3xx Redirect, probably not applicable to testing 4xx Failure 5xx Server/System Error Then, if I'm writing a test engine with an http interface for running tests, little or no translation is necessary. :)
Re: Should TAP capture exit codes
On 7 Mar 2007, at 19:39, Eric Hacker wrote: Now, I know you are thinking about exit status on test scripts and I'm thinking individual tests (of which running another test script might be an instance), but in the distributed functional testing space, one really can't rely on independent test scripts to do much. OK, I see where you're going. We may be talking at cross purposes slightly. TAP already has syntax to capture the status of an individual test. As far as TAP is concerned tests are binary - they succeed or fail. There may be additional diagnostic information that'll help someone to work out why they failed but in terms of the test harness' view of what happened they either pass or fail. Now if all TAP can handle is Perl, or even just code output, then those of us pushing the envelope will eventually migrate to some other test output format. TAP is a simple protocol which has implementations in C, Python, Javascript, PHP. It sounds as if you're doing monitoring rather than testing though. Although they're related the requirements are quite different. You need real time capture of as much information as you can get; the tests we're talking about aren't time dependent and it's quite acceptable to run any test multiple times until the source of the problem is found. The ability to capture more nuanced information about the outcome of a particular test would burden the tests with producing that information. In summary I think you're saying "don't just tell me it failed, tell me /why/ it failed dammit" and we're saying pretty much the opposite: "just tell me if it worked or not and leave it to me to find out why". Functional testing is years behind, especially testing distributed applications like network security. I'm trying to leverage what I can to get something useful working. Perl and Test::Harness seem to be good tools to try. I'd say not actually. Perl's certainly an excellent tool for your application but I don't think Perl's testing framework is a good match for what you're doing. Use the testing framework to prove that your code works of course but don't try to make it do something it wasn't intended to do and then blame it when it fights back :) -- Andy Armstrong, hexten.net
RE: Should TAP capture exit codes
> OK, well we can talk about that now and at least get an idea of what
> kind of future we're proofing ourselves against. What do people
> envisage that we might want / be able to capture about a test run?
1. Access to both STDERR and STDOUT, in proper order ('prove' jumbles them),
capturable into a variable (T::B snatches away STDERR)
2. Option to inject a clearcut delimiter between tests
i.e. A program for programmers--instead of human-consumption output to
screen--offers output back to the programmer.
Gary
The information contained in this message may be privileged, confidential, and
protected from disclosure. If the reader of this message is not the intended
recipient, you are hereby notified that any dissemination, distribution, or
copying of this communication is strictly prohibited. If you have received this
communication in error, please notify us immediately by replying to the message
and deleting it from your computer.
Re: Should TAP capture exit codes
On 7 Mar 2007, at 19:21, demerphq wrote: I guess it comes down to whether you can anticipate the possibility that you will need new codes, and having a framework to put them into. OK, well we can talk about that now and at least get an idea of what kind of future we're proofing ourselves against. What do people envisage that we might want / be able to capture about a test run? -- Andy Armstrong, hexten.net
Re: Should TAP capture exit codes
On 3/7/07, Andy Armstrong <[EMAIL PROTECTED]> wrote: On 7 Mar 2007, at 18:59, demerphq wrote: > Neither to me to be a very convincing reason to redesign something as > well thought out as the HTTP response code schema. With it you have a > well documented, well designed language agnostic response structure. > It seems to me youd have to work hard to come up with something > better. We already have an interface that's even closer to home: emit an error message and return non-zero status. As I say I'm not thoroughly opposed to using HTTP like responses; just not convinced that even that level of complexity is necessary. I guess it comes down to whether you can anticipate the possibility that you will need new codes, and having a framework to put them into. The nice thing about the HTTP scheme is it gives you a way to add new codes that can be interpreted more or less correctly by something that wasnt designed with those codes in mind. (For instance while a particular client might not know what 569 is exactly, they should know its a server error.) If you do go so far as to design a new protocol for this please take a minute to read the RFC's for the HTTP response code schema and the SMTP response schemas. The SMTP schema goes a bit further than the HTTP one, in that it also defines special meaning for the second digit as well as for the first. Yves -- perl -Mre=debug -e "/just|another|perl|hacker/"
Re: Should TAP capture exit codes
On 7 Mar 2007, at 18:59, demerphq wrote: Neither to me to be a very convincing reason to redesign something as well thought out as the HTTP response code schema. With it you have a well documented, well designed language agnostic response structure. It seems to me youd have to work hard to come up with something better. We already have an interface that's even closer to home: emit an error message and return non-zero status. As I say I'm not thoroughly opposed to using HTTP like responses; just not convinced that even that level of complexity is necessary. -- Andy Armstrong, hexten.net
Re: Should TAP capture exit codes
On 3/7/07, Andy Armstrong <[EMAIL PROTECTED]> wrote:
On 7 Mar 2007, at 18:18, demerphq wrote:
>> If you want to say "Temporary Redirect" don't say "307" say
>> "Temporary
>> Redirect"! If you want to put lots of information into one value,
>> like
>> categorization, use a hash! { type => "Redirect", permanent => 0 }
>
> Numeric response codes have the advantage that they are language
> agnostic.
Indeed. That doesn't mean we have to coerce our status information to
map onto HTTP response codes though or even that we have to use a
numeric scheme.
Personally I see this is as a wheel-reinvention issue. Reusing the
HTTP response code seems to me to be a logical and natural step. As a
framework it strikes me that you will be unlikely to come up with
something truely better, so why not just reuse it and not worry about
it?
So far the objections we have are:
1) It uses numeric error codes and not english ones.
2) It wasnt custom designed for test responses.
Neither to me to be a very convincing reason to redesign something as
well thought out as the HTTP response code schema. With it you have a
well documented, well designed language agnostic response structure.
It seems to me youd have to work hard to come up with something
better.
Anyway, just my $0.02
cheers,
Yves
--
perl -Mre=debug -e "/just|another|perl|hacker/"
Re: Should TAP capture exit codes
On 7 Mar 2007, at 18:18, demerphq wrote:
If you want to say "Temporary Redirect" don't say "307" say
"Temporary
Redirect"! If you want to put lots of information into one value,
like
categorization, use a hash! { type => "Redirect", permanent => 0 }
Numeric response codes have the advantage that they are language
agnostic.
Indeed. That doesn't mean we have to coerce our status information to
map onto HTTP response codes though or even that we have to use a
numeric scheme.
So we have broad categories of "test script couldn't be run for some
reason" and "test script failed for some reason". In either case we /
could/ distil the actual source of the problem (t/foo.t not found,
connection to testbox.mydomain timed out) into an error code but then
we'd be discarding information that'd presumably be useful.
How gross would it be just to have a logical channel in TAP that
could represent output to STDERR? That plus the exit status of the
test script is pretty much all you have at the moment. Would that be
so bad?
--
Andy Armstrong, hexten.net
Re: Should TAP capture exit codes
On 3/7/07, Michael G Schwern <[EMAIL PROTECTED]> wrote:
Andy Armstrong wrote:
> On 7 Mar 2007, at 16:26, Eric Hacker wrote:
> [snip]
>> The first digit can be a grouping by success/failure.
>
> Yes, I see where you're going with this :)
>
>> So then if I'm not too far off base with the above, then to use
>> something different than HTTP::Status type codes would be reinventing.
>>
>> 1xx Info
>> 2xx Success
>> 3xx Redirect, probably not applicable to testing
>> 4xx Failure
>> 5xx Server/System Error
>
> As I say I'm broadly in favour of something /like/ this - but we have a
> clean slate here and it seems kind of arbitrary to commit to using
> HTTP-like status codes when we don't have to.
Any time you start writing a system that involves representing states as
numbers and doing bitmasks and math to add extra meaning, step back and
remind yourself that its 2007 and this is not C and you're not writing a
network protocol. You shouldn't have to memorize a table or do math in your
head to figure out the basics of what a message means.
And god forbid we had more than 100 failure types!
If you want to say "Temporary Redirect" don't say "307" say "Temporary
Redirect"! If you want to put lots of information into one value, like
categorization, use a hash! { type => "Redirect", permanent => 0 }
Numeric response codes have the advantage that they are language agnostic.
Yves
--
perl -Mre=debug -e "/just|another|perl|hacker/"
Re: Should TAP capture exit codes
Andy Armstrong wrote:
On 7 Mar 2007, at 16:26, Eric Hacker wrote:
[snip]
The first digit can be a grouping by success/failure.
Yes, I see where you're going with this :)
So then if I'm not too far off base with the above, then to use
something different than HTTP::Status type codes would be reinventing.
1xx Info
2xx Success
3xx Redirect, probably not applicable to testing
4xx Failure
5xx Server/System Error
As I say I'm broadly in favour of something /like/ this - but we have a
clean slate here and it seems kind of arbitrary to commit to using
HTTP-like status codes when we don't have to.
Any time you start writing a system that involves representing states as
numbers and doing bitmasks and math to add extra meaning, step back and
remind yourself that its 2007 and this is not C and you're not writing a
network protocol. You shouldn't have to memorize a table or do math in your
head to figure out the basics of what a message means.
And god forbid we had more than 100 failure types!
If you want to say "Temporary Redirect" don't say "307" say "Temporary
Redirect"! If you want to put lots of information into one value, like
categorization, use a hash! { type => "Redirect", permanent => 0 }
Re: Should TAP capture exit codes
On 7 Mar 2007, at 16:26, Eric Hacker wrote: [snip] The first digit can be a grouping by success/failure. Yes, I see where you're going with this :) So then if I'm not too far off base with the above, then to use something different than HTTP::Status type codes would be reinventing. 1xx Info 2xx Success 3xx Redirect, probably not applicable to testing 4xx Failure 5xx Server/System Error As I say I'm broadly in favour of something /like/ this - but we have a clean slate here and it seems kind of arbitrary to commit to using HTTP-like status codes when we don't have to. -- Andy Armstrong, hexten.net
Re: Should TAP capture exit codes
On 3/7/07, Andy Armstrong <[EMAIL PROTECTED]> wrote: On 7 Mar 2007, at 13:01, Eric Hacker wrote: > Exit code or Status code? Well let's generalise it and discuss the specifics: "any useful information that's available when the test script terminates" Ok > The RFC Status codes might not be a perfect fit for test status, but > like the SIP protocol, they can be reworked to something compatible > and generally recognizable. Everyone knows that a 404 is bad. That's an extra layer of complexity. What benefits does it bring? any useful information ... :) If the testing is distributed, who ordered the test? If the result collector ordered the test, then it would likely want to know the difference between the test failed from a network problem than from a code problem than from a permissions problem. Look at CPAN testers. The lack of an installed module vs all the prereqs are installed and it still won't work on this system is not distinguished. Granted, the module test suite should handle that better itself, so maybe this shouldn't come up, but it does. I think this extra layer of complexity is required for functional testing. It may not be for just testing Perl. I have way more experience testing and breaking functionality than developing. I'm sure if you ever get a look at my code, this will be clear. I'll leave it up to the pros to decide what is really necessary for Perl. I think it was Ovid who recently called it the Test Anything Protocol. If really what is desired, then some additional complexity is required. Meanwhile, I'll adapt HTTP::Status::is_success and is_error to formulate a correct test result from my test status codes, and if more is ever needed, I've already got it. Peace, Hacker
Re: Should TAP capture exit codes
On 3/7/07, Andy Armstrong <[EMAIL PROTECTED]> wrote: Ovid's post about TAP::Tests has reminded me: would it be useful to have a TAP statement that conveys the exit code of a test script? At the moment in a hypothetical situation where there's some distance between the harness and the test script - like perhaps they're on different machines - the test script's exit status has to be sent out- of-band. If you add exit to the grammar then the TAP transcript becomes a complete log of the test. Exit code or Status code? In a distributed environment one often wants a bit more than the information that an Exit code provides depending one what's driving the test. I'm not sure what might be best, but it would be worth considering something like HTTP::Status, RFC 2616 and RFC 2518. I'm working on a distributed functional testing system where the individual tests are distributed, not an entire script. Here it is necessary to know the difference between a Command not found vs a failure, a successful create test vs the later successful pass in order to provide useful diagnostics. The RFC Status codes might not be a perfect fit for test status, but like the SIP protocol, they can be reworked to something compatible and generally recognizable. Everyone knows that a 404 is bad. Regards, -- Eric Hacker, CISSP aptronym (AP-troh-NIM) noun A name that is especially suited to the profession of its owner I _can_ leave well enough alone, but my criteria for well enough is pretty darn high.
Re: Should TAP capture exit codes
On 7 Mar 2007, at 13:48, Eric Hacker wrote: I think it was Ovid who recently called it the Test Anything Protocol. If really what is desired, then some additional complexity is required. Sure - I'm completely in favour of being able to test anything and capture everything that might be useful. But if we want more nuanced exit information (did the test even run? did a connection fail?) we could still capture that without having to massage everything into HTTP style status codes. I'm not /strongly/ opposed to your proposal - just trying to understand what it might do that we can't do more simply by other means. -- Andy Armstrong, hexten.net
Re: Should TAP capture exit codes
On 7 Mar 2007, at 13:01, Eric Hacker wrote: Exit code or Status code? Well let's generalise it and discuss the specifics: "any useful information that's available when the test script terminates" The RFC Status codes might not be a perfect fit for test status, but like the SIP protocol, they can be reworked to something compatible and generally recognizable. Everyone knows that a 404 is bad. That's an extra layer of complexity. What benefits does it bring? -- Andy Armstrong, hexten.net
