Re: Custom extensions to META.yml
Ricardo SIGNES wrote: * brian d foy <[EMAIL PROTECTED]> [2007-03-04T12:09:26] I'm not talking about the particular field name, but the idea that I'd want to say in META.yml "Don't send me mail", or whatever setting I want. Instead of having to disable (or enable) CC for every new tool, I'd want a setting that new tools could look at without me having to change the META.yml in all of my distributions then re-uploading them all. So, for some subset of META.yml settings, you could consult the module's author settings, found at (say) $MIRROR/authors/.../RJBS/METAMETA.yml That would be on your local mirror, be it minicpan or CPAN-over-HTTP. Something like that? I feel a potentially irrational sense of dread. My gut agrees with your gut on that one... I have "The Doom Song" playing in the back of my mind... Adam K
Re: Custom extensions to META.yml
Graham Barr wrote: On Mar 5, 2007, at 1:56 PM, Eric Wilhelm wrote: * brian d foy <[EMAIL PROTECTED]> [2007-03-04T12:09:26] I'm not talking about the particular field name, but the idea that I'd want to say in META.yml "Don't send me mail", or whatever setting I want. Instead of having to disable (or enable) CC for every new tool, I'd want a setting that new tools could look at without me having to change the META.yml in all of my distributions then re-uploading them all. ... I'm just saying that setting configuration options per tool isn't the way to handle global preferences. Are you saying that you want a per-author META.yml or that you don't want to have to say "don't send me mail" in two places in each distribution, or both? What I think brian is saying, and I agree, is that he does not want to have to say "don't send me mail" N times. Where N is between 1 and, um I don't know. Haw many tools will there ever be that want to send mail back to the author and what will their names be ? So as well as having per-tool settings in META.yml there should be some global settings that all tools agree to use as a default. If we are going to do something like this, I wonder if it might be better to somehow integrate it into PAUSE... (although granted that wouldn't be particular useful for extentions-type functionality) Adam K
Re: TAPx::Parser -> TAP::Parser?
On 5 Mar 2007, at 22:30, Ovid wrote: (Resent from the address I've actually subscribed from!) Hi all, Per an email from Schwern, he does not object to renaming TAPx::Parser to TAP::Parser. Hence, we have an official 'blessing' from him for claiming this namespace. Does anyone have any thoughts/objections to this? I realize that enough people are using TAPx::Parser that it might be a tiny speed bump for some, but I can't see any major difficulties here. Any concerns/objections? Nope. Adrian
Re: a safer way to use no_plan?
Adam Kennedy wrote: Its O(1) vs O(n). We do the work once to make "no plan" safer, everybody benefits for happily ever after. Your argument sounds like the things I had to deal with when I introduced Test::More. "Why should we load a module with all these functions when I can just write 'print ...my test... ? "ok 1\n" : "not ok 1\n";'?" But the opposite argument of when I suggested new_ok :) No, I have no problem with new_ok(). It just doesn't belong in Test::More. Not every testing function have to be lumped into one module, write your own. I try to make it as easy as possible.
Paying for TAP 2.0
(Grr ... one day I'll remember to post from the correct email address) This is me, beating this camel again, hoping that money might revive it. This discussion of no_plan work and fixing other testing functions reminds me that creating a new testing module (TAP::Tests) would still be useful. We could incorporate our new thoughts on how to handle testing and not have to worry as much about backwards compatibility with testing modules, only with TAP itself. This would also be a nice development path for TAP 2.0. In fact, I wouldn't mind a grant proposal being submitted to the grant committee for such work and if it's a reasonable proposal (preferably with enough people on this list giving it a ++), I'd be likely to vote to approve funding for such work, though I can't speak for the rest of the grant committee. This would be the next logical step in our work to improve Perl's testing tools. Cheers, Ovid -- Buy the book -- http://www.oreilly.com/catalog/perlhks/ Perl and CGI -- http://users.easystreet.com/ovid/cgi_course/
Paying for TAP 2.0
This is me, beating this camel again, hoping that money might revive it. This discussion of no_plan work and fixing other testing functions reminds me that creating a new testing module (TAP::Tests) would still be useful. We could incorporate our new thoughts on how to handle testing and not have to worry as much about backwards compatibility with testing modules, only with TAP itself. This would also be a nice development path for TAP 2.0. In fact, I wouldn't mind a grant proposal being submitted to the grant committee for such work and if it's a reasonable proposal (preferably with enough people on this list giving it a ++), I'd be likely to vote to approve funding for such work, though I can't speak for the rest of the grant committee. This would be the next logical step in our work to improve Perl's testing tools. Cheers, Ovid -- Buy the book -- http://www.oreilly.com/catalog/perlhks/ Perl and CGI -- http://users.easystreet.com/ovid/cgi_course/
Re: Paying for TAP 2.0
On 7 Mar 2007, at 11:15, Ovid wrote: This discussion of no_plan work and fixing other testing functions reminds me that creating a new testing module (TAP::Tests) would still be useful. We could incorporate our new thoughts on how to handle testing and not have to worry as much about backwards compatibility with testing modules, only with TAP itself. This would also be a nice development path for TAP 2.0. Where does TAP::Tests fit in? D'you mean as an alternative to Test::More or whatever? -- Andy Armstrong, hexten.net
Should TAP capture exit codes
Ovid's post about TAP::Tests has reminded me: would it be useful to have a TAP statement that conveys the exit code of a test script? At the moment in a hypothetical situation where there's some distance between the harness and the test script - like perhaps they're on different machines - the test script's exit status has to be sent out- of-band. If you add exit to the grammar then the TAP transcript becomes a complete log of the test. At the moment in TAP::Harness we have the option of spooling TAP to a file - which captures everything useful /apart/ from exit status. -- Andy Armstrong, hexten.net
Re: Paying for TAP 2.0
--- Andy Armstrong <[EMAIL PROTECTED]> wrote:
> Where does TAP::Tests fit in? D'you mean as an alternative to
> Test::More or whatever?
Yes, an alternative to Test::More. Some thoughts:
* Output everything to STDOUT (thereby avoiding the buffering
problems that many experience)
* Eliminate legacy cruft like eq_array
* Improve how SKIP is handled:
if ( ! $alien->DOES('genocide') ) {
skip 1, "We have morals :(";
}
else {
ok $alien->kill_all_humans, 'Muhahahahah!';
}
* Incorporate a few more standard test functions (exception tests,
eq_or_diff, etc)
* Output TAP 2.0 (line numbers for failures, clear diagnostics, etc)
* Wash my dishes
Cheers,
Ovid
--
Buy the book -- http://www.oreilly.com/catalog/perlhks/
Perl and CGI -- http://users.easystreet.com/ovid/cgi_course/
Re: Paying for TAP 2.0
On 7 Mar 2007, at 11:56, Ovid wrote: * Wash my dishes OK :) And what about its relationship to all the other existing testing modules? Are they cut adrift? Does it replace Test::Builder? -- Andy Armstrong, hexten.net
Re: Paying for TAP 2.0
On 07/03/07, Ovid <[EMAIL PROTECTED]> wrote:
--- Andy Armstrong <[EMAIL PROTECTED]> wrote:
> Where does TAP::Tests fit in? D'you mean as an alternative to
> Test::More or whatever?
Yes, an alternative to Test::More. Some thoughts:
Where is it?
* Output everything to STDOUT (thereby avoiding the buffering
problems that many experience)
Sounds like a replacement for Test::Builder not just Test::More.
* Eliminate legacy cruft like eq_array
* Improve how SKIP is handled:
if ( ! $alien->DOES('genocide') ) {
skip 1, "We have morals :(";
}
else {
ok $alien->kill_all_humans, 'Muhahahahah!';
}
* Incorporate a few more standard test functions (exception tests,
eq_or_diff, etc)
* Output TAP 2.0 (line numbers for failures, clear diagnostics, etc)
* Wash my dishes
Here's what I want from TAP 2.0
* be able to associate diagnostics with a specifc test (currently they
appear after the test by convention)
* nested blocks, each with it's own plan (or no plan)
eg
1..3
1 OK
2 1..2 # start a block
2.1 1..3 # start another block
2.1.1 OK
2.1.2 OK
2.1.3 OK
2.1 OK # inner block was all good
2.2 OK
2 OK # outer block was all good
3 1..3
3.1 OK
3.2 OK
3 NOT OK # planned for 3 but only ran 2 tests
this makes it real easy to count tests even when you do loops and
functions, so it makes almost of all the reasons for using no_plan go
away and when it doesn't it allows you to localise your no_plan to a
specific block while continuing to count all your others.
* tags, I'm not hugely keen on this but being able to provide tags is
interesting. Imagine a huge test suite where you tag eveyr test it's
dependencies. Then something breaks and you have 1000 failures in all
kinds of places, if they all have a tag in common then you have a lead
on what broke. Kinda far fetched...
F
Cheers,
Ovid
--
Buy the book -- http://www.oreilly.com/catalog/perlhks/
Perl and CGI -- http://users.easystreet.com/ovid/cgi_course/
Re: Paying for TAP 2.0
On 7 Mar 2007, at 12:10, Fergal Daly wrote: * nested blocks, each with it's own plan (or no plan) eg 1..3 1 OK 2 1..2 # start a block 2.1 1..3 # start another block 2.1.1 OK 2.1.2 OK 2.1.3 OK 2.1 OK # inner block was all good 2.2 OK 2 OK # outer block was all good 3 1..3 3.1 OK 3.2 OK 3 NOT OK # planned for 3 but only ran 2 tests this makes it real easy to count tests even when you do loops and functions, so it makes almost of all the reasons for using no_plan go away and when it doesn't it allows you to localise your no_plan to a specific block while continuing to count all your others. I like that. I quite often rub up against inconvenience that would be solved by being able to have nested groups of tests in a single test script. -- Andy Armstrong, hexten.net
Re: Paying for TAP 2.0
--- Fergal Daly <[EMAIL PROTECTED]> wrote: > On 07/03/07, Ovid <[EMAIL PROTECTED]> wrote: > > --- Andy Armstrong <[EMAIL PROTECTED]> wrote: > > > > > Where does TAP::Tests fit in? D'you mean as an alternative to > > > Test::More or whatever? > > > > Yes, an alternative to Test::More. Some thoughts: > > Where is it? It's all in my head. You don't want to go there. > > * Output everything to STDOUT (thereby avoiding the buffering > > problems that many experience) > > Sounds like a replacement for Test::Builder not just Test::More. Well, I guess it would be a replacement for Test::Builder and various testing modules. It would be rather important to try and make it work with existing test modules, though. Not sure how workable that would be since we'd have to build in support for things not wanted. > Here's what I want from TAP 2.0 > > * be able to associate diagnostics with a specifc test (currently > they appear after the test by convention) Agreed. That's a major requirement of TAP 2.0. > * nested blocks, each with it's own plan (or no plan) I like this thought. > * tags, I think tags would be extremely useful Right now they can be done informally with the current TAP grammar and still work with existing tools, but built in support would be nice. Cheers, Ovid -- Buy the book -- http://www.oreilly.com/catalog/perlhks/ Perl and CGI -- http://users.easystreet.com/ovid/cgi_course/
Re: Paying for TAP 2.0
On 07/03/07, Andy Armstrong <[EMAIL PROTECTED]> wrote: On 7 Mar 2007, at 12:10, Fergal Daly wrote: > * nested blocks, each with it's own plan (or no plan) > eg > 1..3 > 1 OK > 2 1..2 # start a block > 2.1 1..3 # start another block > 2.1.1 OK > 2.1.2 OK > 2.1.3 OK > 2.1 OK # inner block was all good > 2.2 OK > 2 OK # outer block was all good > 3 1..3 > 3.1 OK > 3.2 OK > 3 NOT OK # planned for 3 but only ran 2 tests > > this makes it real easy to count tests even when you do loops and > functions, so it makes almost of all the reasons for using no_plan go > away and when it doesn't it allows you to localise your no_plan to a > specific block while continuing to count all your others. I like that. I quite often rub up against inconvenience that would be solved by being able to have nested groups of tests in a single test script. On Monday night I finally broke down and implemented nested blocks for Test::Builder (second time to do this, first time required a patch to TB, this time it's easier cos TB uses objects internally now). The module is broken in many ways but I'll post it to CPAN later on anyway because it'll probably be another week before I can work on it again, F -- Andy Armstrong, hexten.net
Re: Paying for TAP 2.0
Ovid wrote: Sounds like a replacement for Test::Builder not just Test::More. Well, I guess it would be a replacement for Test::Builder and various testing modules. It would be rather important to try and make it work with existing test modules, though. Not sure how workable that would be since we'd have to build in support for things not wanted. There's nothing I've seen here so far that seems incompatible with Test::Builder's API. It could be as simple as flipping a switch to change what version of TAP its outputting. Two different TAP builder backends can live together in the same process as long as certain things are coordinated. The most obvious things are coordinating the test counter and where output is going. I'm already planning on implementing this in Test::Builder: http://perl-qa.yi.org/index.php/TAP_diagnostic_syntax
Re: Paying for TAP 2.0
Ovid wrote: This would also be a nice development path for TAP 2.0. Minor nit... please please please use integer version numbers. Please. http://perl-qa.yi.org/index.php/TAP_version
Re: Paying for TAP 2.0
On 7 Mar 2007, at 12:38, Michael G Schwern wrote: Minor nit... please please please use integer version numbers. Please. http://perl-qa.yi.org/index.php/TAP_version Indeed. It's not like we're going to run out of positive integers. Schema versions should always be integers. -- Andy Armstrong, hexten.net
Re: Should TAP capture exit codes
On 7 Mar 2007, at 13:01, Eric Hacker wrote: Exit code or Status code? Well let's generalise it and discuss the specifics: "any useful information that's available when the test script terminates" The RFC Status codes might not be a perfect fit for test status, but like the SIP protocol, they can be reworked to something compatible and generally recognizable. Everyone knows that a 404 is bad. That's an extra layer of complexity. What benefits does it bring? -- Andy Armstrong, hexten.net
Re: TAPx::Parser -> TAP::Parser?
On 3/7/07, Adrian Howard <[EMAIL PROTECTED]> wrote: On 5 Mar 2007, at 22:30, Ovid wrote: > (Resent from the address I've actually subscribed from!) > > Hi all, > > Per an email from Schwern, he does not object to renaming TAPx::Parser > to TAP::Parser. Hence, we have an official 'blessing' from him for > claiming this namespace. Does anyone have any thoughts/objections to > this? I realize that enough people are using TAPx::Parser that it > might be a tiny speed bump for some, but I can't see any major > difficulties here. > > Any concerns/objections? Nope. Also fine by me, as Test::Run only uses TAPx::Parser on a small number of places. Regards, Shlomi Fish Adrian -- -- Shlomi Fish http://www.shlomifish.org/ If his programming is anything like his philosophising, he would find 10 imaginary bugs in the "Hello World" program.
Re: shameless Test::Group plug [was: Paying for TAP 2.0]
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 7 Mar 2007, at 13:04, Dominique Quatravaux wrote: I like that. I quite often rub up against inconvenience that would be solved by being able to have nested groups of tests in a single test script. Then you might be interested in checking out Test::Group. I like that but it'd be even better if grouping was understood by the harness too. - -- Andy Armstrong, hexten.net -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (Darwin) iD8DBQFF7ryIwoknRJZQnCERAmHOAKDXUGVLBkU5Y+xz1quYndkcOscgHACggZ0u qCn73uYZ5map0d/GvH3LE/4= =+Ii5 -END PGP SIGNATURE-
Re: another shameless Test::Group plug [was Re: Paying for TAP 2.0]
On 07/03/07, Dominique Quatravaux <[EMAIL PROTECTED]> wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Fergal Daly a écrit : > > On Monday night I finally broke down and implemented nested blocks > for Test::Builder [...] The module is broken in many ways but I'll > post it to CPAN later on anyway because it'll probably be another > week before I can work on it again, Please take the time to investigate Test::Group first? Interesting, I don't remember looking at this before. It's quite similar interface but it doesn't seem to include plans for the blocks and it looks like it doesn't handle groups within groups, is that correct? F Regards, Dom - -- << Tout n'y est pas parfait, mais on y honore certainement les jardiniers >> Dominique Quatravaux <[EMAIL PROTECTED]> -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iQCVAwUBRe64zfTYH7KfeIIFAQJUkAQAyOcYHNUnDiS1lZXcgfFK1IVT0mPJmfTq wAdXcEsqQRenkjaTAe0R3vQAbw607cyuJF52SQAYUdasfCYuSeBIfWORlarMu97X bgjONWeB3XYAuO/jKaZiN2cP5gy6+SLYTlugARpnOiOfeal1UYXWyducW2GRYavs Oxu3YaKUY3I= =WYJB -END PGP SIGNATURE-
another shameless Test::Group plug [was Re: Paying for TAP 2.0]
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Fergal Daly a écrit : > > On Monday night I finally broke down and implemented nested blocks > for Test::Builder [...] The module is broken in many ways but I'll > post it to CPAN later on anyway because it'll probably be another > week before I can work on it again, Please take the time to investigate Test::Group first? Regards, Dom - -- << Tout n'y est pas parfait, mais on y honore certainement les jardiniers >> Dominique Quatravaux <[EMAIL PROTECTED]> -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iQCVAwUBRe64zfTYH7KfeIIFAQJUkAQAyOcYHNUnDiS1lZXcgfFK1IVT0mPJmfTq wAdXcEsqQRenkjaTAe0R3vQAbw607cyuJF52SQAYUdasfCYuSeBIfWORlarMu97X bgjONWeB3XYAuO/jKaZiN2cP5gy6+SLYTlugARpnOiOfeal1UYXWyducW2GRYavs Oxu3YaKUY3I= =WYJB -END PGP SIGNATURE-
Re: Should TAP capture exit codes
On 7 Mar 2007, at 13:48, Eric Hacker wrote: I think it was Ovid who recently called it the Test Anything Protocol. If really what is desired, then some additional complexity is required. Sure - I'm completely in favour of being able to test anything and capture everything that might be useful. But if we want more nuanced exit information (did the test even run? did a connection fail?) we could still capture that without having to massage everything into HTTP style status codes. I'm not /strongly/ opposed to your proposal - just trying to understand what it might do that we can't do more simply by other means. -- Andy Armstrong, hexten.net
Re: Should TAP capture exit codes
On 3/7/07, Andy Armstrong <[EMAIL PROTECTED]> wrote: Ovid's post about TAP::Tests has reminded me: would it be useful to have a TAP statement that conveys the exit code of a test script? At the moment in a hypothetical situation where there's some distance between the harness and the test script - like perhaps they're on different machines - the test script's exit status has to be sent out- of-band. If you add exit to the grammar then the TAP transcript becomes a complete log of the test. Exit code or Status code? In a distributed environment one often wants a bit more than the information that an Exit code provides depending one what's driving the test. I'm not sure what might be best, but it would be worth considering something like HTTP::Status, RFC 2616 and RFC 2518. I'm working on a distributed functional testing system where the individual tests are distributed, not an entire script. Here it is necessary to know the difference between a Command not found vs a failure, a successful create test vs the later successful pass in order to provide useful diagnostics. The RFC Status codes might not be a perfect fit for test status, but like the SIP protocol, they can be reworked to something compatible and generally recognizable. Everyone knows that a 404 is bad. Regards, -- Eric Hacker, CISSP aptronym (AP-troh-NIM) noun A name that is especially suited to the profession of its owner I _can_ leave well enough alone, but my criteria for well enough is pretty darn high.
Re: shameless Test::Group plug [was: Paying for TAP 2.0]
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Andy Armstrong wrote: > > I like that. I quite often rub up against inconvenience that would > be solved by being able to have nested groups of tests in a single > test script. Then you might be interested in checking out Test::Group. Regards, Dom - -- << Tout n'y est pas parfait, mais on y honore certainement les jardiniers >> Dominique Quatravaux <[EMAIL PROTECTED]> -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iQCVAwUBRe64efTYH7KfeIIFAQIxvwP/XdsbKUcDD+jd19wQB1ZgSe2VE8y8p7hw hWcj8oWPto3Gan0rOoQvxQaFDgxfrGwA3Pltu0gQCt6A4dByh10NL9ti3llS+y+X fn6F1emwR7NSz99JYtvAhtaXhnOVPK1DvXnzITbMKaditMYPvl7OHFGrUBbxzPHd CQ3hRks6ykE= =TlP+ -END PGP SIGNATURE-
Re: another shameless Test::Group plug [was Re: Paying for TAP 2.0]
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Fergal Daly wrote: > it doesn't seem to include plans for the blocks Indeed (but Test::Class does). Patches welcome. > and it looks like it doesn't handle groups within groups, It does: grep for "nest" in t/* Best regards, - -- << Tout n'y est pas parfait, mais on y honore certainement les jardiniers >> Dominique Quatravaux <[EMAIL PROTECTED]> -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iQCVAwUBRe7JN/TYH7KfeIIFAQKQLwP/RWeAbeTrtjGB1rWte9hwXUyk5g9307Ns EIjjQzhOD8xllmJH94tVzkOMsUs0S1MLqIvliqYCZWQDNktszs9pi9R8VZSZ/ZW7 gfh1Rym7C0thjLvoXq/RbUBpC//3+Lh+d2flk5hJO/StVm3k2sRQEmuHq+uxlaLg G3kUlzLCMRE= =/lef -END PGP SIGNATURE-
Re: Should TAP capture exit codes
On 3/7/07, Andy Armstrong <[EMAIL PROTECTED]> wrote: On 7 Mar 2007, at 13:01, Eric Hacker wrote: > Exit code or Status code? Well let's generalise it and discuss the specifics: "any useful information that's available when the test script terminates" Ok > The RFC Status codes might not be a perfect fit for test status, but > like the SIP protocol, they can be reworked to something compatible > and generally recognizable. Everyone knows that a 404 is bad. That's an extra layer of complexity. What benefits does it bring? any useful information ... :) If the testing is distributed, who ordered the test? If the result collector ordered the test, then it would likely want to know the difference between the test failed from a network problem than from a code problem than from a permissions problem. Look at CPAN testers. The lack of an installed module vs all the prereqs are installed and it still won't work on this system is not distinguished. Granted, the module test suite should handle that better itself, so maybe this shouldn't come up, but it does. I think this extra layer of complexity is required for functional testing. It may not be for just testing Perl. I have way more experience testing and breaking functionality than developing. I'm sure if you ever get a look at my code, this will be clear. I'll leave it up to the pros to decide what is really necessary for Perl. I think it was Ovid who recently called it the Test Anything Protocol. If really what is desired, then some additional complexity is required. Meanwhile, I'll adapt HTTP::Status::is_success and is_error to formulate a correct test result from my test status codes, and if more is ever needed, I've already got it. Peace, Hacker
Re: Should TAP capture exit codes
On 7 Mar 2007, at 16:26, Eric Hacker wrote: [snip] The first digit can be a grouping by success/failure. Yes, I see where you're going with this :) So then if I'm not too far off base with the above, then to use something different than HTTP::Status type codes would be reinventing. 1xx Info 2xx Success 3xx Redirect, probably not applicable to testing 4xx Failure 5xx Server/System Error As I say I'm broadly in favour of something /like/ this - but we have a clean slate here and it seems kind of arbitrary to commit to using HTTP-like status codes when we don't have to. -- Andy Armstrong, hexten.net
Re: Should TAP capture exit codes
Andy Armstrong wrote:
On 7 Mar 2007, at 16:26, Eric Hacker wrote:
[snip]
The first digit can be a grouping by success/failure.
Yes, I see where you're going with this :)
So then if I'm not too far off base with the above, then to use
something different than HTTP::Status type codes would be reinventing.
1xx Info
2xx Success
3xx Redirect, probably not applicable to testing
4xx Failure
5xx Server/System Error
As I say I'm broadly in favour of something /like/ this - but we have a
clean slate here and it seems kind of arbitrary to commit to using
HTTP-like status codes when we don't have to.
Any time you start writing a system that involves representing states as
numbers and doing bitmasks and math to add extra meaning, step back and
remind yourself that its 2007 and this is not C and you're not writing a
network protocol. You shouldn't have to memorize a table or do math in your
head to figure out the basics of what a message means.
And god forbid we had more than 100 failure types!
If you want to say "Temporary Redirect" don't say "307" say "Temporary
Redirect"! If you want to put lots of information into one value, like
categorization, use a hash! { type => "Redirect", permanent => 0 }
Re: Should TAP capture exit codes
On 3/7/07, Michael G Schwern <[EMAIL PROTECTED]> wrote:
Andy Armstrong wrote:
> On 7 Mar 2007, at 16:26, Eric Hacker wrote:
> [snip]
>> The first digit can be a grouping by success/failure.
>
> Yes, I see where you're going with this :)
>
>> So then if I'm not too far off base with the above, then to use
>> something different than HTTP::Status type codes would be reinventing.
>>
>> 1xx Info
>> 2xx Success
>> 3xx Redirect, probably not applicable to testing
>> 4xx Failure
>> 5xx Server/System Error
>
> As I say I'm broadly in favour of something /like/ this - but we have a
> clean slate here and it seems kind of arbitrary to commit to using
> HTTP-like status codes when we don't have to.
Any time you start writing a system that involves representing states as
numbers and doing bitmasks and math to add extra meaning, step back and
remind yourself that its 2007 and this is not C and you're not writing a
network protocol. You shouldn't have to memorize a table or do math in your
head to figure out the basics of what a message means.
And god forbid we had more than 100 failure types!
If you want to say "Temporary Redirect" don't say "307" say "Temporary
Redirect"! If you want to put lots of information into one value, like
categorization, use a hash! { type => "Redirect", permanent => 0 }
Numeric response codes have the advantage that they are language agnostic.
Yves
--
perl -Mre=debug -e "/just|another|perl|hacker/"
Test::Files messes with TODO tests in Test::Harness?
Hi,
I've found that using Test::Files in a test script changes the output of
TODO tests in Test::Harness.
== begin test.pl==
use strict;
use warnings;
use lib '../../perl/lib';
use Test::More;
use Test::Files;
plan tests => 2;
TODO: {
local $TODO = "TODO Testing";
is(1, 2, "a failing test");
}
== end test.pl ==
The script to run this test using test::harness
== begin harness.pl==
use strict;
use warnings;
use lib 'perl/lib';
use Test::Harness;
runtests('test.pl');
== end harness.pl ==
Output is:
C:\perl>perl harness.pl
AggregatorTest/test/test
# Failed (TODO) test 'a failing test'
# at AggregatorTest/test/test.pl line 10.
# got: '1'
# expected: '2'
AggregatorTest/test/testok
All tests successful.
Files=1, Tests=1, 0 wallclock secs ( 0.00 cusr + 0.00 csys = 0.00 CPU)
Now when I remove the "use Test::Files" line, then the output of harness.plis
C:\perl>perl harness.pl
AggregatorTest/test/testok
All tests successful.
Files=1, Tests=1, 0 wallclock secs ( 0.00 cusr + 0.00 csys = 0.00 CPU)
It seems to me that using Test::Files creates noise when running TODO tests
in Test::Harness. This is a problem because it pretty much makes TODO tests
look like failures!
I am using v0.13 of Test::Files and v2.64 of Test::Harnesss. I am also
running on windows (I have no other choice :) )
Julien
Re: Should TAP capture exit codes
On 7 Mar 2007, at 18:18, demerphq wrote:
If you want to say "Temporary Redirect" don't say "307" say
"Temporary
Redirect"! If you want to put lots of information into one value,
like
categorization, use a hash! { type => "Redirect", permanent => 0 }
Numeric response codes have the advantage that they are language
agnostic.
Indeed. That doesn't mean we have to coerce our status information to
map onto HTTP response codes though or even that we have to use a
numeric scheme.
So we have broad categories of "test script couldn't be run for some
reason" and "test script failed for some reason". In either case we /
could/ distil the actual source of the problem (t/foo.t not found,
connection to testbox.mydomain timed out) into an error code but then
we'd be discarding information that'd presumably be useful.
How gross would it be just to have a logical channel in TAP that
could represent output to STDERR? That plus the exit status of the
test script is pretty much all you have at the moment. Would that be
so bad?
--
Andy Armstrong, hexten.net
Re: Should TAP capture exit codes
On 3/7/07, Andy Armstrong <[EMAIL PROTECTED]> wrote:
On 7 Mar 2007, at 18:18, demerphq wrote:
>> If you want to say "Temporary Redirect" don't say "307" say
>> "Temporary
>> Redirect"! If you want to put lots of information into one value,
>> like
>> categorization, use a hash! { type => "Redirect", permanent => 0 }
>
> Numeric response codes have the advantage that they are language
> agnostic.
Indeed. That doesn't mean we have to coerce our status information to
map onto HTTP response codes though or even that we have to use a
numeric scheme.
Personally I see this is as a wheel-reinvention issue. Reusing the
HTTP response code seems to me to be a logical and natural step. As a
framework it strikes me that you will be unlikely to come up with
something truely better, so why not just reuse it and not worry about
it?
So far the objections we have are:
1) It uses numeric error codes and not english ones.
2) It wasnt custom designed for test responses.
Neither to me to be a very convincing reason to redesign something as
well thought out as the HTTP response code schema. With it you have a
well documented, well designed language agnostic response structure.
It seems to me youd have to work hard to come up with something
better.
Anyway, just my $0.02
cheers,
Yves
--
perl -Mre=debug -e "/just|another|perl|hacker/"
Re: Should TAP capture exit codes
On 7 Mar 2007, at 18:59, demerphq wrote: Neither to me to be a very convincing reason to redesign something as well thought out as the HTTP response code schema. With it you have a well documented, well designed language agnostic response structure. It seems to me youd have to work hard to come up with something better. We already have an interface that's even closer to home: emit an error message and return non-zero status. As I say I'm not thoroughly opposed to using HTTP like responses; just not convinced that even that level of complexity is necessary. -- Andy Armstrong, hexten.net
Re: Should TAP capture exit codes
On 3/7/07, Andy Armstrong <[EMAIL PROTECTED]> wrote: On 7 Mar 2007, at 18:59, demerphq wrote: > Neither to me to be a very convincing reason to redesign something as > well thought out as the HTTP response code schema. With it you have a > well documented, well designed language agnostic response structure. > It seems to me youd have to work hard to come up with something > better. We already have an interface that's even closer to home: emit an error message and return non-zero status. As I say I'm not thoroughly opposed to using HTTP like responses; just not convinced that even that level of complexity is necessary. I guess it comes down to whether you can anticipate the possibility that you will need new codes, and having a framework to put them into. The nice thing about the HTTP scheme is it gives you a way to add new codes that can be interpreted more or less correctly by something that wasnt designed with those codes in mind. (For instance while a particular client might not know what 569 is exactly, they should know its a server error.) If you do go so far as to design a new protocol for this please take a minute to read the RFC's for the HTTP response code schema and the SMTP response schemas. The SMTP schema goes a bit further than the HTTP one, in that it also defines special meaning for the second digit as well as for the first. Yves -- perl -Mre=debug -e "/just|another|perl|hacker/"
Re: Paying for TAP 2.0
I'd contribute $200, if that would help. -M
Re: Should TAP capture exit codes
On 7 Mar 2007, at 19:21, demerphq wrote: I guess it comes down to whether you can anticipate the possibility that you will need new codes, and having a framework to put them into. OK, well we can talk about that now and at least get an idea of what kind of future we're proofing ourselves against. What do people envisage that we might want / be able to capture about a test run? -- Andy Armstrong, hexten.net
RE: Should TAP capture exit codes
> OK, well we can talk about that now and at least get an idea of what
> kind of future we're proofing ourselves against. What do people
> envisage that we might want / be able to capture about a test run?
1. Access to both STDERR and STDOUT, in proper order ('prove' jumbles them),
capturable into a variable (T::B snatches away STDERR)
2. Option to inject a clearcut delimiter between tests
i.e. A program for programmers--instead of human-consumption output to
screen--offers output back to the programmer.
Gary
The information contained in this message may be privileged, confidential, and
protected from disclosure. If the reader of this message is not the intended
recipient, you are hereby notified that any dissemination, distribution, or
copying of this communication is strictly prohibited. If you have received this
communication in error, please notify us immediately by replying to the message
and deleting it from your computer.
Re: Should TAP capture exit codes
On 7 Mar 2007, at 19:39, Eric Hacker wrote: Now, I know you are thinking about exit status on test scripts and I'm thinking individual tests (of which running another test script might be an instance), but in the distributed functional testing space, one really can't rely on independent test scripts to do much. OK, I see where you're going. We may be talking at cross purposes slightly. TAP already has syntax to capture the status of an individual test. As far as TAP is concerned tests are binary - they succeed or fail. There may be additional diagnostic information that'll help someone to work out why they failed but in terms of the test harness' view of what happened they either pass or fail. Now if all TAP can handle is Perl, or even just code output, then those of us pushing the envelope will eventually migrate to some other test output format. TAP is a simple protocol which has implementations in C, Python, Javascript, PHP. It sounds as if you're doing monitoring rather than testing though. Although they're related the requirements are quite different. You need real time capture of as much information as you can get; the tests we're talking about aren't time dependent and it's quite acceptable to run any test multiple times until the source of the problem is found. The ability to capture more nuanced information about the outcome of a particular test would burden the tests with producing that information. In summary I think you're saying "don't just tell me it failed, tell me /why/ it failed dammit" and we're saying pretty much the opposite: "just tell me if it worked or not and leave it to me to find out why". Functional testing is years behind, especially testing distributed applications like network security. I'm trying to leverage what I can to get something useful working. Perl and Test::Harness seem to be good tools to try. I'd say not actually. Perl's certainly an excellent tool for your application but I don't think Perl's testing framework is a good match for what you're doing. Use the testing framework to prove that your code works of course but don't try to make it do something it wasn't intended to do and then blame it when it fights back :) -- Andy Armstrong, hexten.net
Re: Should TAP capture exit codes
On 3/7/07, Andy Armstrong <[EMAIL PROTECTED]> wrote: On 7 Mar 2007, at 13:48, Eric Hacker wrote: > I think it was Ovid who recently called it the Test Anything Protocol. > If really what is desired, then some additional complexity is > required. Sure - I'm completely in favour of being able to test anything and capture everything that might be useful. But if we want more nuanced exit information (did the test even run? did a connection fail?) we could still capture that without having to massage everything into HTTP style status codes. Not massage, leverage. :) I think this is a situation where you want a code rather than just using text. The conciseness eases automatic interpretation and assures clarity of what was said. If so, then how many codes? Probably not as many as hundreds. Best to group codes so that similar codes are easily identified. Best not to do this using some cool binary or hex code which pushes the calculation to people rather than the big calculator/space heater. So lets say that the codes will be 3 digits, so we'll have enough room to grow for unforseen needs. The first digit can be a grouping by success/failure. They'll be another status group for situations where the test didn't run at all, because of an operation issue. Might want a status group for keepalive info and whatnot on long tests. Sure the Harness might not ever get them, but why have some entirely different way of communication this type of status. I'm not /strongly/ opposed to your proposal - just trying to understand what it might do that we can't do more simply by other means. So then if I'm not too far off base with the above, then to use something different than HTTP::Status type codes would be reinventing. 1xx Info 2xx Success 3xx Redirect, probably not applicable to testing 4xx Failure 5xx Server/System Error Then, if I'm writing a test engine with an http interface for running tests, little or no translation is necessary. :)
Re: Should TAP capture exit codes
On 7 Mar 2007, at 19:50, Gary Hawkins wrote:
1. Access to both STDERR and STDOUT, in proper order ('prove'
jumbles them), capturable into a variable (T::B snatches away STDERR)
TAP's a line oriented protocol so I imagine the best we can do is to
keep /lines/ from STDERR and STDOUT in the right order. I imagine we
can just dump lines of STDERR prefixed by '!' or whatever into the
TAP output - much the same way we handle diagnostic output currently.
2. Option to inject a clearcut delimiter between tests
Distinct from, say, outputting a diagnostic between groups of tests?
--
Andy Armstrong, hexten.net
RE: Should TAP capture exit codes
> > 2. Option to inject a clearcut delimiter between tests > Distinct from, say, outputting a diagnostic between groups of tests? By 'tests' I'm thinking 'file' with its subtests, so-to-speak, so yes, anything that is clearly delineatable (LOL) programmatically where one 'file' output stops and another starts is fine. Thanks, Gary The information contained in this message may be privileged, confidential, and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the message and deleting it from your computer.
Re: Should TAP capture exit codes
On 7 Mar 2007, at 20:35, Gary Hawkins wrote: 2. Option to inject a clearcut delimiter between tests Distinct from, say, outputting a diagnostic between groups of tests? By 'tests' I'm thinking 'file' with its subtests, so-to-speak, so yes, anything that is clearly delineatable (LOL) programmatically where one 'file' output stops and another starts is fine. Different in what way from the current behaviour of Test::Harness? Each test script gets its own line in the output currently. -- Andy Armstrong, hexten.net
RE: Should TAP capture exit codes
2. Option to inject a clearcut delimiter between tests >>> >>> Distinct from, say, outputting a diagnostic between groups of tests? >> >> By 'tests' I'm thinking 'file' with its subtests, so-to-speak, so yes, >> anything that is clearly delineatable (LOL) programmatically where one >> 'file' output stops and another starts is fine. > >Different in what way from the current behaviour of Test::Harness? >Each test script gets its own line in the output currently. Not if verbose output. Surely you jest. It is not reliably parsable, there can be garbage text on the end of those summary lines at times. If the harness was run from one level above a 't' directory, you'll have t/ at the beginning of the line, but what if not? No way to predict the relative path someone will use, how many '.', or whether the last character on the line will be a digit, ok, NOK, dubious, or whatever else, including just a dot. Add to that failed or died diagnostics. Parse this to find the results per test to write them to db. (This does not take verbose into account where the challenge is more desperate). somegreattestfoobars_exceptions..ok xwhateverr_exceptionsok 35/71 # Failed test (xwhateverr_exceptions.t at line 1011) xwhateverr_exceptionsok 71/71# Looks like you failed 1 test of 71. xwhateverr_exceptionsdubious Test returned status 1 (wstat 256, 0x100) DIED. FAILED test 36 Failed 1/71 tests, 98.59% okay foobar_limit.ok 3/16 # Failed test (foobar_limit.t at line 321) foobar_limit.ok 16/16# Looks like you failed 1 test of 16. foobar_limit.dubious Test returned status 1 (wstat 256, 0x100) DIED. FAILED test 4 Failed 1/16 tests, 93.75% okay It would be healthy, smart, and advantagous for the good of TAP to add an option for delimiter. _..._ ~_~ _..._ <- kilroy? somegreattestfoobars_exceptions..ok _..._ ~_~ _..._ xwhateverr_exceptionsok 35/71 # Failed test (xwhateverr_exceptions.t at line 1011) xwhateverr_exceptionsok 71/71# Looks like you failed 1 test of 71. xwhateverr_exceptionsdubious Test returned status 1 (wstat 256, 0x100) DIED. FAILED test 36 Failed 1/71 tests, 98.59% okay _..._ ~_~ _..._ foobar_limit.ok 3/16 # Failed test (foobar_limit.t at line 321) foobar_limit.ok 16/16# Looks like you failed 1 test of 16. foobar_limit.dubious Test returned status 1 (wstat 256, 0x100) DIED. FAILED test 4 Failed 1/16 tests, 93.75% okay _..._ ~_~ _..._ That's all I have to say about that. -Forest Gump The information contained in this message may be privileged, confidential, and protected from disclosure. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the message and deleting it from your computer.
Re: Should TAP capture exit codes
On 7 Mar 2007, at 21:15, Eric Hacker wrote: Monitoring would check that the http server is up, and check that the database server is up, but this is functional testing. Does the application work, end to end. OK, /now/ I understand. That's what Test::More does. Otherwise, everything would just be Ok. Sure - but that's human readable information. As far as the harness is concerned it's succeed or fail plus a bunch of text it passes through verbatim. In summary I think you're saying "don't just tell me it failed, tell me /why/ it failed dammit" and we're saying pretty much the opposite: "just tell me if it worked or not and leave it to me to find out why". is_deeply and cmp_ok start to say why. Now why is a matter of degree, and if a test could tell that the developer meant && instead of || or that the new HTTP firewall was dropping all POSTS with "hacker" in the data, then we'd be out of work. I thought the spirit was to present as much info as practical. Sure, absolutely. Correct me if I'm wrong (again) but you were advocating returning an HTTP-like status code per individual test, yes? Given that there innumerable ways for a particular test to fail I can't quite see how you can distil that down to a status code. I really don't care too much, because I am already making what's there work for me now. But if you are asking what to improve... :) Thanks for taking the time. I appreciate that even if I can't actually work out what you're advocating :) I took a look at http://www.testobsessed.com the other day. Check out the functional test tools next generation dream. That makes status codes look like Lego Duplos. I'm clearly having a stupid day. Is this what I'm supposed to be looking at: What Problem Would Next-Generation Functional Testing Tools Solve? http://lyxus.net/gdc I can't see anything there that's much of a reach for us. -- Andy Armstrong, hexten.net
Re: Should TAP capture exit codes
On 7 Mar 2007, at 21:24, Gary Hawkins wrote: Not if verbose output. Surely you jest. It is not reliably parsable, there can be garbage text on the end of those summary lines at times. [snip] If you want /parsed/ test results you'll be able to use TAPx::Parser and/or TAPx::Harness to get at them directly. No need to parse the output from the harness. -- Andy Armstrong, hexten.net
Re: Should TAP capture exit codes
How gross would it be just to have a logical channel in TAP that could represent output to STDERR? That plus the exit status of the test script is pretty much all you have at the moment. Would that be so bad? Perhaps, for non-Unix testing, perhaps not. Here is what I have. There are bots that are running on various systems. One asks the bot to do something, and it responds. My bots are being set up to do testing things. Can you ping the server, can you open a TCP connection to port 80, can you get a URL over HTTP, can you run this script, etc. The idea is that without a traditional debugger, there should be a command line user interface that does these things as well as an RPC method for automated tests. When I send an RPC response, it has the human readable reply and the response code. Sure, I could have written a regex to figure out if the response is correct of failed, and I allow that. It is easier to process the codes though, and that suffices for most tests. Sure all of this could have been done in SSH, if everyone had account on and the machines and etc. There are other ways, this is what I'm trying to do and I may fail. Then you all can learn from my 409, 416 or 417. Now, I know you are thinking about exit status on test scripts and I'm thinking individual tests (of which running another test script might be an instance), but in the distributed functional testing space, one really can't rely on independent test scripts to do much. I need to set up a tail with regex on a log file on one bot, hit it from another bot, and know that both were successful real time. Then keep going. Now if all TAP can handle is Perl, or even just code output, then those of us pushing the envelope will eventually migrate to some other test output format. Functional testing is years behind, especially testing distributed applications like network security. I'm trying to leverage what I can to get something useful working. Perl and Test::Harness seem to be good tools to try.
Re: Should TAP capture exit codes
On 3/7/07, Michael G Schwern <[EMAIL PROTECTED]> wrote:
Any time you start writing a system that involves representing states as
numbers and doing bitmasks and math to add extra meaning, step back and
remind yourself that its 2007 and this is not C and you're not writing a
network protocol. You shouldn't have to memorize a table or do math in your
head to figure out the basics of what a message means.
I agree, except we are writing a network protocol. Once the test
results are being shipped around to different systems, a network
protocol is in the works.
And god forbid we had more than 100 failure types!
If you want to say "Temporary Redirect" don't say "307" say "Temporary
Redirect"!
How about "temporär adressieren Sie um" or "provisoire réorientez" or
"de tijdelijke werkkracht richt opnieuw". :)
A decent coding system simplifies the translation between machines and
humans and human languages. It is a means of data sanitizing. It
establishes a scheme of the important details within an area that are
agreed upon by the participants. This provides guidance to new
participants, who can see from the coding system what is relevant.
If an existing coding system that many of the user population is
already familiar with can be leveraged, all the better.
If you want to put lots of information into one value, like
categorization, use a hash! { type => "Redirect", permanent => 0 }
Ok. YAML or XML for serialization? :)
Re: Should TAP capture exit codes
On 3/7/07, Andy Armstrong <[EMAIL PROTECTED]> wrote:
It sounds as if you're doing monitoring rather than testing though.
Although they're related the requirements are quite different.
Poor explaining on my part then. Monitoring has similar needs, but us
usually much more shallow. Consider a web app test (simplified):
# Set tail before running test
$database->ok("tail test add $data");
$client->ok("submit $data to url http://www.example.com/form.cgi";);
Monitoring would check that the http server is up, and check that the
database server is up, but this is functional testing. Does the
application work, end to end.
Now stick some new web app security thingy in from of the web server,
and a database security tool in between the web app server and the
database. We need to run a regression test on the changes, but how.
This is not monitoring. This is scripted testing.
You
need real time capture of as much information as you can get; the
tests we're talking about aren't time dependent and it's quite
acceptable to run any test multiple times until the source of the
problem is found. The ability to capture more nuanced information
about the outcome of a particular test would burden the tests with
producing that information.
That's what Test::More does. Otherwise, everything would just be Ok.
In summary I think you're saying "don't just tell me it failed, tell
me /why/ it failed dammit" and we're saying pretty much the opposite:
"just tell me if it worked or not and leave it to me to find out why".
is_deeply and cmp_ok start to say why.
Now why is a matter of degree, and if a test could tell that the
developer meant && instead of || or that the new HTTP firewall was
dropping all POSTS with "hacker" in the data, then we'd be out of
work. I thought the spirit was to present as much info as practical.
> Functional testing is years behind, especially testing distributed
> applications like network security. I'm trying to leverage what I can
> to get something useful working. Perl and Test::Harness seem to be
> good tools to try.
I'd say not actually. Perl's certainly an excellent tool for your
application but I don't think Perl's testing framework is a good
match for what you're doing. Use the testing framework to prove that
your code works of course but don't try to make it do something it
wasn't intended to do and then blame it when it fights back :)
I really don't care too much, because I am already making what's there
work for me now. But if you are asking what to improve... :)
I took a look at http://www.testobsessed.com the other day. Check out
the functional test tools next generation dream. That makes status
codes look like Lego Duplos.
Re: Should TAP capture exit codes
On Wednesday 07 March 2007 11:39, Eric Hacker wrote: > Now if all TAP can handle is Perl, or even just code output, then > those of us pushing the envelope will eventually migrate to some other > test output format. TAP can certainly handle more than just Perl and code output. In my mind, the questions are: 1) Do existing TAP features already provide this feature? 2) Do existing TAP features almost already provide this feature, with a little bit of work? 3) Is this a feature that TAP should support somehow? 4) Is this generalizable to different domains? 5) Is there a clean way to add this to TAP while retaining backwards compatibility? I'm still not sure what you're doing that requires anything other than a machine-parsable diagnostic, but I have a failure of imagination sometimes. -- c
Re: Should TAP capture exit codes
demerphq wrote:
If you want to say "Temporary Redirect" don't say "307" say "Temporary
Redirect"! If you want to put lots of information into one value, like
categorization, use a hash! { type => "Redirect", permanent => 0 }
Numeric response codes have the advantage that they are language agnostic.
So are strings. Wow, I'm having deja vu.
I should have been more clear, you wouldn't use a literal hash. You'd use
YAML or JSON or something.
error:
type: Redirect
permanent: 0
Re: Should TAP capture exit codes
Gary Hawkins wrote: 2. Option to inject a clearcut delimiter between tests Distinct from, say, outputting a diagnostic between groups of tests? By 'tests' I'm thinking 'file' with its subtests, so-to-speak, so yes, anything that is clearly delineatable (LOL) programmatically where one 'file' output stops and another starts is fine. Different in what way from the current behaviour of Test::Harness? Each test script gets its own line in the output currently. Not if verbose output. Surely you jest. It is not reliably parsable, there can be garbage text on the end of those summary lines at times. There's already a proposal to handle per-test diagnostics. http://perl-qa.yi.org/index.php/TAP_diagnostic_syntax
You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
Eric Hacker wrote: I think this is a situation where you want a code rather than just using text. The conciseness eases automatic interpretation and assures clarity of what was said. If so, then how many codes? Probably not as many as hundreds. Best to group codes so that similar codes are easily identified. Best not to do this using some cool binary or hex code which pushes the calculation to people rather than the big calculator/space heater. So lets say that the codes will be 3 digits, so we'll have enough room to grow for unforseen needs. Stop. Stop stop stop! Stop right there. Repeat after me, "We cannot predict how TAP will be used". We can't. Anything we design now will seem silly and narrow 5, 10, 15 years from now. TAP, after all, is 20 years old. 20 years ago would be have considered HTTP status codes? No, they didn't exist. Would we even have cared about other operating systems? No, it would be Unix exit codes... just like it is now. If we reduce exit status down to a series of hard coded 3 digit codes we're erecting an iron crib around the cradle. So limiting. (999 status codes|64K) will be enough for anyone, right? Extensibility is the key. This is why I like a YAML or JSON format so much. We can always add to it. http://perl-qa.yi.org/index.php/TAP_diagnostic_syntax 1..2 ok 1 ok 2 done (or whatever we wind up using for "end of tests") --- exit status: 23 wait status: 0 http status: 404 human readable status: Yer screwed. psychological status: I'm feeling fine, thanks ... We can leverage any existing status system we want. HTTP status. Exit status. Throw them all in! Don't find TAP's existing statuses rich enough? Add your own extension keys! A particular status code not make sense for your application? Leave it out! And uses the same mechanism as the TAP diagnostics proposal so little extra work on the part of the TAP parser is necessary. And we don't have to plan everything out now and be so very, very, very wrong.
Re: Test::Files messes with TODO tests in Test::Harness?
Julien Beasley wrote: Hi, I've found that using Test::Files in a test script changes the output of TODO tests in Test::Harness. Here's the problem. $ perl -wle 'use Test::Files; print Test::Builder->new->exported_to' Test::Files exported_to() is the mechanism Test::Builder uses to guess in what package to look for the $TODO flag. It tries to guess right but is sometimes wrong. http://rt.cpan.org/Ticket/Display.html?id=24725 Anyhow, here's the problem: package Test::Files; use Test::Builder; use Test::More; <- what's this doing there? use Text::Diff; use File::Find; use File::Spec; Just remove that and everything goes smoothly. It can make use of Test::Builder::Module to make things even more foolproof.
Re: a safer way to use no_plan?
Adam Kennedy wrote: Still no obvious place to put it, other than on it's own in Test::NewOk... Umm, your own Test::Object? Though that seems to be a reimplementation of Test::Class with a clumsy (sorry, looks that way to me having to write 3-4 lines of code per test method) test registration interface.
Re: You cannot predict what TAP will be used for (was Re: Should TAP capture exit codes)
On Mar 7, 2007, at 6:40 PM, Michael G Schwern wrote: Repeat after me, "We cannot predict how TAP will be used". That's why it's the "Test Anything Protocol". The A is Anything! -- Andy Lester => [EMAIL PROTECTED] => www.petdance.com => AIM:petdance
Re: per-author META.yml
On Mar 5, 2007, at 1:58 PM, Eric Wilhelm wrote: # from Ricardo SIGNES # on Monday 05 March 2007 10:09 am: * brian d foy <[EMAIL PROTECTED]> [2007-03-04T12:09:26] ... without me having to change the META.yml in all of my distributions then re-uploading them all. So, for some subset of META.yml settings, you could consult the module's author settings, found at (say) $MIRROR/authors/.../RJBS/METAMETA.yml ... Something like that? I feel a potentially irrational sense of dread. It could just be META.yml, or maybe AUTHOR.yml. Why the dread? If we're going to have per-author settings, I think that would be the place for them. In the specific instance we're talking about, cc_author, I wouldn't want it in any such file, whether per-author or per-distribution; I'd want it as a preference I can set within the cpan-testers system. Because if I change my mind and decide I'd rather see reports via RSS and not email, I wouldn't want to have to upload a new file somewhere. -Ken
