Re: Doubts about TB2 and the TAP stack
On 2010.10.29 8:36 AM, Pedro Melo wrote: > All the process got me thinking about Test::Builder. I spent a couple > of hours tonight trying to see how would I output JUnit with TB2. I > tried to write a Formatter::JUnit and was mostly successful, but I > could not see how TB2 ties into the current TAP::Harness stack, that > powers App::Prove. > > I don't know all the pieces that well yet, but seems to me that TB2 > has no need for formatters. Those should be at the Harness level. TB2 > scripts would generate a more robust stream of TAP events, and the > TAP2::Harness would feed them into one or more formatters to produce > the file format. > > Right now, I failed to see how would a Harness module would even setup > a different format for the test scripts, given that the TB2 setup > happens in a different process space, after a IPC::Open3 call in the > TAP::Parser::Iterator::Process. > > I'm sure I'm missing something and there is a plan to glue the Harness > with TB2 formatters. If some kind soul could take a little bit of time > to point me in the right direction (maybe there is a TAP2 tree > somewhere?), I could go and look and try to make use of my hacked > TB2::Formatter::JUnit. Hi Pedro, Sorry, this message got lost along the way. I'm glad you're interested in TB2 and want to put together a JUnit formatter for it. Lots has changed since October. Fortunately it should be a fairly rote change from the old system of to accepting events. I'll be glad to help you fix it up for the new event system and bring your formatter into the fold, but probably as an example because of the XML::Generator dep. Or you can release it as its own CPAN module. We definitely need a maintained JUnit formatter. Also you've noticed that Test::Builder really only works with the TAP formatter due to all the extra methods. I'm working on tearing them apart. Test::Builder does use Test::Builder2::Formatter almost completely as of alpha 2. I still have to make diagnostics work. And then Test::More will come along. Now, to whether you're missing something, you're only missing it because it isn't there. TAP::Harness explicitly does not know anything about what it's executing (beyond enough information to execute it). It just expects TAP. If you want to use TB2 tests with a different format you don't use TAP::Harness. [1] Ta da! What do you use? Presumably you use whatever test runner wants the differently formatted output. Let that run your test programs directly. This might seem like it's creating a redundant system of formatters... and in some way it is. But forcing (for example) a JUnit runner to instead run prove which then takes test results, translates them into TAP and then translates them into JUnit is awkward. There's a lot of information JUnit can't handle that TAP can, so information is lost in the translation. Furthermore, TB1 was having difficulty adapting to TAP extensions. Extensions to TAP turn into pretty much different formatters, so internally it needed to exist anyway. Right now, making a TB2 test program use a different format is awkward. You have to do it per script. There needs to be a way to control this outside the script, perhaps an environment variable one can set to swap out formatters. I haven't been doing much convenience features, more focusing on just getting the design solid and things working. It is a SMOP to make your own test programs respond to an environment variable if you need something now. In the future, discussions about Test::Builder2 are more likely to be noticed on test-more-us...@groups.google.com. [1] Not entirely true. You could, for example, use Test::Builder2::Formatter::Multi and attach two formatters, one which outputs TAP to stdout and another which writes JUnit to a file. This is handy for something like Hudson. -- "I went to college, which is a lot like being in the Army, except when stupid people yell at me for stupid things, I can hit them." -- Jonathan Schwarz
Re: Doubts about TB2 and the TAP stack
Hi, On Fri, Oct 29, 2010 at 6:24 PM, Graham TerMarsch wrote: > On October 28, 2010, Pedro Melo wrote: >> All the process got me thinking about Test::Builder. I spent a couple >> of hours tonight trying to see how would I output JUnit with TB2. I >> tried to write a Formatter::JUnit and was mostly successful, but I >> could not see how TB2 ties into the current TAP::Harness stack, that >> powers App::Prove. > [.snip.] >> I'm sure I'm missing something and there is a plan to glue the Harness >> with TB2 formatters. If some kind soul could take a little bit of time >> to point me in the right direction (maybe there is a TAP2 tree >> somewhere?), I could go and look and try to make use of my hacked >> TB2::Formatter::JUnit. > > Hack or otherwise, I'd love to see what you've done w.r.t. this; we've got > "TAP::Formatter::JUnit" on CPAN and if there's any experience/info that can be > gleaned from your hacking that could improve that, I'd love to get it in > there. I mostly stole your code :) You can see the result here: http://github.com/melo/test-more/commits/junit-formatter/ It works with some of my tests with the following caveats: * the TAP plan is printed as the first line: Test::Builder outputs the plan directly, without going through the Formatter object; * I don't have access to the pretty failure explanations like those of 'is()': Test::More still uses Test::Builder and those explanations are diag()'ed after the test result is reported. Also, you cannot use prove. I've placed a script in examples/, junit.pl that you must use to run each file of your testsuite. I need this to setup the Test::Builder and Test::Builder2 to use the JUnit formatter. I would say that Test::Builder2 API is very very nice, but Test::Builder still assumes everything is TAP all the way. For now, if you want JUnit, TAP::Harness::JUnit or TAP::Formatter::JUnit are still the best choices. At least until Test::Builder leverages Test::Builder2 more profoundly. Bye, -- Pedro Melo http://www.simplicidade.org/ xmpp:m...@simplicidade.org mailto:m...@simplicidade.org
Re: Doubts about TB2 and the TAP stack
On October 28, 2010, Pedro Melo wrote: > All the process got me thinking about Test::Builder. I spent a couple > of hours tonight trying to see how would I output JUnit with TB2. I > tried to write a Formatter::JUnit and was mostly successful, but I > could not see how TB2 ties into the current TAP::Harness stack, that > powers App::Prove. [.snip.] > I'm sure I'm missing something and there is a plan to glue the Harness > with TB2 formatters. If some kind soul could take a little bit of time > to point me in the right direction (maybe there is a TAP2 tree > somewhere?), I could go and look and try to make use of my hacked > TB2::Formatter::JUnit. Hack or otherwise, I'd love to see what you've done w.r.t. this; we've got "TAP::Formatter::JUnit" on CPAN and if there's any experience/info that can be gleaned from your hacking that could improve that, I'd love to get it in there. -- Graham TerMarsch
Doubts about TB2 and the TAP stack
Hi, today I spent a couple of hours setting up my test script to output JUnit, to run them under Hudson. This experience was mostly pain free. I started with the tap-to-junit script but then switched to the TAP::Harness::JUnit to get a prettier JUnit report. All the process got me thinking about Test::Builder. I spent a couple of hours tonight trying to see how would I output JUnit with TB2. I tried to write a Formatter::JUnit and was mostly successful, but I could not see how TB2 ties into the current TAP::Harness stack, that powers App::Prove. I don't know all the pieces that well yet, but seems to me that TB2 has no need for formatters. Those should be at the Harness level. TB2 scripts would generate a more robust stream of TAP events, and the TAP2::Harness would feed them into one or more formatters to produce the file format. Right now, I failed to see how would a Harness module would even setup a different format for the test scripts, given that the TB2 setup happens in a different process space, after a IPC::Open3 call in the TAP::Parser::Iterator::Process. I'm sure I'm missing something and there is a plan to glue the Harness with TB2 formatters. If some kind soul could take a little bit of time to point me in the right direction (maybe there is a TAP2 tree somewhere?), I could go and look and try to make use of my hacked TB2::Formatter::JUnit. Thanks, -- Pedro Melo http://www.simplicidade.org/ xmpp:m...@simplicidade.org mailto:m...@simplicidade.org
