Re: Do we need subtests in TAP?
Hiya, On 29 Oct 2011, at 10:20, Michael G Schwern wrote: > On 2011.10.29 1:51 AM, Adrian Howard wrote: >> On 29 Oct 2011, at 09:18, Michael G Schwern wrote: >> [snip] >>> Do you find *blocks with their own name and plan* convenient, or subtests >>> which have their own separate test state (as currently implemented) >> >> This may be me being dim - but I'm not really groking the distinction you're >> making. Could you give an example of the kind of difference that you're > proposing? > > It's *mostly* internal, but here's an example to explain the difference. > > Here's how subtests currently work: [snip] >ok 1 - First test >1..2 >ok 1 - This is a subtest >ok 2 - So is this >ok 2 - An example subtest >ok 3 - Third test >1..3 [snip] > What I'm asking would it make much difference to how subtests are used if it > was like this: > >This would produce. > >ok 1 - First test ># An example of a subtest >ok 2 - This is a subtest >ok 3 - So is this >ok 4 - Last test >1..4 [snip] Thanks - understand now. I prefer the current subtests system for a few reasons: * With the new system I would have to re-write TAP streams from other sources to match the numbering system of the current stream. This makes more work for folk who are pulling in TAP streams from other boxes/processes. If we do go this route you would, ideally, need to have a "standard" system for renumbering/counting streams. * I cannot, in a general way, tell the difference between a TAP stream with subtests and a linear TAP stream. This means I can do things like easily report whether certain levels of subtests are passing/failing. I'm using nesting because the groups make logical sense. Throwing that information away in the TAP output seems a bad thing to me - it's useful to analyse/view the results as well as organising the tests. * With no "standard" way of representing the nesting in the TAP stream output - everybody who needs to manage any kind of test grouping / hierarchy has to invent their own way of doing it. * Test::Block kinda uses this system. Never really used by anybody. subtests are already used by lots of folk. I think part of the reason why may be coz of the more intention revealing output? * How would a no_plan subtest merge into a planned stream? ... and probably some more if I carry on thinking about it. But that's enough for now ;-) Cheers, Adrian -- http://quietstars.com adri...@quietstars.com twitter.com/adrianh t. +44 (0)7752 419080 skype adrianjohnhoward del.icio.us/adrianh
Re: Do we need subtests in TAP?
On 2011.10.30 2:58 AM, Adrian Howard wrote: > I prefer the current subtests system for a few reasons: > > * With the new system I would have to re-write TAP streams from other sources > to match the numbering system of the current stream. This makes more work for > folk who are pulling in TAP streams from other boxes/processes. If we do go > this > route you would, ideally, need to have a "standard" system for renumbering/counting > streams. No, TAP::Harness would continue to parse subtest TAP. TAP from other tests could still be embedded by simple indentation. > * I cannot, in a general way, tell the difference between a TAP stream with > subtests and a linear TAP stream. This means I can do things like easily > report > whether certain levels of subtests are passing/failing. I'm using nesting > because the groups make logical sense. Throwing that information away in the > TAP output seems a bad thing to me - it's useful to analyse/view the results > as > well as organising the tests. Agreed. > * With no "standard" way of representing the nesting in the TAP stream output > - everybody who needs to manage any kind of test grouping / hierarchy has to > invent their own way of doing it. True. > * Test::Block kinda uses this system. Never really used by anybody. subtests > are already used by lots of folk. I think part of the reason why may be coz > of the more intention revealing output? That's part of what I'm trying to discover. > * How would a no_plan subtest merge into a planned stream? Just fine, thanks. It would require no work at all. Without the TAP formatting, a no_plan subtest is equivalent to just running some tests. -- ...they shared one last kiss that left a bitter yet sweet taste in her mouth--kind of like throwing up after eating a junior mint. -- Dishonorable Mention, 2005 Bulwer-Lytton Fiction Contest by Tami Farmer
Re: Do we need subtests in TAP?
On Sun, Oct 30, 2011 at 3:23 PM, Michael G Schwern wrote: > On 2011.10.30 2:58 AM, Adrian Howard wrote: >> I prefer the current subtests system for a few reasons: >> >> * With the new system I would have to re-write TAP streams from other sources >> to match the numbering system of the current stream. This makes more work for >> folk who are pulling in TAP streams from other boxes/processes. If we do go >> this >> route you would, ideally, need to have a "standard" system for > renumbering/counting >> streams. > > No, TAP::Harness would continue to parse subtest TAP. TAP from other tests > could still be embedded by simple indentation. Michael, I haven't followed the T::B 2 work closely enough, so could I ask you to please step back and explain the benefits of T::B 1.5 that is worth stepping backwards in terms of capabilities? What I mean is that we have TAP::Harness now that processes subtest TAP and we have a T::B now that produces subtest TAP, so why stop? This quote from the TB 2 docs scares me a little: "Test::Builder2 is very generic and doesn't do a lot of the work you've probably come to expect a test framework to do. This reduction of assumptions increases flexibility and ensures that TB2 can remain the core of Perl testing for another decade to come." That sounds an awful lot like second system syndrome. And -- hey, that's great if it works -- I mean no offense is saying that. But something that is *half* of a second system that loses a nice feature of the first system seems a suboptimal outcome. -- David
Re: Do we need subtests in TAP?
On 2011.10.30 7:21 PM, David Golden wrote: > I haven't followed the T::B 2 work closely enough, so could I ask you > to please step back and explain the benefits of T::B 1.5 that is worth > stepping backwards in terms of capabilities? What I mean is that we > have TAP::Harness now that processes subtest TAP and we have a T::B > now that produces subtest TAP, so why stop? I'm not advocating one side or the other just yet. I'm investigating what subtests are used for and if they're worth the bother, because I don't use them. Ovid always took care of them. They're a lot of bother, but it looks like it would be even more bother to do them without nested TAP. The current Test::Builder implementation is a mess and its design cannot go forward. They have to be gotten just right to ensure that not just nested TAP is supported, but nesting in other formats. Or if those formats don't have nesting, then linearizing the subtests in those formats. And event watcher (ie. plugins) authors have to be shielded from the complexity. The end result is looking to be fairly simple, but that doesn't mean a lot of work didn't go into it. > This quote from the TB 2 docs scares me a little: "Test::Builder2 is > very generic and doesn't do a lot of the work you've probably come to > expect a test framework to do. This reduction of assumptions increases > flexibility and ensures that TB2 can remain the core of Perl testing > for another decade to come." > > That sounds an awful lot like second system syndrome. And -- hey, > that's great if it works -- I mean no offense is saying that. But > something that is *half* of a second system that loses a nice feature > of the first system seems a suboptimal outcome. D O N ' T P A N I C What you read is about the Test::Builder2 *class*. The class is separate from the Test::Builder2 framework. Sorry if that's not clear. The framework can do what Test::Builder does (with the current exception of subtests) and more. The Test::Builder2 class does a lot less than the Test::Builder class. I intended to provide a provide a pack of roles to add in more functionality, and people could write more. But I'm not worrying about the Test::Builder2 class right now. Test::Builder2 is a user friendly shell to generate test events. Much of the Test::Builder functionality has been (or will be) pushed down into event watchers. Test::Builder 1.5 will use that event system. The benefit is we can continue to have a unified testing framework, but write tests using whatever interface you like and output in whatever format you like. Plugins which effect the whole state of the test (such as Test::NoWarnings) become safer to write and continue to work even as testing styles become different. You can even write your own builder. TB2 is the opposite of the second system syndrome. The second system is traditionally overbuilt with too many features that nobody needs. While the event system might qualify, the Test::Builder2 class itself does not. -- 10. Not allowed to purchase anyone's soul on government time. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/
Re: Do we need subtests in TAP?
# from Michael G Schwern # on Sunday 30 October 2011 20:30: >The current Test::Builder implementation is a mess and its design > cannot go forward. They have to be gotten just right to ensure that > not just nested TAP is supported, but nesting in other formats. Or > if those formats don't have nesting, then linearizing the subtests in > those formats. And event watcher (ie. plugins) authors have to be > shielded from the complexity. Maybe have a null or default handler for the subtests -- possibly allowing authors to use a base class or role which gives them a subtest_start() returning the null handler and subtest_end() which just forwards the summary as a single test event. Is there a second format being implemented to test this API? >The end result is looking to be fairly simple, but that doesn't mean a >lot of work didn't go into it. Don't you just take the infinite space of bad ideas and cut away everything that doesn't look like a good API? Sounds easy. ;-) --Eric -- "Beware of bugs in the above code; I have only proved it correct, not tried it." --Donald Knuth --- http://scratchcomputing.com ---
