I'm using Test::xxx as a tool for testing the functioning of a rather large C program (currently at 112K+ tests). Many of the tests consist of running combinations of parameters in many nested loops. Sometimes some of these tests fail. For example, out of a test-script that has about 8000 tests, only 20 will fail along the way.


I would like these tests to be marked as "TODO". But that's virtually impossible with the current way you specify TODO tests, as the failures only happen with a specific combination of parameters, usually at least 3 levels deep in loops.



Now, the test output tells me the test number it failed. What the exact combination of parameters is, is less important to me in many cases. The fact that the test (unexpectedly) fails is more important to me.


Now I only have the option of skipping the entire set of nested loops, as I don't want it to produce any test failures on expected errors. What I would like to do is just somehow give it a list of test numbers to be marked as "TODO". And almost everything is there already: setting $TODO to a non-empty string is the only thing needed to make all subsequent tests marked as "todo". I just lack the method to set $TODO at the right moment (or I have missed it somehow).

So, what I'd like to add for myself is something like:

todo_ok( test,{
 1001 => "a b c still fails, wonder why",
 2345 => "d e gf to be investigated",
},"ok text" );


The conundrum I'm facing with this is that the "current_test" method of Test::Builder is not available from Test::More. And the Test::Builder object being used in a Test::More run is also not available in Test::More. And I don't want to make another Test::More.



So I see basically three solutions to this problem:


1. patch Test::More so that the Test::Builder object can be obtained from a test.

Something like adding "sub Test { $Test }" to Test::More


2. patch Test::More to export current_test


Something like adding "sub current_test { $Test->current_test }" to Test::More


3. patch Test::More to export "todo_ok"


From within Test::More it should be trivial to create "todo_ok", but does this itch of mine warrant includion in Test::More? And why wouldn't then "todo_is", "todo_cmp_ok" also be made?


I think I prefer 1 as it will allow you to possibly do other things in the future apart from accessing "current_test". The solution is more generic and only accessible to those people who actually takie a look at the pod / source of Test::Builder. Solution 2 would on one hand be too specific and on the other hand not specific enough. Option 3 introduces bloat (is that a problem?).



If there is another way I could do this, I'm open to that as well... And I wonder where I should post patches to Test::More... ;-)



Liz

Reply via email to