Chris Dolan wrote: > On Dec 2, 2007, at 1:34 PM, nadim khemir wrote: >> Because a TODO means that it is not done not: it might happend to be >> done but >> I'm not really sure, maybe I get lucky. >> >> Either one removes the TODO and all is fine. Or it might just be a >> side effect >> that you haven't planned that makes the test pass. Calling "unexpected >> things" "features" does not make me feel more sure about quality.
(I missed the above part of the original post) The red herring is "unexpected", and it's good that wording was removed from TH 3. The feature is expected, but that it worked is unexpected. You're talking like the TODO test is some slap-dash bit of code. The TODO test is crafted to test a specific feature or bug, just like any other test. What you're asking, essentially, is how do you know that the passing TODO test is testing what it's supposed to test? Well, remove "TODO" from that question and it reduces to the more general question: how do you know a passing test is testing what it's supposed to test? That is a question with plenty of answers. >> This is exactely why I complained loud about Test::Exception::dies_ok. >> It died >> but not for the problem I expected. Other may be better or have more luck >> than me though. There's nothing wrong with dies_ok(), if I understand your complaint. It does what it says on the tin, tests that the code died. And that's useful, sometimes you don't know or don't care or can't accurately predict what the exception will be. In these cases, without dies_ok() you'd have to fake up some static exception for throws_ok(). This doesn't mean that people don't use dies_ok() when they should use throws_ok(). Every tool is open to abuse. The solution is not to remove the tool, but figure out why it's being abused. Maybe the answer is as simple as putting throws_ok() first in the Test::Exception documentation? As an analogy, look at ok() vs is() in Test::More. ok() just checks that the statement is true, it doesn't care what it actually is. is() gets more specific. In general, it's better to use is() but often you don't know (or care) what the value actually is, just that it's true. Removing ok() from Test::More would increase the resolution but also remove a lot of flexibility, just as removing dies_ok() would. >> I really would like to have -Werror for my tests. A warning is "hey, you might have done something wrong!" A passing TODO test is "hey, you might have done something right!" :) -- If at first you don't succeed--you fail. -- "Portal" demo