chromatic wrote:
3) False negatives are EVER acceptable in tests
False negatives have proven to be a useful tool for finding bugs *in the code doing the testing*. They have helped me to find, report, and *get fixed* bugs in several important modules. I find a miniscule number of incorrect test failures (and we're talking no more than a dozen brought to my attention, out of over 20,000 test reports I've sent) to be a price worth paying.
Looking specifically at test suites that fail solely because of documentation failures, the only one I can think of off the top of my head is the little-used p5-Palm, version 1.008. Brian Foy fixed it within hours of me sending the test results to him.
I'm far more concerned by incorrectly passing tests. Those are harder to spot, firstly because testers don't report passes to authors, and secondly because even if they did they'd be ignored. I've not had any reported to me, but I'm absolutely certain that I've sent some. And then there's "unknown" test results, which make CPAN.pm go ahead and install the module anyway, despite the reason for the result being "unknown" being because the user had to hit C-c to get control back after the module's test suite just got stuck.
-- David Cantrell
