chromatic wrote: > On Saturday 22 December 2007 16:48:29 Michael G Schwern wrote: >> The "I installed to a directory with a space in the path" is an example of >> CPAN Testers working as expected. It found and highlighted an annoying bug >> that the rest of us either ignore or work around. > > CPAN Testers reporting failures in every module they test and not stopping to > ask "Hey, is it possible that not everything else in the world is broken?" is > *not* an example of CPAN Testers working as expected. > > Environments where it's impossible even to *build* Perl modules are > unsuitable > for smoketest reporting, as they don't provide any useful information and > they make true failures much more difficult to see and believe.
One of the drawbacks of extensive automation is that special cases which require human intervention cannot be handled and can often go from minor annoyances to pandemics. The proper response is to either fix the automation to eliminate the necessity of human interaction or, if its a bug, fix the bug. A little work now and the system will run even more efficiently than before. These sorts of systemic failures might not be providing *new* information, but I wouldn't say it's not useful. One of the greatest problems facing collecting bug reports (or, in fact, any survey technique) is getting honest, unfiltered feedback. Humans have a tendency to filter out negative feedback, especially if it's perceived to be a known problem with a narrow focus. CPAN testers is giving us honest, unfiltered feedback. [1] We get to see the all the problems and the breadth of them. This makes it difficult to ignore long standing problems, like configuration level dependencies or non-Perl dependencies or failures on BSD or CPANPLUS fighting with Module::Build or all the other things WE know how to work around but others don't and make the end user experience annoying. Spaces in filenames are just the next problem to fix. Even in cases where Perl is broke, it's nice to know how it got into that state and if we can do anything about it. The price we pay is a little more email in the inbox. [2] I'm willing to pay that. [1] Within the range of the set of people doing the testing, of course. [2] Rather than each individual emailing the author how about CPAN Testers sends out a daily/weekly digest? Still push and mandatory, it's important that authors see this information, but at least it's just one email and it can skip a lot of the boilerplate text and provide a nice, tight summary. -- I have a date with some giant cartoon robots and booze.
