Hi all, So here's the situation as it I see it. There's a general consensus that we need more flexibility in how we run our tests. This is because there are two broad categories of tests:
1. Tests which absolutely, definitely, unquestionably, beyond a shadow of a doubt, MUST NEVER FAIL. I'll call these MNF tests. 2. Everything else. MNF tests mean that there's a real, honest-to-god problem in the software. It doesn't mean that someone forgot =back tag. It doesn't mean that network tests fail because the network went down partway through the test. It doesn't mean that yes, they have your favorite GUI module installed but aren't running X. If an MNF test fails, you can't trust the module and the author should be notified. We might disagree with what constitutes an MNF test. I say this to avoid email from folks who might disagree with some of what I describe above, but that's *not* the issue here and I would hate for us to get sidetracked. The issue is that we generally agree that there are MNF tests and all other tests. What we appear to disagree on is how to handle non-MNF tests. The main dividing line seems to be whether we should have a separate test directory. There are suggestions requiring that people consider things like Test::Manifest, YAML, possibly XML or Ant-style tools. These are fine suggestions, but I don't think they're our best option. Too many folks have trouble installing extra modules or balk at installing extra modules or have strange business rules regarding what they can and cannot install. You can argue about how stupid this is, but there's a difference between an ideal world and the real one. There's also the the question of recursive tests. Right now, how do people run tests? Sometimes via prove, sometimes via runtests, sometimes via './Build test', sometims via 'make test', sometimes via smolder, sometimes via custom tools built in-house or via TAP::Parser's extensibility. I've had Adrian Howard report a bug in TAP::Parser because he runs tests recursively by default (Adrian, correct me if I'm wrong) and I'm sure he's not the only one. By insisting that all tests must always be in 't/', we're either insisting upon maintenance of the admittedly unsatisfactory status quo or we have the potential of having to build new tools and possibly create more bugs and then the daunting task of bringing the Perl community up to speed. We cannot control how people are going to run tests and suggestions on changing how we currently do things has a great potential to impact that. We need to accept that. If we can swing it, whatever we do needs to be as seamless as possible and backwards compatile. By historical accident, virtually all tools expect that tests are in test.pl or the 't/' directory. Merely by taking advantage of this and using a different, standard directory, we avoid breaking any of those tools. A new directory: 1. addresses the current problem. 2. doesn't require new tools. 3. doesn't break old installations. 4. still allows us to adopt solutions previously recommended! We know the downside of just sticking to the 't/' directory is that having expectations that we can and must change large portions of the Perl community is a lousy expectation. I'm sure we can think of plenty of examples of this :) So please, if you want to shove all MNF and non-MNF tests in 't/', list its benefits (considering the problems I've listed above) and then list reasons why separating them into a new directory is such a bad idea. Once you can list the strengths and weaknesses of each proposal, you can try and weigh them. So far I've not seen a single compelling argument *for* sticking everything in the 't/' directory and I haven't seen a single compelling argument *against* a new directory. Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/
