Eric Wilhelm writes: > # from Smylers > # on Thursday 16 August 2007 11:40 pm: > > > > I am certain that more than one 'extra tests' directory is needed, > > > > Why are you certain of this? > > Because I already have a use for three 'profiles' in one project and I > can name a few others from elsewhere: > > 1. author (kwalitee, pod, etc)
Yep. > 2. gui Why can't gui tests simply be in t/ and be skipped if the appropriate environment isn't available? That way all users who are in that particular environment run the tests, rather than only those who've discovered the xt/gui/ directory. > 3. network Ditto -- why isn't skipping enough? > 4. you must have an account/password on $external_service > 5. postgres/mysql/whatever availability/setup/permissions There are existing modules user tests which cope with those, either by prompting or environment variables -- defaulting to skipping the tests, of course. Possibly we need a better (or standardized) way of getting information from users running tests in these situations, and for safely skipping by default. (When trying to get a bunch of Cpan modules to install unattendedly I discovered they collectively had many ways of seeking attention; having none of them waiting at a prompt involved all three of setting an environment variable, passing an option to Makefile.PL, and piping yes '' into it.) > 6. no modem attached to /dev/ttyS0 That could probably be treated similarly to the website case, defaulting to presuming no modem is there unless explicitly told so. > What I would ultimately like to see happen is that these > subdirectories fall into some form of standard(ish) naming scheme > which would allow testers and qa efforts to run particular subsets. > That is, the smoke box (or rigorous end-user upgrade verification > process) could set variables to test the author, 'gui', 'network', and > 'postgres' tests[1] without stumbling into e.g. "must have an X.com > web-services account". That requires a central standardized repository of permitted subdirectories and what needs to be done to run the tests there. Surely it's better to let each distribution keep control of this, by specifying in code what's needed to run those particular tests (and skipping otherwise)? Smylers
