Re: a standard place for extra tests
Eric Wilhelm writes: Returning to something that came up in the 'testing pod' thread... # from Adam Kennedy on Tuesday 31 July 2007 08:23 pm: A small nit, if we end up converging around the use of a relatively standard directory name, can we at least use something relatively self-evident? For example author/? I'm definitely with Adam on this one. Nobody is intuitively know what at means, and it isn't obvious that there's a need for the directory name to be short. I would like to suggest that we adopt, recommend, (and support with tools, etc) a standard (convention) for storing additional tests. MyModule-v0.0.1/ xt/ author/ network/ Or something along those lines. At this point, 'author' is the only child directory I'm fairly certain of. I am certain that more than one 'extra tests' directory is needed, Why are you certain of this? It seems far from obvious to me. Anything under this directory isn't going to get run with 'normal' tests, which is an unusual requirement. As for the 'xt' name: Nobody is intuitively know what xt means, and it isn't obvious that there's a need for the directory name to be short. I think anything starting with 't' would get annoying as far as tab-complete goes. You'd only have to type 2 characters for the new directory name, and t itself would obviously remain a single character that doesn't even require pressing Tab. Smylers
Re: a standard place for extra tests
--- Smylers [EMAIL PROTECTED] wrote: child directory I'm fairly certain of. I am certain that more than one 'extra tests' directory is needed, Why are you certain of this? It seems far from obvious to me. Because there are plenty of extra tests we could conceivably have which might represent completely different sets of tests we want to run. Even if *you* don't have a need for it, others might. Imagine, for example: xt/author xt/smoke The latter of which is designed to run smoke tests against a wide variety of Perl versions. That should not be in the t/ directory, but it's also clearly not an author test. Regardless of the examples we come up with, I'm terribly leery of starting with the assumption that we will only ever need one extra test directory and therefore should not plan for more. Enforcing arbitrary limits at the beginning is not a good idea. As for the 'xt' name: Nobody is intuitively know what xt means Because 't/' is self-explanatory? Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/
Re: a standard place for extra tests
# from Smylers # on Thursday 16 August 2007 11:40 pm: I am certain that more than one 'extra tests' directory is needed, Why are you certain of this? Because I already have a use for three 'profiles' in one project and I can name a few others from elsewhere: 1. author (kwalitee, pod, etc) 2. gui 3. network 4. you must have an account/password on $external_service 5. postgres/mysql/whatever availability/setup/permissions 6. no modem attached to /dev/ttyS0 Thus: more than one. What I'm saying is that author is not enough (and/or not descriptive enough of a top-level directory of extra tests.) Anything under this directory isn't going to get run with 'normal' tests, which is an unusual requirement. It's not *that* unusual in the context of CPAN. Anything that requires intervention, setup, or access to a particular resource needs to be skipped or bypassed during unattended pre-installation testing. What I would ultimately like to see happen is that these subdirectories fall into some form of standard(ish) naming scheme which would allow testers and qa efforts to run particular subsets. That is, the smoke box (or rigorous end-user upgrade verification process) could set variables to test the author, 'gui', 'network', and 'postgres' tests[1] without stumbling into e.g. must have an X.com web-services account. Which brings up a good point: I think xt/foo.t should be discouraged. That is, if the tests in xt might assume any number of pre-configured things, the only safe assumption is test what you know you are prepared to test. Thus: prove -rb xt/author prove -rb xt/network prove -rb xt/exhaustive not `prove -rb xt/` -- which might crash for lack of gui before it gets around to testing the network bit. Therefore, 'xt/foo.t' is a problem because it has no category besides it is extra. [1] Yes, setup gets slightly tricky on the 'database', 'account', and 'devices' resources. I like to imagine that those could be fed information via some standard mechanism such as a per-distro config data-source -- the target usage there is more in the context of distributed teams of developers and rigorous end-users than cpan testers. --Eric -- The only thing that could save UNIX at this late date would be a new $30 shareware version that runs on an unexpanded Commodore 64. --Don Lancaster (1991) --- http://scratchcomputing.com ---
Re: a standard place for extra tests
# from Ovid # on Friday 17 August 2007 12:40 am: As for the 'xt' name: Nobody is intuitively know what xt means Because 't/' is self-explanatory? Yeah dude, it's like 't', but the x makes it eXtreme ;-) --Eric -- I eat your socks and you pay me. --The business sense of a very small goat. --- http://scratchcomputing.com ---
Re: a standard place for extra tests
On Thursday 16 August 2007 20:14:31 Eric Wilhelm wrote: I am certain that more than one 'extra tests' directory is needed, thus the thought to make them into subdirectories (objections?) (They cannot live under 't/' due to compatibility issues.) Which compatibility issues? I've used t/author/ for quite a while without any compatibility issues. -- c
Re: a standard place for extra tests
On 8/17/07, Eric Wilhelm [EMAIL PROTECTED] wrote: 1. author (kwalitee, pod, etc) 2. gui 3. network 4. you must have an account/password on $external_service 5. postgres/mysql/whatever availability/setup/permissions 6. no modem attached to /dev/ttyS0 Thus: more than one. What I'm saying is that author is not enough (and/or not descriptive enough of a top-level directory of extra tests.) I'm for author_t personally. xt is meaningless to me. Brainstorming other variations: * optional_t * extra_t (or just extra_tests) * devel_t I really like the idea of highlighting _t because it alludes to the convention of tests being in a t directory. (And, not that's not intuitive, but it's long established convention.) As for different profiles, there's no reason that one couldn't have: author_t/release author_t/kwalitee author_t/gui author_t/network author_t/runs_for_27_hours etc. Assuming that those are tested with prove anyway, there's no need for any standardization below the top level. And what percent of CPAN distros will really include so many author variations anyway? Let's not let the perfect design be the enemy of the good. There really isn't a need to standardize the top level, either, unless there is a goal of adding support for it to Module::Build and/or ExtUtils::MakeMaker. I think the important thing is establishing a *convention* of keeping tests intended for developers and not end users out of the t/ directory. David
Re: a standard place for extra tests
chromatic wrote: On Thursday 16 August 2007 20:14:31 Eric Wilhelm wrote: I am certain that more than one 'extra tests' directory is needed, thus the thought to make them into subdirectories (objections?) (They cannot live under 't/' due to compatibility issues.) Which compatibility issues? I've used t/author/ for quite a while without any compatibility issues. -- c Right, accept when your t/author tests are tests for you, the author and my t/author tests are for my Author class/classes/pages/etc. -=Chris signature.asc Description: OpenPGP digital signature
Re: a standard place for extra tests
chromatic wrote: On Friday 17 August 2007 09:31:50 Christopher H. Laco wrote: chromatic wrote: I've used t/author/ for quite a while without any compatibility issues. Right, accept when your t/author tests are tests for you, the author and my t/author tests are for my Author class/classes/pages/etc. Fair enough, but I'm not sure that this is a place where any heuristic will work perfectly. At some point, I suspect that someone will have to encode some information about which tests are functional and which tests are bonus, nice-to-know fun. -- c I'm always a fan of meta info. I'd have to say, I'd much rather there be a way to specify which tests are author tests within the t files themselves. Then let the files roam where they may, and let the tools, or the test runners figure out when to run the author tests rather than looking for a directory. -=Chris signature.asc Description: OpenPGP digital signature
Re: a standard place for extra tests
--- chromatic [EMAIL PROTECTED] wrote: Which compatibility issues? I've used t/author/ for quite a while without any compatibility issues. Because some people run recursive tests (what a bad name) by default and descend into t/ subdirectories. As a result, they'd be running author tests by default. That was the problem we're trying to get away from. I know that TAP::Parser was bitten by this when I had tests in t/sample-tests which were not intended to be run directly. I renamed them to not have .t extensions to get around this. Naturally, this breaks things in my vim setup :) It's a subtly different problem, but the same symptoms. Cheers, Ovid -- Buy the book - http://www.oreilly.com/catalog/perlhks/ Perl and CGI - http://users.easystreet.com/ovid/cgi_course/ Personal blog - http://publius-ovidius.livejournal.com/ Tech blog - http://use.perl.org/~Ovid/journal/
Re: a standard place for extra tests
On 8/17/07, Christopher H. Laco [EMAIL PROTECTED] wrote: chromatic wrote: On Thursday 16 August 2007 20:14:31 Eric Wilhelm wrote: I am certain that more than one 'extra tests' directory is needed, thus the thought to make them into subdirectories (objections?) (They cannot live under 't/' due to compatibility issues.) Which compatibility issues? I've used t/author/ for quite a while without any compatibility issues. -- c Right, accept when your t/author tests are tests for you, the author and my t/author tests are for my Author class/classes/pages/etc. Testing some Author stuff would be rarer than having author tests. So maybe we could standardize on something like t/author and when other value is desirable, a key/value pair may be specified in META.yml (and in Makefile.PL/Build.PL). # META.yml --- # the default location for author tests (not needed) author_tests: t/author # when something different is wanted author_tests: t/developer
Re: a standard place for extra tests
Adriano Ferreira wrote: On 8/17/07, Christopher H. Laco [EMAIL PROTECTED] wrote: chromatic wrote: On Thursday 16 August 2007 20:14:31 Eric Wilhelm wrote: I am certain that more than one 'extra tests' directory is needed, thus the thought to make them into subdirectories (objections?) (They cannot live under 't/' due to compatibility issues.) Which compatibility issues? I've used t/author/ for quite a while without any compatibility issues. -- c Right, accept when your t/author tests are tests for you, the author and my t/author tests are for my Author class/classes/pages/etc. Testing some Author stuff would be rarer than having author tests. So maybe we could standardize on something like t/author and when other value is desirable, a key/value pair may be specified in META.yml (and in Makefile.PL/Build.PL). # META.yml --- # the default location for author tests (not needed) author_tests: t/author # when something different is wanted author_tests: t/developer That convinces me even more that me other email was spot on imho. Looking for a directory and assuming what's in it are an authors' tests is pure guesswork. It's rude that you assume whatever directory I have isn't used for other purposes and will ignore those tests on make test. *.t files need a way to declare themselves as an authors' test. Let the runner/harness inspect/ask each *.t file if it's an author test or not, and run it or not...regardless of what directory it's in. signature.asc Description: OpenPGP digital signature
Re: a standard place for extra tests
On Friday 17 August 2007 11:17:10 Christopher H. Laco wrote: To whit: Who knows better whether foo.t is an author test or not [in the absence of the actual human author]... The harness, or foo.t? The author. The harness asking the t files will always be more accurate than the harness just find t files somewhere. It doesn't matter where the metadata is, as long as someone who knows (the author) enters that metadata in a way that makes sense to the tools... ... sort of like I've been doing for at least one and probably two years now with Module::Build and my custom disttest action, which I know I've mentioned a few times now, which leads me to wonder why people so conveniently forget it every time this discussion comes up. -- c
Re: a standard place for extra tests
Adriano Ferreira wrote: Testing some Author stuff would be rarer than having author tests. So maybe we could standardize on something like t/author and when other value is desirable, a key/value pair may be specified in META.yml (and in Makefile.PL/Build.PL). # META.yml --- # the default location for author tests (not needed) author_tests: t/author # when something different is wanted author_tests: t/developer Now you need a YAML parser to run tests, which may be fine.
Re: a standard place for extra tests
On 8/17/07, brian d foy [EMAIL PROTECTED] wrote: In article [EMAIL PROTECTED], Adriano Ferreira [EMAIL PROTECTED] wrote: Testing some Author stuff would be rarer than having author tests. So maybe we could standardize on something like t/author and when other value is desirable, a key/value pair may be specified in META.yml (and in Makefile.PL/Build.PL). ...smells a lot like Test::Manifest, which is just test. In t/test_manifest, you just list the files that you want. Optionally, you can set a test level and mark some tests only happening at certain levels. In some cases, I even auto-generate the t/test_manifest so it only has the files that I want to test in that situation. Other than t/test_manifest, I don't worry about magical names or changing all the tools for yet more special cases. It makes a lot of sense. Patterns such as t/*.t , t/**/*.t t/**/*.t minus t/author/*.t helps a quick startup, but it looks like a good idea to have an automated tool to expand this into longer and explicit lists of tests. From then on, additions may be manual like some usually do with MANIFEST. (Even though there is make manifest, you may not like the way it works because you have a bunch of files which you don't to get shipped in the distribution.) The only drawback is redundant information: there are the tests and the list of tests. Keeping tests in t/ with some fine way to discover what they are good for (according to Chris' idea) may be promising if not too magical.
Re: a standard place for extra tests
chromatic wrote: On Friday 17 August 2007 11:17:10 Christopher H. Laco wrote: To whit: Who knows better whether foo.t is an author test or not [in the absence of the actual human author]... The harness, or foo.t? The author. The harness asking the t files will always be more accurate than the harness just find t files somewhere. It doesn't matter where the metadata is, as long as someone who knows (the author) enters that metadata in a way that makes sense to the tools... ... sort of like I've been doing for at least one and probably two years now with Module::Build and my custom disttest action, which I know I've mentioned a few times now, which leads me to wonder why people so conveniently forget it every time this discussion comes up. *cough* selective memory *cough*...myself included. :-) signature.asc Description: OpenPGP digital signature
Re: a standard place for extra tests
In article [EMAIL PROTECTED], Adriano Ferreira [EMAIL PROTECTED] wrote: Testing some Author stuff would be rarer than having author tests. So maybe we could standardize on something like t/author and when other value is desirable, a key/value pair may be specified in META.yml (and in Makefile.PL/Build.PL). ...smells a lot like Test::Manifest, which is just test. In t/test_manifest, you just list the files that you want. Optionally, you can set a test level and mark some tests only happening at certain levels. In some cases, I even auto-generate the t/test_manifest so it only has the files that I want to test in that situation. Other than t/test_manifest, I don't worry about magical names or changing all the tools for yet more special cases.
Re: a standard place for extra tests
# from chromatic # on Friday 17 August 2007 11:28 am: ... sort of like I've been doing for at least one and probably two years now with Module::Build and my custom disttest action, which I know I've mentioned a few times now, which leads me to wonder why people so conveniently forget it every time this discussion comes up. Because 1. it doesn't play nicely with recursive tests[1] 2. it requires a custom disttest action[2] [1] Unless you filter out t/author (and t/gui (and t/network (and ...))) It seems to me that having a whole directory of and also would more easily allow one to cherry-pick which extras to run while leaving t/ as the run all of these, recursively spot. [2] And that wouldn't be so bad if Module::Build had a plugin mechanism, but it doesn't. Thus, avoiding custom actions for something where they could be avoided is nice because it reduces the (currently necessary) copy/paste code reuse. --Eric -- So malloc calls a timeout and starts rummaging around the free chain, sorting things out, and merging adjacent small free blocks into larger blocks. This takes 3 1/2 days. --Joel Spolsky --- http://scratchcomputing.com ---
Re: a standard place for extra tests
# from Michael G Schwern # on Friday 17 August 2007 11:13 am: # when something different is wanted author_tests: t/developer Now you need a YAML parser to run tests, which may be fine. But you also need to update all of the tools, which may not. --Eric -- But as soon as you hear the Doppler shift dropping in pitch, you know that they're probably going to miss your house, because if they were on a collision course with your house, the pitch would stay the same until impact. As I said, that one's subtle. --Larry Wall --- http://scratchcomputing.com ---
Re: a standard place for extra tests
On Friday 17 August 2007 12:52:54 Eric Wilhelm wrote: # from chromatic # on Friday 17 August 2007 11:28 am: ... sort of like I've been doing for at least one and probably two years now with Module::Build and my custom disttest action, which I know I've mentioned a few times now, which leads me to wonder why people so conveniently forget it every time this discussion comes up. Because 1. it doesn't play nicely with recursive tests[1] It's okay to run everything in t/ recursively is a heuristic. You can tell it's a heuristic because sometimes it's wrong. Plus, I can't think of a single time when I've ever needed it. That doesn't mean it's unnecessary, but rather that there's an interesting discussion about test organization that we seem to be failing to have. 2. it requires a custom disttest action[2] Avoiding heuristics requires some sort of metadata. Now a custosm build action may be *bad* for other reasons, but it's not ambiguous, like a heuristic. Perhaps we need to define: 1) what tests should run by default 2) who gets to decide what by default means 3) what other groups need to run non-default tests 3a) and what those non-default tests are 3b) and why they need to run them 4) what options are there for distinguishing between default and non-default tests 5) which answer to #4 sucks the least I respectfully suggest that any answer to #4 which includes a heuristic will fail #5 spectacularly. -- c
Re: a standard place for extra tests
# from chromatic # on Friday 17 August 2007 01:05 pm: On Friday 17 August 2007 12:52:54 Eric Wilhelm wrote: 1. it doesn't play nicely with recursive tests[1] It's okay to run everything in t/ recursively is a heuristic. You can tell it's a heuristic because sometimes it's wrong. That's not what I meant by doesn't play nicely. I mean, my project cannot use recursive tests *and* have author-only tests in t/author without jumping through hoops. Hoops include: a custom test action, special options (or patches) to prove and runtests, etc. Avoiding heuristics requires some sort of metadata. Now a custosm build action may be *bad* for other reasons, but it's not ambiguous, like a heuristic. Let's forget for a second that I'm trying to assume anything about t/ or any other directory. The simple fact is that the current tools will behave more nicely if everything in $the_one_directory is of one sort and everything in $the_other_directory is of the other sort. Without that, we need code to sort the one from the other because they were all thrown into the same bag. Then we need an XML config file for that code and etc, etc. Perhaps we need to define: snip yet-another discussion entirely 4) what options are there for distinguishing... 5) which answer to #4 sucks the least ...any answer to #4 which includes a heuristic will fail #5 Any answer which requires a config-file and modifications to existing tools will also fail. So, maybe we have to say extra_tests: xt (or even some deeper data-structure) to avoid a heuristic. Fine. What would that data-structure look like? extra_tests: base_dir: xt profiles: author: author network: network gui: gui exhaustive: exhaustive The human will want the directory structure to mirror the names of the profiles, right? So, now the human has to type more, but there's no heuristic (well, except for the names of the keys in the config file.) --Eric -- Speak softly and carry a big carrot. --- http://scratchcomputing.com ---
Re: a standard place for extra tests
# from David Golden # on Friday 17 August 2007 10:03 am: On 8/17/07, Eric Wilhelm [EMAIL PROTECTED] wrote: 1. author (kwalitee, pod, etc) 2. gui 3. network 4. you must have an account/password on $external_service 5. postgres/mysql/whatever availability/setup/permissions 6. no modem attached to /dev/ttyS0 ... As for different profiles, there's no reason that one couldn't have: author_t/release author_t/kwalitee author_t/gui author_t/network author_t/runs_for_27_hours etc. There really isn't a need to standardize the top level, either, unless there is a goal of adding support for it to Module::Build and/or ExtUtils::MakeMaker. Yes, that is the goal. The toplevel directory should be the same. While the config-file approach would allow the name to vary, I think that is over-engineering given that 't/' is an invariant assumed directory name. there's no need for any standardization below the top level. Except in that it would conveniently allow smokers to sign up for testing *specific* extra functionality. That is, yes, I have network connectivity means it is ok to run the xtra_t/network/ tests. I'm not saying we have to have a committee establish *all* of the possible profile names. If we can just say 'author', 'network', 'exhaustive', and 'gui' for now, I'll be happy. Are we going to have to argue over those names too? I think the important thing is establishing a *convention* of keeping tests intended for developers and not end users out of the t/ directory. Yes. What I would like to do to motivate that is to provide the incentive of a recommended alternative (one which is supported by tools, smoke testers, etc.) And what percent of CPAN distros will really include so many author variations anyway? Let's not let the perfect design be the enemy of the good. As long as we don't hastily shoot ourselves in the foot. --Eric -- To succeed in the world, it is not enough to be stupid, you must also be well-mannered. --Voltaire --- http://scratchcomputing.com ---
Re: a standard place for extra tests
# from chromatic # on Friday 17 August 2007 01:57 pm: On Friday 17 August 2007 13:20:39 Eric Wilhelm wrote: my project cannot use recursive tests *and* have author-only tests in t/author without jumping through hoops. How are you getting recursive tests by default anywhere? I'm not getting them by default. I'm *using* the recursive functionality for my test suite. I cannot use t/author and t/normal_stuff without filtering-out t/author. I don't want to have to filter-out t/author and t/gui and t/network and t/account. I want it to be convenient to run all of the normal tests and all of the extra tests and a few profiles from the extra tests and all of the tests I want it to be convenient with `./Build` and `make` and `prove` and `runtests`. prove -rb t prove -rb xt prove -rb xt/author xt/network xt/gui prove -rb t xt I just temporarily changed the Build.PL for Test::MockObject (which stores author-only tests in t/developer/) not to filter out that directory in the find_test_files() files method, and the developer tests still didn't run. Now set 'recursive_test_files = 1' (like I do) and try it. Then run `prove -rb t` and `runtests -rb t`. --Eric -- ...the bourgeoisie were hated from both ends: by the proles, because they had all the money, and by the intelligentsia, because of their tendency to spend it on lawn ornaments. --Neal Stephenson --- http://scratchcomputing.com ---
Re: a standard place for extra tests
Eric Wilhelm writes: # from Smylers # on Thursday 16 August 2007 11:40 pm: I am certain that more than one 'extra tests' directory is needed, Why are you certain of this? Because I already have a use for three 'profiles' in one project and I can name a few others from elsewhere: 1. author (kwalitee, pod, etc) Yep. 2. gui Why can't gui tests simply be in t/ and be skipped if the appropriate environment isn't available? That way all users who are in that particular environment run the tests, rather than only those who've discovered the xt/gui/ directory. 3. network Ditto -- why isn't skipping enough? 4. you must have an account/password on $external_service 5. postgres/mysql/whatever availability/setup/permissions There are existing modules user tests which cope with those, either by prompting or environment variables -- defaulting to skipping the tests, of course. Possibly we need a better (or standardized) way of getting information from users running tests in these situations, and for safely skipping by default. (When trying to get a bunch of Cpan modules to install unattendedly I discovered they collectively had many ways of seeking attention; having none of them waiting at a prompt involved all three of setting an environment variable, passing an option to Makefile.PL, and piping yes '' into it.) 6. no modem attached to /dev/ttyS0 That could probably be treated similarly to the website case, defaulting to presuming no modem is there unless explicitly told so. What I would ultimately like to see happen is that these subdirectories fall into some form of standard(ish) naming scheme which would allow testers and qa efforts to run particular subsets. That is, the smoke box (or rigorous end-user upgrade verification process) could set variables to test the author, 'gui', 'network', and 'postgres' tests[1] without stumbling into e.g. must have an X.com web-services account. That requires a central standardized repository of permitted subdirectories and what needs to be done to run the tests there. Surely it's better to let each distribution keep control of this, by specifying in code what's needed to run those particular tests (and skipping otherwise)? Smylers
Re: a standard place for extra tests
# from Adriano Ferreira # on Friday 17 August 2007 11:49 am: The only drawback is redundant information: there are the tests and the list of tests. Keeping tests in t/ with some fine way to discover what they are good for (according to Chris' idea) may be promising if not too magical. I think keeping extra tests in t/ is too magical for the existing tools. After all, I have tried it. In order to have *both* recursive tests (which many larger projects do) *and* extra tests, you have to have magic. My earlier attempt at this was to add test profiles to Module::Build via file extensions which were declared in the Build.PL. The trouble is that those don't work with EU::MM or prove or runtests (or ack (and even vi in some configurations.)) My conclusion from that, was this xt/foo/ thing, which requires zero coding (or pasting) and works fine with all of the existing tools. The xt/foo/ thing solves that redundant information problem too. I thought to myself, if this works so well for what I'm doing, wouldn't it be great if it became a sort of standard? --Eric -- Ignorance more frequently begets confidence than does knowledge. -- Charles Darwin --- http://scratchcomputing.com ---
