a "standard" place for extra tests

[Eric Wilhelm]

Returning to something that came up in the 'testing pod' thread...

# from Adam Kennedy on Tuesday 31 July 2007 08:23 pm:
>A small nit, if we end up converging around the use of a relatively
>standard directory name, can we at least use something relatively
>self-evident? For example "author/"?

I would like to suggest that we adopt, recommend, (and support with 
tools, etc) a "standard" (convention) for storing additional tests.

  MyModule-v0.0.1/
xt/
  author/
  network/

Or something along those lines.  At this point, 'author' is the only 
child directory I'm fairly certain of.

I am certain that more than one 'extra tests' directory is needed, thus 
the thought to make them into subdirectories (objections?)

(They cannot live under 't/' due to compatibility issues.)

As for the 'xt' name:  it tends to sort pretty soon after 't', as in 
'these come next'.  I think anything starting with 't' would get 
annoying as far as tab-complete goes.  Similarly, 't2' might conflict 
with primitive version-control habits.

Better suggestions?

Aside:  I've been playing with using non-'.t' file extensions for this 
and storing them under t/, but this requires extra support from 
Module::Build, runtests, prove, ack, etc.  The only drawback to putting 
them outside of the 't/' directory seems to be that refactoring habits 
like 'grep $method -r t' might need adjustment.  Vs. this "more .t 
files one directory to the right" approach where you can easily get a 
bug report by asking the user to simply run `prove xt/network/`.

--Eric
-- 
"Matter will be damaged in direct proportion to its value."
--Murphy's Constant
---
http://scratchcomputing.com
---

Re: a "standard" place for extra tests

[Smylers]

Eric Wilhelm writes:

> Returning to something that came up in the 'testing pod' thread...
> 
> # from Adam Kennedy on Tuesday 31 July 2007 08:23 pm:
>
> > A small nit, if we end up converging around the use of a relatively
> > standard directory name, can we at least use something relatively
> > self-evident? For example "author/"?

I'm definitely with Adam on this one.  Nobody is intuitively know what
"at" means, and it isn't obvious that there's a need for the directory
name to be short.

> I would like to suggest that we adopt, recommend, (and support with 
> tools, etc) a "standard" (convention) for storing additional tests.
> 
>   MyModule-v0.0.1/
> xt/
>   author/
>   network/
> 
> Or something along those lines.  At this point, 'author' is the only 
> child directory I'm fairly certain of.  I am certain that more than
> one 'extra tests' directory is needed,

Why are you certain of this?  It seems far from obvious to me.  Anything
under this directory isn't going to get run with 'normal' tests, which
is an unusual requirement.

> As for the 'xt' name:

Nobody is intuitively know what "xt" means, and it isn't obvious that
there's a need for the directory name to be short.

> I think anything starting with 't' would get annoying as far as
> tab-complete goes.

You'd only have to type 2 characters for the new directory name, and "t"
itself would obviously remain a single character that doesn't even
require pressing Tab.

Smylers

Re: a "standard" place for extra tests

[Ovid]

--- Smylers <[EMAIL PROTECTED]> wrote:

> > child directory I'm fairly certain of.  I am certain that more than
> > one 'extra tests' directory is needed,
> 
> Why are you certain of this?  It seems far from obvious to me. 

Because there are plenty of "extra" tests we could conceivably have
which might represent completely different sets of tests we want to
run.  Even if *you* don't have a need for it, others might.  Imagine,
for example:

 xt/author
 xt/smoke

The latter of which is designed to run smoke tests against a wide
variety of Perl versions.  That should not be in the t/ directory, but
it's also clearly not an author test.

Regardless of the examples we come up with, I'm terribly leery of
starting with the assumption that "we will only ever need one extra
test directory and therefore should not plan for more".  Enforcing
arbitrary limits at the beginning is not a good idea.

> > As for the 'xt' name:
> 
> Nobody is intuitively know what "xt" means

Because 't/' is self-explanatory?

Cheers,
Ovid

--
Buy the book  - http://www.oreilly.com/catalog/perlhks/
Perl and CGI  - http://users.easystreet.com/ovid/cgi_course/
Personal blog - http://publius-ovidius.livejournal.com/
Tech blog - http://use.perl.org/~Ovid/journal/

Re: a "standard" place for extra tests

[Eric Wilhelm]

# from Smylers
# on Thursday 16 August 2007 11:40 pm:

>> I am certain that more than one 'extra tests' directory is needed,
>
>Why are you certain of this?

Because I already have a use for three 'profiles' in one project and I 
can name a few others from elsewhere:

  1.  author (kwalitee, pod, etc)
  2.  gui
  3.  network
  4.  you must have an account/password on $external_service
  5.  postgres/mysql/whatever availability/setup/permissions
  6.  no modem attached to /dev/ttyS0

Thus: "more than one".  What I'm saying is that "author" is not enough 
(and/or not descriptive enough of a top-level directory of "extra 
tests".)

>  Anything under this directory isn't going to get run with 'normal'
> tests, which is an unusual requirement.

It's not *that* unusual in the context of CPAN.  Anything that requires 
intervention, setup, or access to a particular resource needs to be 
skipped or bypassed during unattended pre-installation testing.

What I would ultimately like to see happen is that these subdirectories 
fall into some form of standard(ish) naming scheme which would allow 
testers and qa efforts to run particular subsets.  That is, the smoke 
box (or rigorous end-user upgrade verification process) could set 
variables to test the author, 'gui', 'network', and 'postgres' tests[1] 
without stumbling into e.g. "must have an X.com web-services account".

Which brings up a good point:  I think "xt/foo.t" should be discouraged.  
That is, if the tests in xt might assume any number of pre-configured 
things, the only safe assumption is "test what you know you are 
prepared to test".

Thus:
  prove -rb xt/author
  prove -rb xt/network
  prove -rb xt/exhaustive

not `prove -rb xt/` -- which might crash for lack of gui before it gets 
around to testing the network bit.  Therefore, 'xt/foo.t' is a problem 
because it has no category besides "it is extra".

[1] Yes, setup gets slightly tricky on the 'database', 'account', and 
'devices' resources.  I like to imagine that those could be fed 
information via some standard mechanism such as a per-distro config 
data-source -- the target usage there is more in the context of 
distributed teams of developers and rigorous end-users than cpan 
testers.

--Eric
-- 
The only thing that could save UNIX at this late date would be a new $30
shareware version that runs on an unexpanded Commodore 64.
--Don Lancaster (1991)
---
http://scratchcomputing.com
---

Re: a "standard" place for extra tests

[Eric Wilhelm]

# from Ovid
# on Friday 17 August 2007 12:40 am:

>> > As for the 'xt' name:
>>
>> Nobody is intuitively know what "xt" means
>
>Because 't/' is self-explanatory?

Yeah dude, it's like 't', but the x makes it eXtreme ;-)

--Eric
-- 
I eat your socks and you pay me.
--The business sense of a very small goat.
---
http://scratchcomputing.com
---

Re: a "standard" place for extra tests

[chromatic]

On Thursday 16 August 2007 20:14:31 Eric Wilhelm wrote:

> I am certain that more than one 'extra tests' directory is needed, thus
> the thought to make them into subdirectories (objections?)
>
> (They cannot live under 't/' due to compatibility issues.)

Which compatibility issues?  I've used t/author/ for quite a while without any 
compatibility issues.

-- c

Re: a "standard" place for extra tests

[chromatic]

On Friday 17 August 2007 09:31:50 Christopher H. Laco wrote:

> chromatic wrote:

> > I've used t/author/ for quite a while without any compatibility issues.

> Right, accept when your t/author tests are tests for you, the author and
> my t/author tests are for my Author class/classes/pages/etc.

Fair enough, but I'm not sure that this is a place where any heuristic will 
work perfectly.  At some point, I suspect that someone will have to encode 
some information about which tests are functional and which tests are bonus, 
nice-to-know fun.

-- c

Re: a "standard" place for extra tests

[Smylers]

Ovid writes:

> --- Smylers <[EMAIL PROTECTED]> wrote:
> 
> > > child directory I'm fairly certain of.  I am certain that more
> > > than one 'extra tests' directory is needed,
> > 
> > Why are you certain of this?  It seems far from obvious to me. 
> 
> for example:
> 
>  xt/smoke
> 
> The latter of which is designed to run smoke tests against a wide
> variety of Perl versions.  That should not be in the t/ directory, but
> it's also clearly not an author test.

Just to be clear (because this is an area I'm ignorant about, not
because there's anything wrong with your explanation) this is for the
situation when testing a module with several Perl versions requires
something other than merely running the standard tests on each Perl
version, and as such these tests wouldn't routinely be run by the author
(because he's just developing with one version of Perl to hand) but are
available for use by people specifically doing this kind of smoke
testing -- have I got that right?

> Regardless of the examples we come up with, I'm terribly leery of
> starting with the assumption that "we will only ever need one extra
> test directory and therefore should not plan for more".

Sure.  I didn't say I was _against_ this level of hierarchy, nor that I
was certain we wouldn't need.  I merely asked why whoever-it-was-whose-
name-you-snipped-from-the-above was certain we would -- so as to provide
guidance to those of us who haven't yet worked it out.

> Enforcing arbitrary limits at the beginning is not a good idea.

Indeed not, but nor is creating lots of structure that isn't going to be
needed; it's a balance.

> > > As for the 'xt' name:
> > 
> > Nobody is intuitively know what "xt" means
> 
> Because 't/' is self-explanatory?

Of course not, but that's no reason to make things worse when there are
plenty of other more descriptive potential directory names out there.

Smylers

Re: a "standard" place for extra tests

[David Golden]

On 8/17/07, Eric Wilhelm <[EMAIL PROTECTED]> wrote:
>   1.  author (kwalitee, pod, etc)
>   2.  gui
>   3.  network
>   4.  you must have an account/password on $external_service
>   5.  postgres/mysql/whatever availability/setup/permissions
>   6.  no modem attached to /dev/ttyS0
>
> Thus: "more than one".  What I'm saying is that "author" is not enough
> (and/or not descriptive enough of a top-level directory of "extra
> tests".)

I'm for "author_t" personally. "xt" is meaningless to me.
Brainstorming other variations:

* optional_t
* extra_t (or just extra_tests)
* devel_t

I really like the idea of highlighting "_t" because it alludes to the
convention of tests being in a "t" directory.  (And, not that's not
intuitive, but it's long established convention.)

As for different profiles, there's no reason that one couldn't have:

author_t/release
author_t/kwalitee
author_t/gui
author_t/network
author_t/runs_for_27_hours
etc.

Assuming that those are tested with prove anyway, there's no need for
any standardization below the top level.  And what percent of CPAN
distros will really include so many author variations anyway?  Let's
not let the perfect design be the enemy of the good.

There really isn't a need to standardize the top level, either, unless
there is a goal of adding support for it to Module::Build and/or
ExtUtils::MakeMaker.

I think the important thing is establishing a *convention* of keeping
tests intended for developers and not end users out of the "t/"
directory.

David

Re: a "standard" place for extra tests

[Christopher H. Laco]

chromatic wrote:
> On Thursday 16 August 2007 20:14:31 Eric Wilhelm wrote:
> 
>> I am certain that more than one 'extra tests' directory is needed, thus
>> the thought to make them into subdirectories (objections?)
>>
>> (They cannot live under 't/' due to compatibility issues.)
> 
> Which compatibility issues?  I've used t/author/ for quite a while without 
> any 
> compatibility issues.
> 
> -- c
> 
> 

Right, accept when your t/author tests are tests for you, the author and
my t/author tests are for my Author class/classes/pages/etc.

-=Chris



signature.asc
Description: OpenPGP digital signature

Re: a "standard" place for extra tests

[Christopher H. Laco]

chromatic wrote:
> On Friday 17 August 2007 09:31:50 Christopher H. Laco wrote:
> 
>> chromatic wrote:
> 
>>> I've used t/author/ for quite a while without any compatibility issues.
> 
>> Right, accept when your t/author tests are tests for you, the author and
>> my t/author tests are for my Author class/classes/pages/etc.
> 
> Fair enough, but I'm not sure that this is a place where any heuristic will 
> work perfectly.  At some point, I suspect that someone will have to encode 
> some information about which tests are functional and which tests are bonus, 
> nice-to-know fun.
> 
> -- c
> 
> 

I'm always a fan of meta info. I'd have to say, I'd much rather there be
a way to specify which tests are author tests within the t files
themselves. Then let the files roam where they may, and let the tools,
or the test runners figure out when to run the author tests rather than
looking for a directory.

-=Chris



signature.asc
Description: OpenPGP digital signature

Re: a "standard" place for extra tests

[Ovid]

--- chromatic <[EMAIL PROTECTED]> wrote:

> Which compatibility issues?  I've used t/author/ for quite a while
> without any 
> compatibility issues.

Because some people run "recursive" tests (what a bad name) by default
and descend into t/ subdirectories.  As a result, they'd be running
author tests by default.  That was the problem we're trying to get away
from.

I know that TAP::Parser was bitten by this when I had tests in
t/sample-tests which were not intended to be run directly. I renamed
them to not have .t extensions to get around this.  Naturally, this
breaks things in my vim setup :)  It's a subtly different problem, but
the same symptoms.

Cheers,
Ovid

--
Buy the book  - http://www.oreilly.com/catalog/perlhks/
Perl and CGI  - http://users.easystreet.com/ovid/cgi_course/
Personal blog - http://publius-ovidius.livejournal.com/
Tech blog - http://use.perl.org/~Ovid/journal/

Re: a "standard" place for extra tests

[Adriano Ferreira]

On 8/17/07, chromatic <[EMAIL PROTECTED]> wrote:
> On Friday 17 August 2007 09:31:50 Christopher H. Laco wrote:
>
> > chromatic wrote:
>
> > > I've used t/author/ for quite a while without any compatibility issues.
>
> > Right, accept when your t/author tests are tests for you, the author and
> > my t/author tests are for my Author class/classes/pages/etc.
>
> Fair enough, but I'm not sure that this is a place where any heuristic will
> work perfectly.  At some point, I suspect that someone will have to encode
> some information about which tests are functional and which tests are bonus,
> nice-to-know fun.

There was no heuristics before, but running t/*.t (which will leave
alone subdirectories). But according to Ovid some run "recursive"
tests by default
and descend into t/ subdirectories, which is not perfect as chromatic
pointed. They should not do that without discrimination. They could
use some method to determine the full set of tests ( t/**/*.t ) minus
special tests (which could be found in meta-data), constructing a
file-set like:






That can be done with Ant. Why not do it with Perl?

Re: a "standard" place for extra tests

[Adriano Ferreira]

On 8/17/07, Christopher H. Laco <[EMAIL PROTECTED]> wrote:
> chromatic wrote:
> > On Thursday 16 August 2007 20:14:31 Eric Wilhelm wrote:
> >
> >> I am certain that more than one 'extra tests' directory is needed, thus
> >> the thought to make them into subdirectories (objections?)
> >>
> >> (They cannot live under 't/' due to compatibility issues.)
> >
> > Which compatibility issues?  I've used t/author/ for quite a while without 
> > any
> > compatibility issues.
> >
> > -- c
> >
> >
>
> Right, accept when your t/author tests are tests for you, the author and
> my t/author tests are for my Author class/classes/pages/etc.

Testing some Author stuff would be rarer than having author tests. So
maybe we could standardize on something like "t/author" and when other
value is desirable, a key/value pair may be specified in META.yml (and
in Makefile.PL/Build.PL).

# META.yml
---

# the default location for author tests (not needed)
author_tests: t/author

# when something different is wanted
author_tests: t/developer

Re: a "standard" place for extra tests

[Christopher H. Laco]

Adriano Ferreira wrote:
> On 8/17/07, Christopher H. Laco <[EMAIL PROTECTED]> wrote:
>> chromatic wrote:
>>> On Thursday 16 August 2007 20:14:31 Eric Wilhelm wrote:
>>>
 I am certain that more than one 'extra tests' directory is needed, thus
 the thought to make them into subdirectories (objections?)

 (They cannot live under 't/' due to compatibility issues.)
>>> Which compatibility issues?  I've used t/author/ for quite a while without 
>>> any
>>> compatibility issues.
>>>
>>> -- c
>>>
>>>
>> Right, accept when your t/author tests are tests for you, the author and
>> my t/author tests are for my Author class/classes/pages/etc.
> 
> Testing some Author stuff would be rarer than having author tests. So
> maybe we could standardize on something like "t/author" and when other
> value is desirable, a key/value pair may be specified in META.yml (and
> in Makefile.PL/Build.PL).
> 
> # META.yml
> ---
> 
> # the default location for author tests (not needed)
> author_tests: t/author
> 
> # when something different is wanted
> author_tests: t/developer
> 
> 

That convinces me even more that me other email was spot on imho.
Looking for a directory and assuming what's in it are an authors' tests
is pure guesswork. It's rude that you assume whatever directory I have
isn't used for other purposes and will ignore those tests on make test.

*.t files need a way to declare themselves as an authors' test. Let the
runner/harness inspect/ask each *.t file if it's an author test or not,
and run it or not...regardless of what directory it's in.



signature.asc
Description: OpenPGP digital signature

Re: a "standard" place for extra tests

[chromatic]

On Friday 17 August 2007 11:17:10 Christopher H. Laco wrote:

> To whit: Who knows better whether foo.t is an author test or not [in the
> absence of the actual human author]... The harness, or foo.t?

The author.

> The harness asking the t files will always be more accurate than the harness
> just find t files somewhere.

It doesn't matter where the metadata is, as long as someone who knows (the 
author) enters that metadata in a way that makes sense to the tools...

... sort of like I've been doing for at least one and probably two years now 
with Module::Build and my custom disttest action, which I know I've mentioned 
a few times now, which leads me to wonder why people so conveniently forget 
it every time this discussion comes up.

-- c

Re: a "standard" place for extra tests

[Christopher H. Laco]

Christopher H. Laco wrote:
> Adriano Ferreira wrote:
>> On 8/17/07, Christopher H. Laco <[EMAIL PROTECTED]> wrote:
>>> chromatic wrote:
 On Thursday 16 August 2007 20:14:31 Eric Wilhelm wrote:

> I am certain that more than one 'extra tests' directory is needed, thus
> the thought to make them into subdirectories (objections?)
>
> (They cannot live under 't/' due to compatibility issues.)
 Which compatibility issues?  I've used t/author/ for quite a while without 
 any
 compatibility issues.

 -- c


>>> Right, accept when your t/author tests are tests for you, the author and
>>> my t/author tests are for my Author class/classes/pages/etc.
>> Testing some Author stuff would be rarer than having author tests. So
>> maybe we could standardize on something like "t/author" and when other
>> value is desirable, a key/value pair may be specified in META.yml (and
>> in Makefile.PL/Build.PL).
>>
>> # META.yml
>> ---
>>
>> # the default location for author tests (not needed)
>> author_tests: t/author
>>
>> # when something different is wanted
>> author_tests: t/developer
>>
>>
> 
> That convinces me even more that me other email was spot on imho.
> Looking for a directory and assuming what's in it are an authors' tests
> is pure guesswork. It's rude that you assume whatever directory I have
> isn't used for other purposes and will ignore those tests on make test.
> 
> *.t files need a way to declare themselves as an authors' test. Let the
> runner/harness inspect/ask each *.t file if it's an author test or not,
> and run it or not...regardless of what directory it's in.
> 

To whit: Who knows better whether foo.t is an author test or not [in the
absence of the actual human author]... The harness, or foo.t? The
harness asking the t files will always be more accurate than the harness
just find t files somewhere.

But I digress...



signature.asc
Description: OpenPGP digital signature

Re: a "standard" place for extra tests

[Michael G Schwern]

Adriano Ferreira wrote:
> Testing some Author stuff would be rarer than having author tests. So
> maybe we could standardize on something like "t/author" and when other
> value is desirable, a key/value pair may be specified in META.yml (and
> in Makefile.PL/Build.PL).
> 
> # META.yml
> ---
> 
> # the default location for author tests (not needed)
> author_tests: t/author
> 
> # when something different is wanted
> author_tests: t/developer

Now you need a YAML parser to run tests, which may be fine.

Re: a "standard" place for extra tests

[Adriano Ferreira]

On 8/17/07, brian d foy <[EMAIL PROTECTED]> wrote:
> In article
> <[EMAIL PROTECTED]>, Adriano
> Ferreira <[EMAIL PROTECTED]> wrote:
>
>
> > Testing some Author stuff would be rarer than having author tests. So
> > maybe we could standardize on something like "t/author" and when other
> > value is desirable, a key/value pair may be specified in META.yml (and
> > in Makefile.PL/Build.PL).
>
> ...smells a lot like Test::Manifest, which is just test. In
> t/test_manifest, you just list the files that you want. Optionally, you
> can set a test level and mark some tests only happening at certain
> levels.
>
> In some cases, I even auto-generate the t/test_manifest so it only has
> the files that I want to test in that situation. Other than
> "t/test_manifest", I don't worry about magical names or changing all
> the tools for yet more special cases.
>

It makes a lot of sense. Patterns such as t/*.t , t/**/*.t  " t/**/*.t
minus t/author/*.t " helps a quick startup, but it looks like a good
idea to have an automated tool to expand this into longer and explicit
lists of tests. From then on, additions may be manual like some
usually do with MANIFEST. (Even though there is "make manifest", you
may not like the way it works because you have a bunch of files which
you don't to get shipped in the distribution.)

The only drawback is redundant information: there are the tests and
the list of tests. Keeping tests in "t/" with some fine way to
discover what they are good for (according to Chris' idea) may be
promising if not too magical.

Re: a "standard" place for extra tests

[Christopher H. Laco]

chromatic wrote:
> On Friday 17 August 2007 11:17:10 Christopher H. Laco wrote:
> 
>> To whit: Who knows better whether foo.t is an author test or not [in the
>> absence of the actual human author]... The harness, or foo.t?
> 
> The author.
> 
>> The harness asking the t files will always be more accurate than the harness
>> just find t files somewhere.
> 
> It doesn't matter where the metadata is, as long as someone who knows (the 
> author) enters that metadata in a way that makes sense to the tools...
> 
> ... sort of like I've been doing for at least one and probably two years now 
> with Module::Build and my custom disttest action, which I know I've mentioned 
> a few times now, which leads me to wonder why people so conveniently forget 
> it every time this discussion comes up.

*cough* selective memory *cough*...myself included.  :-)



signature.asc
Description: OpenPGP digital signature

Re: a "standard" place for extra tests

[brian d foy]

In article
<[EMAIL PROTECTED]>, Adriano
Ferreira <[EMAIL PROTECTED]> wrote:


> Testing some Author stuff would be rarer than having author tests. So
> maybe we could standardize on something like "t/author" and when other
> value is desirable, a key/value pair may be specified in META.yml (and
> in Makefile.PL/Build.PL).

...smells a lot like Test::Manifest, which is just test. In
t/test_manifest, you just list the files that you want. Optionally, you
can set a test level and mark some tests only happening at certain
levels.

In some cases, I even auto-generate the t/test_manifest so it only has
the files that I want to test in that situation. Other than
"t/test_manifest", I don't worry about magical names or changing all
the tools for yet more special cases.

Re: a "standard" place for extra tests

[Eric Wilhelm]

# from chromatic
# on Friday 17 August 2007 11:28 am:

>... sort of like I've been doing for at least one and probably two
> years now with Module::Build and my custom disttest action, which I
> know I've mentioned a few times now, which leads me to wonder why
> people so conveniently forget it every time this discussion comes up.

Because

  1.  it doesn't play nicely with recursive tests[1]
  2.  it requires a custom disttest action[2]

[1]  Unless you filter out t/author (and t/gui (and t/network 
(and ...)))  It seems to me that having a whole directory of "and also" 
would more easily allow one to cherry-pick which extras to run while 
leaving "t/" as the "run all of these, recursively" spot.

[2]  And that wouldn't be so bad if Module::Build had a plugin 
mechanism, but it doesn't.  Thus, avoiding custom actions for something 
where they could be avoided is nice because it reduces the (currently 
necessary) copy/paste code reuse.

--Eric
-- 
So malloc calls a timeout and starts rummaging around the free chain,
sorting things out, and merging adjacent small free blocks into larger
blocks. This takes 3 1/2 days.
--Joel Spolsky
---
http://scratchcomputing.com
---

Re: a "standard" place for extra tests

[Eric Wilhelm]

# from Michael G Schwern
# on Friday 17 August 2007 11:13 am:

>> # when something different is wanted
>> author_tests: t/developer
>
>Now you need a YAML parser to run tests, which may be fine.

But you also need to update all of the tools, which may not.

--Eric
-- 
But as soon as you hear the Doppler shift dropping in pitch, you know
that they're probably going to miss your house, because if they were on
a collision course with your house, the pitch would stay the same until
impact. As I said, that one's subtle.
--Larry Wall
---
http://scratchcomputing.com
---

Re: a "standard" place for extra tests

[chromatic]

On Friday 17 August 2007 12:52:54 Eric Wilhelm wrote:
> # from chromatic

> # on Friday 17 August 2007 11:28 am:
> >... sort of like I've been doing for at least one and probably two
> > years now with Module::Build and my custom disttest action, which I
> > know I've mentioned a few times now, which leads me to wonder why
> > people so conveniently forget it every time this discussion comes up.

> Because
>
>   1.  it doesn't play nicely with recursive tests[1]

"It's okay to run everything in t/ recursively" is a heuristic.  You can tell 
it's a heuristic because sometimes it's wrong.

Plus, I can't think of a single time when I've ever needed it.  That doesn't 
mean it's unnecessary, but rather that there's an interesting discussion 
about test organization that we seem to be failing to have.

>   2.  it requires a custom disttest action[2]

Avoiding heuristics requires some sort of metadata.  Now a custosm build 
action may be *bad* for other reasons, but it's not ambiguous, like a 
heuristic.

Perhaps we need to define:

1) what tests should run by default
2) who gets to decide what "by default means
3) what other groups need to run non-default tests
3a) and what those non-default tests are
3b) and why they need to run them
4) what options are there for distinguishing between default and non-default 
tests
5) which answer to #4 sucks the least

I respectfully suggest that any answer to #4 which includes a heuristic will 
fail #5 spectacularly.

-- c

Re: a "standard" place for extra tests

[Eric Wilhelm]

# from chromatic
# on Friday 17 August 2007 01:05 pm:

>On Friday 17 August 2007 12:52:54 Eric Wilhelm wrote:
>>   1.  it doesn't play nicely with recursive tests[1]
>
>"It's okay to run everything in t/ recursively" is a heuristic.  You
> can tell it's a heuristic because sometimes it's wrong.

That's not what I meant by "doesn't play nicely".  I mean, "my project 
cannot use recursive tests *and* have author-only tests in t/author 
without jumping through hoops."  Hoops include:  a custom test action, 
special options (or patches) to prove and runtests, etc.

>Avoiding heuristics requires some sort of metadata.  Now a custosm
> build action may be *bad* for other reasons, but it's not ambiguous,
> like a heuristic.

Let's forget for a second that I'm trying to assume anything about t/ or 
any other directory.  The simple fact is that the current tools will 
behave more nicely if everything in $the_one_directory is of one sort 
and everything in $the_other_directory is of the other sort.  Without 
that, we need code to sort the one from the other because they were all 
thrown into the same bag.  Then we need an XML config file for that 
code and etc, etc.

>Perhaps we need to define:
>

>4) what options are there for distinguishing...
>5) which answer to #4 sucks the least

>...any answer to #4 which includes a heuristic will fail #5

Any answer which requires a config-file and modifications to existing 
tools will also fail.

So, maybe we have to say "extra_tests: xt" (or even some deeper 
data-structure) to avoid a heuristic.  Fine.  What would that 
data-structure look like?

  extra_tests:
base_dir: xt
profiles:
  author: author
  network: network
  gui: gui
  exhaustive: exhaustive

The human will want the directory structure to mirror the names of the 
profiles, right?  So, now the human has to type more, but there's no 
heuristic (well, except for the names of the keys in the config file.)

--Eric
-- 
Speak softly and carry a big carrot.
---
http://scratchcomputing.com
---

Re: a "standard" place for extra tests

[Eric Wilhelm]

# from David Golden
# on Friday 17 August 2007 10:03 am:

>On 8/17/07, Eric Wilhelm <[EMAIL PROTECTED]> wrote:
>>   1.  author (kwalitee, pod, etc)
>>   2.  gui
>>   3.  network
>>   4.  you must have an account/password on $external_service
>>   5.  postgres/mysql/whatever availability/setup/permissions
>>   6.  no modem attached to /dev/ttyS0
>...
>As for different profiles, there's no reason that one couldn't have:
>
>author_t/release
>author_t/kwalitee
>author_t/gui
>author_t/network
>author_t/runs_for_27_hours
>etc.
>
>
>There really isn't a need to standardize the top level, either, unless
>there is a goal of adding support for it to Module::Build and/or
>ExtUtils::MakeMaker.

Yes, that is the goal.  The toplevel directory should be the same.  
While the config-file approach would allow the name to vary, I think 
that is over-engineering given that 't/' is an invariant "assumed 
directory name".

> there's no need for any standardization below the top level.  

Except in that it would conveniently allow smokers to "sign up" for 
testing *specific* extra functionality.  That is, "yes, I have network 
connectivity" means it is ok to run the "xtra_t/network/" tests.

I'm not saying we have to have a committee establish *all* of the 
possible profile names.  If we can just say 'author', 'network', 
'exhaustive', and 'gui' for now, I'll be happy.  Are we going to have 
to argue over those names too?

>I think the important thing is establishing a *convention* of keeping
>tests intended for developers and not end users out of the "t/"
>directory.

Yes.  What I would like to do to motivate that is to provide the 
incentive of a recommended alternative (one which is supported by 
tools, smoke testers, etc.)

>And what percent of CPAN 
>distros will really include so many author variations anyway?  Let's
>not let the perfect design be the enemy of the good.

As long as we don't hastily shoot ourselves in the foot.

--Eric
-- 
To succeed in the world, it is not enough to be stupid, you must also be
well-mannered.
--Voltaire
---
http://scratchcomputing.com
---

Re: a "standard" place for extra tests

[chromatic]

On Friday 17 August 2007 13:20:39 Eric Wilhelm wrote:

> >>   1.  it doesn't play nicely with recursive tests[1]

> >"It's okay to run everything in t/ recursively" is a heuristic.  You
> > can tell it's a heuristic because sometimes it's wrong.

> That's not what I meant by "doesn't play nicely".  I mean, "my project
> cannot use recursive tests *and* have author-only tests in t/author
> without jumping through hoops."  Hoops include:  a custom test action,
> special options (or patches) to prove and runtests, etc.

How are you getting recursive tests by default anywhere?

Why you would need to patch prove or runtests?

I just temporarily changed the Build.PL for Test::MockObject (which stores 
author-only tests in t/developer/) not to filter out that directory in the 
find_test_files() files method, and the developer tests still didn't run.

I don't understand why you're having trouble with recursive tests, and that's 
maybe why I don't think this is a problem at all.  I've not encountered it, 
and I'm curious how you have.

-- c

Re: a "standard" place for extra tests

[Eric Wilhelm]

# from chromatic
# on Friday 17 August 2007 01:57 pm:

>On Friday 17 August 2007 13:20:39 Eric Wilhelm wrote:
>> "my project cannot use recursive tests *and* have author-only tests
>> in t/author without jumping through hoops."

>How are you getting recursive tests by default anywhere?

I'm not getting them by default.  I'm *using* the recursive 
functionality for my test suite.  I cannot use t/author and 
t/normal_stuff without filtering-out "t/author".  I don't want to have 
to filter-out t/author and t/gui and t/network and t/account.

I want it to be convenient to run
  "all of the normal tests" and
  "all of the extra tests" and
  "a few profiles from the extra tests" and
  "all of the tests"

I want it to be convenient with `./Build` and `make` and `prove` and 
`runtests`.

  prove -rb t
  prove -rb xt
  prove -rb xt/author xt/network xt/gui
  prove -rb t xt

>I just temporarily changed the Build.PL for Test::MockObject (which
> stores author-only tests in t/developer/) not to filter out that
> directory in the find_test_files() files method, and the developer
> tests still didn't run.

Now set 'recursive_test_files => 1' (like I do) and try it.  Then run 
`prove -rb t` and `runtests -rb t`.

--Eric
-- 
"...the bourgeoisie were hated from both ends: by the proles, because
they had all the money, and by the intelligentsia, because of their
tendency to spend it on lawn ornaments."
--Neal Stephenson
---
http://scratchcomputing.com
---

Re: a "standard" place for extra tests

[Smylers]

Eric Wilhelm writes:

> # from Smylers
> # on Thursday 16 August 2007 11:40 pm:
> 
> > > I am certain that more than one 'extra tests' directory is needed,
> > 
> > Why are you certain of this?
> 
> Because I already have a use for three 'profiles' in one project and I
> can name a few others from elsewhere:
> 
>   1.  author (kwalitee, pod, etc)

Yep.

>   2.  gui

Why can't gui tests simply be in t/ and be skipped if the appropriate
environment isn't available?  That way all users who are in that
particular environment run the tests, rather than only those who've
discovered the xt/gui/ directory.

>   3.  network

Ditto -- why isn't skipping enough?

>   4.  you must have an account/password on $external_service
>   5.  postgres/mysql/whatever availability/setup/permissions

There are existing modules user tests which cope with those, either by
prompting or environment variables -- defaulting to skipping the tests,
of course.

Possibly we need a better (or standardized) way of getting information
from users running tests in these situations, and for safely skipping by
default.  (When trying to get a bunch of Cpan modules to install
unattendedly I discovered they collectively had many ways of seeking
attention; having none of them waiting at a prompt involved all three of
setting an environment variable, passing an option to Makefile.PL, and
piping yes '' into it.)

>   6.  no modem attached to /dev/ttyS0

That could probably be treated similarly to the website case, defaulting
to presuming no modem is there unless explicitly told so.

> What I would ultimately like to see happen is that these
> subdirectories fall into some form of standard(ish) naming scheme
> which would allow testers and qa efforts to run particular subsets.
> That is, the smoke box (or rigorous end-user upgrade verification
> process) could set variables to test the author, 'gui', 'network', and
> 'postgres' tests[1] without stumbling into e.g. "must have an X.com
> web-services account".

That requires a central standardized repository of permitted
subdirectories and what needs to be done to run the tests there.  Surely
it's better to let each distribution keep control of this, by specifying
in code what's needed to run those particular tests (and skipping
otherwise)?

Smylers

Re: a "standard" place for extra tests

[Eric Wilhelm]

# from Smylers
# on Friday 17 August 2007 03:13 pm:

>Eric Wilhelm writes:
>Why can't gui tests simply be in t/ and be skipped if the appropriate
>environment isn't available?  That way all users who are in that
>particular environment run the tests, rather than only those who've
>discovered the xt/gui/ directory.

"Can't you just" paste some 'if($ENV{WHATEVER})' code into all of your 
tests?

Yes, "I can just".

I don't want to.

I don't want the test suite to start 100 test processes and skip 90 of 
them.

I don't want to maintain some randomish, pasted 'if($ENV{WHATEVER})' 
code in 20+ test suites.

I don't want other CPAN authors to have to do these things either.

I don't want other CPAN authors to have to discover "the recommended 
way" to do these things.

>> What I would ultimately like to see happen is that these
>> subdirectories fall into some form of standard(ish) naming scheme
>> which would allow testers and qa efforts to run particular subsets.
>> That is, the smoke box (or rigorous end-user upgrade verification
>> process) could set variables to test the author, 'gui', 'network',
>> and 'postgres' tests[1] without stumbling into e.g. "must have an
>> X.com web-services account".
>
>That requires a central standardized repository of permitted
>subdirectories and what needs to be done to run the tests there. 

It might be useful, but is not strictly required.  I would still prefer 
that to "everything is turing-complete and self-selecting".  Plus, 
self-selecting tests bring us back to the same point which started all 
of this which is that not all of them are absolutely critical to the 
installation.

Part of my original point is that it would be useful[1] to define a 
standardized mapping of what functionality or resources are required to 
run typical subsets:

  author: some pod testing, kwalitee, etc prereqs
  network: network access
  gui: a gui

Yes, the rest is a bit more complicated.  Here, I'll define the 
arbitrary invent-your-own-thing ad-hoc asylum location right now:

  misc: everything else

> Surely it's better to let each distribution keep control of this, by
> specifying in code what's needed to run those particular tests (and
> skipping otherwise)?

It is only better in specific cases of "totally unlike everything else".  
Self-selecting tests are clearly (to me) worse for things as simple and 
typical as pod, network, and gui because it is ad-hoc, 
non-discoverable, and requires pasting code or modifying all of the 
tools.

There is nothing about categorization which prohibits ad-hoc from 
occurring within it.  However, you cannot easily have categorization 
within ad-hoc.

[1] useful for developers, smoker testers, qa-testers, and end-users.  
Each of which have various reasons to want to run or not run various 
categories of tests (and do it in a standard way.)

--Eric
-- 
If the above message is encrypted and you have lost your pgp key, please
send a self-addressed, stamped lead box to the address below.
---
http://scratchcomputing.com
---

Re: a "standard" place for extra tests

[Eric Wilhelm]

# from Adriano Ferreira
# on Friday 17 August 2007 11:49 am:

>The only drawback is redundant information: there are the tests and
>the list of tests. Keeping tests in "t/" with some fine way to
>discover what they are good for (according to Chris' idea) may be
>promising if not too magical.

I think keeping extra tests in "t/" is too magical for the existing 
tools.

After all, I have tried it.  In order to have *both* recursive tests 
(which many larger projects do) *and* extra tests, you have to have 
magic.  My earlier attempt at this was to add test profiles to 
Module::Build via file extensions which were declared in the Build.PL.  
The trouble is that those don't work with EU::MM or prove or runtests 
(or ack (and even vi in some configurations.))

My conclusion from that, was this "xt/foo/" thing, which requires zero 
coding (or pasting) and works fine with all of the existing tools.

The "xt/foo/" thing solves that redundant information problem too.

I thought to myself, "if this works so well for what I'm doing, wouldn't 
it be great if it became a sort of standard?"

--Eric
-- 
"Ignorance more frequently begets confidence than does knowledge."
-- Charles Darwin
---
http://scratchcomputing.com
---

Re: a "standard" place for extra tests

[A. Pagaltzis]

* Eric Wilhelm <[EMAIL PROTECTED]> [2007-08-18 02:25]:
> Part of my original point is that it would be useful[1] to
> define a standardized mapping of what functionality or
> resources are required to run typical subsets:
> 
>   author: some pod testing, kwalitee, etc prereqs
>   network: network access
>   gui: a gui

What about author tests for the GUI? Or tests that need both the
network and a GUI?

Regards,
-- 
Aristotle Pagaltzis // 

Re: a "standard" place for extra tests

[A. Pagaltzis]

* Eric Wilhelm <[EMAIL PROTECTED]> [2007-08-17 05:15]:
> As for the 'xt' name: it tends to sort pretty soon after 't',
> as in 'these come next'. I think anything starting with 't'
> would get annoying as far as tab-complete goes. Similarly, 't2'
> might conflict with primitive version-control habits.

Those arguments would indicate that it should be `tt`. That’s
equally short, sorts far closer to `t` than `xt`, precludes the
need for tab completion, and does not get in the way of primitive
version control.

Regards,
-- 
Aristotle Pagaltzis // 

Re: a "standard" place for extra tests

[Eric Wilhelm]

# from A. Pagaltzis
# on Saturday 18 August 2007 06:59 am:

>* Eric Wilhelm <[EMAIL PROTECTED]> [2007-08-18 02:25]:
>> Part of my original point is that it would be useful[1] to
>> define a standardized mapping of what functionality or
>> resources are required to run typical subsets:
>>
>>   author: some pod testing, kwalitee, etc prereqs
>>   network: network access
>>   gui: a gui
>
>What about author tests for the GUI? Or tests that need both the
>network and a GUI?

OR tests for pumping screenshots from the gui (run via a network 
connection to the author's machine) into a postgres database and 
uploading them to flicker?

Yes, it can get complicated.

No, it doesn't have to always be complicated.

What we have now sucks.

So, more complicated scenarios need more complicated configuration.  
This could be done via "the thing to be discussed in a different 
thread" (e.g. "a config-file plus 'use skipme' in the test" under the 
misc directory ("xt/misc/gui/foo.t" -- misc meaning that they are not 
easily classifiable and/or are externally/internally configured.))

The main goal (well, my main goal) is to have some kind of 
easy-to-understand recommendation for new authors (or even experienced 
ones) on how to organize their tests when the tests are considered 
optional[1].

I think a way to cover the basics with existing tools (without having 
everything totally ad-hoc) is important.

I would prefer that this doesn't involve skipping tests.  That is, there 
is less noise in the output if an optional test is never run than if it 
starts and then says "oops, no ignore me."  The tools should decide 
which tests to execute.

I would like to be able to categorize/sort/select tests without 
executing them.  I keep harping on this point because I think it could 
enable a much richer distributed testing environment.

[1] It seems that "optional" is a bit of a sticking point around here.  
Essentially, when resources required by those tests are likely to cause 
spurious failure or when failure of a given test is otherwise 
irrelevant to functionality or when the tests are testing optional 
features.

--Eric
-- 
"Matter will be damaged in direct proportion to its value."
--Murphy's Constant
---
http://scratchcomputing.com
---

Re: a "standard" place for extra tests

[brian d foy]

In article
<[EMAIL PROTECTED]>, Adriano
Ferreira <[EMAIL PROTECTED]> wrote:

> On 8/17/07, brian d foy <[EMAIL PROTECTED]> wrote:
> > In article
> > <[EMAIL PROTECTED]>, Adriano
> > Ferreira <[EMAIL PROTECTED]> wrote:


> > ...smells a lot like Test::Manifest, which is just test. In
> > t/test_manifest, you just list the files that you want. Optionally, you
> > can set a test level and mark some tests only happening at certain
> > levels.


> The only drawback is redundant information: there are the tests and
> the list of tests. Keeping tests in "t/" with some fine way to
> discover what they are good for (according to Chris' idea) may be
> promising if not too magical.

I've yet to provide the "make testmanifest" homolog to "make manifest"
because so far I just `find . -name "*.t" > test_manifest".

There is a bit more work to maintain that, but it's a small bit of work
to the current alternatives, especially when I want to repeatedly run
a subset of tests by replacing test_manifest temporarily.

Even if people don't like Test::Manifest, the solution is probably the
same for any other suggestions: override parts of Test::Harness to
discover the file names you want to test, or override runtests. Those
are both very short routines that hand off things to the heavy lifters.

Re: a "standard" place for extra tests

[Eric Wilhelm]

# from brian d foy
# on Saturday 18 August 2007 01:33 pm:

> the solution is probably the
>same for any other suggestions: override parts of Test::Harness to
>discover the file names you want to test, or override runtests.

Or override nothing and simply group the tests into directories.

--Eric
-- 
Turns out the optimal technique is to put it in reverse and gun it.
--Steven Squyres (on challenges in interplanetary robot navigation)
---
http://scratchcomputing.com
---

Re: a "standard" place for extra tests

[Ovid]

--- Smylers <[EMAIL PROTECTED]> wrote:
 
> >   3.  network
> 
> Ditto -- why isn't skipping enough?

What if they have the correct modules installed, they have a net
connection, but it fails partway through the tests?  That doesn't mean
the module's bad.

Admittedly, some might disagree with whether or not this is an
appropriate analysis (there will be lots of agreement over which are
"Must Not Fail" tests), but there's consensus that the current system
needs to change.  Just because the author forgot an =back in the POD
doesn't mean that the module should fail to install.  The main thing we
need to focus on is letting people Get Stuff Done, not worrying about
tiny details in a quest for purity (I'm not saying you're doing that! 
I assume that was an honest question of yours :)

Cheers,
Ovid

--
Buy the book  - http://www.oreilly.com/catalog/perlhks/
Perl and CGI  - http://users.easystreet.com/ovid/cgi_course/
Personal blog - http://publius-ovidius.livejournal.com/
Tech blog - http://use.perl.org/~Ovid/journal/

Re: a "standard" place for extra tests

[brian d foy]

In article <[EMAIL PROTECTED]>, Eric Wilhelm
<[EMAIL PROTECTED]> wrote:

> # from brian d foy
> # on Saturday 18 August 2007 01:33 pm:
> 
> > the solution is probably the
> >same for any other suggestions: override parts of Test::Harness to
> >discover the file names you want to test, or override runtests.
> 
> Or override nothing and simply group the tests into directories.

Well, if you want some directories to be special, simply grouping tests
into directories isn't going to get you anything. :)