Re: s/FAIL/welcome basket/

[Gabor Szabo]

On Sat, Sep 6, 2008 at 6:33 AM, Eric Wilhelm <[EMAIL PROTECTED]> wrote:
> Until PAUSE starts doing that, how do you let new authors know about
> cpantesters?  Also note that creation of an account may be separated
> from uploading a module by several years.

IMHO it is easy to add a few lines about the resources to every
message that goes out from PAUSE when you upload a module.

You get those anyway.
Some people might even look at it.

Gabor

Re: passing the baton onwards

[Gabor Szabo]

On Sat, Sep 6, 2008 at 3:15 AM, brian d foy <[EMAIL PROTECTED]> wrote:
> I'll do the work to handle the ones the authors give up without a
> maintainer, and my first idea was that a virtual user than we
> advertised as "free modules" (free as in kittens) would move modules
> int willing homes faster. But then, maybe not.

Use case 1:
  I have two modules I would like to give up.
  Occassionally I might still update it (e.g. if someone sends me a good patch)
  but in general I'd like to put it in the "take this module" basket.
  IMHO this means the module needs to stay in my pause id or it will
get back there
  if I upload a new version but it should be visible that
  "this module needs a new primary maintainer".

Use case 2: (quite similar)
  I see a module that seems to be unmaintained and needs a fix but something
  I don't really want to maintain.
  I can ask the author and if she is not responsive then you to take it over.
 Once I got the module I upload my fix but I'd also would like to *easily* set
 the "this module needs a new primary maintainer" flag.

Use case 3:
  Someone passes away or just disappears for a long period.
  The CPAN maintainers should set the flag "this module needs a new
primary maintainer".


IMHO Instead of encouraging people to upload new modules we should
encourage them
to take over existing ones.

Gabor

Re: s/FAIL/welcome basket/

[Eric Wilhelm]

# from David Golden
# on Friday 05 September 2008 20:19:

>> You know, a "hello" that doesn't start with "FAIL!"
>
>Unless the result is 11 FAILs.  ;-)

Still.  The subject line says "Welcome", not "FAIL".

>An author welcome with resources is probably best handled by PAUSE,
>not CPAN Testers.

Until PAUSE starts doing that, how do you let new authors know about 
cpantesters?  Also note that creation of an account may be separated 
from uploading a module by several years.

--Eric
-- 
If the above message is encrypted and you have lost your pgp key, please
send a self-addressed, stamped lead box to the address below.
---
http://scratchcomputing.com
---

Re: Plans for CPAN Testers notification when author CC's go away

[Aristotle Pagaltzis]

* David Golden <[EMAIL PROTECTED]> [2008-09-05 21:10]:
> * After a period of time to allow people to opt-in, the default
>   policy for authors without a stated preference will be
>   changed to "no mail".
> 
>   From that point on, CPAN Testers will be a purely opt-in
>   service. Hopefully, the design of the notification system
>   will be such that people who want to innovate new ways of
>   filtering notifications to particular distributions or
>   platforms of interest can contribute to the code base to do
>   so.

I would hope that this still leaves room for Eric Wilhelm’s
proposed “welcome basket” mail getting sent at least once to new
authors, so they have a chance to learn about the things that the
CPAN Testers can do for them, should they so desire.

In fact, if that is in the cards, then maybe the default policy
change should simply switch authors without a stated preference
to “has never received a welcome message” so that they get a
chance to hear about the new developments that are transpiring
and the new options they have, rather than having CPAN Testers
quietly go radio silent for them. Not everyone reads perl-qa.

Also, I would stipulate that if an author has not specifically
stated their preference, maybe mail them another welcome basket
in a year or two. No other mail, of course – they still have to
opt in if they want FAIL/PASS mail. But reiterating the welcome
basket every once in a very long time (in case it slipped through
a crack in their TODO list, got purged during an inbox bankruptcy
start-over, or whatever) seems like a gentle enough reminder that
it would not annoy. The goal, again, is to reach people at the
outskirts of the community who do not read 30 Perl mailing lists
and all of the use.perl journals (yes, that’s me).

Regards,
-- 
Aristotle Pagaltzis // 

Re: s/FAIL/welcome basket/

[Andy Lester]

On Sep 5, 2008, at 10:19 PM, David Golden wrote:


You know, a "hello" that doesn't start with "FAIL!"


Unless the result is 11 FAILs.  ;-)

An author welcome with resources is probably best handled by PAUSE,
not CPAN Testers.



Why not?  A one-time "Hi, we're watching your code, and if you'd like  
to follow our monitoring of your code, go here, and if you want  
messages all the time, you can do such-and-such."


--
Andy Lester => [EMAIL PROTECTED] => www.petdance.com => AIM:petdance

Re: s/FAIL/welcome basket/

[David Golden]

On Fri, Sep 5, 2008 at 9:15 PM, Eric Wilhelm <[EMAIL PROTECTED]> wrote:
>  Hi.  This mail is from the cpantesters.  We are a group of helpful
>  volunteers who automatically download and test modules from the CPAN.
>  You are receiving this mail because we've just tested your first ever
>  CPAN distribution, on $n platforms, [congratulations and so on, etc.]
>
>  Results:  11 PASSes!
>
>  To receive mail notifications or subscribe to RSS feeds, click ...
>
>
> You know, a "hello" that doesn't start with "FAIL!"

Unless the result is 11 FAILs.  ;-)

An author welcome with resources is probably best handled by PAUSE,
not CPAN Testers.

-- David

Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)

[brian d foy]

On Fri, Sep 5, 2008 at 6:17 PM, chromatic <[EMAIL PROTECTED]> wrote:
> On Friday 05 September 2008 16:52:05 brian d foy wrote:
>
>> In article <[EMAIL PROTECTED]>, chromatic
>
>> > If I could see somehow that my distribution implicitly runs on
>> > Perl 5.001 (or explicitly runs only on 5.11.0), or that it has no
>> > Makefile.PL or Build.PL, or any of the other dozens of packaging quirks
>> > that can cause problems, I could fix them before uploading and before
>> > triggering a wave of testing.
>
>> That's what Module::Release does, and since the QA Hackathon in Oslo,
>> it even tests under multiple perls. It can check kwalitee (or not), or
>> anything else that you like.
>
> Hm.  What's your thought on turning that into something which a Module::Build
> or ExtUtils::MakeMaker plugin could run on ./Build distcheck or make
> distcheck?

Module::Release already does that, too. You don't run it from make,
you just type 'release' at the command line and it checks the `make
test` and `make disttest`. There is a Module::Build plugin, but I
haven't verified that it works for my latest versions yet (it probably
doesn't). If someone wants to integrate it some other way, they can,
but I have a different workflow that works a lot better for me.

Module::Release tests a bunch of things, and if anything fails, it
stops and doesn't release anything. The major bits are the distro
tests, prereqs (I'm moving Test::Prereq out of my dists and into
Module::Release), kwalitee, and source control status.

Although you can look at the 'release' script in search.cpan.org,
here's the output of a dry run to show you how it works (even more
stuff happens for a real release after all this stuff passes)l:

macbookpro_brian[686]$ release -t
Testing with /usr/local/bin/perl5.10.0
Cleaning directory...  no Makefile---skipping
Recreating make file... done
Running make... done
Checking make test... all tests pass
Checking prereqs... done
Making dist... done
Checking make disttest... all tests pass
Testing with /usr/local/bin/perl
Cleaning directory... done
Recreating make file... done
Running make... done
Checking make test... all tests pass
Checking prereqs... done
Making dist... done
Checking make disttest... all tests pass
Testing for kwalitee
Cleaning directory... done
Recreating make file... done
Running make... done
Making dist... done
Checking kwalitee... done
Checking source repository
Checking state of Git... Git up-to-date on branch 1
Cleaning directory... done





-- 
brian d foy <[EMAIL PROTECTED]>
http://www.pair.com/~comdog/

Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)

[David E. Wheeler]

On Sep 5, 2008, at 18:17, chromatic wrote:

Hm.  What's your thought on turning that into something which a  
Module::Build

or ExtUtils::MakeMaker plugin could run on ./Build distcheck or make
distcheck?

I'm happy to write the Module::Build plugin, if anyone else might  
find this
idea useful.  (I'd very happily explain to new CPAN contributors the  
value of

running those commands against distributions before upload.)


I think we need to implement plugins for Module::Build, first. It's in  
the cards, but no one has written the code, yet.


Best,

David

Re: s/FAIL/welcome basket/

[Andy Lester]

On Sep 5, 2008, at 8:15 PM, Eric Wilhelm wrote:


You know, a "hello" that doesn't start with "FAIL!"



Yes, beautiful.  We need to remember that not everyone is a grizzled  
veteran.


--
Andy Lester => [EMAIL PROTECTED] => www.petdance.com => AIM:petdance

Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)

[chromatic]

On Friday 05 September 2008 16:52:05 brian d foy wrote:

> In article <[EMAIL PROTECTED]>, chromatic

> > If I could see somehow that my distribution implicitly runs on
> > Perl 5.001 (or explicitly runs only on 5.11.0), or that it has no
> > Makefile.PL or Build.PL, or any of the other dozens of packaging quirks
> > that can cause problems, I could fix them before uploading and before
> > triggering a wave of testing.

> That's what Module::Release does, and since the QA Hackathon in Oslo,
> it even tests under multiple perls. It can check kwalitee (or not), or
> anything else that you like.

Hm.  What's your thought on turning that into something which a Module::Build 
or ExtUtils::MakeMaker plugin could run on ./Build distcheck or make 
distcheck?

I'm happy to write the Module::Build plugin, if anyone else might find this 
idea useful.  (I'd very happily explain to new CPAN contributors the value of 
running those commands against distributions before upload.)

-- c

Re: s/FAIL/welcome basket/

[Eric Wilhelm]

# from Andy Lester on Friday 05 September 2008 17:12:

>On Sep 5, 2008, at 7:07 PM, brian d foy wrote:
>> CPAN Testers is what happens in a world of open source. Anyone gets
>> to look at and comment on your code, whether you agree with them or
>> not, and they don't need anyone's permission.
>
>That's a very helpful explanation you gave there.
>
>What they do need permission to do is send me the mass mail, and
>fortunately that's what's going to happen.

Yes.  Though I wouldn't be opposed to them (or any service) mailing new 
authors once with a welcome basket.

  Subject: Welcome from CPAN testers

  Hi.  This mail is from the cpantesters.  We are a group of helpful
  volunteers who automatically download and test modules from the CPAN. 
  You are receiving this mail because we've just tested your first ever
  CPAN distribution, on $n platforms, [congratulations and so on, etc.]

  Results:  11 PASSes!

  To receive mail notifications or subscribe to RSS feeds, click ...


You know, a "hello" that doesn't start with "FAIL!"

And the same goes for any new CPAN-related service or big change or 
whatever.  A nice message telling me about something I might like is 
probably ok to get ~12 random times per year (in the absence of a 
PAUSE newsletter or some other method of managing preferences 
globally.)  And I would love to see all of that collected together in 
one single nice welcome basket from CPAN ("hi, here is a list of 
helpful resources"), but not if it means we have to argue about what 
goes in the basket.

Of course, you can only be a new author once, so I'm just using my 
imagination.

--Eric
-- 
But you can never get 3n from n, ever, and if you think you can, please
email me the stock ticker of your company so I can short it.
--Joel Spolsky
---
http://scratchcomputing.com
---

Re: passing the baton onwards

[brian d foy]

In article <[EMAIL PROTECTED]>, (Andreas J. Koenig)
<[EMAIL PROTECTED]> wrote:

> > On Fri, 05 Sep 2008 10:54:40 -0700, brian d foy <[EMAIL PROTECTED]>
> > said:
> 
>   > Or, Andreas could change PAUSE, which is a bit more involved :)
> 
> Do you not know the abandoned flag? Or not considering it appropriate?

*I* know about it, but you also have to register the module, don't you?

Beyond that, is there a way for everyone to see the list of those
modules? That's what I meant about you changing pause, much like you
did for 06.perms.

It would also be nice to see that bit set somewhere like CPAN Search,
maybe with a button that says "I want to take over maintenance".  

None of this happens any differently than before. Either we need
fore-knowledge that the author has decided to move on in life, or our
waiting period for public notification.

I'll do the work to handle the ones the authors give up without a
maintainer, and my first idea was that a virtual user than we
advertised as "free modules" (free as in kittens) would move modules
int willing homes faster. But then, maybe not.

-- 
brian d foy (one of many PAUSE admins), http://pause.perl.org
archives at http://www.xray.mpe.mpg.de/mailing-lists/modules
please send all messages back to [EMAIL PROTECTED]

Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)

[Andy Lester]

On Sep 5, 2008, at 7:07 PM, brian d foy wrote:


CPAN Testers is what happens in a world of open source. Anyone gets to
look at and comment on your code, whether you agree with them or not,
and they don't need anyone's permission.



That's a very helpful explanation you gave there.

What they do need permission to do is send me the mass mail, and  
fortunately that's what's going to happen.


--
Andy Lester => [EMAIL PROTECTED] => www.petdance.com => AIM:petdance

Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)

[brian d foy]

In article <[EMAIL PROTECTED]>, Andy
Lester <[EMAIL PROTECTED]> wrote:


> Where's the warning on the front page of PAUSE saying "Your upload had  
> better match certain criteria or else we will send you mass email  
> about it"?

PAUSE sends you email too, but only for PAUSE things, such as indexer
failure, upload, or deletion reports. I've received more mail from
PAUSE in the last month than from CPAN Testers.

PAUSE, however, has an explicit policy of not caring about the quality
of whatever people upload, and we encourage people to upload early
rather than later. We don't make those judgements, and we're not going
to.

CPAN Testers is what happens in a world of open source. Anyone gets to
look at and comment on your code, whether you agree with them or not,
and they don't need anyone's permission.

Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)

[brian d foy]

In article <[EMAIL PROTECTED]>, chromatic
<[EMAIL PROTECTED]> wrote:

> If I could see somehow that my distribution implicitly runs on 
> Perl 5.001 (or explicitly runs only on 5.11.0), or that it has no Makefile.PL 
> or Build.PL, or any of the other dozens of packaging quirks that can cause 
> problems, I could fix them before uploading and before triggering a wave of 
> testing.


That's what Module::Release does, and since the QA Hackathon in Oslo,
it even tests under multiple perls. It can check kwalitee (or not), or
anything else that you like.

Re: Module::Build 0.2809 release coming, should we test it?

[Eric Wilhelm]

# from David Cantrell
# on Friday 05 September 2008 05:56:

>> Perhaps 2286 is still a lot.  A one-liner tells me there are 474
>> authors.  I wonder if starting with one dist from each author would
>> be a useful sampling, since often the weird stuff happens when an
>> author found a way to do some undocumented thing with M::B and we
>> didn't know about it.  Should I split-out two lists that way?
>
>Please.

Here it is with the whole thing and the two parts (once per author and 
then the rest)

  http://scratchcomputing.com/tmp/generated_by.module_build.list
  http://scratchcomputing.com/tmp/generated_by.module_build.onceper
  http://scratchcomputing.com/tmp/generated_by.module_build.therest

Once we get this release out, if anyone is interested in smoking the 
M::B svn I could easily regen this stuff daily.

Thanks,
Eric
-- 
Those who cannot remember the past are condemned to repeat it.
--George Santayana
---
http://scratchcomputing.com
---

Re: revokable FAIL (was Reporting Bugs Where ... (was ...))

[David Golden]

On Fri, Sep 5, 2008 at 5:48 PM, Eric Wilhelm <[EMAIL PROTECTED]> wrote:
> So, whether a 2.0 design consideration or whatever, please help me get
> the things I don't have to remember out of my brain and into the
> computer.  And maybe I'll even be able to help keep other authors from
> scratching their head wondering what happened.

It's on the list.  The one thing that concerns me is that some authors
object to FAIL reports showing up on their search.cpan.org page and
will just want them all blanked out.

So my thought at the moment is that we let people "dispute" reports --
or some synonym -- and let people filter them if they don't want to
see them.  (Authors or end users.)

> Is there a machine/configuration identifier of some sort and/or other
> ways to characterize a "report source" such that once the cause of a
> false report is identified every report caused by that same issue could
> be automatically crossed off?

Yes and no.  "Modern" reports have toolchain info, environment info,
etc.  But to be really complete, we'd need reports to bundle up
testers CPAN/CPANPLUS/CPAN::Reporter/YACSmoke/etc configuration files
and so on.  Benefit probably not worth the cost, hassle and privacy
concerns.  Most of the obvious problem "Build -j3" can probably be
caught with a regex.

> And now, feature creep:  optional notifications of "please disregard
> these reports: ... " for affected authors who have just started
> ambitiously scratching their heads at the time the issue is
> identified. ;-)

When we have 2.0 up and running, patches will be welcome.  ;-)

-- David

Re: revokable FAIL (was Reporting Bugs Where ... (was ...))

[Eric Wilhelm]

# from Aristotle Pagaltzis
# on Friday 05 September 2008 06:07:

>> UNIVERSAL::isa and UNIVERSAL::can are examples of applying the
>> design principle of Report Bugs Where They Are, Not Where They
>> Appear.
>
>How do you propose doing that in the general case? I am certainly
>interested in what technology you have invented so that computer
>programs can automatically debug themselves and detect the real
>source of any problems.

I too am interested in this technology and would like to subscribe to 
its rss feed.

But to the extent that we must solve technology problems with people, 
I'm interested in using technology to leverage the efforts of those 
people.

In the case of e.g. the CPAN.pm bug/issue with -j or tar, we have 
identified these issues at the cost of perhaps dozens of spurious FAIL 
reports to various authors.  If it is possible to track down all of 
these reports and "cross them out", this would eliminate confusion and 
possibly duplicated effort -- and would make the annoyance of the false 
FAIL less of a "take one for the team" (because uh, "take what where?")

Now, this "cross them out" might mean deletion, but possibly just simply 
mumble~strikethrough with linking to the resultant CPAN.pm RT ticket or 
etc. (because "Hey, RSS said FAIL. *click*  um... where did it go?")

It also keeps the author from looking at this in e.g. 6 months and 
thinking "FAIL? ... oh, right.  I forgot about that silly thing, well 
can't do anything about that."

  http://testers.cpan.org/show/Error.html

So, whether a 2.0 design consideration or whatever, please help me get 
the things I don't have to remember out of my brain and into the 
computer.  And maybe I'll even be able to help keep other authors from 
scratching their head wondering what happened.

And yes, it will take some people some effort.  But everyone else will 
appreciate it and if I put in the effort myself I even get my 100% 
greens back on my own page, which is a nice motivator because I can do 
something about the evil fake stupid wrong reds besides complain. :-D

Is there a machine/configuration identifier of some sort and/or other 
ways to characterize a "report source" such that once the cause of a 
false report is identified every report caused by that same issue could 
be automatically crossed off?

And now, feature creep:  optional notifications of "please disregard 
these reports: ... " for affected authors who have just started 
ambitiously scratching their heads at the time the issue is 
identified. ;-)

Thanks,
Eric
-- 
"It is a mistake to allow any mechanical object to realize that you are 
in a hurry."
--Ralph's Observation
---
http://scratchcomputing.com
---

Re: Plans for CPAN Testers notification when author CC's go away

[David E. Wheeler]

On Sep 5, 2008, at 12:08, David Golden wrote:


There is sufficient outrage now over email volumes that waiting for
the preference system seems pointless and hopefully, in exchange for
quick action now, those that are most annoyed will be willing to be
patient during the transition from opt-out to opt-in.


+1 Sounds like a good stop-gap plan.

Best,

David

Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)

[Aristotle Pagaltzis]

* chromatic <[EMAIL PROTECTED]> [2008-09-05 19:50]:
> If I could see somehow that my distribution implicitly runs on
> Perl 5.001 (or explicitly runs only on 5.11.0), or that it has
> no Makefile.PL or Build.PL, or any of the other dozens of
> packaging quirks that can cause problems, I could fix them
> before uploading and before triggering a wave of testing.

That’s CPANTS, the useful part of it anyway… except that the
feedback cycle is even *much* slower there than with the Testers.

CPANTS is really miscast as a service; the useful bits should
really be extracted and made part of the `distcheck` targets
of the various Perl distro toolchains.

Regards,
-- 
Aristotle Pagaltzis // 

Re: Plans for CPAN Testers notification when author CC's go away

[chromatic]

On Friday 05 September 2008 12:08:10 David Golden wrote:

> There is sufficient outrage now over email volumes that waiting for
> the preference system seems pointless and hopefully, in exchange for
> quick action now, those that are most annoyed will be willing to be
> patient during the transition from opt-out to opt-in.

+1

-- c

Plans for CPAN Testers notification when author CC's go away

[David Golden]

For anyone still following the other threads, I plan to make changes
to Test::Reporter that will stop authors from being copied on reports.
 These changes will likely happen on Sunday and then I will be
encouraging CPAN Testers to upgrade.

As testers upgrade, the immediate effect is that authors will stop
getting notifications.  However, that's a "cold turkey" approach and
I've heard about as many authors complain that they don't want to stop
getting emails as I've heard authors complain that they don't want to
keep getting emails.

I can't please all of you all the time, so these are my thoughts on
how to mildly piss-off both groups while migrating to a better
long-term solution.

* Barbie and I are discussing a centralized email notification
service.  It will likely provide either an aggregate of recent reports
for an author or will provide first notification for an dist/perl/arch
tuple.  (Come join cpan-testers-discuss if you want to weigh in with
opinions as to which way it should be.)

* We'll turn the email notification service on, hopefully within a
week or two, tuits permitting.  It will be opt-out, probably manually
administered at least at first, but will restore email notification
for those that never wanted to see it go away.  Those who don't want
email at all will probably hate us again, but at least the overall
volume of mail will be way down (max 1 per day).  Those that we know
don't want email (like Andy Lester) we'll add to the opt-out list at
the start.

* We'll launch some sort of author notification preferences system.
(This is also in the design stage).  We'll ask authors to start
confirming their desire to be notified by email --> i.e. we'll ask
people to opt in

* After a period of time to allow people to opt-in, the default policy
for authors without a stated preference will be changed to "no mail".

>From that point on, CPAN Testers will be a purely opt-in service.
Hopefully, the design of the notification system will be such that
people who want to innovate new ways of filtering notifications to
particular distributions or platforms of interest can contribute to
the code base to do so.

In my view this approach has a degree of fairness to all parties:

* Authors who don't want email will see substantially reduced volumes
of email quickly and will have a single place to request an opt-out

* Authors who do want email will continue to get notifications after a
brief interruption and will eventually be forced to opt-in to continue
to do so

* Authors who only want notifications for some things will have no
choice initially, but will be able contribute to add such features

There is sufficient outrage now over email volumes that waiting for
the preference system seems pointless and hopefully, in exchange for
quick action now, those that are most annoyed will be willing to be
patient during the transition from opt-out to opt-in.

-- David

Re: reasonable reporting

[Eric Wilhelm]

# from Andy Lester
# on Friday 05 September 2008 09:34:

>> Well, yeah, I have too. And sometimes I make a tweak to get things  
>> working on 5.005, and other times I tell my users that it runs 5.006
>>   or later by saying so in Build.PL. Seems reasonable to me to
>> specify such dependencies.
>
>"Seems reasonable to me" is EXACTLY my frustration.  That is YOUR
>standard that YOU are specifying.
>
>How about if I start sending email to everyone, whether they want it
> or not, if their code doesn't run under -T checking?  It seems
> reasonable to me that it should.

Yes.  The mail thing is clearly a rub (especially if it isn't 
comprehensive or centrally configurable.)

Putting a big red flag on a dist for not running under taint would also 
be a bit severe - but a yellow "notaint" flag might be appropriate and 
it is useful information.

But I'm talking about the view of the reports.

>> hope you'll tell your users about a Perl version dependency
>
>This is beyond me and my frustration about getting worthless emails.
>  It is about the presumption of telling CPAN authors ... 
>  punishing them for not bending to the whims of the CPAN Testers 

So, let's please move on from complaining about the mail.  I hope we all 
can agree that opt-out is annoying.

I think the presentation of the reports is what needs improvement, and I 
think we need to come to some consensus on what is a reasonable default 
presentation.  There is indeed some aspect of reward and punishment to 
what you present about an author's code.

Note:  Notifications are just one aspect of presentation (and are not 
limited to the author for that matter.)

It should not be a "big red deal" if an author omits the declaration of 
requiring perl 5.8.mumble+.  At this point, it is a reasonable 
assumption that code "should" run on 5.8.8 and 5.10.+, but an 
unreasonable assumption that it runs on 5.6.x or less.

It should not be a "big red deal" if a distribution doesn't run on VMS.  
This is not a system many people have ready access to or interest in.

It should not be a "big red deal" if a distribution doesn't install on a 
non-updated toolchain.

I'm not saying that you should not test for these things.  Try to run it 
on a toaster or whatever.  That's useful data to some people (and hey -
"dude!  My code passes all tests on a toaster!" is a great conversation 
starter.)  But a big stack of red FAIL's in the default presentation 
from all toasters and VMS is just unreasonable, right?

As I mentioned before, I would love to have any user or author be able 
to configure their own various modes of presentation.  A VMS user 
really does want to see a big stack of FAIL when that's the case.

And 5.6.x- is the same way.  Maybe it works.  Maybe the author will tag 
the next distro that 5.6 doesn't work, whatever.  It's just not that 
important to the majority of users and authors at this point.  (IF it 
is, that's a different problem and there will never be a funpan.)

And I hope perhaps that we've finally reached the point where a user can 
reasonably be expected to upgrade their toolchain before trying to 
install the latest and greatest of *everything else*.  If we must shove 
a toolchain update down the user's throat, we have got the entire rest 
of the CPAN completely wrong.)


Now, currently, the presentation of notifications is both out of the 
cpantesters control *and* out of the author's control.  That causes 
conflict that nobody is really in a position to immediately resolve.


Now, also currently, there is only one web-view presentation of the 
reports on search.cpan.org and cpantesters.perl.org.  This also causes 
conflict:

  * If anybody starts a VMS smoker, the entire CPAN is likely to
light-up red, which (I hope we can agree) is unreasonable.

  * An author using Module::Build is still in a rock vs hard place

  * An author writing in 5.8ese has to add "no, not perl4" checks to
keep from seeing red.

  * Win32?  What if half of the smokes were on Win32?  How red would
that be?  Hey, a valid usage of the OS from the browser identifier?

So, what *is* a reasonable set of expectations from a modern perl user?  
We should expect that to be an evolving set of expectations 
as "reasonable" adjusts to keep up with new releases, etc.  If the 
presentation is fluid, the defaults can evolve and if it is 
configurable, the holdouts can set their own view of it.

The question then becomes a matter of what you (by default) present to 
the world about an author's code and not just which data has been 
gathered.  Thus:  What does the default fail-o-meter look like and what 
does it mean?

That should basically encapsulate what a reasonable author would be 
willing to shoot for as a mark of quality.  Everything else should be 
something they *want* to shoot for as a badge of excellence.

Thanks,
Eric
-- 
"Beware of bugs in the above code; I have only proved it correct, not
tried it."
--Donald Knuth
---

Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)

[David E. Wheeler]

On Sep 5, 2008, at 11:36, chromatic wrote:

They are annoying, but I'm not sure it's my biggest complaint.   
There's also
the arbitrariness of the upload/debug/revise cycle of trying to  
please a
black box full of testers.  I'm not willing to say that this is  
primarily the

fault of CPAN Testers, but it does expose a lot of cracks in the CPAN
plumbing.


Yes. Some easily-accessed documentation of best practices wold be  
welcome, linked to from the pause upload page. That would help. I  
recently updated all of my modules to work on various platforms and  
specify versions of Perl, and it was pretty annoying to do the upload/ 
debug/revise bit (though it was mainly Windows that gave me the pain,  
not old versions of Perl).


It's a little bit like trying to have a discussion with someone  
who's upset
but won't tell you why, and you have to guess and hope you don't  
make things

worse before you get a useful answer.


Yes, better diagnostics would be welcome, especially for those who  
suffer from action-at-a-distance failures like you do for your  
UNIVERSAL:: modules.



Well, you can upload a dev version to CPAN and the testing bots will
test it, I believe. It'd be nice if there was a separate place to
upload code to be tested before you actually released it. That'd be
very handy indeed.


Even being able to identify from a distribution which CPAN Testers  
platforms
will even try to run tests would help.  (Oh dear, this'll get all of  
those

5.005 boxes running my code.)

I do like how CPANTS lists the original dozen or so Kwalitee metrics  
and their

solutions on the individual distribution Kwalitee pages.


Yeah, it's a bit more advanced that way, in that it has specific  
metrics, whereas CPAN testers is just ("does it build" and "do the  
tests pass"). It's the former that seems to cause the most  
aggravation, as there are many reasons a build could fail and it's  
difficult to tell which reason or reasons are the underlying cause.


Best,

David

Re: What do you want? (was Re: The relation between CPAN Testers and quality)

[David Golden]

On Fri, Sep 5, 2008 at 2:03 PM, Eirik Berg Hanssen
<[EMAIL PROTECTED]> wrote:
>> You keep saying spam, but that's not the right term. You're being an
>> ass characterizing it like that.
>
>  Yeah, it's not the right term.  While it meets the other criteria to
> qualify as spam by the classical definition, it would also have to be
> bulk.  Which this is not.

For what it's worth, *I* consider it not quite spam, but not really
not far from it and don't object to people using that term. The more
successful CPAN Testers is at getting more testers and more platforms
tested, the more it becomes bulk.

For example, in my stress test of CPAN::Reporter::Smoker, when I
submitted more than 138,000 in a single month, I made a point to turn
off author CCing, because I didn't think that sending 20,000 fail
reports was a particularly friendly thing to do.

-- David

Re: What do you want? (was Re: The relation between CPAN Testers and quality)

[Andy Lester]

On Sep 5, 2008, at 1:36 PM, David Golden wrote:


I will be changing Test::Reporter to stop all author CC'ing which will
take effect when/if we convince existing testers to upgrade.



Thank you, sir.

--
Andy Lester => [EMAIL PROTECTED] => www.petdance.com => AIM:petdance

Re: What do you want? (was Re: The relation between CPAN Testers and quality)

[David Golden]

On Fri, Sep 5, 2008 at 12:14 PM, David E. Wheeler <[EMAIL PROTECTED]> wrote:
> On Sep 5, 2008, at 09:10, Andy Lester wrote:
>
>>> I'd hate to lose those in my email because other people don't want to
>>> filter their mail.
>>
>> I'd hate to get spammed because other people don't want to sign up to
>> receive them.
>
> I think that adding changing things so that authors opt-in to getting
> reports is something that's generally been agreed on, though it might have
> to wait until the Web service is done. In the meantime, David Golden, at
> least, has tweaked his reporting toolchain so that it no longer submits
> reports to you, Andy.

Clarification -- David *Cantrell* has tweaked his tools so it doesn't
submit to Andy already.  Too many Davids in this discussion
apparently.

I will be changing Test::Reporter to stop all author CC'ing which will
take effect when/if we convince existing testers to upgrade.  I have
vague plans to create a "nag bot" to detect testers using old tools
and nag them to upgrade.

I'll post about the near-term replacement notification plan in a separate email.

-- David

Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)

[chromatic]

On Friday 05 September 2008 11:23:07 David E. Wheeler wrote:

> And if you have to opt-in, I imagine that would solve the biggest
> complaint, yes? It's the unsolicited email reports that are annoying,
> right?

They are annoying, but I'm not sure it's my biggest complaint.  There's also 
the arbitrariness of the upload/debug/revise cycle of trying to please a 
black box full of testers.  I'm not willing to say that this is primarily the 
fault of CPAN Testers, but it does expose a lot of cracks in the CPAN 
plumbing.

It's a little bit like trying to have a discussion with someone who's upset 
but won't tell you why, and you have to guess and hope you don't make things 
worse before you get a useful answer.

> Well, you can upload a dev version to CPAN and the testing bots will
> test it, I believe. It'd be nice if there was a separate place to
> upload code to be tested before you actually released it. That'd be
> very handy indeed.

Even being able to identify from a distribution which CPAN Testers platforms 
will even try to run tests would help.  (Oh dear, this'll get all of those 
5.005 boxes running my code.)

I do like how CPANTS lists the original dozen or so Kwalitee metrics and their 
solutions on the individual distribution Kwalitee pages.

-- c

Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)

[David E. Wheeler]

On Sep 5, 2008, at 11:28, Andy Lester wrote:

Getting fail report emails is annoying and should be changed to be  
opt-in. Would that solve your problem?


Oh, and yes.  Once we stop spamming people, CPAN Testers then  
becomes the Consumer Reports model, not the police model.


Thank you. I think we're all in agreement on the benefit of an opt-in  
model, and that it would go a long way towards eliminating the  
complaints. At least, that's my reading of this thread.


So what needs to happen to get this working sooner rather than later?

Best,

David

Re: What do you want? (was Re: The relation between CPAN Testers and quality)

[David E. Wheeler]

On Sep 5, 2008, at 10:57, chromatic wrote:

Full credit (and many thanks) to David Golden and others who are  
moving away
from this model, but if I'm an ass for saying "You know, that has a  
lot in
common with spam" and "CPAN-related services with good intentions  
should
carefully consider the effects of their actions on their  
constituents", then

I'm proudly an ass as well.


I think that such diplomatic descriptions are useful, and don't make  
you an ass. But simply calling it "spam" is not very useful. There is  
an underlying reason one says its spam or spam-like, and that's the  
important thing.


Best,

David

Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)

[Andy Lester]

On Sep 5, 2008, at 12:24 PM, David E. Wheeler wrote:

Getting fail report emails is annoying and should be changed to be  
opt-in. Would that solve your problem?



Oh, and yes.  Once we stop spamming people, CPAN Testers then becomes  
the Consumer Reports model, not the police model.


xoa

--
Andy Lester => [EMAIL PROTECTED] => www.petdance.com => AIM:petdance

Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)

[David E. Wheeler]

On Sep 5, 2008, at 11:27, Andy Lester wrote:



On Sep 5, 2008, at 12:24 PM, David E. Wheeler wrote:

Punishing? Punishing would be removing a module from CPAN. Getting  
fail report emails is annoying and should be changed to be opt-in.  
Would that solve your problem?



One person's "annoying" is another person's "punishment."

The key here is that my reward for uploading to CPAN is to get mass  
unsolicited email, unless my upload conforms to some arbitrary  
standard that I was not informed of in advance.


Where's the warning on the front page of PAUSE saying "Your upload  
had better match certain criteria or else we will send you mass  
email about it"?


Am I the only one looking at this from the point of view of others?


You did not answer my question.

Best,

David

Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)

[Andy Lester]

On Sep 5, 2008, at 12:24 PM, David E. Wheeler wrote:

Punishing? Punishing would be removing a module from CPAN. Getting  
fail report emails is annoying and should be changed to be opt-in.  
Would that solve your problem?



One person's "annoying" is another person's "punishment."

The key here is that my reward for uploading to CPAN is to get mass  
unsolicited email, unless my upload conforms to some arbitrary  
standard that I was not informed of in advance.


Where's the warning on the front page of PAUSE saying "Your upload had  
better match certain criteria or else we will send you mass email  
about it"?


Am I the only one looking at this from the point of view of others?

xoa

--
Andy Lester => [EMAIL PROTECTED] => www.petdance.com => AIM:petdance

Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)

[David E. Wheeler]

On Sep 5, 2008, at 10:46, chromatic wrote:


I don't like the check testers/grumble/upload new distribution with no
functional changes just niggly little packaging bits you hope will  
opt out of
testers tests you don't care about/sleep/repeat cycle.  It's a slow,  
clunky
black box game where the rules aren't always clear and you have to  
upload new
versions of your distributions that don't do much for most of your  
users.


Yeah, but that's a symptom of the voluneerism of the organization. We  
don't have a dedicated serve farm you can upload a potential  
distribution to and get results in an hour. It'd be nice to have, but  
uploading development releases to CPAN and waiting 24-48 hours is as  
close as it gets right now.


I would find some sort of *optional* distribution packaging best  
practices
scanner more useful here than getting CPAN Testers reports (by  
whatever
mechanism).  If I could see somehow that my distribution implicitly  
runs on
Perl 5.001 (or explicitly runs only on 5.11.0), or that it has no  
Makefile.PL
or Build.PL, or any of the other dozens of packaging quirks that can  
cause
problems, I could fix them before uploading and before triggering a  
wave of

testing.


Not a bad idea. Surely someone could write a test (or Perl Critic  
plugin) for that, yes?


Best,

David

Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)

[David E. Wheeler]

On Sep 5, 2008, at 10:32, chromatic wrote:

You're right, there's no "compel".  If reports don't come by email  
to people
who haven't asked for them, then they'll only get reported via an  
RSS feed I

can choose to read or not, and on the search.cpan.org pages of my
distributions, which I don't visit.  That's not compulsion, that's
just "offering public data points" in yet another location I can  
choose to

visit or disregard.  Thus now I can get bug reports via:

* personal mail
* RT web/RSS
* CPAN Forum
* CPAN Testers (email/RSS)
* CPAN Annotations
* CPANTS
* Usenet
* PerlMonks
* countless mailing lists


And if you have to opt-in, I imagine that would solve the biggest  
complaint, yes? It's the unsolicited email reports that are annoying,  
right?


In my ideal world, CPAN Testers would be a series of platform  
combinations
where I could submit a development version of a distribution and get  
test
results with the latest version of dependent modules and toolchain  
modules

*before* uploading to the CPAN.


Well, you can upload a dev version to CPAN and the testing bots will  
test it, I believe. It'd be nice if there was a separate place to  
upload code to be tested before you actually released it. That'd be  
very handy indeed.



* It's opt-in
* It answers my most important question when I want it answered
* It can provide a reference configuration for CPAN installers
* It gives me something I don't already have
* It focuses tester resources where (I believe) they're most  
appropriate


Yeah, it's a reasonable feature request. Will probably have to wait  
for the Web service, though.



I may do so because I take the quality and utility of my software
seriously,but do not mistake that for anything which may instill  
in you
any sort of entitlement.  That is an excellent way not to get what  
you

want from me.

I don't think anyone would argue that. Straw man, dude.


I refer you to Greg Sabino Mullane's posts, in which the gentlest  
expectation

is:

 I recognize that CPAN is a volunteer effort, but it does seem to me  
there
 is a implicit responsibility on the part of the author to maintain  
the
 module going forward, or to pass the baton to someone else. Call it  
a Best

 Practice, if you will. The end-user simply wants the module to work.
 Maintainers not paying attention, and the subsequent bitrot that is
 appearing on CPAN, is one of Perl's biggest problems at the moment.
 Careful attention and responsiveness to CPAN testers and to  
rt.cpan.org is

 the best cure for this.


Nothing to do with entitlement. I have hopes for the quality of code,  
and hope that you'll maintain it, but I don't feel entitled to it.



Obviously it's not always easy to identify the source of the bug. It
is the responsibility of CPAN Testers to run the most recent module  
in

order to minimize such circumstances.


That would increase the utility of CPAN Testers to me if it were  
true (at

least, if by "most recent module" you mean toolchain modules).


Yes, I do. And the main CPAN testers do try to stay up-to-date. But  
getting them all to do so is of course an uphill battle. But certainly  
those who use bots tend to stay on top of things, IME.



It's not my "job" to fix bugs in *my own* distributions.  I do it
because I care about quality and, contrary to what appears to be  
near-

universal belief around here, I care that people can use my code.
Straw man again. Do you really believe anyone is actually saying  
that?


I can't find the link, but someone here on Wednesday said "You don't  
care if
people actually use your code", and it's not the first time I've  
heard it.


Yeah, people can be dicks on mail lists (/me raises his hand), but you  
don't get that in CPAN Testers reports, do you? You can ignore the  
assholes. I (usually) do (and FWIW, I didn't notice anything like that  
in this thread, and David Golden apologized for his out-of-line  
comment).


Best,

David

Re: What do you want? (was Re: The relation between CPAN Testers and quality)

[Andy Lester]

On Sep 5, 2008, at 12:46 PM, brian d foy wrote:


You keep saying spam, but that's not the right term. You're being an
ass characterizing it like that.



I knew before even opening this mail that it would contain a personal  
attack.


Why is it necessary to insult people who have different opinions?

--
Andy Lester => [EMAIL PROTECTED] => www.petdance.com => AIM:petdance

Re: Reporting Bugs Where they Belong (was Re: The relation between CPAN Testers and quality)

[David Golden]

On Fri, Sep 5, 2008 at 1:36 PM, chromatic <[EMAIL PROTECTED]> wrote:
>> Just in the last couple of days, David Golden reported making at
>> least two (did I count correctly?) substantial changes to how
>> CPAN::Reporter grades tests, in order to prevent particular
>> classes of bogus FAILs. Isn't that a demonstration of exactly the
>> same care?
>
> Provided it also identifies the problems on tester machines to help them
> reconfigure/update to produce relevant reports, yes.

It will identify it, but testers may or may not see the warning as it
scrolls by in the CPAN output. Not much I can do about that.  But it
will suppress the reports at least.

David

Re: What do you want? (was Re: The relation between CPAN Testers and quality)

[Eirik Berg Hanssen]

brian d foy <[EMAIL PROTECTED]> writes:

> In article <[EMAIL PROTECTED]>, Andy Lester
> <[EMAIL PROTECTED]> wrote:
>
>> > I'd hate to lose those in my email because other people don't want to
>> > filter their mail.
>> 
>> I'd hate to get spammed because other people don't want to sign up to
>> receive them.
>
> You keep saying spam, but that's not the right term. You're being an
> ass characterizing it like that. 

  Yeah, it's not the right term.  While it meets the other criteria to
qualify as spam by the classical definition, it would also have to be
bulk.  Which this is not.

  Or at least, not precisely.

  But then, languages evolve; it's been a long time since the
classical definition was the only one used.  And frankly, unsolicited
mass e-mail by any other name would smell as bad.


Eirik
-- 
Statistics means never having to say you're certain.

Re: What do you want? (was Re: The relation between CPAN Testers and quality)

[chromatic]

On Friday 05 September 2008 10:46:59 brian d foy wrote:

> You keep saying spam, but that's not the right term. You're being an
> ass characterizing it like that.

It's unsolicited, opt-out, bulk mail generated by an army of robots trying to 
get me to care about things I don't necessarily care about, and many 
responses to complaints about receiving it is "Just delete it!" or "You have 
email filters, right?" or "You really should be thankful about getting this 
great information!" or "Most people who get it aren't complaining, so 
obviously we're doing something right!" or "I don't have time and don't care 
to verify that I'm sending useful information, so that's the job of thousands 
of recipients."

Is that neither pinkish nor meatish?

Full credit (and many thanks) to David Golden and others who are moving away 
from this model, but if I'm an ass for saying "You know, that has a lot in 
common with spam" and "CPAN-related services with good intentions should 
carefully consider the effects of their actions on their constituents", then 
I'm proudly an ass as well.

-- c

Re: passing the baton onwards (was Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it))

[brian d foy]

In article <[EMAIL PROTECTED]>, Nicholas Clark
<[EMAIL PROTECTED]> wrote:

> On Thu, Sep 04, 2008 at 02:19:26PM -0400, Greg Sabino Mullane wrote:
> 
> > I recognize that CPAN is a volunteer effort, but it does seem to me there
> > is a implicit responsibility on the part of the author to maintain the
> > module going forward, or to pass the baton to someone else. Call it a Best
> 
> Is there an easy central visible way to flag up a module as "up for adoption"?
> What should have been the right list to ask that question on?

A couple of the PAUSE admins have been talking about that, but we
haven't really decided anything about how it should happen. There would
probably be some virtual PAUSE ID that people could pass primary
maintainership too and once those modules are there, someone could
request maintainership of them without a waiting period.

That's the way that would work with what is already in place, although
someone has to upload a new dist for it to show up in the new account.
I was thinking we'd want to do that anyway to at least modify the docs
to note its status.

Or, Andreas could change PAUSE, which is a bit more involved :)

Re: What do you want? (was Re: The relation between CPAN Testers and quality)

[brian d foy]

In article <[EMAIL PROTECTED]>, Andy Lester
<[EMAIL PROTECTED]> wrote:

> > I'd hate to lose those in my email because other people don't want to
> > filter their mail.
> 
> I'd hate to get spammed because other people don't want to sign up to
> receive them.

You keep saying spam, but that's not the right term. You're being an
ass characterizing it like that. 

Right now there is not a way to opt in, but there is a way to ignore it
if you don't want to see it. I'll sign up for it when there is a way to
do it. It's not a matter of me not wanting to do something, but jsut
dealing effectively with the current situation.

So, Andy, put up or shut up. Send in the patches.

Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)

[chromatic]

On Friday 05 September 2008 10:31:29 Aristotle Pagaltzis wrote:

> I would be interested to know that you don’t care about supporting
> my configuration, but as you don’t even care enough to declare
> your non-support explicitly, I have to find out otherwise.

I don't like the check testers/grumble/upload new distribution with no 
functional changes just niggly little packaging bits you hope will opt out of 
testers tests you don't care about/sleep/repeat cycle.  It's a slow, clunky 
black box game where the rules aren't always clear and you have to upload new 
versions of your distributions that don't do much for most of your users.

I would find some sort of *optional* distribution packaging best practices 
scanner more useful here than getting CPAN Testers reports (by whatever 
mechanism).  If I could see somehow that my distribution implicitly runs on 
Perl 5.001 (or explicitly runs only on 5.11.0), or that it has no Makefile.PL 
or Build.PL, or any of the other dozens of packaging quirks that can cause 
problems, I could fix them before uploading and before triggering a wave of 
testing.

-- c

Re: Reporting Bugs Where they Belong (was Re: The relation between CPAN Testers and quality)

[chromatic]

On Friday 05 September 2008 06:07:53 Aristotle Pagaltzis wrote:

> * chromatic <[EMAIL PROTECTED]> [2008-09-04 23:15]:

> > UNIVERSAL::isa and UNIVERSAL::can are examples of applying the
> > design principle of Report Bugs Where They Are, Not Where They
> > Appear.

> How do you propose doing that in the general case? I am certainly
> interested in what technology you have invented so that computer
> programs can automatically debug themselves and detect the real
> source of any problems.

Who's trying to solve it in the general case?  It's a design *principle*, sort 
of like the architectural design principle that Buildings Should Not Fall 
Down.  You can't make a general rule as to building materials and 
construction techniques used for every building everywhere, but you use that 
principle to help you decide which materials and techniques to use.

See also the Carp module.  (croak() from the point of view of the caller to 
identify misuse of a function or method.)

> > Earlier versions had one tremendous flaw in that they reported
> > all *potential* failures, rather than actual actionable
> > failures explicitly worked around. This was a huge mistake to
> > which I clung to stubbornly for far too long, and I've
> > corrected it in recent versions. However good my intentions in
> > maintaining that feature, the effects worked against my goals.

> Just in the last couple of days, David Golden reported making at
> least two (did I count correctly?) substantial changes to how
> CPAN::Reporter grades tests, in order to prevent particular
> classes of bogus FAILs. Isn’t that a demonstration of exactly the
> same care?

Provided it also identifies the problems on tester machines to help them 
reconfigure/update to produce relevant reports, yes.

-- c

Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)

[chromatic]

On Friday 05 September 2008 08:48:36 David E. Wheeler wrote:

> On Sep 4, 2008, at 10:09, chromatic wrote:

> Well, you can ignore the FAILs. Or you can evaluate each one to
> determine if you could change something your code to make it easier
> for your users. No one compels you to do anything.

You're right, there's no "compel".  If reports don't come by email to people 
who haven't asked for them, then they'll only get reported via an RSS feed I 
can choose to read or not, and on the search.cpan.org pages of my 
distributions, which I don't visit.  That's not compulsion, that's 
just "offering public data points" in yet another location I can choose to 
visit or disregard.  Thus now I can get bug reports via:

 * personal mail
 * RT web/RSS
 * CPAN Forum
 * CPAN Testers (email/RSS)
 * CPAN Annotations
 * CPANTS
 * Usenet
 * PerlMonks
 * countless mailing lists

I may have missed a few.

In my ideal world, CPAN Testers would be a series of platform combinations 
where I could submit a development version of a distribution and get test 
results with the latest version of dependent modules and toolchain modules 
*before* uploading to the CPAN.

 * It's opt-in
 * It answers my most important question when I want it answered
 * It can provide a reference configuration for CPAN installers
 * It gives me something I don't already have
 * It focuses tester resources where (I believe) they're most appropriate

> > I may do so because I take the quality and utility of my software
> > seriously,but do not mistake that for anything which may instill in you
> > any sort of entitlement.  That is an excellent way not to get what you
> > want from me.
> I don't think anyone would argue that. Straw man, dude.

I refer you to Greg Sabino Mullane's posts, in which the gentlest expectation 
is:

  I recognize that CPAN is a volunteer effort, but it does seem to me there
  is a implicit responsibility on the part of the author to maintain the
  module going forward, or to pass the baton to someone else. Call it a Best
  Practice, if you will. The end-user simply wants the module to work.
  Maintainers not paying attention, and the subsequent bitrot that is
  appearing on CPAN, is one of Perl's biggest problems at the moment.
  Careful attention and responsiveness to CPAN testers and to rt.cpan.org is
  the best cure for this.

> > However, by what possible logic can you conclude that the
> > appropriate way to get that bug fixed is to report it to people who, given 
> > all of the information detected automatically, *do not* maintain CPAN.pm?

> Obviously it's not always easy to identify the source of the bug. It
> is the responsibility of CPAN Testers to run the most recent module in
> order to minimize such circumstances.

That would increase the utility of CPAN Testers to me if it were true (at 
least, if by "most recent module" you mean toolchain modules).

> > It's not my "job" to fix bugs in *my own* distributions.  I do it
> > because I care about quality and, contrary to what appears to be near-
> > universal belief around here, I care that people can use my code.
> Straw man again. Do you really believe anyone is actually saying that?

I can't find the link, but someone here on Wednesday said "You don't care if 
people actually use your code", and it's not the first time I've heard it.

-- c

Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)

[Aristotle Pagaltzis]

* Andy Lester <[EMAIL PROTECTED]> [2008-09-05 18:15]:
> > >"Here are test reports reporting on failures for these
> > >things that  we care about you caring about."
> > 
> > Again, this is CPANTS, not CPAN Testers.
> 
> Getting failure reports for a module not running on Perl 5.005
> is a test about something I don't care about.  I don't give
> Shit One if my code runs on 5.005, and yet, I've had failures
> for them.

Sure. You have had failures because your code doesn’t run on 5.5
means it doesn’t run on 5.5 means it doesn’t run on 5.5. If I am
using 5.5 for whatever reason and am considering trying your code
for whatever reason [chromatic: let’s not get into that right
here], then to me it is interesting to know that your code is
going to fail. I don’t give Shit One about how you feel about me
running 5.5, I just want to know if the code works or not. I
would be interested to know that you don’t care about supporting
my configuration, but as you don’t even care enough to declare
your non-support explicitly, I have to find out otherwise.

Regards,
-- 
Aristotle Pagaltzis // 

Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)

[David E. Wheeler]

On Sep 5, 2008, at 09:34, Andy Lester wrote:


Well, yeah, I have too. And sometimes I make a tweak to get things
working on 5.005, and other times I tell my users that it runs 5.006
or later by saying so in Build.PL. Seems reasonable to me to specify
such dependencies.


"Seems reasonable to me" is EXACTLY my frustration.  That is YOUR
standard that YOU are specifying.


Well, how else do your users know what versions of Perl you support?

How about if I start sending email to everyone, whether they want it  
or

not, if their code doesn't run under -T checking?  It seems reasonable
to me that it should.


That's different, isn't it? Your code makes no claims about external  
dependencies. You can control that. Making code run under -T might be  
a decent Kwalitee metric, but it has nothing to do with whether or not  
your tests pass. You can control that in your tests by putting -T on  
the shebang line of individual test scripts.



those of us interested in quality hope you'll tell your users about a
Perl version dependency, but no one is demanding anything.


And you will spam me if I don't provide that dependency.


Spam is in the eye of the beholder, I guess. David Golden won't send  
you any reports anymore, and I completely agree that email reports  
should be opt-in.


This is beyond me and my frustration about getting worthless  
emails.  It

is about the presumption of telling CPAN authors what they can upload.


No one is telling CPAN authors what they can upload.


No, we're not preventing people from uploading anything, but we're
punishing them for not bending to the whims of the CPAN Testers  
ideals.


Punishing? Punishing would be removing a module from CPAN. Getting  
fail report emails is annoying and should be changed to be opt-in.  
Would that solve your problem?


Best,

David

Re: "FAIL Error" - please fix your smoker configuration

[Shlomi Fish]

Hi!

On Thursday 04 September 2008, David Golden wrote:
> On Thu, Sep 4, 2008 at 4:56 AM, Shlomi Fish <[EMAIL PROTECTED]> wrote:
> > "-j2" is invalid for ./Build and you shouldn't use it with it.
> > Alternatively, you can use "perl Makefile.PL ; make ; ", etc., which is
> > also supported by the Error distribution.
> >
> > But as it stands, you're giving many false positives.
>
> FYI, that's a CPAN.pm bug:
>
> http://rt.cpan.org/Public/Bug/Display.html?id=32823
>

Thanks for letting us know.

> Fixed in CPAN 1.92_57.
>
> I strongly encourage testers to upgrade to a recent CPAN dev version
> if you're going to use "-jN" flags.
>
> I've added the need to detect and discard that case to the
> CPAN::Reporter TODO list.
>

Thanks again.

Regards,

Shlomi Fish

-
Shlomi Fish   http://www.shlomifish.org/
Funny Anti-Terrorism Story - http://xrl.us/bjn7t

Shlomi, so what are you working on? Working on a new wiki about unit testing 
fortunes in freecell? -- Ran Eilam

Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)

[Andy Lester]

> Well, yeah, I have too. And sometimes I make a tweak to get things  
> working on 5.005, and other times I tell my users that it runs 5.006  
> or later by saying so in Build.PL. Seems reasonable to me to specify  
> such dependencies.

"Seems reasonable to me" is EXACTLY my frustration.  That is YOUR
standard that YOU are specifying.

How about if I start sending email to everyone, whether they want it or
not, if their code doesn't run under -T checking?  It seems reasonable
to me that it should.


> those of us interested in quality hope you'll tell your users about a  
> Perl version dependency, but no one is demanding anything.

And you will spam me if I don't provide that dependency.

This is beyond me and my frustration about getting worthless emails.  It
is about the presumption of telling CPAN authors what they can upload.
No, we're not preventing people from uploading anything, but we're
punishing them for not bending to the whims of the CPAN Testers ideals.

xoa

-- 
Andy Lester => [EMAIL PROTECTED] => www.petdance.com => AIM:petdance

Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)

[David E. Wheeler]

On Sep 5, 2008, at 09:13, Andy Lester wrote:


"Here are test reports reporting on failures for these things that
we care about you caring about."


Again, this is CPANTS, not CPAN Testers.


Getting failure reports for a module not running on Perl 5.005 is a  
test

about something I don't care about.  I don't give Shit One if my code
runs on 5.005, and yet, I've had failures for them.


Well, yeah, I have too. And sometimes I make a tweak to get things  
working on 5.005, and other times I tell my users that it runs 5.006  
or later by saying so in Build.PL. Seems reasonable to me to specify  
such dependencies.



These are not in test reports. They were in this thread. And they're
suggestions to you. Do what you want with them.


And yet they're encapsulations of this entire problem.  It's  
unsolicited

advice.  "You should make your code handle 5.005.  You should have
checks for such-and-such a platform."  Who is anyone to say?


No one says that. They say, "Hey, this fails on 5.05." No one is  
suggesting what you should do about it. That's your choice. Of course,  
those of us interested in quality hope you'll tell your users about a  
Perl version dependency, but no one is demanding anything.



There is far too much bile floating in this thread considering that I
believe we all have a shared interest in the quality of our code.


That "quality" slider is long and multidirectional.


That doesn't make it any less real.

Best,

David

Re: What do you want? (was Re: The relation between CPAN Testers and quality)

[David E. Wheeler]

On Sep 5, 2008, at 09:10, Andy Lester wrote:


I'd hate to lose those in my email because other people don't want to
filter their mail.


I'd hate to get spammed because other people don't want to sign up to
receive them.


I think that adding changing things so that authors opt-in to getting  
reports is something that's generally been agreed on, though it might  
have to wait until the Web service is done. In the meantime, David  
Golden, at least, has tweaked his reporting toolchain so that it no  
longer submits reports to you, Andy.


Does that work for you?

Best,

David

Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)

[Andy Lester]

> >"Here are test reports reporting on failures for these things that  
> >we care about you caring about."
> 
> Again, this is CPANTS, not CPAN Testers.

Getting failure reports for a module not running on Perl 5.005 is a test
about something I don't care about.  I don't give Shit One if my code
runs on 5.005, and yet, I've had failures for them.


> >"Maybe you should add a co-maintainer."
> >
> >"Your responsibility as an author is to..."
> 
> These are not in test reports. They were in this thread. And they're  
> suggestions to you. Do what you want with them.

And yet they're encapsulations of this entire problem.  It's unsolicited
advice.  "You should make your code handle 5.005.  You should have
checks for such-and-such a platform."  Who is anyone to say?


> There is far too much bile floating in this thread considering that I  
> believe we all have a shared interest in the quality of our code.

That "quality" slider is long and multidirectional.

xoa

-- 
Andy Lester => [EMAIL PROTECTED] => www.petdance.com => AIM:petdance

Re: What do you want? (was Re: The relation between CPAN Testers and quality)

[Andy Lester]

> I'd hate to lose those in my email because other people don't want to
> filter their mail.

I'd hate to get spammed because other people don't want to sign up to
receive them.

xoa

-- 
Andy Lester => [EMAIL PROTECTED] => www.petdance.com => AIM:petdance

Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)

[David E. Wheeler]

On Sep 4, 2008, at 11:53, Andy Lester wrote:

Maybe what's so frustrating to me, and perhaps to chromatic, and  
whoever else ignores CPAN Testers but doesn't discuss it, is that  
we're being fed things that we should be thankful for and goddammit  
why aren't we appreciative??!?


"Here are the things that we have determined are quality."

"Here are test reports reporting on failures for these things that  
we care about you caring about."


Again, this is CPANTS, not CPAN Testers.


"Maybe you should add a co-maintainer."

"Your responsibility as an author is to..."


These are not in test reports. They were in this thread. And they're  
suggestions to you. Do what you want with them.


There is far too much bile floating in this thread considering that I  
believe we all have a shared interest in the quality of our code.


Best,

David

Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)

[David E. Wheeler]

On Sep 4, 2008, at 15:21, David Cantrell wrote:


What I'm not willing to do, however, is to manually check every report
and ensure perfection that way.  Why?  Because it takes too long,  
and I

have a job and a life.


How about checking a random sample of them, just as a sanity check for  
your toolchain?


Best,

David

Re: Reporting Bugs Where they Belong (was Re: The relation between CPAN Testers and quality)

[David E. Wheeler]

On Sep 4, 2008, at 13:32, chromatic wrote:

... but my concern is that no matter how well I document the idea  
that if

T::MO and T::MO::E appear not to work correctly and that there may be
method-as-function bugs causing the problem, I'll again get a flurry  
of bug
reports that I'll have to shunt to other distributions.  More  
likely, they'll
linger in a mail folder for a while and I'll delete them, months  
later.  I am
*not* the person you want reporting bugs that don't affect me.   
Attempts to

make them affect me do not work.  That's just how my brain works.


The emailing of reports clearly has to stop unless a user requests  
them. Reports should be submitted as structured data to a Web service,  
and that Web service should decide what to do with the reports (put  
them into various RSS/Atom feeds, submit requested email reports to  
authors, etc.). I think this is the main thing that will make CPAN  
Testers reports easier to swallow, as it were.


Best,

David

Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)

[David E. Wheeler]

On Sep 4, 2008, at 10:50, David Cantrell wrote:


Change the record, please.  This one's getting boring.

Maybe I should start being equally loud and obnoxious about obviously
stupid and broken things like the existence of UNIVERSAL-isa.  It  
might

give you some appreciation for how you're coming across here.


Let's keep it civil, shall we?

David

Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)

[David E. Wheeler]

On Sep 4, 2008, at 10:09, chromatic wrote:

My job is editor, not programmer.  Also novelist -- but again, not  
programmer.

Certainly not CPAN programmer.


What's your novel? Can I read it?

Paying attention is not my job.  Releasing software I've written  
under a free
and open license does not instill in me the obligation to jump  
whenever a

user snaps his or her fingers.


Well, you can ignore the FAILs. Or you can evaluate each one to  
determine if you could change something your code to make it easier  
for your users. No one compels you to do anything.


I may do so because I take the quality and utility of my software  
seriously,
but do not mistake that for anything which may instill in you any  
sort of
entitlement.  That is an excellent way not to get what you want from  
me.


I don't think anyone would argue that. Straw man, dude.

I don't like this:  failure by any other name would smell just as  
bad. In

other words, if an end user is not going to have a happy, functional
module after typing install Foo::Bar at the CPAN prompt, this is a  
failure

that should be noted as such and fixed by the author.


Then CPAN Testers reports should come with login instructions so  
that I can
resurrect decade-old skills and perform system administration to fix  
broken
installations and configurations -- oh, and as you say, a truly  
*modern*

reporting system should publish these logins and passwords publicly in
syndication feeds.


Huh? I don't think he's referring to configuration issues on the  
tester's box. Clearly that's not the author's responsibility. It's the  
job of CPAN Testers to try to minimize the FAIL reports for such a  
situation, but not your job to change anything when the occasional  
invalid FAIL gets through. That's not to say that CPAN Testers  
couldn't suggest a way for you as an author to cut down on those  
FAILs, but you're not compelled to do anything.


However, by what possible logic can you conclude that the  
appropriate way to

get that bug fixed is to report it to people who, given all of the
information detected automatically, *do not* maintain CPAN.pm?


Obviously it's not always easy to identify the source of the bug. It  
is the responsibility of CPAN Testers to run the most recent module in  
order to minimize such circumstances.


"Oh," perhaps you think, "it's easy for them to read the reports and  
diagnose

the problem remotely on machines they have never seen before, did not
configure, and cannot probe -- and it's so easy for them to file a  
bug in the
right place!"  If you don't think that, precisely what *do* you  
think to

produce such a bold assertion that it is Shlomi's job to install and
reconfigure a new version of CPAN.pm for a CPAN Tester -- or for  
that matter,
everyone with a misconfigured version of CPAN.pm which contains this  
bug?


Straw man again. When I get such a report, I email the CPAN Tester and  
say, "WTF?" He usually gets back to me with, "My bad, sorry, won't  
happen again."


It's not my "job" to fix bugs in *my own* distributions.  I do it  
because I
care about quality and, contrary to what appears to be near- 
universal belief

around here, I care that people can use my code.


Straw man again. Do you really believe anyone is actually saying that?

Best,

David

Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)

[David E. Wheeler]

On Sep 4, 2008, at 01:19, Eric Wilhelm wrote:

But with the per-tester direct mail, the recipient is powerless to  
stop

it, and feeling powerless tends to make people angry.


This, to me, demonstrates better than most points how CPAN Testers is  
being crushed by its own success. A few years ago, I got very few  
reports, so it was no big deal. But nowadays, when I release a module,  
I can expect it to be tested a *lot* in the 24-36 hours after release.  
If I fucked something up, that can be a lot of FAIL mails appearing in  
my inbox.


Inspiration: So for CPAN testers site 404s, we should display a "FAIL  
MAIL". Kinda like the fail whale and the fail snail. :-P


Best,

David

Re: What do you want? (was Re: The relation between CPAN Testers and quality)

[David E. Wheeler]

On Sep 4, 2008, at 21:42, Andy Lester wrote:


I want nothing in my inbox that I have not explicitly requested.


Yes, for email reports, it'd be nice to subscribe to a "list" of your  
own reports -- and to be able to request which reports you want (fail  
only, non-pass, all, etc.).



I want to choose how I get reports, if at all, and at what frequency.

I want aggregation of reports, so that when I send out a module with  
a missing dependency in the Makefile.PL, I don't get a dozen  
failures in a day.  (Related, but not a want of mine, it could  
aggregate by platform so I could see if I had patterns of failure in  
my code).


This makes sense for email reports, but if you use the RSS feed, it's  
not so important. As CPAN testers moves away from using email to  
submit reports, this should get a lot better. I can't wait, frankly.


I want to be able to sign up for some of this, some of that, on some  
of those platforms.


Schedule that for CPAN Testers 3.0. ;-)

I want suggestions, not mandates, in how I might improve my code.  I  
want explanations on my CPAN Testers dashboard that explains why I  
would be interested in having such-and-such an option checked on my  
distributions.  See how the Perl::Critic policies have explanations  
of the problem, and why it can be a problem, in the docs for the code.


I want CPAN Testers to be as flexible as Perl::Critic, and even  
easier to do that flexing.


For a volunteer effort, this could be quite tricky.

I want the understanding that not everyone shares the same coding  
ideals.


Done. Boy, that was easy! ;-P

I want to select what kwalitee benchmarks I choose my code to be  
verified under, so that I can proudly say "My modules meet these  
criteria across these platforms."  I want a couple dozen checkboxes  
of things that could be checked where I say "All my modules had  
better match test X, Y and Z, and these specific modules had also  
better past A, B and C, too."


That's CPANTS, not CPAN Testers.

I want easily selected Kwalitee settings which group together  
options.  Slacker level means you pass these 10 tests, and Lifeguard  
level means you are Slacker + these other 15 tests, and Stringent  
level means something else, all the way up to Super Duper Batshit  
Crazy Anal Perfection level.


Also more like CPANTS. CPAN Testers is all about two things: Does the  
module build and do all the tests pass. Nothing more. For all those  
metrics, you'd have to include them in your tests, methinks.


I want CPAN Testers to do what I can not easily do, which is test my  
code on other platforms on other versions of Perl.


That already happens. My modules are a lot better for it (mainly  
thanks to David Golden and Slaven Rezic.


I do NOT want CPAN Testers to do what I could easily do if I wanted,  
but do not, which is run tests that I don't care about.


Don't ship those tests, then.

I want CPAN Testers to be a service where people say "Hey, have you  
seen CPAN Testers?  You've got to check it out, it will help out  
your code so much" and then they tell their friends and they tell  
their friends, and passing a certain batter of CPAN Testers tests  
consistently is a badge of honor.


I want the Ruby guys go "holy shit, I wish we had something like  
that."


Heh.

Best,

David

passing the baton onwards (was Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it))

[Nicholas Clark]

On Thu, Sep 04, 2008 at 02:19:26PM -0400, Greg Sabino Mullane wrote:

> I recognize that CPAN is a volunteer effort, but it does seem to me there
> is a implicit responsibility on the part of the author to maintain the
> module going forward, or to pass the baton to someone else. Call it a Best

Is there an easy central visible way to flag up a module as "up for adoption"?
What should have been the right list to ask that question on?

> Practice, if you will. The end-user simply wants the module to work.
> Maintainers not paying attention, and the subsequent bitrot that is
> appearing on CPAN, is one of Perl's biggest problems at the moment.
> Careful attention and responsiveness to CPAN testers and to rt.cpan.org is
> the best cure for this.

Alternatively, I could just not upload code to CPAN, and not have this
problem. You're right that it's a problem. I'm not convinced that your
cure will work with all real world volunteers.

Nicholas Clark

Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)

[brian d foy]

In article <[EMAIL PROTECTED]>, David
Cantrell <[EMAIL PROTECTED]> wrote:


> What I'm not willing to do, however, is to manually check every report
> and ensure perfection

I don't think CPAN Testers should manually check every report, but I
would like them to do something a bit more cautious with new setups or
when the failure rate in a certain time period is too high (for
whatever high is).  Once a setup seems to work, just let it go on its
own. :)

Re: What do you want? (was Re: The relation between CPAN Testers and quality)

[brian d foy]

In article
<[EMAIL PROTECTED]>, David
Golden <[EMAIL PROTECTED]> wrote:

> On Fri, Sep 5, 2008 at 12:42 AM, Andy Lester <[EMAIL PROTECTED]> wrote:
> > I want nothing in my inbox that I have not explicitly requested.
> >
> > I want to choose how I get reports, if at all, and at what frequency.
> 
> I'm going to take the first steps towards this over the weekend by
> deprecating author CC's in the code and encouraging top testers to
> upgrade their tools.

I remember some discussion about setting some thing to opt-in or -out
of those.

I'd hate to lose those in my email because other people don't want to
filter their mail.

Re: Reporting Bugs Where they Belong (was Re: The relation between CPAN Testers and quality)

[David Golden]

On Fri, Sep 5, 2008 at 9:07 AM, Aristotle Pagaltzis <[EMAIL PROTECTED]> wrote:
> Just in the last couple of days, David Golden reported making at
> least two (did I count correctly?) substantial changes to how
> CPAN::Reporter grades tests, in order to prevent particular
> classes of bogus FAILs. Isn't that a demonstration of exactly the
> same care?

Technically, I only promised to make them.  ;-)

I have time blocked out on Sunday for doing the actual coding work.

David

Re: Reporting Bugs Where they Belong (was Re: The relation between CPAN Testers and quality)

[Aristotle Pagaltzis]

* chromatic <[EMAIL PROTECTED]> [2008-09-04 23:15]:
> UNIVERSAL::isa and UNIVERSAL::can are examples of applying the
> design principle of Report Bugs Where They Are, Not Where They
> Appear.

How do you propose doing that in the general case? I am certainly
interested in what technology you have invented so that computer
programs can automatically debug themselves and detect the real
source of any problems.

> Earlier versions had one tremendous flaw in that they reported
> all *potential* failures, rather than actual actionable
> failures explicitly worked around. This was a huge mistake to
> which I clung to stubbornly for far too long, and I've
> corrected it in recent versions. However good my intentions in
> maintaining that feature, the effects worked against my goals.

Just in the last couple of days, David Golden reported making at
least two (did I count correctly?) substantial changes to how
CPAN::Reporter grades tests, in order to prevent particular
classes of bogus FAILs. Isn’t that a demonstration of exactly the
same care?

Regards,
-- 
Aristotle Pagaltzis // 

Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)

[Aristotle Pagaltzis]

* Andy Lester <[EMAIL PROTECTED]> [2008-09-04 17:45]:
> Who's to say what my job as an author is?

No one, but at the same time, you as an author of libre software
have no moral right to dictate what your users want from your
code, and if your job according to your understanding does not
extend to satisfying the users’ expectations according to *their*
understanding, then the users have a legitimate case for knowing
that your distributions are made of FAIL as far as they are
concerned. Whether you choose to care is then your prerogative,
obviously.

Regards,
-- 
Aristotle Pagaltzis // 

Re: Module::Build 0.2809 release coming, should we test it?

[David Cantrell]

> > Just a big long list of AUTHOR/dist-1.23.tar.gz lines would be
> > sufficient.
> Thanks.  Does this work?
>   http://scratchcomputing.com/tmp/generated_by.module-build.txt

Perfect.

> Perhaps 2286 is still a lot.  A one-liner tells me there are 474 
> authors.  I wonder if starting with one dist from each author would be 
> a useful sampling, since often the weird stuff happens when an author 
> found a way to do some undocumented thing with M::B and we didn't know 
> about it.  Should I split-out two lists that way?

Please.

> Now, of course setting-up your smokes with MB as the preferred installer 
> and such is important.
> 
> It would also be necessary to be able to check fails from this against 
> the previous MB version.

Yep.  I'll run them all first with the MB that ships with 5.10.0, then
upgrade and do it again.  Do you have a more recent dev release than
what's on the CPAN right now?

-- 
David Cantrell | top google result for "topless karaoke murders"

  All principles of gravity are negated by fear
-- Cartoon Law V

Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)

[Nicholas Clark]

On Thu, Sep 04, 2008 at 09:50:01AM -0700, chromatic wrote:
> On Thursday 04 September 2008 01:19:44 Eric Wilhelm wrote:
> 
> > Let's pretend that I'm a real jerk of an author and I only care about
> > whether my code installs on a perl 5.8.8+ (a *real* perl -- no funky
> > vendor patches) with a fully updated and properly configured toolchain
> > and the clock set to within 0.65s of NTP time.
> 
> Oh great, yet another check I have to add to my Build.PL.  What's the magic 
> cantrip for THAT one?
> 
> (Why yes, I *have* seen bugs related to time skew on network-mounted paths),

So have I. I think that there's at least one stat test in the core that will
fail if you're testing on an NFS mount from a machine where the clock
differs. IIRC it's because file creation time is stamped by the server, and
file modification time is stamped with a time from the client, and if they're
out the "impossible" can happen. Which when you're sanity testing the ops
that read these sorts of things, you're looking out for.

Although I suspect that there can be more general problems with make getting
(correctly) confused by timestamps on files it touched.

Nicholas Clark

Re: What do you want? (was Re: The relation between CPAN Testers and quality)

[David Golden]

On Fri, Sep 5, 2008 at 12:42 AM, Andy Lester <[EMAIL PROTECTED]> wrote:
> I want nothing in my inbox that I have not explicitly requested.
>
> I want to choose how I get reports, if at all, and at what frequency.

I'm going to take the first steps towards this over the weekend by
deprecating author CC's in the code and encouraging top testers to
upgrade their tools.

That has the disadvantage of stopping FAIL reports for everyone,
including those who want them so in the mean time, I or one of the
other CPAN Testers plan to work up a centralized notification that
either limits a FAIL email to one per dist/arch/perl-version tuple or
else puts all reports into a single daily digest.

That isn't entirely "nothing in your inbox", but it should be a
dramatic reduction while we figure out where and how to let authors
set preferences of this sort.  I would imagine that in the meantime we
will have an exclusions list for authors to skip, but it will have to
be manually maintained until we set up a system to automate preference
management.

> I want aggregation of reports, so that when I send out a module with a
> missing dependency in the Makefile.PL, I don't get a dozen failures in a
> day.  (Related, but not a want of mine, it could aggregate by platform so I
> could see if I had patterns of failure in my code).

Have you seen http://matrix.cpantesters.org ?  That's the answer to
the related part of your question.

> I want to be able to sign up for some of this, some of that, on some of
> those platforms.

That will probably be possible once we figure out how to let authors
specify preferences of that type.

> I want CPAN Testers to do what I can not easily do, which is test my code on
> other platforms on other versions of Perl.

Well, this, at least, we're doing today.

> I do NOT want CPAN Testers to do what I could easily do if I wanted, but do
> not, which is run tests that I don't care about.

Fortunately, we're not doing this.  (As someone mentioned, CPAN
Testers is not CPANTS.)  Unless you count "testing for prerequisites"
which we're doing only because they're part of the PL/make/test cycle.

> I want the Ruby guys go "holy shit, I wish we had something like that."

+1

-- David

Re: What do you want? (was Re: The relation between CPAN Testers and quality)

[Aristotle Pagaltzis]

* Andy Lester <[EMAIL PROTECTED]> [2008-09-05 06:45]:
> I want nothing in my inbox that I have not explicitly
> requested.
>
> I want to choose how I get reports, if at all, and at what
> frequency.
>
> I want aggregation of reports, so that when I send out a module
> with a  missing dependency in the Makefile.PL, I don't get a
> dozen failures in a day.  (Related, but not a want of mine, it
> could aggregate by platform so I could see if I had patterns of
> failure in my code).
>
> I want to be able to sign up for some of this, some of that, on
> some of those platforms.

These are currently difficult as a matter of architecture, which
is “every tester sends mail directly to every author.” The design
overhaul will centralise the gathering and issuing of reports, so
all of these things will become possible in the medium term. You
are not the only one to ask for them, FWIW.

> I want to select what kwalitee benchmarks I choose my code to
> be verified under, so that I can proudly say "My modules meet
> these criteria across these platforms." I want a couple dozen
> checkboxes of things that could be checked where I say "All my
> modules had better match test X, Y and Z, and these specific
> modules had also better past A, B and C, too."
>
> I want easily selected Kwalitee settings which group together
> options.  Slacker level means you pass these 10 tests, and
> Lifeguard level means you are Slacker + these other 15 tests,
> and Stringent level means something else, all the way up to
> Super Duper Batshit Crazy Anal Perfection level.

It seems that you are confusing the CPAN Testers with the CPAN
Testing Service (CPANTS), Domm’s pet project. The only relation
between the two is their confusingly similar naming. CPANTS tries
to lint-check your distribution and code without running any of
it; the CPAN Testers download your releases, install any prereqs
and then run your test suite. The goals and designs of the two
projects as well as their participants are entirely different.

Regards,
-- 
Aristotle Pagaltzis // 

Re: The relation between CPAN Testers and quality (or why CPAN Testers sucks if you don't need it)

[chromatic]

On Thursday 04 September 2008 15:21:19 David Cantrell wrote:

> What I'm not willing to do, however, is to manually check every report
> and ensure perfection that way.  Why?  Because it takes too long, and I
> have a job and a life.  And anyway, I'd still make mistakes - and even if
> I don't make mistakes, people will still think I have.  Everyone makes
> mistakes when they're doing a boring job, doubly so without the prospect
> of reward.  So anyone who insists that I read every report I send to them
> will just get no reports from me.

With respect, the people receiving and manually checking those unsolicited, 
automated, opt-out bulk reports also have jobs and lives.

-- c