Re: About tidying up Kwalitee metrics

[Gabor Szabo]

On Thu, Jun 26, 2008 at 2:23 AM, Hilary Holz <[EMAIL PROTECTED]> wrote:
> On 6/25/08 10:24 AM, "chromatic" <[EMAIL PROTECTED]> wrote:
>
>> On Wednesday 25 June 2008 03:15:59 Thomas Klausner wrote:
>>
>>> One comment regarding 'each devel sets his/her own kwalitee metrics':
>>> This could be quite easy for the various views etc. But I'm not sure how
>>> to calculate a game score then. Do we end up with lots of different
>>> games? But then, it's only the game (which still motivates a few
>>> people..)
>>
>> Removing the game score completely would fix a lot of what I consider wrong
>> with CPANTS.
>>
>> -- c
> second!

It seems that the game theme is after all turned into "fierce competition" or
"lack of interest" depending on ... I don't know on what, but neither
is good for
CPAN.
In some cases - me included - people fix the symptom to get the metric point
while the underlying code does not really change. So the indicator stops being
an indicator.

I don't know how to fix that.
Maybe the suggestions above and elsewhere to get rid of the game theme
and the "top N" "bottom N" authors would help.

Maybe what we need to do is
1) remove the game
2) fix the current metrics (e.g. license is not correct now)
3) Add detailed explanations for each metric, or maybe to create a page on
the TPF Perl 5 wiki for each metric where it would be easier to provide
pro and contra explanations for each metric.
4) add more metrics (including those that collect data from external sources)
5) categorize the metrics as suggested by Salve
6) get the search engines to start to use some of the metrics
 in their search results.

Not necessarily in that order

Gabor

Re: About tidying up Kwalitee metrics

[chromatic]

On Saturday 28 June 2008 08:54:34 Gabor Szabo wrote:

> It seems that the game theme is after all turned into "fierce competition"
> or "lack of interest" depending on ... I don't know on what, but neither is
> good for CPAN.
> In some cases - me included - people fix the symptom to get the metric
> point while the underlying code does not really change. So the indicator
> stops being an indicator.

Exactly -- and in other cases, the metric point is actively harmful to the 
CPAN.

> Maybe what we need to do is
> 1) remove the game
> 2) fix the current metrics (e.g. license is not correct now)
> 3) Add detailed explanations for each metric, or maybe to create a page on
> the TPF Perl 5 wiki for each metric where it would be easier to provide
> pro and contra explanations for each metric.
> 4) add more metrics (including those that collect data from external
> sources) 5) categorize the metrics as suggested by Salve
> 6) get the search engines to start to use some of the metrics
>  in their search results.
>
> Not necessarily in that order

Full support from me on these.

-- c

Re: About tidying up Kwalitee metrics

[Aristotle Pagaltzis]

* chromatic <[EMAIL PROTECTED]> [2008-06-25 19:25]:
> Removing the game score completely would fix a lot of what I
> consider wrong with CPANTS.

I think the game is actually an excellent idea. The problem is
with the metrics. Here are some metrics that are inarguably
good:

• has_buildtool
• extracts_nicely
• metayml_conforms_to_known_spec
• no_pod_errors
• has_tests_in_t_dir
• is_prereq

Here are some that have proven harmful:

• has_test_pod
• has_test_pod_coverage

What’s the difference? It’s whether the things that are measured
are used as a proxy for some other ostensibly desirable property.

Kwalitee is helpful when it measures good form directly and
unhelpful when it measures an arbitrarily chosen indicator of one
of many ways to adhere to good form. (Good style, in contrast,
cannot be measure. Kwalitee is not quality.)

The game merely amplifies this: helpful metrics become actively
helpful and unhelpful metrics become actively unhelpful.

I am in favour of keeping the good effects of the game and
removing the bad ones. That means cleaning up the metrics roster
– for the game. Some not universally applicable metrics may still
be of interest, after all. They can stay – as long as they are
removed from the game. And by this I mean removing them entirely,
that is, they should not figure into the score at all, not even
as uncounted bonus points or such, nor even shown on any page
that ranks authors.

Take Paul Fenwick’s recent proposal in this light: he wants to
know whether a module accidentally untaints values or not, but
what he proposed is to measure whether a module does `use re
'taint';` – nevermind that this is inapplicable to a large class
of modules OT1H and doesn’t even cover all the ground OTO.
(Iterating a hash’s keys returns strings that are never tained.)
Thus it might be useful to see a page listing the distributions
whose modules do `use re 'taint';`, but it’s less than helpful
to rank module authors by how many of their modules do so.

Regards,
-- 
Aristotle Pagaltzis //