I haven't looked at what's going on in CPANTS for a while but Andy's post made me have a look and oh dear. There's a problem. CPANTS is not a game. If you make it a game, the system does not work.
Let's review. CPANTS is not a measure of module quality since module quality is not well defined and difficult to measure. CPANTS is a measure of those things which can be measured about quality modules. To use a real world example, fast cars tend to be red. There is no direct relationship between a CPANTS kwalitee indicator and the actual quality of the module, it is merely a statistical indicator. A car can be red without being fast. Conversely, a blue car can be fast. If I alter my distribution to increase its CPANTS score I have not reduced the bug count, made it more efficient, improved the documentation or given it an easier interface. I have not done any of the difficult to measure things which we associate with improving the real quality of a module. [1] If I paint my car red, it does not go faster. If people *think* red cars are faster, and I want to sell my car, I'm going to paint my car red. People will now be more likely to buy my car thinking it goes faster, but they have been fooled. I have gamed the system, used my knowledge of its rules to my own gain. The more people game CPANTS, the more they alter their modules specificly to increase their CPANTS score, the less useful the CPANTS indicators will be. If everyone paints their car red then its no longer a valid indicator of whether its a fast car. The indicator is now useless. CPANTS is already easy to game since its rules are published. This is hard enough to deal with, but gaming is gleefully encouraged! Its called "The CPANTS Game". There's a scoreboard with a top 40, hall of fame and shame. Failure to have a CPANTS indicator is expressed as a "shortcoming" which must be "remedied". In order to best perform its function and reduce the urge to game, CPANTS should... * not express itself as a game or competition * not widely publish its rules * express its indicators in a neutral fashion without indication as to which state of the indicator is better * not suggest ways to "fix" your module * not publish a numerical score which one may be compelled to raise, instead a less precise indicator such as color * not publish a scoreboard, top ten, hall of fame or shame * add a feedback loop so that certain human-checked low and high quality modules are checked against their CPANTS indicators to see which still have value. Adjust the indicator's weights accordingly. Sorry if it sucks all the fun out of it, and I don't mean to poo-poo the work Thomas and others have done, but I think we've forgotten what CPANTS is. [1] I may have improved the real quality of the distribution, but not the payload contained within. Using CPANTS as a distribution improvement tool is another story and may ultimately be its best destiny.
