On Sun, Apr 17, 2005 at 12:17:17AM +0000, Smylers wrote: > Remember that we aren't measuring quality, but kwalitee. Kwalitee is > supposed to provide a reasonable indication of quality, so far as that's > possible. So what matters in determining whether a kwalitee heuristic > is appropriate is whether there is a correlation between modules that > pass the heuristic and those that humans would consider to be of high > quality.
I don't think this is what kwalitee is. (Of course, I may be wrong.) If we consider your example of Bayesian spam detectors, they only work once you've trained them. And that's not what's going on here. We have no indications of which are Quality modules in the first place, such that we can run tests to see what the, perhaps surprising, common features are that we could then set up as Kwalitee checks. Rather, Kwalitee is that subset of Quality which can be measured in an automated manner. You should be able to look at a Kwalitee check and agree that it does indeed constitute Quality. > so even if a neural net (or whatever) did come up > with the above substring heuristic, once it's know then authors can game > the system by artificially crowbarring into their modules' sources, at > which point the heuristic loses value. I thought the idea was that we /wanted/ people to increase their Kwalitee, and thus their Quality. The things we designate as Kwalitee indicators should be things that module authors are encouraged to do. "Gaming" the system in this environment is to be welcomed. Tony
