On Tue, 26 Nov 2002, Jan Velterop wrote: > Scientometrics and other metrics are about counting what can be > counted... So 'quantity' is dealt with. What about 'quality'? > Quality is relative, and based on judgement... utterly subjective, > so what we count is 'votes'. Do more votes mean a higher 'quality' > than fewer votes? Does it matter who does the voting?
All good scientometric questions, it seems to me (even the one about how to identify and weight voting "authorities"). How to answer, if not scientometrically? (Or do you think it should just be a matter of individual opinion or taste?) > I think it [matters who does the voting], at least in these matters, > and therefore a review process is needed that ranks things like > originality, fundamental new insights, and yes, contributions to > wider dissemination and understanding as well, in order to base > important decisions on more than just quasi-objective measurements. Is this not among the things peer review is supposed to do? These are almost literally the questions that appear in many referee evaluation forms. Are you proposing a second round of review, a few years after a paper appears? By all means, if you have the time and resources. And certainly the RAE should include such secondary review data in its scientometric equation too, if they are available in time. But in what way is any of this an alternative to the quantitative, scientometric assessment of research quality and impact? The only ones who are not doing it scientometrically are the reviewers themselves (whether in the primary peer review or in the second one Jan recommends). But their judgments are just votes (i.e., scientometric data) too, just as the journal-names are, in 1st-round peer review. Perhaps reviewer-names could accrue some objective scientometric weight too, for the second round. But this is all speculation about what the future scientometric analyses will yield, once we have these (open access) data available to do all these analyses on. For the RAE, unless Jan is recommending that the assessors do a 3rd round of direct review of all their submission themselves, scientometrics (yes, counting!) seems to be the only way they can do their ranking (which is likewise counting). > Fortunately, in biology such secondary review is beginning to take shape: > Faculty of 1000 (www.facultyof1000.com). It shows that the subjective > importance of articles is often unconnected, or only very loosely connected, > to established scientometrics. It constantly brings up 'hidden jewels', > articles in pretty obscure journals that are nonetheless highly interesting > or significant. I would certainly want to use Faculty of 1000 ratings and citations in my multiple regression equation for impact, perhaps even giving them a special weight (if analysis shows they earn it!). But what is the point? This is just a further source of scientometric data! > I am sure that automated, more inclusive, counting of votes made possible by > open and OAI-compliant online journals and repositories will help the > visibility of those currently outside the ISI Impact Factory universe, such > as the journals from Bhutan. But it can't replace judgement. No, it can't replace judgment. Like all other analyses, it can merely quantify the outcomes of judgments, and weigh them, against one another and against other measures. What else is there? Even the decision to browse, read, and cite is just a set of human judgments we are counting and trying to use to predict with. Predict what? Later human performance, and findings, and judgments, i.e., research impact. I don't think that in reminding us that all of this is based on human judgment (and, of course, on empirical reality, in the case of science), Jan is not giving us an alternative to scientometric quantification. He is just reminding us of what it is that we are quantifying! Stevan Harnad
