I consider the impact factor (IF) properly used as a valid measure in comparing of journals; I also consider the IF properly used as a possibly valid measure of article quality. But either use has many possible interfering factors to consider, and these measurements have been used in highly inappropriate ways in the past, most notoriously in previous UK RAEs.
Stevan mentions one of the problems. Certainly the measure of the impact of an individual article is more rational for assessing the quality of the article than measuring merely the impact of the journal in which it appears. This can be sufficiently demonstrated by recalling that any journal necessarily contains articles of a range of quality. More attention is needed to the comparison of fields. The citation patterns in different subject fields varies, not just between broad subject fields but within them. In the past, UK RAEs used a single criterion of journal impact factor in ALL academic fields; this was patently absurd (just compare the impact factors of journals in math with those in physics, or those in ecology with those in biochemistry). To the best of my knowledge they have long stopped this. (This incorrect use did much to decrease the repute of this measure, even when correctly used.) In comparing different departments, the small scale variation between subjects specialisms can yield irrelevant comparisons, because few departments have such a large number of individuals that they cover the entire range of their subject field. I'll use ecology as an example: essentially all the members of my university's department [Ecology and Evolutionary Biology] work in mathematical ecology, and we think we are the leading department in the world. Most ecologists work in more applied areas. The leading journals of mathematical ecology have relatively lower impact factors, as this is a very small field. This can be taken into account, but in a relatively small geopolitical area like the UK, there may be very few truly comparable departments in many fields. It certainly cannot be taken into account in a mechanical fashion, and the available scientometric techniques are not adequate to this level of analysis. The importance of a paper is certainly reflected in its impact, but not directly in its impact factor. It is not the number of publications that cite it which is the measure, but the importance of the publications that cite it. This is inherently not a process that can be analyzed on a current basis. There is a purpose in looking at four papers only: in some fields of the biomedical sciences in particular, it is intended to discourage the deliberate splitting of papers into many very small publications, with the consequence that in some fields of biomedicine a single person might have dozens in a year, adding to the noise in the literature. One could also argue that a researcher should be judged by the researcher's best work, because the best work is what primarily contributes to the progress of science. In most other respects I agree with Stevan. I will emphasize that the publication of scientific papers in the manner he has long advocated will lead to the posiblility of more sophisticated scientometrics. This will provide data appropriate for analysis by those who know the techniques, the subject, and the academic organization. The data obtainable from the current publication system are of questionable usefullness for this. Dr. David Goodman Biological Sciences Bibliographer Princeton University Library [email protected]
