>bb> "Where an article is published is an irrelevant issue. A top >bb> quality piece of work, in a freely available medium, should get >bb> top marks. The issue is really that many assessment panels use >bb> the medium of publication, and in particular the difficulty of >bb> getting accepted after peer review, as a proxy for quality. But >bb> that absolutely does not mean that an academic who chooses to >bb> publish his work in an unorthodox medium should be marked down. >bb> At worst it should mean that the panel will have to take rather >bb> more care in assessing it."
On Monday, November 25, 2002, at 03:04 PM, Stevan Harnad wrote: > A rather complicated statement, but meaning, I guess, that the RAE, is > assessing quality, and does not give greater weight to paper journal > publications than to online journal publications. Funny how the same words can be read in different ways. When Bahram says 'medium', I read it as 'journal', 'channel of communication': a 'carrier-neutral' notion, so to speak, not making the distinction at all between 'print' and 'electronic'. His concern is that the journal, or more in particular, the journal's perceived acceptance policy upon peer-review, is used as a proxy for quality. This acceptance policy can be as strict in on-line journals as in print ones, so there would be no reason for him to equate strict policies with those employed by print journals. > But I would be more skeptical about the implication that it is the RAE > assessors who review the quality of the submissions, rather than the > peer-reviewers of the journals in which they were published. Some > spot-checking there might occasionally be, but the lion's share of the > assessment burden is borne by the journals' quality levels and impact > factors, not the direct review of the papers by the RAE panel! > (So the *quality* of the journal still matters: it is the *medium* of > the journal -- on-paper or online -- that is rightly discounted by > the RAE as irrelevant.) The quality of journals matters, but quality is not the same as impact factor. Possibly, journals with the highest impact factors can be seen to be -- in general -- of higher quality than those with low impact factors, but, as one often sees, rankings on the basis of differences that run into the single digit percentage points (e.g. IF 2.35 vs IF 2.27) are utterly meaningless. It is a known phenomenon that impact factors are highly vulnerable to manipulation and that in just about any given journal a minority of articles is commonly responsible for the bulk of the citations on which the impact factors are based. An American medical journal will almost always have a very much higher impact factor than its European qualitative equivalent, simply because in the medical areas the culture 'dictates' that American authors publish in the main in American journals and do not cite their European colleagues, whereas European authors publish as much in American as in European journals and usually cite all relevant literature, be it American or European. Quality in this example is not easily measurable in terms of impact factors. > (Hence the suggestion that a "top-quality" work risks nothing in being > submitted to an "unorthodox medium" -- apart from reiterating that > the medium of the peer-reviewed journal, whether on-line or on-paper, > is immaterial -- should certainly not be interpreted by authors as RAE > license to bypass peer-review, and trust that the RAE panel will review > all (or most, or even more than the tiniest proportion of submissions > for spot-checking) directly! Not only would that be prohibitively expensive > and time-consuming, but it would be an utter waste, given that peer > review has already performed that chore once already! However, if 'unorthodox medium' means 'new journal with an unorthodox publishing model' (after all, since most journals have an on-line edition nowadays, being electronic by itself would hardly have been described by Bahram as 'unorthodox'), then authors of top-quality work are perceived to take a risk by publishing in them, for these unorthodox new journals will not have an impact factor yet. This is not to say that articles in the new open access journals are not cited as often as in conventional journals -- on the contrary: we have strong indications at BioMed Central that they are actually cited a great deal more often than similar articles published conventionally. The system of impact factors, however, is stacked against new journals and has a considerable bias toward entrenched journals and their toll-gate models. Fortunately, this is only an irritating, but temporary problem as the rates at which articles published in BMC open access journals are cited, will ensure high impact factors once the Impact Factory deems the time ripe to calculate them. >jv> HEFCE clearly recognises the flaws of the RAE methodology used >jv> hitherto, which is the first step towards a more satisfactory >jv> assessment system. What is not clear to me is the question whether >jv> your suggested reform will indeed be saving time and money. It seems >jv> to me that just adding Impact Factors of articles is indeed the shortcut >jv> (proxy for quality) that Bahram refers to, and that anything else will >jv> take more effort. I don't pretend to have any contribution to make >jv> to that discussion on efficiency of the assessment methodology, though. > > I couldn't quite follow this. Right now, most of the variance in the > RAE rankings is predictable from the journal impact factors of the > submitted papers. That, in exchange for each university department's > preparing a monstrously large portfolio at great time and expense > (including photocopies of each paper!). I agree that the preparation of large portfolios is a waste of time and expense if all that happens is straightforwardly adding impact factors of the journals in which the papers have been published. So it is not so much that RAE rankings are 'predictable' from the impact factors, they are *based on* the impact factors. > Since I seriously doubt that Bahram meant replacing impact ranking by > direct re-review of the all the papers by RAE assessors, I am not quite > sure what you think he had in mind! (You say "just adding Impact Factors > of articles is indeed the shortcut" but adding them to what, how?) Adding as in 'adding up', Stevan, 'tallying'. > At Southampton we are harvesting the RAE 2001 returns > http://www.hero.ac.uk/rae/ into a demo -- RAEprints -- to give a taste > of what having a global national open-access research archive would > be like, and what possibilities it would open up for research access, > impact, and assessment. (For a preview, see: http://www.hyphen.info/ ) We agree that open access could help, even with research impact assessment. Jan.
