On Wed, 6 Nov 2002, Andrew Odlyzko wrote: >sh> (R1) Your primary motivation is (R) to reform research >sh> communication, mainly through the posting of all papers online, >sh> unrefereed, and then relying on self-selected "vetters," in >sh> place of of classical peer review, to evaluate, improve and >sh> thereby sign-post paper quality.
>ao> I would not phrase it this way. I would say that my primary motivation >ao> is indeed to reform research communication, mainly through the posting >ao> of all papers online, unrefereed, and then relying on whatever mixture >ao> of classical peer review, or contributions of self-selected "vetters," >ao> the community decides to rely on. I do not hold dogmatic views of how >ao> peer review will be handled, and predict only rough trends. It is gratifying to know that you are only predicting rough trends. It is important to make it quite explicit here just how close our position is, so as to pinpoint exactly what it is that we disagree on (i.e., the trends you predict): (1) My motivation is freeing access to all research, both pre- and post peer review (i.e., nut just unrefereed preprints), through self-archiving. (2) If we could (for some arbitrary reason!) free only one of the two, I would choose the refereed final draft rather than the unrevised preprint, but there is no reason not to free both (with a few special exceptions: see below). (3) Given success in convincing the world research community to free access to their research by self-archiving both their preprints and their postprints, we would eo ipso also have all the benefits of "self-selected vetting" that you (and I!) both value. But this would be a *supplement* not a *substitute* for classical peer review. (Nothing whatever would be lost, and everything would be gained!) (4) If I were you, and I believed self-selected vetting will eventually replace classical peer review, I would not feel any need to add anything to (1) - (3). Self-archiving, and the open access it brings, would be all I would need to evangelize for. The rest (if my belief that vetting could and would replace peer review was correct) would then take care of itself. (5) But self-archiving is (we both agree) taking place far too slowly. (6) Among the reasons self-archiving is taking place so slowly are worries that researchers have that hold them back from self-archiving. The two main ones are: (i) the worry that self-archiving would compromise or destroy peer review http://www.eprints.org/self-faq/#7.Peer and (ii) the worry that self-archiving would violate copyright http://www.eprints.org/self-faq/#10.Copyright (7) As we both regard open access as optimal, inevitable, and long overdue, we should both be concerned with relieving worries (i) and (ii). (8) Your belief that self-vetting will eventually replace classical peer review is one that would *reinforce* rather than relieving researchers' worry on that score. Hence, unless you could persuade them not only that it will happen, but that it too is optimal, voicing your belief (and it is only a belief, a hypothesis), even as the prediction of a "rough trend" seems more likely to slow rather than speed the transition to what we both agree is the optimal and the inevitable outcome (open access). (9) I believe strongly that your belief (that self-vetting will replace classical peer review) is wrong, and that there is no trend, rough or smooth, in that direction, now till there be. (I believe that, not dogmatically, but for the very concrete reasons I have repeatedly adduced in this series of exchanges.) But if someone like me -- who believes fully in self-archiving and the transition to open-access, and disbelieves in any causal connection between that and any risk to classical peer review -- is not persuaded by your contrary belief, nor the arguments you adduce in its support, then how likely is it that someone who does not yet believe in self-archiving *and* worries that it would be a risk to peer review will be emboldened (to self-archive) by your hypothesis (even formulated as a "rough trend")? (10) The optimality of open access to the research literature is a certainty, not a hypothesis. We both agree about that, and about the ample evidence that it maximizes research visibility, accessibility, uptake, usage, citation, and impact, as well as scope, speed, and interactivity, in short, that it greatly benefits research and researcher productivity. (11) The transition from classical peer review to self-selected vetting, in contrast, is merely a hypothesis (and one with -- in my view -- much a priori evidence and many reasons to conclude that it is incorrect, but never mind) -- a hypothesis that, on the face of it, looks as if it would retard the transition to open access through self-archiving by reinforcing the worries of those who do not self-archive precisely because they are afraid it would destroy classical peer review! (11) So why even voice the hypothesis? For if it is true, then it will be a causal consequence of self-archiving anyway; hence, since self-archiving is the necessary and sufficient condition for it, everything should be done to hasten self-archiving. Yet voicing this (fallible) hypothesis is very likely to slow, rather than hasten self-archiving. (12) As to my (arbitrary, desert-island) forced choice of open access for the peer-reviewed final draft over the unrefereed preprint, not only is it not a choice anyone has to make (since there is in most cases no reason not to have both), but I even have an explicit commitment to the self-archiving of unrefereed preprints, for two reasons: (13) The first benefit of self-archiving the unrefereed preprint is one on which we both agree -- that it hastens and strengthens the progress, interactivity and self-correctiveness of research. (14) The second benefit of self-archiving the unrefereed preprint is that it is a means of legally self-archiving even in the face of the most restrictive copyright transfer agreement: The self-archiving of the preprint pre-dates any agreement or even any submission to a journal; and after submission, refereeing, revision, and acceptance, if the copyright agreement forbids self-archiving the final draft, one need only self-archive the corrections that would have to be made by the user on the already public preprint to make it conform to the final draft. http://www.eprints.org/self-faq/#publisher-forbids (15) Hence, besides being highly beneficial in its own right, the self-archiving of the unrefereed preprint is an ally in ensuring open access to the contents (if not the form) of the refereed final draft too. (16) So we agree on the importance and sure benefits of self-archiving preprints and we agree on the importance and sure benefits of open access. We disagree only on your hypothesis that self-archiving will eventually lead to the replacement of classical peer review by self-selected vetting (or that there is any "rough trend" in that direcrion.) (17) Yet we agree that even if the hypothesis is correct, all it needs is self-archiving -- and we both believe fully in the importance and benefits of self-archiving. (18) Perhaps what self-archiving needs now is facts rather than hypotheses. The benefits of self-archiving are demonstrated facts, whereas your hypothesis remains completely untested (and the likelihood of success for any eventual experiment would seem to be contradicted by a number of prima facie logical and practical considerations that I have repeatedly listed in these exchanges). (19) There is only one remaining problem with emphasizing (as you do), only the self-archiving of pre-refereeing preprints, rather than also the refereed postprints: (20) There are special cases -- I couldn't say how many, or how their distribution varies by field, but certainly a non-zero number, as the "preprint culture" of physics is by no means universal yet -- in which the author would prefer not to make his research public before it is (classically) peer-reviewed. (There are also cases, particularly in the medical literature, where the public posting of unrefereed findings might represent a danger to public health.) http://www.nih.gov/about/director/ebiomed/com0509.htm#harn45 (21) This is a further reason for emphasizing the self-archiving of the refereed postprint. Otherwise, such authors/papers are also lost to self-archiving altogether. (22) If such authors go on to sign over-restrictive copyright transfer agreements after refereeing then they may still be lost to self-archiving for the time-being, because for them the preprint-plus-corrigenda strategy above (14) will not work (there being no already-archived preprint); but there seems no reason to lose open access to the research of all preprint non-archivers by restricting self-archiving to preprints alone, rather than preprints and postprints. (23) Finally, many sceptics about the benefits of open access will only be won over once the refereed, published postprints are openly accessible, and not merely the unrefereed preprints. (24) Having said all this, I have sufficient confidence in the self-correctiveness of open online communication (such as this exchange) not to worry too much about the possible untoward effects of the airing of your hypothesis on those who are reluctant to self-archive because of worries about peer review. As long as both sides are aired, let us trust the outcome to (self-corrective) human judgment. (I hope Fiona Godlee will agree to co-publish my reply along with your hypothesis in the collection in which it will appear. Thanks for agreeing to post this exchange to the American Scientist Forum.) >sh> (A1) My primary motivation is (A) to make the classically peer-reviewed >sh> research literature we have now openly accessible online, such as >sh> it is, to maximize its accessibility, usage and impact. Qualified >sh> journal editors continue to select the referees, referees and author >sh> remain answerable to the editor, and the journal-name and its quality >sh> track-record sign-post for potential users that an article has met >sh> that journal's quality-standards. >sh> >sh> (R2) You see a causal connection between A and R: A will lead to R. > >ao> Yes, provided that (A) involves open access to preprints. (The way you >ao> have phrased (A), it could encompass a system in which the Ingelfinger >ao> rule would be universal, it's just that after publication by a journal, >ao> articles would be freely available. That form of (A) would not yield >ao> to (R).) I agree, and I hope that what I wrote above now shows that this is not at all what I meant. I am both strongly in favor of the self-archiving of preprints and strongly opposed to the Ingelfinger rule and have published critiques of it. However, I am at least equally in favor of the self-archiving of the refereed postprints too: Harnad, S. (2000) Ingelfinger Over-Ruled: The Role of the Web in the Future of Refereed Medical Journal Publishing. Lancet Perspectives 256 (December Supplement): s16. http://cogprints.soton.ac.uk/documents/disk0/00/00/17/03/ Harnad, S. (2000) E-Knowledge: Freeing the Refereed Journal Corpus Online. Computer Law & Security Report 16(2) 78-87. [Rebuttal to Bloom Editorial in Science and Relman Editorial in New England Journal of Medicine] http://cogprints.soton.ac.uk/documents/disk0/00/00/17/01/ >sh> (R4) You think that the research advances that occur before peer review >sh> through the online posting of pre-refereeing preprints today are evidence >sh> that peer review is unnecessary and can be replaced by spontaneous vetting >sh> (R) without loss (and perhaps even with a gain) in quality. > >ao> Not that "peer review is unnecessary," but that "classical peer review is >ao> unnecessary." I would also quarrel with the phrasing of the last part >ao> of this point, but will let it go for lack of time. It would be useful to know precisely what you mean -- especially since this exchange is really meant to clarify things for those who may be worried that self-archiving poses a risk to classical peer review. I am convinced that there is no causal connection whatsoever between self-archiving and changes in classical peer review (except that online pre-refereeing feedback is a useful supplement to peer review, and that the online medium will make it possible to implement classical peer review more quickly, cheaply, efficiently and equitably). Hence there is no reason to refrain from self-archiving because of worries about peer review: http://www.eprints.org/self-faq/#7.Peer On the other hand, a system consisting exclusively of self-selected online vetting is no form of peer review, classical or otherwise (except in those cases where it happens to happen by chance!). >sh> There is not only this total empirical gap between the data you >sh> use and the conclusions you draw, but there are also logical >sh> gaps: You have not replied when I have asked how, in a system >sh> where classical peer review and journal-names with track-records >sh> are no longer there as the back-up and bottom line -- as they are >sh> universally and without exception now -- how the annual 2,000,000 >sh> papers (which are today refereed and sign-posted by refereed >sh> journals) will find their proper vetting, and how this will be >sh> sign-posted for potential users? This question does not even come >sh> up in the case of pre-refereeing preprints, because those are a >sh> "parallel economy," backed up by the classical peer-review and >sh> then sign-posted by the names and track-records of the journals to >sh> which just about every one of those preprints has been submitted, >sh> and in which they will all appear eventually (though perhaps only >sh> after several rounds of revision and refereeing, and perhaps not >sh> always at the level of the journal to which they were submitted >sh> first.) > >ao> I did not reply because I did not have time to reply to all of your >ao> points. Since you make this such a central point, let me respond >ao> now (although very briefly and so inadequately). How will all >ao> those papers "find their proper vetting"? Well, how do they find >ao> proper refereeing now, under your vaunted classical peer review? By being submitted to a journal, whose editor (presumably a recognized expert in the field) is responsible for (1) selecting qualified referees (busy, reluctant, but willing-if-asked by the right journal/editor), (2) deciding which of the referees' recommendations are valid and need to be satisfied in order to meet the journal's established quality standards, and (3) making sure the accepted, final draft has satisfied them. The result is then recognizably (4) tagged (sign-posted) as having been thus quality-controlled by the journal-name (and associated track-record). That's all there is to classical peer review: Human expert judgment, systematized, answerable, and reliably labelled accordingly. How does anarchic self-selected vetting ensure an equivalent outcome, and how is it to be sign-posted? >ao> We know that serious frauds like the Jan Hendrik Schoen slip through. I have replied about fraud in earlier postings. In brief, fraud will be found out anyway, because one cannot build on it, and research progress is about building cumulatively on findings, not merely reporting them. But, in any case, with open-access preprints and postprints as a supplement to peer review, all the benefits of the extra lines of defence are there, over and above peer review, without any need to change classical peer review in any way (other than to make it faster and more efficient in finding and reaching referees and in distributing the refereeing load more broadly and evenly). Re: A Note of Caution About "Reforming the System" http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/2322.html >ao> We also have plenty of evidence that lots of simply >ao> not very solid science that is not fraudulent gets through. Yes, human judgment, even expert peer judgment, is fallible. And supplementing its systematic, answerable, labelled application with open feedback will be very useful: But why construe this supplement as a substitute? Why would it not simply co-exist with the classical line of defense? >ao> (I don't have time to dig up references, but there was a paper >ao> quite a while ago that looked at the statistical methodology used >ao> in a large sample of medical papers. It found a horrendously high >ao> rate of misapplications of statistics. Horrendous compared to what? Compared to how many would have been found if the raw preprints had all simply been publicly posted, hoping that the right vetters would find and test them all for us all spontaneously? What evidence is there that they could or would, without the systematic mediation of someone who was qualified and responsible for the outcome (such as a journal editor)? Or would attention have to be drawn by someone having poisoned himself on the basis of an unrefereed remedy first? http://www.nih.gov/about/director/ebiomed/com0509.htm#harn45 On the other hand, to repeat, open feedback on an open access classically refereed literature, pre-, during, and post- refereeing, would certainly be a welcome second line of defence, and would no doubt help improve the quality of the research literature. >ao> There are lots more examples.) The point is that classical peer >ao> review does not provide much of a signal, especially for journals >ao> in the lower quality tiers. And you think self-selected feedback would provide at least as much of a "signal," especially for journals in the lower quality tiers? (Sometimes I think what you are saying is that the elite work does not really need peer review and for the rest it doesn't really matter...!) >ao> So how does science progress? Well, there are all sorts of checks >ao> that are applied post-publication. (And none of them are infallible. >ao> Even a few Nobel prizes are now regarded as having been given in error.) >ao> Basically classical peer review is just one noisy and uncertain signal >ao> that the scholarly community relies on. Classical peer review is not a signal; it is a dynamic, interactive quality-control and tagging system -- and the only one that is systematic and answerable. I cannot see any way that anarchic self-selected feedback can replace this (other than by re-inventing classical peer review under another name) -- though I can see how it can (and does) complement it. >sh> If self-archiving had (mirabile dictu) begun instead with refereed >sh> postprints [rather than unrefereed preprints], we might have >sh> spared ourselves these misconstruals [about peer review] and >sh> we might have been further along the road to open access by now... > >ao> The incentives were not there to do this. The authors, who after all >ao> control the information flow, could see the benefits to themselves >ao> of quick circulation of preprints. Open access to published journal >ao> articles was of much less value to them, since they typically had access >ao> to those journals in their libraries. I agree that those were the initial conditions that turned it into a historical fact that self-archiving en masse began with physicists, self-archiving their pre-refereeing preprints first (though soon also their published postprints too). They were already a (paper) "preprint culture" and (the elite among them) did not lack access to the toll-based journal literature. That's why I said it would have been "mirabile dictu" if it had begun instead with refereed postprints. But that is all history now, and our eyes are opened, and the data are in, and it should now be clear that open access to the entire research literature, before and after refereeing, is what will be optimal for all. And that what peer-reviewed research needs now is to be freed from access-blocking tolls, not from quality-controlling peer review. Stevan Harnad
