On Sun, 12 Mar 2000, Andrew Kenneth Fletcher wrote: > I have a real problem with the current Peer Review System. It is biased > towards in-house publications and outsiders are ignored. > > I had an idea to set up a new newsgroup titled "Peer Review Sci" I am > certain that it would attract many professional contributors, who would > normally have been ignored by publishers and therefore provide independent > researchers with an unbiased review of their work. > It would also be a far better way to make sure that nothing false arrives in > print, because it would be an open peer review system and anyone > contributing either a paper or a review of a paper would be open to comment > from other reviewers. This would generate a tremendous amount of new science > and encourage the people with the ideas to come forward. > > What say ye to this?
The notion of replacing peer review by some form of open commentary has been proposed many times, in this Forum and elsewhere (and it is being experimented with by several sites on the Web). See the other threads on this in this Forum (1999, 1999, & 2000) and: Harnad, S. (1998) The invisible hand of peer review. Nature [online] (c. 5 Nov. 1998) http://helix.nature.com/webmatters/invisible/invisible.html Longer version below to appear in Exploit Interactive <http://www.exploit-lib.org/>: http://www.ecs.soton.ac.uk/~harnad/nature2.html Here are some relevant excerpts from the above: Self Policing? Alternatives have of course been proposed, but to propose is not to demonstrate viability. Most proposals have envisioned weakening the constraints of classical peer review in some way or other. the most radical way being to do away with it altogether: Let authors police themselves; let every submission be published, and let the reader decide what is to be taken seriously. This would amount to discarding the current hierarchical filter -- both its active influence, in directing revision, and its ranking of quality and reliability to guide the reader trying to navigate the ever-swelling literature (Hitchcock et al. 2000). There is a way to test our intuitions about the merits of this sort of proposal a priori, using a specialist domain that is somewhat more urgent and immediate than abstract "learned inquiry"; then if we are not prepared to generalise this intuitive test's verdict to scholarly/scientific research in general, we really need to ask ourselves how seriously we take the acquisition of knowledge: If someone near and dear to you were ill with a serious but potentially treatable disease, would you prefer to have them treated on the basis of the refereed medical literature or on the basis of an unfiltered free-for-all where the distinction between reliable expertise and ignorance, incompetence or charlatanism is left entirely to the reader, on a paper by paper basis? A variant on this scenario is currently being tested by the British Medical Journal <http://www.bmj.com/cgi/shtml/misc/peer/index.shtml>, but instead of entrusting entirely to the reader the quality control function performed by the referee in classical peer review, this variant, taking a cue from some of the developments and goings-on on both the Internet and Network TV chat-shows, plans to publicly post submitted papers unrefereed on the Web and to invite any reader to submit a commentary; these commentaries will then be used in lieu of referee reports as a basis for deciding on formal publication. Expert Opinion or Opinion Poll? Is this peer review? Well, it is not clear whether the self-appointed commentators will be qualified specialists (or how that is to be ascertained). The expert population in any given speciality is a scarce resource, already overharvested by classical peer review, so one wonders who would have the time or inclination to add journeyman commentary services to this load on their own initiative, particularly once it is no longer a rare novelty, and the entire raw, unpoliced literature is routinely appearing in this form first. Are those who have nothing more pressing to do with their time than this really the ones we want to trust to perform such a critical QC/C function for us all? And is the remedy for the possibility of bias or incompetence in referee-selection on the part of editors really to throw selectivity to the winds, and let referees pick themselves? Considering all that hangs on being published in refereed journals, it does not take much imagination to think of ways authors could manipulate such a public-polling system to their own advantage, human nature being what it is. Peer Commentary vs. Peer Review And is peer commentary (even if we can settle the vexed "peer" question) really peer review? Will I say publicly about someone who might be refereeing my next grant application or tenure review what I really think are the flaws of his latest raw manuscript? (Should we then be publishing our names alongside our votes in civic elections too, without fear or favour?) Will I put into a public commentary -- alongside who knows how many other such commentaries, to be put to who knows what use by who knows whom -- the time and effort that I would put into a referee report for an editor I know to be turning specifically to me and a few other specialists for our expertise on a specific paper? If there is anyone on this planet who is in a position to attest to the functional difference between peer review and peer commentary (Harnad 1982, 1984), it is surely the author of the present article, who has been umpiring a peer-reviewed paper journal of Open Peer Commentary (Behavioral and Brain Sciences [BBS] <http://www.princeton.edu/~harnad/bbs.html>, published by Cambridge University Press) for over 2 decades (Harnad 1979), as well as a brave new online-only journal of Open Peer Commentary, likewise peer-reviewed (Psycoloquy, sponsored by the American Psychological Association, <http://www.princeton.edu/~harnad/psyc.html>), which entered its second decade with the millennium. Both journals are rigorously refereed; only those papers that have successfully passed through the peer review filter go on to run the gauntlet of open peer commentary, an extremely powerful and important SUPPLEMENT to peer review, but certainly no SUBSTITUTE for it. Indeed, no one but the editor sees [or should have to see] the population of raw, unrefereed submissions, consisting of some manuscripts that are eventually destined to be revised and accepted after peer review, but also (with a journal like BBS, having a 75% rejection rate) many manuscripts not destined to appear in that particular journal at all. Referee reports, some written for my eyes only, all written for at most the author and fellow referees, are nothing like public commentaries for the eyes of the entire learned community, and vice versa. Nor do 75% of the submissions justify soliciting public commentary, or at least not commentary at the BBS level of the hierarchy. It has been suggested that in fields such as Physics, where the rejection rate is lower (perhaps in part because the authors are more disciplined and realistic in their initial choice of target journal, rather than trying their luck from the top down), the difference between the unrefereed preprint literature and the refereed reprint literature may not be that great; hence one is fairly safe using the unrefereed drafts, and perhaps the refereeing could be jettisoned altogether. -------------------------------------------------------------------- Stevan Harnad har...@cogsci.soton.ac.uk Professor of Cognitive Science har...@princeton.edu Department of Electronics and phone: +44 23-80 592-582 Computer Science fax: +44 23-80 592-865 University of Southampton http://www.ecs.soton.ac.uk/~harnad/ Highfield, Southampton http://www.princeton.edu/~harnad/ SO17 1BJ UNITED KINGDOM NOTE: A complete archive of this ongoing discussion of providing free access to the refereed journal literature is available at the American Scientist September Forum (98 & 99 & 00): http://amsci-forum.amsci.org/archives/American-Scientist-Open-Access-Forum.html You may join the list at the site above. Discussion can be posted to: american-scientist-open-access-fo...@amsci.org