Re: The Economist: Publish and perish
on Wed, 27 Nov 2002 Arkadiusz Jadczyk a...@cassiopaea.org wrote But the main problem in this thread is the proceder of peer reviewing and what to do about it. For me the action of the editor of Classical and Quantum and Gravity is just funny. If they are really serious, they should re-review all the papers published in the journal, because there will be more equally or even more controversial. Of course they will not do it. Who would do? Such systematic re-reviewing has been done, in physics by Conyers Herring, and in clinical medicine by a Canadian team led by Walter O Spitzer. Both found considerable amounts of bad research. The fact that Spitzer required the assistance of a task force to screen and evaluate all the citations on a common diagnosis is ample evidence in itself that the volume and specialization of research published can be beyond the reach of any individual reader. This, in fact, has been called the major challenge of any research project. Spitzer also aimed his comprehensive review to provide all interested researchers with an authentic fresh starting point. One of the observations made, in connection with the re-review, was that peer evaluations will change over time, as new information is disseminated. The other observation was that lots of poorly designed, poorly executed, and duplicative research is not only done but then published. (Some solace may come from the fact that most authors publish no more than one or two papers, never to be heard from again.) It is far more expensive to do than to publish. Why are so many resources wasted on useless and misleading research? Why waste our attention on nickel and dime issues like publication peer review when grant review panels permit hundred-dollar bills to be blown away by researcher sponsors? The U.S. General Accounting Office studied peer review at the stage where much waste could be avoided. In 1994 it reported: Although most reviewers reported expertise in the general areas of the proposals they reviewed, many were not expert on closely related questions and could cite only a few, if any, references. This lack of proximate expertise was most pronounced at NIH. However, although this raises questions about the relative adequacy of NIH reviews and ratings, the greater proximity of NSF reviewers makes them potentially more vulnerable to apparent or actual self-interest in their reviews. Moreover, the report noted that considerable research is financed with no review, thanks to Congressional earmarks and agency policy. A low point in peer review was probably breached when a research subject at Johns Hopkins died as a result of researchers and referees failing to adequately study the scientific record at the proposal stage. As it is, I wonder if the project had any merit at all. If Hopkins had done a Spitzer review, a life would have been spared and the research would have had a better chance of coming to reliable conclusions. Critics of peer review might well concentrate on the institutional conflict of interest, the motive that makes grant income as more important than productivity. The universities that do the research are also responsible for most reviews. Wouldn't a low tolerance for poor preparation hurt their pockets? My impression is that publishers' peer review is generally no better than the review that supports the research. The scientific record is not perfect. At least it demonstrates an effort to filter out amateurs, quacks, and poorly prepared contributions. If there is a weakness, editors point out, it is their bias against publishing negative results, reports that might save other researchers from going down blind alleys. The open archive movement, on the other hand, welcomes unreviewed contributions, mixing them with the scientific record. While informal exchanges of information -- conference papers, letters, preprints, face-to-face conversations -- are essential, the admission of such material to archives has created some confusion. When they are cited (as if they were published in the scholarly sense) we see the authors, in vain hopes of seeing further, climbing on the backs of little people sinking in the mud. I know physicists who say that 90% of papers published in Phys Rev A is junk. My estimate is 40%. It is
Re: The Economist: Publish and perish
Are referees for second rate journals less likely to steal your article? On Fri, 29 Nov 2002, Arkadiusz Jadczyk wrote: I can't refrain from quoting the pertinent piece from Chance and Chaos by David Ruelle. (Notes 5 and 7 to Chapter 11, p. 179-180) A few words about rejected papers may be appropriate here. A prerequisite for a successful profesional carreer, for many people, is to have published scientific papers in refereed journals. In other words, appointments and promotions are decided on the basis of number of published papers. This situation forces many individuals who have neither interest in nor ability for scientific research, to write papers and submit them to journals. The referees, who are themselves research scientists, are thus flooded with mediocre papers, about which they are required to produce reports. Since they have more interesting work to do, the reports are often hasty and superficial. Reasonable-looking papers are accepted, obviously bad papers are rejected, and good papers that a bit original and out of the norm tend to be rejected too. This is a well known problem, and nobody really knows what to do about it. Fortunately, there are many scientific journals, and a really good paper will eventually get published somewhere. [...] If you are a conscientious scientist, you will acknowledge the sources the sources of all the ideas that you use (supposing you remember). If you are unscrupulous, you will try to present as your own some results obtained by others. For example, if you find a good idea in a paper that you referee, you will try to stop the paper, and rush to publish the idea under your own name (or have one of your students publish it). [...] I have myself worked in some areas in which I could freely discuss ideas with collegues, and other areas in which it was unwise, because of the risk that the idea would be stolen. Ark again: I think Ruelle lists explicitly certain very important issues here. He gives possible reasons why, in some cases, publishing in second rank journals, or just only on arXiv and similar (as advocated by Andrew Odlyzko), is indeed a wise choice. ark http://www.cassiopaea.org/quantum_future/ Dr. David Goodman Biological Sciences Bibliographer Princeton University Library dgood...@princeton.edu
The Economist: Publish and perish
There is an interesting article in The Economist of this week (November 16th-22nd) which raises questions about the scientific peer-review system: http://www.economist.com/science/displayStory.cfm?story_id=1441745 I am afraid that this article is not Open Access but the story and the facts can be found on this (much more specific) page: The Bogdanov Affair, by John Baez http://math.ucr.edu/home/baez/bogdanov.html I wonder: is this affair related to the discussion between Andrew and Stevan on Peer Review and Self-Selected Vetting: Supplement or Substitute? Cheers, Imre Simon
Re: The Economist: Publish and perish
At 10:37 PM 11/18/2002 -0200, you wrote: There is an interesting article in The Economist of this week (November 16th-22nd) which raises questions about the scientific peer-review system: http://www.economist.com/science/displayStory.cfm?story_id=1441745 I am afraid that this article is not Open Access but the story and the facts can be found on this (much more specific) page: The Bogdanov Affair, by John Baez http://math.ucr.edu/home/baez/bogdanov.html I wonder: is this affair related to the discussion between Andrew and Stevan on Peer Review and Self-Selected Vetting: Supplement or Substitute? Cheers, Imre Simon Here's a free story on the Bogdanovs in the New York Times http://www.nytimes.com/2002/11/17/weekinreview/17JOHN.html And here's one in the Chronicle of Higher Education, accessible only to subscribers http://chronicle.com/weekly/v49/i12/12a01601.htm The Bogdanov story is not related to the dialogue between Andrew and Stevan except in the broadest sense. Because it challenges how well peer review is performed today in cutting edge physics, it invites the question how to reform peer review in order to preserve its traditional value and prevent this sort of problem. I find the Bogdanov case fascinating, but I haven't yet seen any direct FOS or open-access implications. Peer review is essential to open-access science just as it is to closed- or toll-access science. Open access doesn't depend on peer-review reform any more (or any less) than toll-access science depends on peer-review reform. Peter -- Peter Suber, Professor of Philosophy Earlham College, Richmond, Indiana, 47374 Email pet...@earlham.edu Web http://www.earlham.edu/~peters Editor, Free Online Scholarship Newsletter http://www.earlham.edu/~peters/fos/ Editor, FOS News blog http://www.earlham.edu/~peters/fos/fosblog.html
