On Sat, 23 Feb 2002, Sergio Della Sala and Jordan Grafman wrote:
> We are convinced that peer-review is central to scientific credibility.
> However, as it stands the process is far from watertight. Is there any
> way we can improve it by suggesting any modification, either radical or
> minimal? Time is ripe for such a discussion to be launched (see the
> JAMA and BMJ four congresses on peer review in biomedical publication:
> www.jama-peer.org).
Many papers presented at the JAMA/BMJ congresses and
other sources point to a problem that I call
"insularity." That is ignorance of, ignoring or
avoiding inconvenient information. This includes
national and language biases as well as the sort of
short-sightedness that led to the death of a subject
at Johns Hopkins last year and commercial biases that
typically omit studies that contradict the desired
conclusion.
To combat insularity, several medical journals adopted
the Consolidated Standards of Reporting Trials [CONSORT].
One of the recommendations was that authors "state
general interpretation of the data in light of the
totality of the available evidence." A study at the
Prague conference showed little evidence that authors
complied or that authors were able to compel it. The
little time reportedly spent by referees, according
to other studies, suggests they would not catch
many omissions and blind spots. The Achilles' Heel
of peer review is that referees are no better informed
than authors.
Of course, it is the sponsors of research who call
the tune. The totality of the literature is
overwhelming. That includes not only primary reports
but review articles. The sponsors appear to tolerate
a shallow review in proposals and preparation, and
little more in conclusions.
More intensive screening, evaluating, digesting, and
review of all lines of research is essential. Many
reviews reflect an erroneous consensus, such as the
notion in the 1940s that research on steriods was at a
dead end.
I have written more on this in SOCIETY 38,2 47-54
(J/F 2001), if anyone is interested. I would also
be happy to provide references to studies of peer
review that actually shed light on the problem and
its solution.
Best wishes,
Albert Henderson
Former Editor, PUBLISHING RESEARCH QUARTERLY 1994-2000
<70244.1...@compuserve.com>
