David has brought up some good points and questions about Peerage of Science, and the attempt to fix some of the obvious problems with the current peer-review system. Below are some responses to paraphrased versions of David's questions [in brackets] and further thoughts, jointly written by Janne-Tuomas Seppänen (PoS founder) and me. "[Wouldn't editors still send a submitted ms out for review anyway]?" -- The idea of Peerage of Science is that authors eventually would no longer need to "submit" anywhere. Instead they *send* (emphasizing this word to distinguish from "submit") manuscripts to be peer reviewed at PoS, and journals with access to the system can proactively send publishing offers, if and when a peer review process indicates a manuscript meets their standards. In their offer to publish, editors can specify that the offer is conditional on another "in-house" peer review, if they wish. But that is extra work, and a journal that trusts Peerage of Science enough to participate probably would not see a need to do that.
An editor of a non-participating journal receiving a manuscript with "exported" peer review reports is another situation. Since PoS is still such a new endeavour, many editors probably will opt to organize an additional "in-house" peer review. We trust that they will soon realize that the reliability and quality of reviews from Peerage of Science is at least as good as, and probably better than, the reviews they get after significant delay and toil through the traditional system, and start to accept the Peerage of Science reviews due to the improved efficiency of decision-making. "[Wouldn't this attract biased reviewers who see themselves as gatekeepers]?" -- Reviewers with strong unjustified point of view, unqualified comments, and nasty wording are common enough in the traditional system. The peer-review-of-peer-review keeps these in check in Peerage of Science, because unjustified comments lead to lower scores. The system seeks to have the best of both anonymity and openness: you have the security of anonymity, but suffer from doing malicious things personally (your performance score takes a hit), and you have incentive to openly publish your peer-reviewed review report once the process is complete. In the traditional system, the worst consequence you can suffer from abusing anonymity is that the journal never requests your reviews again. "[What prevents the system from being gamed]?" -- Gaming is not impossible, but takes some effort and is very risky relative to payoff. You are not allowed to access manuscripts authored by affiliated scientists (from someone in your institution, or your recent co-authors). A pattern of reciprocal altruism with a co-conspirator would be evident in the system logs, and scientific misconduct leads to life-long ban from the service. If necessary, the system will feature a "report abuse" button for reviewers spotting a purposefully biased (be it positive or negative) review, launching an investigation. You can invite imaginary people, but they will not get peer status (i.e. a right to access manuscripts) because this is granted only after service administration has verified identity and qualifications. Naturally, such a "prank" would be clearly identified in the system logs. "[What prevents ideas from getting stolen and scooped]?" -- Peerage of Science actually mitigates the possibility of ideas or papers getting stolen, this is discussed in the service FAQ. We hope this helps clarify a few of the excellent points David raised. Answering these questions is very important - as it illustrates some of the advantages of PoS over the current (and more flawed) traditional peer-review system, which is more subject to many of the maladies mentioned above. Sincerely, Joe Nocera and Janne-Tuomas Seppänen (Member of Board of Governors and Founder of Peerage of Science, respectively) Peerage of Science http://www.peerageofscience.org ----- Original Message ----- From: David C Duffy <ddu...@hawaii.edu> Date: Monday, January 9, 2012 9:22 am Subject: Re: [ECOLOG-L] Is there a referee crisis in ecology? To: ECOLOG-L@LISTSERV.UMD.EDU > The concept of the Peerage of Science is an interesting one and > potentially quite useful but I have a few concerns, perhaps > because I don't understand the system. > > Wouldn't editors still send a submitted ms out for review, to > reviewers they personally know are competent, know the subject, > and are unbiased (although editors have also been known to send > mss to a biased or 'pitbull' reviewer to see what their best > shot at the ms is). Wouldn't this actually add a layer to the > publication process and stretch good reviewers even thinner? > > Wouldn't such a system attract reviewers who have their own > strong point of view and see themselves as gatekeepers for > particular subjects? Climate change deniers could infiltrate > and wreck havoc with reviews of climate change mss. Also would > it constrain those who know little or nothing about a subject > from reviewing? As we know anonymity brings out the worst of > people on the internet; there would be little to constrain them. > Get reviewed by a few nasty people and authors will simply stop > sending mss even if those reviewers got low scores. > > If the system is fully automatic, what keeps it from being > gamed? I'll do a nice review for you if you do one for me-- > remember it's anonymous so we can't detect conflicts of > interest. Similarly what is to keep a prankster from signing up > a dozen or so of his imaginary colleagues to help review his > papers? Or, on the other hand, what is to keep ideas or whole > papers from going walkabout, from an anonymous author to one who > publishes first? Reviewers could self police but Quis custodiet > upsos custodes? > > With my present state of knowledge, the Peerage appears as if > would serve most effectively as a discussion group, an extended > network of colleagues, an extension of the many smaller informal > networks that do the same thing at present. We would submit a > paper to get feedback to improve it, but as you mention, editors > could solicit the mss they see as interesting for consideration > in their journals. This would essentially be a continuous > scientific meeting, held on the Internet. > > > Of course it is easy to see problems, some things I have brought > up may be minor, so it's worth a try. > > > Cheers, > > > David > > David Cameron Duffy Ph.D. > Professor/PCSU Unit Leader/CESU Director > PCSU/CESU/Department of Botany > University of Hawaii Manoa > 3190 Maile Way, St John 410 > Honolulu, HI 96822 USA > Tel 808-956-8218, FAX 808-956-4710 > http://www.botany.hawaii.edu/faculty/duffy/ > > > > > ----- Original Message ----- > From: Joe Nocera <nocer...@queensu.ca> > Date: Saturday, January 7, 2012 12:03 pm > Subject: Re: [ECOLOG-L] Is there a referee crisis in ecology? > To: ECOLOG-L@LISTSERV.UMD.EDU > > > David's message rings clear, but I am happy to report that he > is > > incorrect on one matter. We DO have a system that measures > > and rewards reviewing. A recent initiative, called Peerage > > of Science, has instituted a system that (among other things) > > provides quantitative ratings of review quality. > > > > I encourage you to read the details about this bold new > > endeavour at: http://www.peerageofscience.org/ > > I hope the information there can convince many of you to join, > > or at least breathe easier that attempts are being made to > > divert an (aptly described) referee crisis. > > > > In reference to the issue at hand, that of quantifying referee > > effort, the PoS system works along the following lines: > > 1. A manuscript is submitted to PoS for review > > 2. Members are alerted to the ms, and can sign up to review it > > 3. After the manuscript's first submission is reviewed, the > > reviewers are then allowed to see each other's reviews (all > anonymous)> 4. The reviews are then scored by the other reviewers > > 5. The manuscript continues on in the process... > > > > Each reviewer then accumulates an average "review quality" > score > > over time. Poor reviews are justifiably penalized with low > > scores. Excellent reviews accrue good scores. > > > > I am sure that the benefits here are obvious, and perhaps so > are > > a few drawbacks. But, it is the first attempt of which I > > am aware that is trying to create a currency amongst reviewers > > that is not just an extra bullet on a performance review or CV. > > > > Check it out. Chris, as originator of this thread, I > > especially think you would be interested in this. > > > > Sincerely, > > Joe Nocera > > > > (Member of the Board of Governers for Peerage of Science) > > > > > > > > ----- Original Message ----- > > From: David C Duffy <ddu...@hawaii.edu> > > Date: Saturday, January 7, 2012 2:56 pm > > Subject: Re: [ECOLOG-L] Is there a referee crisis in ecology? > > To: ECOLOG-L@LISTSERV.UMD.EDU > > > > > I haven't the time to develop this, so I'll throw it out > there > > > in hopes someone will run with it. I believe being asked to > > > referee indicates one's standing in a field. Journals will > > > always try to get the best referees possible. We simply > don't > > > have a way to measure or reward reviewing. > > > > > > > > > For authors we have a measure of impact (actually several, > > > see http://en.wikipedia.org/wiki/H-index for a quick start). > I > > > would suggest something similar for referees. Journals would > > > produce an annual list of reviewers and the number of time > > each > > > reviewed. The sum of the number of reviews by a referee > times > > > the impact factor of the journals they review in should > > give a > > > pretty good index of their standing in their field. > Reviewing > > in > > > Science would be rare but earn a high score but more > frequent > > > reviewing in high ranked but more focused journals would > > really > > > drive scores. Reviewing in low ranked journals would not > help > > > one's score much but as at present would be done more as > moral > > > obligation than for one's career. > > > > > > > > > Further indices could correct for time and frequency of > > reviews, > > > or look at mean rank, much as the H-index spawned a wave of > > > refinements. > > > > > > Once each of us has a number (or various), there will be a > > > natural inclination to want to improve one's standing (which > > can > > > be done by more reviewing or by being asked to review by > > higher > > > impact journals). Administrators, obsessed with the > > quantitative > > > will latch onto this like flies onto roadkill for > evaluating. > > > The bottom line would be a competition for opportunities to > > > review rather than a competition among editors for a limited > > > number of reviewers. We would measure those who give back, > not > > > just those who publish. > > > > > > > > > Of course this could be gamed, but the best defense would be > > > editors who don't count reviews unless they reach a certain > > > standard of excellence. Of course if editors were too picky, > > we > > > wouldn't bother to review for that particular journal. > > > > > > > > > We can continue to bemoan the state of reviewing, and dream > up > > > sticks with which to beat reviewers into helping, or we can > > come > > > up with carrots. This carrot is cheap and appeals to > > both our > > > better and worse angels. > > > > > > > > > Anyway, I'd appreciate thoughts on it. If it goes anywhere, > I > > > hope someone will call it the D-Index. > > > > > > > > > Cheers, > > > > > > > > > David Duffy > > > > > > > > > > > > > > > > > > > > > Professor/PCSU Unit Leader/CESU Director > > > PCSU/CESU/Department of Botany > > > University of Hawaii Manoa > > > 3190 Maile Way, St John 410 > > > Honolulu, HI 96822 USA > > > Tel 808-956-8218, FAX 808-956-4710 > > > http://www.botany.hawaii.edu/faculty/duffy/ > > > > > > > > > > > > > > > ----- Original Message ----- > > > From: malcolm McCallum <malcolm.mccal...@herpconbio.org> > > > Date: Saturday, January 7, 2012 4:49 am > > > Subject: Re: [ECOLOG-L] Is there a referee crisis in ecology? > > > To: ECOLOG-L@LISTSERV.UMD.EDU > > > > > > > Recent joint editorial from all herp societies published in > > > > Herpetological Conservation and Biology. > > > > > > > > The "peer" in Peer Review. > > > > > > > http://www.herpconbio.org/Volume_6/Issue_3/Joint_editorial_2011.pdf> > > > > > On Thu, Jan 5, 2012 at 4:02 PM, Chris Lortie > > > > <lor...@yorku.ca> wrote: > > > > > Dear Ecologgers, > > > > > > > > > > Thank you so much for your feedback on the editorial > > 'Money > > > > for nothing and referees for free' > > > > > published in Ideas in Ecology and Evolution in December > > > > > (http://library.queensu.ca/ojs/index.php/IEE/index). > > The > > > most > > > > compelling and common question > > > > > I was asked was is there a referee crisis in ecology (or > > > > tragedy of the 'reviewers common' as > > > > > Hochberg et al. proposed). This is an excellent > > question. > > > I > > > > propose that whilst there are more > > > > > perfect ways to test this (total up number of > submissions > > > and > > > > then estimate total pool of referees, > > > > > tricky), an interesting indicator would instead to be > > > > calculate the decline to review rate (d2rr) in > > > > > ecology. I envision the following two primary > > data streams > > > > to calculate this rate: a per capita > > > > > estimate derived from each of us personally and a mean > > > > estimate of rate from the publishing > > > > > portals (journals). Hence, let's do it. Only > > you know your > > > > decline to (accept doing a) review rate > > > > > across all requests whilst journals track their own net > > > rates > > > > and your specific rate with them too. > > > > > > > > > > So, please take 30 seconds and fill in this short > survey, > > > and > > > > we can then assess, to an extent, > > > > > whether there is a referee crisis in ecology. > > > > > > > > > > https://www.surveymonkey.com/s/VD3K36W > > > > > > > > > > I have also compiled a long list of emails for every > > editor > > > I > > > > could find for all ecology journals and > > > > > have contacted them to see if they would share the rate > at > > > > which individuals decline for each of > > > > > them, i.e. do they have to ask 5 or 6 people to even > > secure > > > > two reviews? I will not share the journal > > > > > names etc. and protect their rates as I recognize the > > > > implications. I would just like to know what > > > > > our overall mean is from a journal perspective too. > > > > > > > > > > Thanks so much for your time and help with these > > > discussions. > > > > I hope you think they are > > > > > important too, but I also want to assure you that this > is > > my > > > > penultimate post on the subject. > > > > > Warm regards, > > > > > Christopher Lortie. > > > > > lor...@yorku.ca > > > > > www.onepoint.ca > > > > > > > > > > > > > > > > -- > > > > Malcolm L. McCallum > > > > Department of Molecular Biology and Biochemistry > > > > School of Biological Sciences > > > > University of Missouri at Kansas City > > > > > > > > Managing Editor, > > > > Herpetological Conservation and Biology > > > > > > > > "Peer pressure is designed to contain anyone with a sense > of > > > > drive" - > > > > Allan Nation > > > > > > > > 1880's: "There's lots of good fish in the sea" W.S. Gilbert > > > > 1990's: Many fish stocks depleted due to overfishing, > > habitat loss, > > > > > > and pollution. > > > > 2000: Marine reserves, ecosystem restoration, and > > pollution > > > > > reduction> MAY help restore populations. > > > > 2022: Soylent Green is People! > > > > > > > > The Seven Blunders of the World (Mohandas Gandhi) > > > > Wealth w/o work > > > > Pleasure w/o conscience > > > > Knowledge w/o character > > > > Commerce w/o morality > > > > Science w/o humanity > > > > Worship w/o sacrifice > > > > Politics w/o principle > > > > > > > > Confidentiality Notice: This e-mail message, including any > > > > attachments, is for the sole use of the intended > > recipient(s) > > > > and may > > > > contain confidential and privileged information. Any > > unauthorized> > review, use, disclosure or distribution is > > prohibited. If you > > > > are not > > > > the intended recipient, please contact the sender by reply > e- > > > > mail and > > > > destroy all copies of the original message. > > >
