What you are leaving out is the quality of the reviews. Just as with papers, some reviews are of much higher quality than others. Perhaps there should also be included in the index how many times the editors had to send reminders to the reviewer - before all the on-line journal review systems were up, this was a major pain in the neck.
> I haven't the time to develop this, so I'll throw it out there in hopes > someone will run with it. I believe being asked to referee indicates one's > standing in a field. Journals will always try to get the best referees > possible. We simply don't have a way to measure or reward reviewing. > > > For authors we have a measure of impact (actually several, > see http://en.wikipedia.org/wiki/H-index for a quick start). I would > suggest something similar for referees. Journals would produce an annual > list of reviewers and the number of time each reviewed. The sum of the > number of reviews by a referee times the impact factor of the journals > they review in should give a pretty good index of their standing in their > field. Reviewing in Science would be rare but earn a high score but more > frequent reviewing in high ranked but more focused journals would really > drive scores. Reviewing in low ranked journals would not help one's score > much but as at present would be done more as moral obligation than for > one's career. > > > Further indices could correct for time and frequency of reviews, or look > at mean rank, much as the H-index spawned a wave of refinements. > > > Once each of us has a number (or various), there will be a natural > inclination to want to improve one's standing (which can be done by more > reviewing or by being asked to review by higher impact journals). > Administrators, obsessed with the quantitative will latch onto this like > flies onto roadkill for evaluating. The bottom line would be a competition > for opportunities to review rather than a competition among editors for a > limited number of reviewers. We would measure those who give back, not > just those who publish. > > > Of course this could be gamed, but the best defense would be editors who > don't count reviews unless they reach a certain standard of excellence. Of > course if editors were too picky, we wouldn't bother to review for that > particular journal. > > > We can continue to bemoan the state of reviewing, and dream up sticks with > which to beat reviewers into helping, or we can come up with carrots. This > carrot is cheap and appeals to both our better and worse angels. > > > Anyway, I'd appreciate thoughts on it. If it goes anywhere, I hope someone > will call it the D-Index. > > > Cheers, > > > David Duffy > > > > > > > Professor/PCSU Unit Leader/CESU Director > PCSU/CESU/Department of Botany > University of Hawaii Manoa > 3190 Maile Way, St John 410 > Honolulu, HI 96822 USA > Tel 808-956-8218, FAX 808-956-4710 > http://www.botany.hawaii.edu/faculty/duffy/ > > > > > ----- Original Message ----- > From: malcolm McCallum <malcolm.mccal...@herpconbio.org> > Date: Saturday, January 7, 2012 4:49 am > Subject: Re: [ECOLOG-L] Is there a referee crisis in ecology? > To: ECOLOG-L@LISTSERV.UMD.EDU > >> Recent joint editorial from all herp societies published in >> Herpetological Conservation and Biology. >> >> The "peer" in Peer Review. >> http://www.herpconbio.org/Volume_6/Issue_3/Joint_editorial_2011.pdf >> >> On Thu, Jan 5, 2012 at 4:02 PM, Chris Lortie >> <lor...@yorku.ca> wrote: >> > Dear Ecologgers, >> > >> > Thank you so much for your feedback on the editorial 'Money >> for nothing and referees for free' >> > published in Ideas in Ecology and Evolution in December >> > (http://library.queensu.ca/ojs/index.php/IEE/index). The most >> compelling and common question >> > I was asked was is there a referee crisis in ecology (or >> tragedy of the 'reviewers common' as >> > Hochberg et al. proposed). This is an excellent question. I >> propose that whilst there are more >> > perfect ways to test this (total up number of submissions and >> then estimate total pool of referees, >> > tricky), an interesting indicator would instead to be >> calculate the decline to review rate (d2rr) in >> > ecology. I envision the following two primary data streams >> to calculate this rate: a per capita >> > estimate derived from each of us personally and a mean >> estimate of rate from the publishing >> > portals (journals). Hence, let's do it. Only you know your >> decline to (accept doing a) review rate >> > across all requests whilst journals track their own net rates >> and your specific rate with them too. >> > >> > So, please take 30 seconds and fill in this short survey, and >> we can then assess, to an extent, >> > whether there is a referee crisis in ecology. >> > >> > https://www.surveymonkey.com/s/VD3K36W >> > >> > I have also compiled a long list of emails for every editor I >> could find for all ecology journals and >> > have contacted them to see if they would share the rate at >> which individuals decline for each of >> > them, i.e. do they have to ask 5 or 6 people to even secure >> two reviews? I will not share the journal >> > names etc. and protect their rates as I recognize the >> implications. I would just like to know what >> > our overall mean is from a journal perspective too. >> > >> > Thanks so much for your time and help with these discussions. >> I hope you think they are >> > important too, but I also want to assure you that this is my >> penultimate post on the subject. >> > Warm regards, >> > Christopher Lortie. >> > lor...@yorku.ca >> > www.onepoint.ca >> >> >> >> -- >> Malcolm L. McCallum >> Department of Molecular Biology and Biochemistry >> School of Biological Sciences >> University of Missouri at Kansas City >> >> Managing Editor, >> Herpetological Conservation and Biology >> >> "Peer pressure is designed to contain anyone with a sense of >> drive" - >> Allan Nation >> >> 1880's: "There's lots of good fish in the sea" W.S. Gilbert >> 1990's: Many fish stocks depleted due to overfishing, habitat loss, >> and pollution. >> 2000: Marine reserves, ecosystem restoration, and pollution reduction >> MAY help restore populations. >> 2022: Soylent Green is People! >> >> The Seven Blunders of the World (Mohandas Gandhi) >> Wealth w/o work >> Pleasure w/o conscience >> Knowledge w/o character >> Commerce w/o morality >> Science w/o humanity >> Worship w/o sacrifice >> Politics w/o principle >> >> Confidentiality Notice: This e-mail message, including any >> attachments, is for the sole use of the intended recipient(s) >> and may >> contain confidential and privileged information. Any unauthorized >> review, use, disclosure or distribution is prohibited. If you >> are not >> the intended recipient, please contact the sender by reply e- >> mail and >> destroy all copies of the original message. >
