Sunday, November 27, 2005

Peer Review and Free Socks

Larry Wasserman at The Academic Curmudgeon has an interesting entry on the peer review process upon which Cosma Shalizi at Three-Toed Sloth has an interesting follow-up. So I'll join the crowd and offer up a few additional thoughts about peer review.

What I find myself wondering about the peer review process is: why does anyone agree to serve as a referee? Referees for journal articles almost always are unpaid. Doing a good job as a referee is rarely rewarded, often goes entirely unnoticed, and sometimes elicits unpleasant responses from authors. Given this, I've got a couple of theories as to why people agree to serve as referees.

  1. It's an honor. Initially, I suspect that for many it's an honor to be asked to review an article -- it seems like an indication that one has "arrived" as a researcher. Being invited to serve as a referee is a sure sign that you're a member of the brother- and sister-hood of physicists (or whatever one's field is.) However, this sense of honor and pride wears off pretty quickly. Refereeing gets old in a hurry, and emails from editors soon elicit mild dread instead of pride.

  2. It's fun. Well, sometimes it's fun. I enjoy refereeing a well written, interesting paper about which I feel I can make some constructive suggestions. Unfortunately, this doesn't happen that frequently.

  3. Duty and obligation. I think many scientists continue to referee out of a sense of professional responsibility: it's just part of the job. We all benefit, the thinking goes, from peer review, since we get the prestige (and jobs and degrees and promotions) that come with having our work published in peer-reviewed journals. So, in a sense, by agreeing to referee we're returning the favor done for us by those who have refereed our papers.

  4. It gives one power. Unarguably, referees wield some power over what gets published and what doesn't get published. These decisions have consequences, both for the overall direction of the field and also for the author(s).

Reason #4 is somewhat problematic for a number of reasons. Cosma and Larry discuss these at some length. I want to focus some on reason #3.

If the peer-review system is to work, we need people to do serve as referees. (We might not want the peer review system to work. Arguably it's inefficient and prone to abuse. But that's another story for another time.) However, right now there's little incentive to referee, and perhaps even less of an incentive to do a good job. This is the case with much academic labor: the reward for doing a good job is a "thank you", followed by more work.

What are some possible solutions? I suppose one option is to pay referees. This would likely help entice people to serve as referees. Presumably, referees who did a bad job would not be invited back. So this could certainly provide a better incentive system: one that rewards good referees. However, the academic publishing industry relies on the unpaid work of vast numbers of unpaid reviewers. If journals were to start paying referees, the system would collapse pretty quickly.

Sometimes journals do provide some small reward for reviewing an article. For example, The European Journal of Physics B once gave me a free, 6-month online subscription to their journal as a thank you for refereeing and article for them. With all due respect for EJPB, this wasn't that exciting. Also, my guess is that most who are potential reviewers for EJPB can already get EJPB through their library.

But what about allowing the editors to chose one or two referees per issue to get some sort of small award? Maybe a coffee mug, or a t-shirt, or even a nice mechanical pencil. Maybe a baseball hat or a pair of socks with the journal name on them. A pair of socks that said "International Journal of Bifurcations and Chaos" or "Advances in Complexity" would be pretty cool. These objects have some utility, and might become prestigious objects in the scientific community. If a journal started giving away cool socks for conscientious reviewers of their articles, I'd be much happier about refereeing for them.

No comments: