Thursday, February 7, 2008

Double-blind reviews

The peer review process in science is typically single-blind, i.e. the authors do not know the identity of the reviewers, but the reviewers are aware of the identities of the authors. This situation always puzzled me; is it not better to have a double-blind system?

My own experience suggested that it is wise. I recently received a manuscript where I was aware of the authors' identities. The final author (of several) is quite well respected, if not eminent. The paper, however, was crap. As I sat down to write the review, I began to second-guess myself. Maybe I don't know what I am talking about and Dr. Eminent has a much better command of the subject, after all I am just a junior faculty and he is a big shot professor at a prominent university. In the end, after talking to colleagues, I decided I was correct in my initial assessment and returned a "reject" review. A few weeks later I received an email showing that my decision was supported by a similar conclusion from the other reviewer.

I can easily imagine a situation where this process could have gone the other way, that is accepting poor quality work because of the eminence of the author.

This week's Nature has an editorial on the subject. If access is denied, it can be accessed here.

The editorial makes the point that, "Knowing author identities also makes it easier to compare the new manuscript with the authors’ previously published work, to ensure that a true advance is being reported."

I don't agree. In my opinion, the reviewer's job is to comment on the quality of the present work, whereas it is the editor's job to judge on the novelty in light of previous work by the same author or by others. Obviously the reviewer can do this too, even without knowledge of the author's identity.

Another issue is that, "referees could identify at least one of the authors on about 40% of the papers, undermining the raison d’ĂȘtre for double-blinding."

Well of course you can play the guessing game (I've often tried to decipher the ID of a reviewer), but you never know for certain.

The editorial also suggests that double-blinding is contrary to openness. This is difficult to assess; however, I might have to agree with it. I've experienced several situations where I've become aware of another team's work by virtue of seeing their work in review. Collaborations were initiated before the work appeared in print. With double-blind, this might not be possible.

Anyway, PLoS abandoned double-blind reviews, supposedly because too few authors chose it. I was unable to find further information. I'd like to know more behind that decision.

You can comment on the editorial via Nature blogs.

6 comments:

  1. I've had similar issues with reviews. I came to realize that one of the steps for me to develop as a young scientist was to have the stones to call crap "crap", with of course all due respect. Chances are good that, if not before the review process, but certainly after, they will realize it's crap too. If they are vindictive or whatever, it's the editor (not you, the anonymous reviewer) dealing with it, and you can take heart in knowing that you held up your end of the scientific compact, and helped the field progress despite their apparent wish for a cult of personality.

    My very first review, I chose not to be anonymous... and I'm still not sure where I stand on that issue specifically. I've occasionally learned after the fact who have been my reviewers/editors, as they've approached me themselves, but I think that's only going to be true for more substantive work.

    ReplyDelete
  2. You did Dr. Eminent a favor. Adding a bad paper to a long list of good and great papers won't help his reputation any. The grad student who was first author may not see it that way, but if rejection leads to rewriting three bad papers into two good ones, the student may benefit as well.

    ReplyDelete
  3. I prefer a totally open process where the reviewers sign their names to the reviews and they are mentioned when the paper is published.

    I don't like secrecy in science.

    ReplyDelete
  4. I agree with you for the most part, Larry. I would be afraid of retaliation for bad reviews, but I'd sign my name to a good review any day.

    ReplyDelete
  5. To follow up your question about why PLoS moved away from double-blind peer review, we did provide this option to authors initially on PLoS Biology, but after a couple of years, we found that very few authors took advantage of it. The editors also felt that double-blind review rarely lives up to its name. Reviewers are inevitably in the same field as the authors, and they might have already come across the work through the grapevine, at a conference, etc. And the citations in a manuscript refer back to the previous work that the researchers have published, which again can make it pretty obvious who the authors are. In the end, we were concerned that by offering double-blind review, we would be misleading authors into thinking that their manuscript can be anoymized simply by removing their names and affiliations. So, after some discussion, we took the option away, and have received little if any comment from the community.

    Looking ahead, we’re more inclined to move towards an open system of review, and like many journals we offer or even encourage reviewers to reveal their identities to authors. Compulsory open review has its issues too of course.

    Mark Patterson, PLoS

    ReplyDelete
  6. There is a flip side to the hesitation to criticize Dr. Eminent -- that harsher reviews might be meted out to junior researchers. Single blind review appears to disfavor female authors (and women tend to be more junior) in behavioral ecology journals, according to a recent TREE paper.

    ReplyDelete