Breaking the peer review blind
June 18, 2012
The manuscript peer review process is supposed to be secret, for the most part. The authors are not to know who reviewed their manuscript…this is generally for the protection against potential retaliation and the corresponding expectation of unfettered evaluation.
Yet one often has conversations with ones fellow scientists at conferences where it becomes obvious the other person reviewed your manuscript. Or that you reviewed theirs.
I find, especially lately, that this is *good* for science. You can discuss the issues with the person. Naturally this is only in cases for which the reviewer wasn’t a total hater…don’t think I’ve had that conversation yet!
DrugMonkey 
June 18, 2012 at 7:19 am
Not sure how that protection is more necessary in this arena when it’s not really used in other fields. …just curious…
LikeLike
June 18, 2012 at 7:28 am
It is not “necessary” in the sense of there being no other way, but it recognizes the reality that there are power differentials in this business of science. Such that a reviewer might fear some sort of retaliation that would unfairly compromise her publication, grant acquisition and/or career advancement.
LikeLike
June 18, 2012 at 7:31 am
The Glamour journals seem to unofficially break the blind for some of their articles and reviewers. When you see a short article commenting on a long article later in that issue, my understanding is that the author of the short article is usually somebody who reviewed the longer article.
LikeLike
June 18, 2012 at 7:50 am
I think this is one of those things that people “grow into” — the more established one is, the more one only sees the positive side to signed reviews. I’m at the stage where I’m willing to sign some reviews, but not quite courageous enough to sign a negative review of weird crap coming from a lab of a famous old fart.
LikeLike
June 18, 2012 at 8:17 am
Right JB, the more you are the person who can hand out the retribution, the more you are likely to want to know who to hand it out to.
LikeLike
June 18, 2012 at 8:52 am
I do sign my reviews, and a couple authors have written back to me. But as a reviewer I never email the authors. It feels strange sometimes to review a manuscript that might not be written as clearly as possible and wonder what the authors are thinking or whether you are missing something — yet it would be violating protocol to email the authors. Is this another example of how peer review is stuck in the 20th century?
LikeLike
June 18, 2012 at 9:22 am
It’s entirely possible to discuss the issues “blinded”. For instance, in the Frontiers journals “Interactive Review Forum”* authors can ask reviewers for clarification, but the identity of the reviewers is only revealed if the paper is published.
*”During the Interactive Review phase, authors and Review Editors can interact with each other through real-time comments in the discussion forum.” (http://www.frontiersin.org/Neuroscience/reviewguidelines)
LikeLike
June 18, 2012 at 11:04 am
Yes, I’m heavily in favor of some forms of interactions between reviewers and authors during review… there are so many small misunderstandings that can snowball. It benefits everyone for these to be resolved (or agreed how to resolve) before an editor has to make a decision. This can still be anonymous, as with Frontiers.
LikeLike
June 18, 2012 at 11:50 am
“Yet one often has conversations with ones fellow scientists at conferences where it becomes obvious the other person reviewed your manuscript. Or that you reviewed theirs.”
this seems so weird to me. So at some point you actual come out and admit it, so for the rest of the conversation it is all out in the open? Does this happen often? It seems like it would. Do you ever come right out and ask someone, or wait til it’s admitted? Are there ever cases where you just come right out and admit it when you run into someone, that you were their reviewer?
So if you are in a pretty small subfield, by the time your grants are reviewed, than your pubs, everybody knows all the details before you ever publish, but it’s all gone on in “secret”.
I wonder how big a threat this fear of reprisal is. Nice that people do sign reviews.
LikeLike
June 18, 2012 at 11:55 am
As a newly-minted PhD, I am completely in favor of both anonymous review (for the obvious reason of not wanting to play Russian Roulette to find out which bigshot holds a grudge), but also want to have more back-and-forth with referees. At one point, we ended up discussing our paper extensively with a referee, who was a friend of my advisor – it cleared up a great deal of miscommunication. The system miko and Mizumi discuss sounds great – there are lots of times when I’d like to take authors aside and say, “no really, *this* is what I meant,” and not have it take two months.
LikeLike
June 18, 2012 at 12:14 pm
I always write my reviews assuming that I will sign. This helps me be straightforward and even-handed in my review. I almost always remove my name when I submit my review. I do like the boundary of a blind review.
But–I usually talk to the authors in person about their papers I review. I’ve never sent an email to an author whose paper I’m reviewing (that to me would seem like violating a boundary) but I usually talk to authors when I see them at the next conference. Sometimes this goes great. Other times, not so much.
I have always appreciated it when reviewers of my papers come to talk with me about my paper and their review. It’s sometimes easier for me to take new ideas & critiques on-board when delivered face to face by a person I know. Also, I know that when people discuss papers face-to-face with me they really care about the science, and my work.
LikeLike
June 18, 2012 at 2:07 pm
So if you are in a pretty small subfield, by the time your grants are reviewed, than your pubs, everybody knows all the details before you ever publish, but it’s all gone on in “secret”.
Not really. If there’s three reviewers per grant (and maybe 20 on the panel who may or may not pay any attention) and three per paper that’s not “everybody”.
Do you ever come right out and ask someone, or wait til it’s admitted?
I don’t ask, no. and I don’t exactly admit it, either. but I’ve had people both ask me directly (I give a noncommittal response) and tell me directly. Most of the time it is more along the lines of “growing realization”.
I wonder how big a threat this fear of reprisal is.
Do you mean the fear or the reality? Either way, the point is to minimize the possibility, not to come up with some accurate accounting of rate.
I know that when people discuss papers face-to-face with me they really care about the science, and my work.
Yeah, this is the best-case. When the system is working well, this can be excellent.
LikeLike
June 18, 2012 at 3:17 pm
For instance, in the Frontiers journals “Interactive Review Forum”* authors can ask reviewers for clarification, but the identity of the reviewers is only revealed if the paper is published.
The Frontiers manuscript management system and its stupid fucken review Web form SUCKES complete shitte, as does “Interactive Review”. When I review a paper, I want to read it, write a couple paragraphs of analysis, give a recommendation to the editor, and be done with it. I have no time or interest in grading the goddamn motherfucken manuscript on a ten point scale on a half dozen different categories, let alone “interacting” with the authors or other reviewers.
LikeLike
June 19, 2012 at 3:05 am
Comrade PP (on Frontiers) – I ignore the stupid web-review form and just put all my comments in the final “other comments” section. That’s what all the reviewers I’ve ever dealt with in my Frontiers publications have done. That’s a different issue from the “interactive” nature of the review. I have had several cases where a reviewer has asked for some big complex thing, we’ve written a quick question “If we did X instead, would that satisfy you?” and gotten an answer. Took us a few minutes to ask the question, took them a few minutes to answer, saved us six months work, got us a paper published. Very efficient all around.
Alex – Although it is true that those GlamourMag Comment statements at the beginning are often the reviewers, they are not always. I know of several cases in which the reviewers were unable or unwilling to write the Comment and suggested someone else. The key is that although you may suspect the reviewer, unless they tell you, you don’t really know.
On study section review – it is definitely dangerous to assume that because someone is on study section they reviewed your grant. Because triaged grants are only seen by three people and those three are assigned by the SRO and chair, it is very possible that the person you think is the reviewer never saw your grant at all.
I am reminded of a comment by one of the editors at one of the GlamourMags (I think it was Sandra Aamodt at Nature Neuroscience, but I’m not sure) that people calling to complain guessed wrong about 40% of the time who the reviewer was.
LikeLike
June 19, 2012 at 5:33 am
Hmm…this is an interesting discussion. I never do anonymous reviews. I guess I figure if I am going to be critical of someone they should know who is doing the criticizing. Perhaps it is different in my discipline (earth sciences), but it is common for the reviewers to reveal themselves. Not as common as blind reviews though. I can understand your points about why to go anonymous, but I guess I hope for the best. Naive, I know.
LikeLike
June 19, 2012 at 5:53 am
I sincerely hope that other journals adopt -some- kind of online forum for reviewing, such that simple questions can be asked and answered … simply. Then the major work and decisionmaking could be set apart from those. Because in an ideal world, accept/reject decisions would not be made based on the simple, easily answered, non-impacting questions. I had a review once where I felt the rejection was based at least in part by being “pecked to death by ducks”, none of which impacted the ideas or the science.
LikeLike
June 19, 2012 at 12:18 pm
Why do some of you use such a fowl language?
LikeLike
June 19, 2012 at 12:27 pm
Why do some of you use such a fowl language?
Because we love fucken birds!!
LikeLike
June 19, 2012 at 1:00 pm
What?
LikeLike
June 20, 2012 at 4:41 am
Only angry birds..
LikeLike