The reviews are in…

May 22, 2007

Lots of bashing of the peer review process lately. Admittedly Orac has a nice counter, directed at forces external to science but highly relevant to on-the-bus complainers. [Update: another comment on peer review from NeuroLogica]

I have some unusually un-cynical thoughts today. I finally got some reviews back on a recent submission and they touch on much that is wrong and much that is right with manuscript review. First of all, we’re talking a normal journal here, Impact Factor in the 3 range, field specific, working scientist as the editor. Meat and potatoes stuff. The topic of the paper is pretty much in the heart of the journal. It does however, reflect a slightly contrarian experimental approach which in some ways violates all the “norms” which were established over the past couple-three decades in this area. nothing earth shaking, just some experiments which converge on a single point of view, suggesting that no, we don’t always have to do things the canonical way and there is room for some improved models.

One reviewer is…critical. obsessively so. detailed point-by-point complaints about the interpretation of results. The flip side is that one reviewer “gets it”. Very laudatory review, really. Almost makes a better argument for publication than I could make myself. Editor comes in with “may be acceptable pending revision” with some additional critique.

Okay, pretty standard stuff, GREAT, I think and start beavering away with the responses and revisions. Why am I not ticked as other seem to be by the divergent opinions of the reviewers? Well, first of all, let’s face it. In contrast to the dismal reinforcement rate of the grant process, paper review has a fantastic effort/reward relationship. As one luminary in my area pointed out a very long time ago, everything gets published eventually. Especially when the editor seems favorably disposed in the face of at least one rather critical review. But in addition we should all admit in these cases that the truth lies somewhere in the middle. The paper is likely not as good as the favorable review indicates and not as bad as the critical one indicates. Editor serves as mediator over where the mean should fall. This is a good thing. Often the bias is for publication, again a good thing for all of us. As the old saw has it, real peer review starts after publication anyway.

Let’s take the ‘bad’ review first. Yes it IS irritating that some idiot questions our brilliant conclusions, seems willfully to miss the point and can’t see the forest for the trees. However, one of my best mentors once said to me that no matter how bad or stupid the reviews seem, it always results in a better paper. I have found this to be true, sure enough. A related point is that we should understand that the reviewers stand in proxy for our eventual audience. There will be critics and nonbelievers reading your paper if it does make it into print, don’t you want to have the chance to head off some of the criticism in advance? So the “idiot reviewer” is useful. Finally, heh, strategically if one wants to make sure the critical review isn’t heeded by the editor, we want them to be as obsessive, critical and idiotic as possible. Personal insults if possible (yeah, I had one of those recently too!) This can’t possible help the editor take his/her side and therefore is a good thing for the authors.

Now the “good” review. This one is tougher. Sure, we all want a cream puff review because after all, our manuscripts are brilliant as submitted right? Is this a reflection of the good-old-boys/girls club that those on the “outside” lament? Well perhaps. The lab group I’m in isn’t big-wiggy for sure but it IS known. Furthermore, the journals are increasingly requesting advice on who the reviewers should be (and should not), so yeah, we took advantage of that to request people we thought might be friendly. The thing is, who knows? I suspect that when it comes to paper and grant review, we don’t hit very high on average estimating who is going to give us an easy time of it and who is going to rip us apart. Just because you have a drink or two with a colleague and bemoan the state of the funding crisis into your beer doesn’t mean they’ll accept crap science from you! Getting back to the point, I just can’t say. Maybe the “good” reviewer was from our suggested list but maybe both of them were too. Maybe the “bad’ reviewer was someone we think of as a friend of the lab and the “good” one was a complete unknown!

Anyway, the system is working today, even if I am spending inordinate amounts of time on a point-by-point rebuttal of idiotic comments….

4 Responses to “The reviews are in…”

  1. Rob Knop Says:

    Bay the way — you link to my post about there being no point in applying to the NSF. That was a grant application there, not a paper submission. As such, you slightly misrepresent my position by implying that I’m one of those ticked at the paper review process. I’m certainly ticked at the inconsistent feedback in the grant review process– because so few grants are accepted, you need everybody who reads it to like it, but when you have inconsistent feedback it’s impossible to go forward. As you note, any paper worthy of publication will get published, whereas we’re a *long* way from every grant worthy of funding receiving funding.



  2. drugmonkey Says:

    Rob, I don’t think I was getting this specific with the links, just pointing to some other posts that I found interesting or motivational with respect to the topic. I linked to your post because you illustrate, quite nicely, the general issue at hand. Namely the irritation that one gets when scientific peer review results in apparently conflicting opinions. This is a very similar process with many of the same considerations in grant and paper review. It is just that the outcome is more painful in grant review!


  3. Rob Knop Says:

    One thing that is interesting — in my field, generally, most papers have a single referee unless it’s a special case. Sometimes after an initial review, the editor will decide, or the authors will request, that the paper be sent to another referee.

    In my experience, most of the time you send your paper in. You get comments back from the referee, which always include some corrections and requests to address issues. Sometimes it’s minor, sometimes there are more serious issues. If the issues are minor, once it’s resubmitted and the authors have addressed the comments, the editor will go ahead and OK it for publication. If the comments were major, or of the authors add something that wasn’t specifically requested in the comments, it will go back to the same referee.

    Since, at least in my experience, there’s just one referee, we haven’t had the problem of conflicting feedback. Sometimes we’ve had issues of the referee not seeming to have “gotten” something, but that can often be handled in the response to the comments. It probably does indicate that the paper did not explain what was done clearly enough.



  4. drugmonkey Says:

    A single referee? Ouch. I’m feeling ticked because what used to be a seeming default for three reviews in my area is rapidly becoming two. And the selection of reviewers is starting to be based on who clicks the “accept” link on the invitation email first- I’ve had a couple of experiences where an invite to review was retracted within hours because the journal got the necessary number of acceptances before I got to the email.

    Thanks for commenting Rob, I find the differences and similarities between the physical and biomedical sciences to be fascinating…


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: