Repost: The reviews are in…

July 29, 2010

This post originally appeared at the old site May 22, 2007.

Lots of bashing of the peer review process lately. Admittedly Orac has a nice counter, directed at forces external to science but highly relevant to on-the-bus complainers. [Update: another comment on peer review from NeuroLogica]
I have some unusually un-cynical thoughts today.

I finally got some reviews back on a recent submission and they touch on much that is wrong and much that is right with manuscript review. First of all, we’re talking a normal journal here, Impact Factor in the 3 range, field specific, working scientist as the editor. Meat and potatoes stuff. The topic of the paper is pretty much in the heart of the journal. It does however, reflect a slightly contrarian experimental approach which in some ways violates all the “norms” which were established over the past couple-three decades in this area. nothing earth shaking, just some experiments which converge on a single point of view, suggesting that no, we don’t always have to do things the canonical way and there is room for some improved models.
One reviewer is…critical. obsessively so. detailed point-by-point complaints about the interpretation of results. The flip side is that one reviewer “gets it”. Very laudatory review, really. Almost makes a better argument for publication than I could make myself. Editor comes in with “may be acceptable pending revision” with some additional critique.
Okay, pretty standard stuff, GREAT, I think and start beavering away with the responses and revisions. Why am I not ticked as other seem to be by the divergent opinions of the reviewers? Well, first of all, let’s face it. In contrast to the dismal reinforcement rate of the grant process, paper review has a fantastic effort/reward relationship. As one luminary in my area pointed out a very long time ago, everything gets published eventually. Especially when the editor seems favorably disposed in the face of at least one rather critical review. But in addition we should all admit in these cases that the truth lies somewhere in the middle. The paper is likely not as good as the favorable review indicates and not as bad as the critical one indicates. Editor serves as mediator over where the mean should fall. This is a good thing. Often the bias is for publication, again a good thing for all of us. As the old saw has it, real peer review starts after publication anyway.
Let’s take the ‘bad’ review first. Yes it IS irritating that some idiot questions our brilliant conclusions, seems willfully to miss the point and can’t see the forest for the trees. However, one of my best mentors once said to me that no matter how bad or stupid the reviews seem, it always results in a better paper. I have found this to be true, sure enough. A related point is that we should understand that the reviewers stand in proxy for our eventual audience. There will be critics and nonbelievers reading your paper if it does make it into print, don’t you want to have the chance to head off some of the criticism in advance? So the “idiot reviewer” is useful. Finally, heh, strategically if one wants to make sure the critical review isn’t heeded by the editor, we want them to be as obsessive, critical and idiotic as possible. Personal insults if possible (yeah, I had one of those recently too!) This can’t possible help the editor take his/her side and therefore is a good thing for the authors.
Now the “good” review. This one is tougher. Sure, we all want a cream puff review because after all, our manuscripts are brilliant as submitted right? Is this a reflection of the good-old-boys/girls club that those on the “outside” lament? Well perhaps. The lab group I’m in isn’t big-wiggy for sure but it IS known. Furthermore, the journals are increasingly requesting advice on who the reviewers should be (and should not), so yeah, we took advantage of that to request people we thought might be friendly. The thing is, who knows? I suspect that when it comes to paper and grant review, we don’t hit very high on average estimating who is going to give us an easy time of it and who is going to rip us apart. Just because you have a drink or two with a colleague and bemoan the state of the funding crisis into your beer doesn’t mean they’ll accept crap science from you! Getting back to the point, I just can’t say. Maybe the “good” reviewer was from our suggested list but maybe both of them were too. Maybe the “bad’ reviewer was someone we think of as a friend of the lab and the “good” one was a complete unknown!
Anyway, the system is working today, even if I am spending inordinate amounts of time on a point-by-point rebuttal of idiotic comments….

20 Responses to “Repost: The reviews are in…

  1. Namnezia Says:

    I agree – unless the bad reviewer is just being mean because he doesn’t like you, it suggests that perhaps the importance of your paper might not be coming through to all readers (because if it did they would all think its brilliant – right?). The rebuttal gives you a chance to more finely hone your message, no matter how annoying it is. I’m currently in the midst of preparing a monster rebuttal, and it IS aggravating to do, but hopefully the paper will be better for it (and hopefully it will get accepted).


  2. another young FSP Says:

    I don’t go any easier on my friendlier colleagues’ papers than the more distant ones. The main advantage that colleagues who know me reasonably well have is that we’ve often talked methods or approaches in the past, so some of the major questions are already addressed.
    Personally, I would much rather that bad reviewer give me the chance to address their complaints prior to publication and to argue my case, as opposed to having them trash talk my work elsewhere. Paper review is a 2-way dialog.


  3. Beaker Says:

    I would personally like to thank the anonymous cranky bastard who reviewed our last paper. We managed to appease after 3 rounds of revision, and those extra experiments and re-writes not only made the study better but also pointed us towards the next discovery for the next paper. Actually, this has been my experience with many papers. As long as the review is a dialogue (even a heated, bitter, cynical dialogue), the final product will be better.


  4. First of all, we’re talking a normal journal here, Impact Factor in the 3 range, field specific, working scientist as the editor.

    Dude, why are you wasting time and resources with shit you’re only gonna publish in some shitball IF=3 journal that no one even fucken reads?


  5. whimple Says:

    Yeah, I was wondering that too. Watch 2 of the 3 IF points be from self-references from your future crap in this nothing journal. This is where your “brilliant conclusions” go?


  6. IF curious Says:

    Since CCP and whimple are giving DM a hard time about the journal, I am wondering… What is a respectable IF for a very good, reputable, disciplinary journal, where you send solid but not flashy or extremely hot work?
    In my field (physics/materials science) the very good disciplinary journals have IF from 3 to 6 (e.g. Physical Review B), IF of over 7 is considered quite high already (Physical Review Letters, Nano Letters) and reserved for hot stuff with broad interest to physicists, while Nature progeny (Nature Materials, Nature Nanotechnology) with IF around 20 are for really flashy stuff.
    So if DM were a physicist, the work he described would go to a journal with IF of around 3-4, which is very respectable. What are the equivalent journals in biomed and their IF’s?


  7. DrugMonkey Says:

    IF curious, see how even your queries are laden with value judgment words? It is not only field dependent but also in the eye of the beholder.
    and FYI CPP is just getting revenge for me carping about how GlamourMagScience is ruining everything and means you are not actually interested in building knowledge, etc.


  8. IF curious Says:

    DM, I didn’t mean to judge. I think the value judgements above hold pretty broadly among scientists in my field so I would consider them unwritten but universally accepted criteria in my field. I listed them here as I am sure many of your biomed readers have no idea what these journals are of what their standing is.
    I was honestly just curious what the equivalent criteria are in biomed.


  9. DSKS Says:

    “Eye of the beholder” my arse. A journal with a kick-ass IF is obviously one that I’m published in. Everything else, above and below that, is peripherally relevant tosh.
    As a great scientist once said (I think it was me, yesterday), “When it comes to science, there’s my stuff and then there’s all that other bullshit.”


  10. Neuro-conservative Says:

    All hand-wringing aside, there is a simple, empirically-derived answer to the question:
    If you enter journal IF’s into a frequency histogram, you will see a clear inflection point just below 5. This holds across most major fields of biomedical research.
    By this method, journals from 5-10 are “solid.” Journal of Neuroscience, for example, is around 7. Once you hit 10 and above, you start to get into the “high impact” range.


  11. whimple Says:

    I’ve always considered JBC as the gatekeeper to the “solid” range.


  12. Neuro-conservative Says:

    Then we’re roughly in agreement — JBC=5.328


  13. Publishing in anything below IF=5 is just fucken jerking yourself off.


  14. LadyDay Says:

    How would you know, CPP, since you’ve clearly never done either of those things?


  15. Jerry Says:

    Off topic. Since you’re the only person I read on ScienceBlogs, have you seen this:


  16. kalisto Says:

    IMO, the author sounds arrogantly angry lessening credibility to her overall analysis.


  17. Dr. Feelgood Says:

    If you have a paper in a IF=3 journal, but your actual paper has an impact score of 15 are you still just jerking off? My view is, if you can bootstrap an OK journal with a paper that is well above the journal’s IF, then you are improving it.
    Getting a boner over an IF is often counterproductive.


  18. DK Says:

    All this misses the most important point. Which is that peer review is not taken seriously and as a rule reviewers don’t do even remotely decent job. Who has time for this when you have several papers to review every week? Unless one has no life, it’s nearly impossible. The result is that the vast majority of reviews I received are downright incompetent, clearly indicating that the manuscript has simply never been read attentively. That goes for both negative and positive reviews.
    The idea of peer review is like the idea of communism. Which is that most often than not humans will do very well something that does not give them immediate advantage. Sounds good in theory, doesn’t work in real life. That the humans in question are anonymous and not subject to any kind of feedback makes achieving the utopia even less likely.
    JBC as the gatekeeper to the “solid” range.
    LOL. For many labs, JBC is a dumping ground to publish lab journals nearly verbatim.


  19. DSKS Says:

    “For many labs, JBC is a dumping ground to publish lab journals nearly verbatim.”
    Man, those labs must be so hip and cutting edge. Can I get an autograph?


  20. Waving not drowning, yet UK Says:

    I do not go in for this blogging craze but I stumbled on this website and found it to be very pertinent to my experience as a PI of a small group struggling against the biases of the UK funding system, Institutional red tape, viscious research peers who refuse to cite our work, Bibliometric performance review (honestly) and blatantly polarised journals. I totally agree that a ‘bad’ review whilst ruining your day and upsetting PhD students who do not understand the un-even playing field does make for a better paper, eventually – and if you have the resources to do those suggested experiments or the time to dismantled your lovingly crafted prose. I have passed on DMs opening passage to my PhD student as an index of the wisdom that is missing from our daily lives. Thanks


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: