Peer review of manuscripts as part of the scientific discussion
June 9, 2011
We occasionally lose track of this fact. Our stance towards the reviewers of our manuscripts can be fairly antagonistic. After all, we wouldn’t have submitted the dang thing in the first place if we didn’t think it was ready for publication as-is, right?
It doesn’t help that one of the manuscripts we have out right now has drawn reviewer fire over some of the more maddening reasons. Basically a difference of opinion on interpretation, background and context- my view (even apart from my own manuscripts, thank you!) is that if the data are sound, well analyzed and placed in a context that is supported it is not my place to hold up publication because my interpretation differs. So these kind of “discussions” during peer review don’t really please me.
Another paper we have in the submission process is another matter. I have a little less than usual confidence that we know what is what when it comes to our findings. I am really keen on seeing what reviewers have to say about it. I am looking forward to what I think is the start of a pretty cool discussion. Hopefully in the sense of additional data, models and papers resulting because I have a sense this little subarea is about to take off. Not that we’re going to be the spark, mind you. Just that we’re getting into some stuff that a bunch of other usual-suspect labs can do and have all the same reasons that we do to delve into the questions. There are, however, a whole lot of ways to get into the question and model the behavior.
We took one approach and I am pretty interested in what the reviewers are going to think. Will they buy that our kewl effect is actually interesting? Will they come up with a whole ‘nother context in which it should be framed or interpreted or do they sign up for our view on the phenomenon?
Can’t wait for the reviews to come in on this one…..
DrugMonkey 
June 9, 2011 at 2:27 pm
I know of researchers who send out manuscripts that are not really “done.” They use the peer review process as their editor/critic, apparently because they are to lazy to do it themselves or to ask colleagues to read it before submission. Chances are good that they get rejected on the first submission, so then they just fix some of the shit the reviewers didn’t like and send to another journal. This process goes on recursively until acceptance, typically in a crap journal.
This lame strategy subverts the peer review process and creates extra work for people who are not the authors. Every submission should be the best you can make it, and busy reviewers should not be expected to waste their time “cleaning up” somebody else’s sloppy manuscript. This is different from honest disagreements and discussion, such as mentioned by DM.
On a related note, why do a surprising number of manuscripts ignore the word limits or the reference limit and work under the assumption that it will get fixed later by somebody else? This is the written equivalent of giving a 1.5 hr seminar for a 50 min time slot. Editors should punish this practice and not send out these manuscripts for review. Period.
LikeLike
June 9, 2011 at 7:15 pm
I agree with the Beak on the former part, and mostly on the latter part, but if you are going to Glamour town, I usually think if you can get with 50% of the limits for S/N then that is good enough to go out to review. You need to get the reviewers to understand the work, and many edited, published S/N papers are almost incomprehensible, so it is likely they got the reviewers to go for it based on a a document with complete sentences.
DM- question- how often in you experience where you review a paper and you know you are one of THE experts to review it, so you have a lot of perhaps, subtle/sophisticated things to say about interpretations and the author’s reading of the literature (perhaps inappropriate), and yet the other reviewers send in 30 word reviews with no evidence they have even read the paper?
This is quite annoying.
LikeLike
June 10, 2011 at 2:11 am
Agree, those are annoying. Not too frequent IME, though. Below 10%?
In return, when you are THE expert do you make allowances for this? Recognize that you may be unfairly picky about your pet hobbyhorse issues? Or do you dig in because you are THE ONE who can protect the literature from this pollution?
LikeLike
June 10, 2011 at 2:17 am
Beaker-
The biggest thing I try to instill in postdocs about reviewing papers is to know how to not fix the paper for the authors. Kind of like teaching them to triage and to identify the key issues without fixing the whole thing.
LikeLike
June 10, 2011 at 3:49 am
You mean the postdocs that aren’t authorized to review these kinds of privileged communications? 🙂
LikeLike
June 10, 2011 at 4:57 am
That is inaccurate, whimple. Where relevant I have added a note to the editor that a postdoc assisted in the review and have never, ever had an Editor object.
LikeLike
June 10, 2011 at 6:20 am
Actually getting postdocs & senior phd students involved in the review process is a win-win-win situation. I often send the editor a note, asking if they mind if I supervise a trainee reviewing the paper. Editors are by and large relieved to get a review, and as long as there are some grown-up eyes on it, are happy to do this. Students & postdocs learn a lot about reviewing, but its also a good part of the skin-thickening process. I remember being devastated when I got one of my first reviews back. Writing a review at the time they are getting their first ones back helps everyone have a little perspective on the situation. As DM, I have never ever had an editor object.
LikeLike
June 10, 2011 at 10:14 am
You clear this with the editor before you share with the postdoc right? Back in the dark ages of my postdoc, the PI would give the manuscripts he wanted killed to the postdocs to review. He didn’t do that with grant applications, but that might be a useful training experience for the staff too.
LikeLike
June 10, 2011 at 12:09 pm
I have reviewed vast numbers of papers with the assistance of my trainees, and editors are always fine with it. Some journals even have a special text box in the review submission Web page for you to list the names of anyone in your lab who has assisted with the review.
LikeLike
June 10, 2011 at 3:46 pm
I definitely always to step away a little bit about being too picky, but it isn’t usually the issue. There are cases where one of the other reviews is lazy (and this class is almost always NOT a post-doc in my guess, but a bigwig PI), but never both other reviews out of 3, as it was just this month. Most of the time in my experience reviews are really good, with the minority being either positive but ultra lazy (like two sentences) or unfair and trying to kill the paper.
LikeLike
June 11, 2011 at 10:30 am
“The biggest thing I try to instill in postdocs about reviewing papers is to know how to not fix the paper for the authors.”
Why the hell not? Yes, people submit shitty papers, but what does it prove for you to withold potentially helpful comments or suggestions? If the paper is bad enough, it should be the editor who shitcans it and sends it back without review. Once it’s in your hands, though (and you agree to review), you are doing the editor a favor (and the authors) by providing fair reasons for rejection. I can’t believe that you qualify this as THE “biggest thing”.
LikeLike
June 11, 2011 at 11:16 am
Giving your fair reasons for rejection is not the same as sorting out all the problems in a really bad manuscript.
and it is my biggest concern because my postdocs usually do fine with the rest of the task
LikeLike
June 13, 2011 at 7:52 am
Could it be that sometimes “giving a fair rejection” actually means “sorting out all the problems”? Let’s assume the authors didn’t do several important experiments that you think are essential to test their hypothesis. If you say, “the authors need to do X and Y and Z”, aren’t you sorting out at least some of the authors’ problems? And if, instead, you just say, “the authors’ experiments are inadequate to test the hypothesis,” is that really being fair to the authors? They may have no idea of what experiments you think are necessary.
LikeLike
June 13, 2011 at 9:05 am
Grumble,
There is a whole world of difference between “sorting out the authors problems” and simply saying “the authors experiments are inadequate to test the hypothesis”. The former is too much work and the latter is half-assed. The in-between spot is the proper amount of review.
CB
LikeLike
June 13, 2011 at 10:37 am
If you say, “the authors need to do X and Y and Z”, aren’t you sorting out at least some of the authors’ problems?
Beyond the fact that it is simply not the responsibility of the reviewers to help the authors do good science (that’s the job of the peers they contact personally for help, imo), there is another problem. Who are the reviewers to tell the investigator what scientific directions they should take? In my experience if you have, say, a paper that is kind of uninteresting there are any number of directions that *could* be taken. Maybe more of the same type of data is required. Maybe they need some different methods or models. Maybe they need to go pharmacological, physiological, molecular or behavioral. Any of these might turn a boring finding into something great. Deciding on that, however, is NOT the job of the reviewers.
The authors have a responsibility to actually, you know, do science. On their own hook and volition.
LikeLike
June 15, 2011 at 11:18 am
“Who are the reviewers to tell the investigator what scientific directions they should take? In my experience if you have, say, a paper that is kind of uninteresting there are any number of directions that *could* be taken.”
Well, yes, but I’m not talking about the issue of whether the paper is *interesting* or not. I’m talking about experiments that should be done to thoroughly test a hypothesis, whether the hypothesis itself is boring or interesting.
Canadian_Brain is probably on the right track: one has to find some in-between, but I think if the reviewer thinks the experiments are inadequate, a thorough review will end up providing a list of necessary experiments anyway – even if just by implication (i.e., “the authors need to show that X is related to Y” – well, there is likely to be only a limited number of ways to show that that particular X is related to that particular Y, so this comment is really just a way of saying, “do this damn experiment.”)
LikeLike
June 15, 2011 at 7:49 pm
I never tell authors what to do. I say the experiment or interpretation is inadequate, or unconvincing. Then I suggest what I think would make it convincing. I realize that some authors and reviewers are philosophically against this, but I’m not really concerned with that. I don’t do it in an offensive way and papers are definitely improved. Of course this is my opinion. The internet is for opinions.
Also, editors for certain journals specifically ask for reviewers to make judgements on the significance or general interest of a story. If the reviewer just says “not interesting enough” that doesn’t seem as collegial as “no, but…”
LikeLike
June 23, 2011 at 4:51 am
I mentioned once to a assistant editor from a low-impact journal that my postdoc would be looking at the paper and review it as part of the review process done by me/my lab and got back a lecture that that is absolutely not acceptable and pertains confidential information bla bla. needles to say I subsequently declined to do the review and put the journal on my blacklist as I don’t think I need to be lectured by an assistant editor if they want me to do something for them.
Speaking about editors – how come that in journals like the Nature ones some assistant editor decides about live or die even before a scientific reviewer from the field has looked at the paper for review (and yes, I am grumpy about a paper not send to review while other crap that has been published in lower tier journals before got in)…??
LikeLike
June 30, 2011 at 5:49 am
I can relate to your curiosity about what the reviews will look like. The experience is different every time.
What do you think about publishing reviewers’ comments, say, on a blog? I know I’d be interested seeing the reviews of various papers and I’ve been contemplating doing this with my papers. Since the reviews are anonymous, there should be no conflicts…right?
LikeLike
June 30, 2011 at 7:44 am
I’m just contrasting, JS, with situations in which I am relatively more confident that I know what is what, where the data fit into the literature, what the alternate interpretations are, etc. Another way to put it is that I am the worlds expert in this stuff and I care relatively less what anyone else thinks 🙂
I don’t really know whether journals have policies about publishing reviewer comments that you receive as an author. It sure would seed the discussion points. Especially when you annotated with your responses “clearly Reviewer three wasthat asshole Dr. Blo who has been trying to squelch our awesome data ever since I asked him that question he couldn’t answer at SFN in 1986…”
LikeLike