On re-reviewing a manuscript
June 25, 2012
I most usually find the job of re-review to be quick and easy. You take a look at the revised manuscript and the answer to criticism and see if you major complaints have been answered. Maybe take a look at the other reviewer comments to see if they are cluing you in to something you’d missed and should be concerned about. But basically I don’t typically get into a dogfight. Either the authors have taken a good stab at it and you say “good enough, accept that sucker” or they have not and….
..well there’s the rub.
When they have basically blown you off and not responded to your insightful and completely necessary comments in a significant way…what do you do?
REJECT! of course. Right? They had a chance to improve the manuscript and they didn’t. So you demand rejection.
But has anything really changed from the initial review to the second one? You said the first time that it should be published, given that the authors fix a couple of nagging details. And you don’t know if the failure to respond to your (insightful and completely necessary) criticism is because they just didn’t want to or because if they did so, there would no longer be a publishable result. Or they simply fear that if they did what you asked (add a control, boost the number of subjects, verify in a slightly different system or wtfever it was) that there would no longer be a publishable result.
If, however, they just didn’t want to do any more work (and believe you me, I’ve been there) and basically disagree with your criticism…what next? In essence your stance should not have changed from the first round of review, right? The data could still be publishable if they would just do. what. you. suggested!
So your stance on accept/reject shouldn’t change, right? Because if it does change to REJECT, doesn’t this just mean you are being a petulant jerk, stamping your feet that you’ve been ignored? You don’t have any additional information on which to base an estimate that the results are, in fact, as … fragile…as you suspected them to be when you suggested more experiments be conducted. Maybe they are, maybe they aren’t.
Your stance should be the same as the first time.
DrugMonkey 
June 25, 2012 at 1:13 pm
Yup. It’s the editor’s call, not the reviewer’s: the latter are there to advise the former.
So you tell the editor, I still think this is publishable, but they haven’t taken care of the issues I brought up last time and of course I still think they should.
LikeLike
June 25, 2012 at 1:19 pm
But what if the editor takes ‘accept with major revisions’ as ‘accept’ the second time around and the paper is accepted without them changing anything? I’ve had this happen enough times that I’m tempted to start doing the ‘reject’ second pass, to get the point across.
LikeLike
June 25, 2012 at 3:05 pm
I have only been blown of maybe once and that was when in response to one of my queries the first time they admitted that thei major datast was n=1 so I said “i can no longer judge the merit of this work as it has not been shown not be reproducible” ugh.
LikeLike
June 25, 2012 at 3:06 pm
The authors have the right to reply to any criticism of the reviewers and try to convince the editor that their position is valid. I have done this successfully a couple of times, and failed a couple of times as well.
LikeLike
June 25, 2012 at 3:09 pm
I am a co-author on a manuscript that just got flat-out rejected after revisions. I have not seen that before. All three of the reviewers were clearly pissed. The experiments that were requested were very reasonable, but the senior author refused to do them. Oh well, his decision really…….
LikeLike
June 25, 2012 at 4:15 pm
I am an associate editor at a society level journal. I like to see that the authors have made a good-faith effort to respond to my comments. If they have, then I forgive them things that they cannot do (provided there is a reasonable explanation). If I don’t feel that they have made a good-faith effort, I reject.
LikeLike
June 25, 2012 at 4:24 pm
I’m generally mild if authors make an effort. But if I get this “I’m so smart and I talked myself out of all critque”-rebuttal I have no mercy. neither as a reviewer nor as an editor. Reject.
LikeLike
June 25, 2012 at 4:43 pm
I am an associate editor at a society level journal. I like to see that the authors have made a good-faith effort to respond to my comments. If they have, then I forgive them things that they cannot do (provided there is a reasonable explanation). If I don’t feel that they have made a good-faith effort, I reject.
This is depraved. What the fucke does “good-faith effort” have to do with whether the manuscript is suitable for publication or not?
LikeLike
June 25, 2012 at 4:51 pm
I expect the authors to respond to my comments. (Not necessarily agree with, but at least respond to them. I have been persuaded by good arguments before.) If they misunderstood my comments, but made a good-faith effort to respond to them, I’ll try to re-explain my problem with the manuscript and I’ll include a secret note to the editor that the authors misunderstood the issue and that I still think the paper is publishable but only if they fix it. (This is an example where I think secret notes to the editor are good, but some journals don’t allow them. For those journals [I’m talkin’ to you JNeurosci], I put the blunt statements in the notes to the authors.)
However, if the response is “F-U, I’m too good for your comments”, then I figure they’re never going to fix it and I recommend reject on the theory that they’re never going to fix it and it’s wrong without the fix and I don’t want to waste anymore of my time.
LikeLike
June 25, 2012 at 5:54 pm
Similar things have been pissing me off recently too…
I hate asshole authors.
LikeLike
June 25, 2012 at 6:17 pm
I may regret jumping in here but I have a bit of anecdotal evidence in regards to the rabbit hole of re-revision. My sub-field is one of the few where you can get away with publishing a a cool result for cool results sake and this paper was one of those. The manuscript was a communication, 3 pages long and ended up with four reviewers. It got kicked back as accept with major revisions and the criticisms were genuinely insightful and completely necessary.
We addressed them as best we could but the result was a weird perfect storm situation which made it difficult to address the comments as directly as they would have liked (not for lack of trying). I composed the rebuttal as thoroughly and respectfully as possible and even offered up the raw data from the requested experiments for review.
Re-review comes back 2 go aheads, one tiny tweak, and one “you need to do X, Y and Z.” Attempted X, Y, and Z and again rebutted with the results for review. This time coming back with 3 go-aheads, one “still not convinced” and a note from the editor that we had removed the one citation that came from the journal we were submitting to and had to put another one in (which pissed me off but that is another discussion entirely).
Rebutted again, this time explaining as politely as humanly possible that repeating the same experiment a fifth time wasn’t going to make the result that they wanted. I included extensive references as to why our interpretation was the only one that didn’t violate the laws of chemistry as we know them. It was only then that reviewer 4 acquiesced but only if we added hedging words into the title.
I have great respect for reviewers and try to treat them with respect that they deserve for donating their time to improve my science. I even can understand reviewer 4’s trepidation but in this case the continued back and forth just ended up biting everyone in the butt.
LikeLike
June 26, 2012 at 2:26 am
Crystal Voodoo, I feel your pain. This sort of a ridiculous time-and-effort-wasting exercise is part of the reason that I have increasing reservations about the whole practice of traditional pre-publication peer-review. Yes, it has significant benefits; but it also has significant costs. And in a digital world where there are no limits on capacity any more the importance of the filtering benefit is shrinking.
LikeLike
June 26, 2012 at 5:04 am
When I review, I try to distinguish between things an author really must change to get an acceptance vs things that I feel would make it a better paper. If I’m clear & the authors don’t make a critical change, I reject. I very rarely require authors to collect more data. I often say they can collect more data OR very clearly note the limitations of interpretation based on their existing data. I lean towards accepting honest papers that state limitations vs waiting for every paper to be a masterpiece. The exact threshold depends on the journal. An interesting, but preliminary result fits in some journals better than others.
My worst reviewing situation was when I insisted some authors correct a severe statistical mistake that made it impossible to draw meaningful conclusions from their data. We went back and forth for months until I finally had to reject their 3rd revision. It didn’t feel good, but I wasn’t asking for much & they still didn’t do it.
LikeLike
June 26, 2012 at 5:56 am
I reviewed a manuscript for PloS One in which the experiments were solid, and the analysis valid. But the authors excessively over-interpreted their data, going so far as to put their excessive speculative conclusions prominently into the title and abstract of the paper. Since this was PloS One and the science was fine, I simply asked that they tone down their excessive speculations and restrict them to the discussion. When the revision came back to me almost one year later, they had not changed their text and the title and abstract still misrepresented their modest discovery. Talk about shooting yourself in the foot. They could have gotten away with one day of re-writing and then submitted the revision. After another round of revision (the editor was totally on my side), the authors finally capitulated. It was a waste of everybody’s time, but the authors brought in on themselves.
LikeLike
June 26, 2012 at 6:21 am
“If, however, they just didn’t want to do any more work (and believe you me, I’ve been there) and basically disagree with your criticism…what next?”
I think reviewers should make some effort to be convinced by authors’ arguments — our cognitive bias is to dig in when argued against. It’s seldom that authors “just” don’t want to do any more work. It’s also that they’ve thought a lot more about the work than anyone else. But no one likes to get blown off, and anything less than doing exactly what was asked sets most reviewers off. Sometimes before sending a “you didn’t fucking do what I asked so fuck you” re-reviews to authors, I will ask the reviewer to carefully consider the journal’s criteria, the author’s main point and rebuttal, and see if they can reconsider (obviously, I don’t do this if it is a technical error, missing control, or failure of interpretation/analysis).
I firmly believe that for every essential or manuscript-improving reviewer experiment there are at least 5 (maybe closer to 10) that are a total waste of everyone’s time. Authors will almost never have written the paper you wish they had, and unless it’s really something that should not be part of the scientific record, a light touch is best. The idea that 2-3 people chosen essentially at random is a rational or effective form of gatekeeping is a fucking joke.
LikeLike
June 26, 2012 at 7:09 am
If I take the responsibility to review a paper, then I am partly responsible for the work being in the scientific record. So, if I wouldn’t allow it to be published under my own name, as a reviewer I don’t intend to allow it to be published under someone else’s. If authors don’t respond to my critique (with either modifications to the paper or sound reasons why they didn’t modify), I see no point in continuing the charade, and I advise the editor to reject.
That said, I’m certainly inclined to give authors some slack for not fully addressing the more minor criticisms. Those criticisms that, if not addressed, would cause a reader at least as astute as me to question the results — those criticisms absolutely need to be dealt with.
LikeLike
June 26, 2012 at 8:50 am
Sometimes it takes the third or fourth time a reviewer reads the paper for s/he to put their finger on exactly what is wrong with it. I defend the right of a reviewer to become more informed and find new faults in a resubmitted manuscript.
As a rotating grad student, I helped my PI with revisions and reanalysis of a manuscript that was submitted without the original researchers understanding the unexpected weirdness they had seen in their results. Admittedly, the conclusions of the paper didn’t rely on understanding the weirdness, but it was TOO WEIRD (and against what they expected, given their methods section) to be ignored. It wasn’t until the 2nd round of revision that one of the reviewers felt confident enough to say “Something is seriously wrong with your methods” and caused the PI to reconsider. I was assigned the case, and along with one of the original researchers, we figured out the problem, explained it in the methods and everything was fine — paper accepted and published. The PI had decided there was just no rational explanation for it, and wasn’t about to devote any further man-hours to understanding it. If the reviewers hadn’t insisted, on the second round of reviews, a paper would have gone into the literature without as much coherence. It solved the problem of multiple independent readers having to waste time trying to understand and derive the solution to the problem.
LikeLike
September 12, 2012 at 4:52 am
I agree with Jim Thomerson that “the authors have the right to reply to any criticism of the reviewers”
I found that some reviewers did not carefully read the manuscript. Some of them also asked to do more experiments on the thing that it was not necessary to do or even that thing already discussed in the manuscript. A good reviewer should listen to the reason of authors when they tried to respond the reason why they cannot do as reviewers suggest.
I play 2 roles as an author and a reviewer and I always open my mind to listen the reason of the authors for the second round of the review.
LikeLike