"I just reviewed this dog of a manuscript for the Journal of…"
August 18, 2008
An interesting discussion on the ethical implications of reviewing a manuscript that one has previously reviewed for a different journal has arisen in another venue. (Hat tip: various participants; not sure which of them want credit but they know who I’m talking about.) Since you, DearReaders, have considerable paper reviewing and submitting experience I thought I’d solicit some viewpoints.
What do you do when receiving a scientific paper to review that you have already reviewed for another journal?
Do you accept it for review?
I think yes. I have done so as a reviewer and as an author, cannot come up with any consistent grounds on which I should object. My score is: no problem.
Does it matter whether you were positive or negative on the prior review?
It shouldn’t make any difference. And in fact if your do use this criterion either way, then you are introducing a bias that should be avoided. As a reviewer, re-reviewing can be a burden but if one felt strongly in the “accept” or “reject” camps, one is motivated to do so. It is the neutral / bored cases that one would be motivated to avoid reviewing again. As an author I would of course prefer the obviously demented negative reviewers did not re-review my work and that the positive ones returned! That is no grounds for a policy however: again, no problem.
Do you have an ethical obligation to inform the editor that you have already reviewed the paper for Journal of Bunny Hopping?
This seems to be a point of some contention in the early going. As far as I am aware, I haven’t seen any explicit instructions to reviewers that they must do so from any journal for which I review. I have, however, attended on-scientific-publishing sessions in which journal editors have come down strongly that this was an obligation of the reviewer. My feeling at this point is that we have no explicit and consistent field guidelines on which to rely and thus it is worth discussing how people feel about the issue.
As an author, I think that on the whole I would prefer that the editor be informed when a reviewer has seen the paper before. As a reviewer all I can say is that I do generally indicate this to the editor. I’m not entirely sure why but I think this is the most above-board and ethical thing to do. I am one that is at least a little concerned about my own biases and feel quite strongly that the most general prescription for combating the bias that is inherent in any situation in which human judgment is rendered is 1) sunlight and 2) competition of biases. The Editor is the one that is supposed to be accounting for biases when making the final decision and s/he should be aware of potential landmines. (You may recall that this is one reason I prefer Editors who are active scientists over the professional-editor variety.) Note that I do not think that we are at a point where there is an effective and general policy on this and thus there is no stench of ethical impropriety if anyone feels differently. Just because I vote in one direction at present does not mean that I’m ready to etch anything in stone.
I do not think that there is any obligation to tell the editor for which journal one has previously reviewed the manuscript, however. In fact, I think that indicating which journal may be a violation of the confidentiality of review. In a way that merely indicating that one has previously reviewed the paper is not. The reason being that the politics of impact factor and journal reputation are so fraught with issues which are irrelevant to the science at hand. Say a society level journal receives a paper which has been sent out for review (but rejected) at a CNS journal and a paper which has been rejected from another society journal of equivalent status or (gasp) a supposed dump journal? Are they going to be treated equivalently? One thinks not. Suppose an author gets a reputation for constantly submitting stuff to a GlamourMag which is marginal and then working their way down the chain- is this going to given them a bad reputation in the eyes of the editor of a society level journal? It might.
Does the nature of the critique matter?
Here we might focus on the two biggies for which a reviewer might recommend rejection of the manuscript. First because there is something scientifically flawed about it- missing controls, weak data, data not supporting the argument, etc. Second if the manuscript is fine, but not viewed as sufficiently exciting, hot, etc for the journal to which it has been submitted. As a reviewer, of course, the case in which a major flaw has not been fixed but the paper has merely been re-submitted unchanged is a BigDeal. Nothing pisses me off as a reviewer quite so much (or as a reader of a paper that finally appears unchanged in another journal). As an author, such criticism is completely off-base, biased and of course it is a fantastic outcome if one gets one’s unchanged manuscript past another set of reviewers! (Ahem.) Actually in full disclosure I have never pulled this. I always try to revise the manuscript (almost) as if I were resubmitting it for two reasons. First, I believe strongly that this process of responding to even the most idiotic reviews as best one might improves the paper- remember that the reviewers stand in proxy for your eventual reader population. Not that you have to take every idiotic suggestion, far from it. Merely that you should chew over each critique as objectively as you can and see where you might improve the manuscript. Second, I operate under the assumption that I may get the same reviewer again!
With respect to the “fine but not hot enough for this journal” critique….hard to say because I’m not a fan of that critique anyway. What I strive to do is to point out why I think a paper is exciting or not exciting but not say anything about not being hot enough for a given journal. The closest I come is criticizing a paper for being a bit too lean on data, not going far enough or something. It has to be really lean for me to say something, in most cases, I only say this when I think it shouldn’t be published anywhere. There is one journal for which I review on occasion that insists on a percentile estimate for overall scientific impact and quality- as in “top 25% of papers” or “top 10% of papers”, something like that. In this case, I do my best to make the estimate. So I suppose if I saw one of these again at a “lower” journal I might have to think a little harder on this topic.
As an author, the question really becomes a question of reviewers taking a “set” against your paper from reviewing it for a higher-reputation journal that they cannot overcome for the lower journal. That’s a tricky one. Suppose you have two reviewer mindsets: one for your society journal and one for your favorite GlamourMag. Let us further suppose that you can apply these standards appropriately under most circumstances. Will using GlamourMag standards on one manuscript bias your ability to use your society journal standards when you see it again? Could be. As an author, score me as “rather not” have a repeat review under these circumstances.
Conclusions
No rousing conclusion here from my present perspective. Obviously I have more uncertainty on this issue than certainty and feel that this is an issue that has not been hashed out to the point of consistent journal policy. Luckily I have a well-experienced and opinionated commentariat to help! Go to it, Dear Readers.
August 18, 2008 at 6:10 pm
A colleague of mine once got the same manuscript (aside from minor changes in formatting) sent to him for review from two different Journals within one week. He informed the editors and they turned it down…..
LikeLike
August 18, 2008 at 6:21 pm
Dude, you trolling for that wackaloon former editor who claimed that editors are “just like PIs, only better”? That schmuck cracked me the fuck up!
LikeLike
August 18, 2008 at 6:25 pm
To clarify, I mean a Situation where an editorial decision has been made in the first review. Simultaneous submission is a no-no.
LikeLike
August 18, 2008 at 6:36 pm
I know – my story about simultaneous submission was slightly off topic but I thought I share it anyway.
LikeLike
August 18, 2008 at 8:19 pm
About four years ago I was asked to review a paper that was crap. I rejected it. So did the other reviewer for pretty much the same reasons as I. Two weeks later I was asked to review the same paper by a second journal. The authors hadn’t made a single change, so I recycled my review. Seemed fair. The second reviewer also rejected it. I’m fairly sure it wasn’t the same second reviewer as the first journal. A month later I got the paper again. Yet another journal. At that time that particular journal would send the paper along with the email asking if you would be willing to review. I had no intention of reviewing it a third time, but was curious, so I had a read. Still no changes. None. Rightly or wrongly I informed the editor that I had reviewed this paper for two other journals, the authors had not made any revisions, and declined to review. As far as I know, that manuscript never made it to press.
While I agree that it’s perfectly reasonable to review the same paper for two successive submissions, three times is too much. Three times with absolutely no changes is bullshit.
LikeLike
August 18, 2008 at 8:46 pm
At one point I worked for Giant Publishing Company as an assistant to the editors (read: the poor sap who answers the phone when you call, flaming pissed about the reviews you just got, ready to eviscerate the editor).
Two observations –
1) I’d estimate that around 5% of the reviews coming in made mention of having seen the manuscript in some form for a previous journal. Clearly the other 95% probably contain some people who simply don’t mention the repetition. I saw no correlation between this and positive or negative review.
2) I cannot even count how many cover letters came in with the intro, “Dear Editors, We are happy to submit our manuscript ‘Analysis of Phenomena X’ for consideration in GlamourMagz #1.” I did not work for GlamourMagz #1; I worked for GlamourMagz #2. Seriously, people. Just take a cursory fucking look at your paper before you resubmit it. The editors were always amused to know they were at least the authors’ second choice.
LikeLike
August 18, 2008 at 9:53 pm
It’s also possible that they just did a bad job of proofreading their edits of a standard boilerplate cover letter. Maybe the last time they used it, it was for some other paper that went to GM#1, and this one really is a first submission to GM#2.
LikeLike
August 18, 2008 at 10:12 pm
I would love to add some witty or insightful comment. But I can’t. I pretty much agree with everything you said PP.
I think if I did see a repeat review I would probably wouldn’t disclose that I had seen the manuscript before. Not because I’m a complete bastard but because I wouldn’t want to damage the chances of the manuscript being accepted just because that particular journal was not first choice for those authors.
I would however cut and paste the previous review into the comments section and let the editor know that’s what I was doing if they had simply resubmitted the same manuscript I’d already taken the time to offer improvement suggestions at the previous journal. That’s just a bit dumb/lazy/rude to ignore completely the previous review.
LikeLike
August 18, 2008 at 11:05 pm
It’s also possible that they just did a bad job of proofreading their edits of a standard boilerplate cover letter.
I think that’s an understatement! Is there any excuse for putting the wrong journal name in your cover letter? It’s a horrendous first impression, regardless of the reason.
LikeLike
August 18, 2008 at 11:18 pm
When I re-submit, I mention in the cover letter that the paper’s been reviewed elsewhere. Correspondingly, should journals that care about re-submits *require* that authors prior submittals? That would clarify things for reviewers.
LikeLike
August 19, 2008 at 2:27 am
Identical submissions to 2nd/3rd/4th choice journals suck! I thought the whole point of having reviewer comments was to improve the paper, but apparently not.
LikeLike
August 19, 2008 at 6:19 am
Depending on the circumstance, this can be completely false. If the paper was rejected, not because it had scientific flaws, but because it was of “insufficient broad interest” to the perceived readership of a particular journal, there is absoulutely nothing whatsoever wrong with resubmitting the identical manuscript to a journal that perceives its readership as being of narrower scope.
And BTW, if you have never had a manuscript rejected for this reason, you are not aiming high enough.
LikeLike
August 19, 2008 at 8:44 am
And BTW, if you have never had a manuscript rejected for this reason, you are not aiming high enough.
************************************
Such reviews should still be rare. You should have a good sense of where your work belongs. Disagreements will occur, which is fine but if you can’t be objective enough about your work, you are just wasting your time. Experiments needed for N/S vs. a more specialist journal do differ. Writing styles are also different. Revisions take times as do additional experiments.
LikeLike
August 19, 2008 at 9:41 am
My concern regarding re-reviewing a manuscript is the case in which a jackass reviewer dominates a particular field. It turns the review process into an unfair monopoly. When a review is simply snarky and unproductive, I take issue with having the same guy/gal look at it for another journal. This happened to me once, and it is discouraging when you are just being bullied and there is no improvement to the science at hand.
LikeLike
August 19, 2008 at 10:17 am
This is totally false. Review at high-end journals is extremely capricious and non-objective, and so one’s ability to predict what will fly and what won’t is exceedingly limited. Accordingly, in order to successfully publish in those journals, you need to expose yourself to the possibility as often as possible.
LikeLike
August 19, 2008 at 12:23 pm
In the rest of the publishing world, PP’s (very good) advice is generally given as “Don’t reject your own work. That’s the editor’s job.” Editors already have a reputation as unreasonable jerks. Make them earn it.
And just so this doesn’t turn into that thread, yes, I know and love many editors and understand that it’s often just the demands of the job that are unreasonable.
LikeLike
August 19, 2008 at 12:42 pm
Review at high-end journals is not capricious and non-objective solely because of the behavior of editors. Peer reviewers for those journals also contribute substantially.
LikeLike
August 19, 2008 at 1:07 pm
True. There are also people who function as pre- or co-editors in non-journal publications, but the tendency there is to talk about “the editor,” whatever the actual review team looks like.
LikeLike
August 19, 2008 at 1:57 pm
When I am given a paper to review, my assumption is that it is not my responsibility or right to tell anyone else, including the editor of another journal, that I’ve done that.
I am usually given the option to ‘uncloak’ (and I always do) but that is still information passed back to the editorial staff, not to anyone else.
Therefore I think it is wrong (but not the end of the world) to pass that information on.
One of the reasons to not pass prior history on is that it produces an arbitrary negative bias against the author. If one really hates the paper, one may be tempted to do this, but what kind of scoundrel would do that?
There are reasons why one should probably consider not reviewing the paper the second time, as I’ve said in answer to this question in the ‘other’ venue. In fact, it may be best to assume that the second submission is a reasonable lateral shift from one venue to another. Your opinion of the validity of this is not what you are being asked for as a reviewer, and intervening or involving oneself further on such grounds would be an inappropriate torpedo, or could be seen as such.
If one feels so strongly that one’s negative opinion (that a paper should not be published) is god’s word, and needs to be reiterated whenever possible even if it involves breaking a presumed trust, then that is an entirely different problem that should be looked into.
LikeLike
August 19, 2008 at 2:03 pm
My concern regarding re-reviewing a manuscript is the case in which a jackass reviewer dominates a particular field. It turns the review process into an unfair monopoly. When a review is simply snarky and unproductive, I take issue with having the same guy/gal look at it for another journal. This happened to me once, and it is discouraging when you are just being bullied and there is no improvement to the science at hand.
I’d like to second this. There are reviewers in some small subfields who always torpedo everything, for instance. They also make some reasonable comments and useful remarks, so they get used, but their suggestion to reject is summarily ignored. (and other advice is sought from other people on that level). But that is rare. More often, you have a handful of “obvious” reviewers who end up seeing more than their share, and exerting their biases to an unfair degree. If I got the same paper from two journals, I’d worry that this was a danger and pull out of the process.
LikeLike
August 19, 2008 at 2:45 pm
Such reviews should still be rare. You should have a good sense of where your work belongs. Disagreements will occur, which is fine but if you can’t be objective enough about your work, you are just wasting your time.
The journal is not a monolithic entity with fixed, clear purpose. It’s a hodgepodge of disconnected reviewers and editors who don’t prep for each review with a heart-to-heart about the journal’s purpose and readership. Sometimes the author’s a better judge of that than the reviewer. A personal example:
As an author, I was certain that Journal X had the right readership and should be the first venue to publish a particular finding. My submittal drew 2 reviewers who thought it was high-impact work but appeared not to know that Journal X published work in that field. I quote: “I doubt this domain is a core interest for Journal X’s readership.” (Strange, that. My ms. cited prior Journal X articles in that very domain.)
One of the reviewers suggested numerous citations to what I assume were his own works. If I’ve correctly identified the reviewer, he has never published in Journal X. How do I know he’s even read Journal X?
The section editor agreed with me, and suggested a new review cycle. As the first reviews had taken 14 months, I declined and published elsewhere.
LikeLike
August 19, 2008 at 2:58 pm
My concern regarding re-reviewing a manuscript is the case in which a jackass reviewer dominates a particular field. It turns the review process into an unfair monopoly.
That’s a *big* problem in my field. The same few people review for every journal I submit to. It’s not that they reject the competition out of direct self-interest. It’s a more intellectual competition and self-interest. If you happen to draw Reviewer A you’ll struggle to get your paper out unless your conclusions fit his viewpoint.
I imagine this happens in other fields with distinct schools of thought. Reviewer A will always think Professor B’s disciples are missing the boat. If there were groups C through Z also working in the field, there would be better odds of getting an open-minded and baggage-free review.
LikeLike
August 19, 2008 at 4:22 pm
The section editor agreed with me, and suggested a new review cycle. As the first reviews had taken 14 months, I declined and published elsewhere.
Why on earth did you wait that long? No journal is worth that long a wait just for reviews.
LikeLike
August 19, 2008 at 5:52 pm
Re: #10 – At least two of the more important journals in my field *do* require authors to identify whether a submission has previously been submitted elsewhere and rejected, and one of those requires that the author submit the reviews. This seems to me a heavy burden on the author, and I don’t understand why it is required (for that particular journal). If the reviews make clear the type of revisions that were necessary (and that they were reasonable), and a reading of the submission makes clear that those revisions weren’t made (such that the paper could be rejected without forwarding to the peers), then I suppose such a policy makes some sense. Given, however, that I regularly receive manuscripts for review that shouldn’t even have made it off the author’s desk, let alone be forwarded by the editor, I find it difficult to believe that things actually go down that way. And I would wonder whether having the previous reviews in hand might bias the evaluation of the (presumably revised) manuscript.
(Disclosure – I’ve resubmitted revised, previously rejected manuscripts to alternative journals, but have always identified them as such.)
LikeLike
August 20, 2008 at 9:14 am
This is totally false. Review at high-end journals is extremely capricious and non-objective, and so one’s ability to predict what will fly and what won’t is exceedingly limited. Accordingly, in order to successfully publish in those journals, you need to expose yourself to the possibility as often as possible.
************************
Well from my experience and the advice I have received from top scientists is to have a good sense where your article belongs. These members of the Academy would disagree with you. These scientists regularly publish in C/N/S journals but also submit to mid-high to mid. level journals as well. My advisor has let grad students and post-docs do exactly what you suggest against his/her judgement. Every time the judgement of the advisor was correct. The papers were rejected based on the research not being “hot/exciting” enough for a top notch journal. In turn this meant spending more time to rewrite the paper for another journal (Or to shorten it to meet PNAS requirements). An article written for Nature or Science is not written in the same manner as one written to a specialist journal. This was a waste of time that could be better spent writing other papers, grants, making additional contacts, and doing other experiments. You can get away with aiming too high if you have the time to waste.
It depends also on how competitive your field is. Publishing first is important. You aim too high and you open yourself up to being scooped. We have scooped many labs exactly because they aimed too high. The articles still went to high journals just not the CNS level and we published first.
LikeLike