Paranoid Perceptions of Peer Perfidy
October 7, 2009
A letter to the blog asked us a question which expresses a paranoia about the peer review process that, while typical, is absolutely corrosive to the practice of science. I’m not casting any stones here, I’ve had suspicions about the motivations of certain reviewers of my grants on at least one occasion myself. The letter reads:
I am submitting an NIH proposal, and the most appropriate study section has a (perceived) conflict of interest: one study section member is at the same institution/department with my only competition. Am I inevitably putting my application in the hands of my competitor’s colleague, who could share it with my competition and steal my ideas before I have a chance to work on them? I could argue to go to a slightly different study section, but I do not think it would be as appropriate. What should I do? Does it involve Jameson?
(Slightly) Paranoid,
A Junior Scientist
PhysioProf had the following comment:
This would not be considered a “conflict of interest” under any fair reading of the NIH rules. If your competitor’s colleague shares your proposal with your competitor, that would be a gross breach of NIH rules. Bottom line: suck it up and hope the reviewer is not morally bankrupt.
I concur. And I hate to break it to you, this is the tepid version. Sometimes it will be the reviewer herself who you think is in scientific conflict because she is the one you think of as your “competitor”.
While we should not be naive there is a point at which suspicion and paranoia over the potential scooping or stealing actions of other scientists make the whole enterprise break down. Agreed that idea stealing will also really screw up science but I argue that the number of suspicions run several orders of magnitude more plentiful than actual nefarious acts.
I return to some of my usual points.
-You do not know who actually reviewed your grant (or paper), nor do you know who was pulling for it and who was crapping on it. My direct experiences suggest that the usual paranoid mind might be very surprised indeed.
-Are you a scummy, unethical idea-stealing scientist? Are your closest scientific homies? No? Then what makes you think those other folks are?
-Ideas are plentiful in science- your peers are a bunch of really, really smart and highly trained individuals who are interested in the exact same things that you are. They read the same literature and can deploy the same model systems. Are you positive you are the only one who could possibly have had this “idea”?
I’ll close on the specific strategy proposed by the letter writer. Should you try a study section which has less expertise but doesn’t contain a scientific competitor? I’d say no way in hell. I think you absolutely must have at least one advocate that really understands your sub-(sub,sub,sub)field and / or the importance of your specific ideas and models. You need someone knowledgeable to convince the other members of the panel of how great your project really is. If you have people with peripheral expertise, they aren’t going to care as much. They aren’t going to understand your goals as well, won’t be able to articulate your significance and are going to have other proposals that they find more familiar and of personal interest.
I tend to think that if you don’t have at least one Bunny Hopper reviewing your bunny hopping grant you are sunk. If you are in the Bunny Hopper study section, at least then you have a chance. A chance that your reviewer will be one of the overwhelming majority of ethical reviewers rather than one of the small minority of moral wasteland reviewers.
DrugMonkey 
October 7, 2009 at 4:18 pm
Right on! We need to quit with the conspiracy theories.
LikeLike
October 7, 2009 at 4:24 pm
Although the description sounds like a good paranoia case study…
In your cover letter to CSR, you can explicitly ask the grant not be reviewed by a list of individual people. You need only minimal justification and as long as you don’t cut out half the field or the only qualified reviewer, these requests are nearly always honored (or so several POs have told me). Request the study section you want and specify not this person. As long as this person isn’t actually assigned to your grant (i.e. forced to read it), it should all be good.
LikeLike
October 7, 2009 at 4:50 pm
So, here’s a somewhat related ethical dilemma I currently have:
I recently got a manuscript that badly needed to be rejected. Very badly. Horrible stuff. Nothing even remotely salvageable in it. Results were tainted from the start.
So I wrote a detailed explanation of the fundamental errors and recommended that the editor reject it.
Then, a day later, I was thinking “You know, this is an interesting question that they were asking. Looking at how A and B go together is certainly important and not obvious to a lot of people. Too bad that they did it all wrong. Hmm, if I looked at the interaction of A and B in this other context, with a different technique, and did it right, and threw in a bunch of other stuff….yeah, that would be some badass science.”
I don’t think I’m stealing anything or betraying the confidentiality of the document. In fact, they posted it on ArXiv.org (where a lot of theoretical physics papers go while under review) in addition to sending it to a journal for consideration, so the science I was thinking about was all public knowledge. Still, I probably would not have thought very deeply about it if I hadn’t been asked to review it and assume some confidential responsibilities.
I don’t want to walk away from this problem, because I’d already been kind of thinking about it anyway, but reading that paper and finding the flaws really stirred the pot for me. I guess I could contact them and say “Hi, read the manuscript on ArXiv, interesting ideas, but needs some fixes, want to collaborate?” except that I don’t really want to collaborate with anybody that sloppy. And I should emphasize that what I have in mind is still pretty different from what they did.
But, I know that a lot of this wouldn’t have come together in my mind if I hadn’t done that review. I know I’m not on the wrong side of the line, but being on the right side of the line isn’t enough. I also want to be very, very, very far from the line so I don’t ever wander into dangerous territory.
Any thoughts?
LikeLike
October 7, 2009 at 5:28 pm
I certainly agree with you DM that the lifting of good ideas is rare but there might be a semi-quantitative way to search for the sort of copying that everyone is worried about. eTBlast is a tool for finding similarity in the text of scientific journals. They made a splash with an article in Nature describing how they searched for (and found) instances of plagiarism in published articles. Someone could do a retrospective study and see if there has any direct plagiarism of sentences between a Grant Reviewer’s publications and the grants that they have previously reviewed.
I think there might be some rare cases but I would be extremely surprised if it was as widespread as some conspiracy theorists believe.
LikeLike
October 7, 2009 at 6:53 pm
“You do not know who actually reviewed your grant (or paper), nor do you know who was pulling for it and who was crapping on it. My direct experiences suggest that the usual paranoid mind might be very surprised indeed.”
I’ve heard the same thing from an independent source (at least, I think they are independent!).
LikeLike
October 7, 2009 at 6:57 pm
@Alex (comment #3); I don;t think there is anything unethocal in that at all. Two reasons: 1) The science was posted to a public domain server. 2) This is part of being a scientist! You get exposed to new ideas and new ways of looking at the issue at hand. To blatantly go and steal something from someone while rejecting their paper/grant is morally reprehensible. But to be inspired by the usually insipid process of peer-review is one of the perks of the job!
LikeLike
October 7, 2009 at 7:22 pm
It is a great question Alex. I’ve gone so far as to hold off one round on a grant submission to let one I reviewed get through the process before mine hit the deck. I’ve also fought extra hard on reviews of projects similar to stuff I was getting close to proposing- guilty or preparing the field of combat for my attack? Or was I just really charged about the ideas I had and stoked that someone else was going that direction too? It gets tricky…
LikeLike
October 7, 2009 at 9:23 pm
The concern of our correspondent doesn’t have jack fucking shit to do with plagiarism of text.
LikeLike
October 7, 2009 at 9:53 pm
Of course, there’s always the distinct possibility that Reviewer X and Reviewer’s Colleague Y, while they work in adjacent labs at the same institution, have hated each other ever since the Great Centrifuge Repair Bill Incident of 1983, and would rather shoot their own mothers than share any kind of information.
Or, just assume that most people are ethical enough to not engage in a form of academic misconduct as clear-cut as helping steal research ideas from a grant application.
LikeLike
October 7, 2009 at 10:04 pm
Or, just assume that most people are ethical enough to not engage in a form of academic misconduct as clear-cut as helping steal research ideas from a grant application.
Yeah, or from a paper they reviewed and trashed. No one would ever do that.
LikeLike
October 7, 2009 at 10:47 pm
Hmmmm. I must know more corrupt scientists than anyone else will admit.
LikeLike
October 7, 2009 at 11:07 pm
(same Jim as above; I am Junior Scientist) I’ve convinced myself that going to the other study section is a good possibility. 2/5 (or so) of my grant is more related to the study section I want to avoid, while 3/5 (or so) of my grant is more related to the other study section. Both study sections have experts in my field.
LikeLike
October 7, 2009 at 11:25 pm
DM,
Your point about postponing something is more or less what I’m thinking. I’ve been thinking about this topic but haven’t done anything on it for a while. I have plenty of other things on my plate to do for a while. Holding off on working on this, while seeing what this group does, seems to be the best approach. If they do something that I think is worth collaborating on I can say (with 100% honesty) that I saw the paper they posted in an open-access preprint server. If they do nothing further with it, but they put the seed out there, I can cite their (admittedly mistake-ridden) preprint as an example of what I’m thinking of.
whimple-
I didn’t steal anything from them. I was already thinking about this problem, A and B had already entered my thinking, and while I wouldn’t combine A and B the way that they did, thinking about where they went wrong did plant seeds in my mind. They put it in an open forum, so it is fair game. But because I encountered their work through a non-open process I am holding off for a while. Maybe they’ll see this through to completion in the next round, and maybe my comments will help them do so. If so, great. If not, well, they put it out in an open forum, and at some point I’ll move on what I saw in an open forum. I’ll just hold off for a while to give them the space to respond to my comments, since my task was to critique them, not to compete with them.
LikeLike
October 8, 2009 at 12:43 am
Are you a scummy, unethical idea-stealing scientist? Are your closest scientific homies? No? Then what makes you think those other folks are?
How’s this?
1. Because almost every single scientist I’ve ever worked with (not my homies by any definition of the word) has either taken credit for my work when they contributed nothing or deliberately sabotaged and/or unfairly attacked my attempts at publication.
2. Because I’ve gotten unfair, unprofessional reviews on the last two papers I tried to publish.
3. Because almost all the grants I was actually eligible to apply for that provided any kind of feedback came back with non-scientific reasons why I should not get them. These “reasons” had to do with scientists hating on other scientists and/or research institutions. They had nothing to do with what I was proposing, whether I was qualified, or whether the quality of what I had written.
I’m with Jim. There are a LOT of corrupt bunny hoppers out there.
Having said that, Jim, I think that you don’t necessarily know who is friendly with whom, even if they are colleagues. They might hate each other! In which case, they would love to see you funded and maybe even recruit you to replace your scumbag competitor. You never know.
@Alex, you probably should have declined to review if you were really already thinking about this problem. I’m just saying. You’re awfully close to the line. It’s nice to try to revise history to make yourself feel better, but that’s just rationalizing bad behavior after the fact.
LikeLike
October 8, 2009 at 8:41 am
It is worth considering that even though it is not scientific misconduct to make use of information presented publicly–either at a conference or on a pre-print server–if you are constantly running around with your nose up other people’s buttholes and shitting out derivative copy-cat work of your own, you are gonna develop a bad reputation as a derivative, non-creative scientist. There are a few people like this in one of my fields, and everyone knows it.
While these people may have reaped some short-term benefits when they first started pulling this shit, they are now reaping their just deserts: getting fucked in the ass during manuscript and grant peer review.
LikeLike
October 8, 2009 at 10:24 am
msphd,
In the future, I think I’ll have to be a lot more careful about what I agree to review.
The chronology is this:
For some time: I’m thinking about how to combine ideas A and B with C, D, and E to answer a certain question.
A month or so ago: I see a preprint that combines ideas A and B with X, Y, and Z to answer a different question. I skim the paper and see that most of the analysis focuses on X, Y, and Z. I put it in my “Interesting but not urgent, read in detail eventually” pile.
Shortly thereafter: Review request. I read the abstract again, skim again, don’t see them as direct competition, so I agree to review.
Review: On close examination, I find lots of serious mistakes, things that don’t require a lot of advanced knowledge to recognize. I reject with a clean conscience.
The day after: I’m thinking about A and B again, and lamenting how awful the attempt was. I start thinking about what the right approach would be. I get ideas.
In a narrow sense, I don’t see any appropriation of their ideas. I don’t even see my stuff as derivative, because I’m answering a different question and using a different technique. In that sense, I think I’m clear.
But I can’t deny that the pot got stirred because of my participation in this process. If the pot had gotten stirred while reading a good preprint or sitting in a conference talk, I could cite the paper or conference abstract as an inspiration, up their citation count, and put my stuff out there with a clear conscience, for my peers to either find it cool or find it derivative. But because the pot got stirred as a result of the review process, I am in more troubling territory.
This is the first time that reviewing has really gotten me thinking. Generally I find it more of a chore that I do to be a good citizen, sometimes an interesting chore, sometimes a boring one, but I do it without getting much stimulation from it. This is the first time I found it stimulating. If I had realized going in that I would find it this stimulating, I would have declined.
The key is what I do in the future. My resolutions for the future are:
1) Be careful to decline to review anything that might get my ideas going. This isn’t always easy because sometimes connections and inspirations are indirect.
2) Stay away from this idea for a while.
LikeLike
October 8, 2009 at 11:25 am
Alex, why don’t you contact the authors and share your ideas with them? That’s the efficient way to move the science forward. There will probably be plenty of credit to go around, particularly if it’s a good idea.
LikeLike
October 8, 2009 at 3:02 pm
“Be careful to decline to review anything that might get my ideas going.”
Heh? There’s nothing wrong with that, so long as this doesn’t also stimulate a nefarious design on your part to sit on their paper while you hurry your own experiments out. On the one hand, if it’s a paper with good ideas and good data, then the fact that it stimulates your own thinking shouldn’t really pose much of an immediate threat to the authors; it’s fair to say they have an adequate head start, and if they don’t exploit that, tough shit. Science is competitive.
OTOH, if the paper is full of good ideas executed with terrible experimental design, then the fact that it stimulates ideas in your head about how to do it better is not a moral quandary for you, but a lesson in following good scientific procedure for them. Seriously, if they put their best ideas out there backed up with nowt but half-baked results, they can hardly cry for sympathy if somebody else jacks the theme of the research and executes it properly.
LikeLike
October 8, 2009 at 3:11 pm
Given your description Alex, I think you are erring way on the side of over-concern. Not every stimulating connection or idea that leads to cool stuff is citable. Random comment at the microphone after a research presentation? You gonna cite that? or stay away from something you subsequently thought of because there is no way to cite the comment? hell no.
obviously you are the one with all the details and have to make the call from your own comfort level but no reason to over think it…
LikeLike
October 9, 2009 at 6:46 pm
Now, I’ve never witnessed or even heard of a situation like that described in comment 14, but that doesn’t make Aëtogate any less real… and there we’re talking about people who owned an entire subsubfield.
If you don’t collaborate with them, they’ll probably never learn it.
Your decision whether it’s worth teaching them.
The last conference I was to, the abstracts were only embargoed till the time of their presentation, but all the rest is top secret unless you have permission from the authors to talk about it.
No. American science funding is competitive.
LikeLike
October 9, 2009 at 6:48 pm
“It” being “how not to be sloppy”.
LikeLike
October 9, 2009 at 9:56 pm
Marjanović has it precisely correct. Science is supposed to be cooperative. You should be good enough to come up with ideas on your own and be able to realize those ideas AND help those around you come up with and realize their own ideas too. You should be trying to help these authors get their new concept boat to float, rather than waiting for it to sink so you can steal the design in good conscience.
LikeLike
October 9, 2009 at 10:50 pm
I’m paranoid. An idea I had that I was advised against proposing is now the foundation of a young colleague of mine’s promising research program.
We can all talk till we’re blue in the face about how “science doesn’t happen like that” but if it’s smart enough and easy enough your idea sure can be stolen. The problem is that truly elegant science often IS easy. I don’t want to be paranoid and I now “ideas are cheap” etc. etc. I still believe that if it’s big enough and easy enough you had better get enough prelim data to convince the slimeballs out there that you have too big of a lead on them.
I do agree with the earlier points that you have to have an advocate in the room when your grant comes up for funding. Life is messy, no?
LikeLike
October 10, 2009 at 8:38 am
But you don’t need anyone’s permission to *use* the information yourself.
LikeLike