Mutually Assured Destruction theory of NIH Grant Review

January 9, 2013

Back in the distant past, younguns, the US was involved in a struggle with the Soviet Union that many felt was an existential threat to our continuation on this planet. Among other features, this Cold War (perhaps better termed Ongoing Proxy War) featured the buildup of ecosphere destroying megaweapon bombs.

The fuzzy blankie we used to keep from going insane was the thought that since both sides could destroy huge amounts of the other side’s population, render much of its territory uninhabitable, and could do so should the other side move first, we were safe.

Since we were mutually assured to destroy each other, the logic of starting some serious beef was an insane one. Nobody in their right mind would actually do such a thing. So this kept certain behaviors (like the hilariously NewSpeak “pre-emptive counter-strike” with nuclear weapons) off the table.

In discussions of NIH Grant review, there is often a certain paranoia voiced that members of the review panel use this position of tremendous power to screw over their scientific rivals. Sounds plausible, does it not? After all, this grant stuff is a zero-sum game and the “peers” of peer review are after the same pool of money that each applicant is eying. These days it is a good bet that the reviewer has her own application under review elsewhere in the CSR…or has one pending funding in this self-same Fiscal Year.

That’s before we get to scientific competition to publish papers in some research area first. We all know that first is best and all others might as well go home, right? And any rational grant funding agency (don’t laugh) like the NIH should diversify their portfolio such that if they fund grant on a topic, the chances of another one on nearly the same topic should be lesser.

Naturally, the closer the reviewer expertise is to the grant in question, the closer this reviewer is to being in direct conflict of interest at some level.

My first approach to comforting the distraught Assistant Professor is to emphasize that our peers are professionals, with some degree of ethical centeredness who are for the most part attempting to do the job as asked.

This doesn’t comfort everyone. So today I offer the Mutually Assured Destruction theory for your consideration.

One of the most surprising things I found about study section service is the rapidity and surety with which payback opportunity was provided. During the early days of my study section service it was the appearance of many grants in my piles to review that were submitted by PIs who had previously appeared on study section panels reviewing my own proposals. After I’d been reviewing for a little bit, it was remarkable how quickly people who’s grants had appeared in study sections that I was on (and in some cases apps to which I had been assigned) were now in a position on panels reviewing other grants of mine.

I came away from all of this with the understanding that what goes around comes around VERY quickly in NIH grant review.

So for the paranoid types…do consider this additional source of pressure on the reviewer. If you don’t trust their professionalism, trust in their self-interest. This Mutual Assurance tends to suggest that reviewers would be crazy to screw with applicants out of pure self-interested bias.

Advertisements

No Responses Yet to “Mutually Assured Destruction theory of NIH Grant Review”

  1. kevin. Says:

    I think this same principle operates during paper peer review. You put X, Y, and Z as recommended reviewers, and while you aren’t 100% that X was Reviewer #1, the tell-tale signs are there. If they were fair to you, you will be fair to them, and everyone (generally) succeeds in getting the paper published without all the drama and back-fighting.

    Like

  2. eeke Says:

    “One of the most surprising things I found about study section service is the rapidity and surety with which payback opportunity was provided.”

    DM, you have warned repeatedly against assuming that a member of a study section (or suggested reviewer) was the douchebagge that shit-canned your grant/manuscript. “Payback” would never be appropriate under any circumstance. I don’t know who gets to serve on study section or review panels these days, but members of the field to choose from has grown much too large to get this “payback opportunity” anytime soon. Also, I’m sure a lot of the applicants disappear off the landscape once they’ve tried and failed a number of times. Mutual assurance my ass.

    Like


  3. We stand face to face in a lake of gasoline, each holding a match.

    Like

  4. zb Says:

    Yes, as pointed out, the anonymity of peer review (for grants & papers) as well as the threshold effect (noob AP’s driven out of the competition) makes the payback/MAD solution ineffective, in practice, and also in comforting noobs.

    I think people who choose to play the game have to believe in the honor of their peers and/or accept that if the system is out to get them they might as well give up (and so, by not giving up, they have to believe the system isn’t out to get them).

    I’m of the belief that systems based on honor become unreliable as resources become critically limited. Therefore, I believe in limiting the entrants by imposing greater demands on PI support from non-federal sources & reducing grad populations by preventing their support on research grants (and relying on training/independent grants, which in turn, can be controlled to limit supply).

    A theoretical alternative would be to increase resources, but I don’t see that happening in the foreseeable future, and, in any case, the “doubling” seems to have created classic overpopulation effects. So, even with greater resources, I’d probably still be advocating for limited entry in the playing field.

    Like


  5. Obviously, you should change the name of this fucken blogge to StrangeloveMonkey.

    Like

  6. drugmonkey Says:

    Obviously.

    Like

  7. Dr Becca Says:

    “How I learned to stop worrying and love peer review.”

    Like

  8. Busy Says:

    I had a very good paper receive the same single really bad review over and over in three different venues (other two reviews were fine). Since we were submitting to top tier journals a single bad review is enough to kill your paper. It eventually was submitted to a lower venue where it was accepted at once (a full year later). To this day people at conferences and scientific meetings still ask us: “how come did you submit it to such a low ranking journal? your result is better than that”.

    The point is that while we can make some educated guesses, we have no way of narrowing down to one or two people. So much for MAD.

    Like

  9. whimple Says:

    …the douchebagge that shit-canned your grant…

    They don’t have to try nearly that hard; just expressing mildly reduced enthusiasm to move that 1 to a 2, or that 2 to a 3 will do the trick nicely.

    Like

  10. zb Says:

    And if they are a direct competitor, and are feeling pressured themselves, it takes a pretty strong person to *not* be a little bit less than over the top enthusiastic.

    when the system works, I suspect it depends on people who don’t feel immediately threatened, groups of folks engaging in mutual support (otherwise known as old boys, though they don’t have to be any more), some super honorable (or naive) people, and mutually beneficial science (I.e. folks whose science is complimentary)

    A memorable feature of the last is when people using one technique support the research of those using other techniques (and not their own technique). I think all of the above exist enough that decent science still gets done and published, at least in some fields. But banking your career on it is a risk.

    Like

  11. Grumble Says:

    “I had a very good paper receive the same single really bad review over and over in three different venues (other two reviews were fine)”

    If I get the same paper from two different journals, after rejecting it once I tell the editor I’ve already seen it and so it wouldn’t be fair to the authors for me to review it again. Seems like common decency. Apparently not everyone thinks so.

    Like

  12. Bam294 Says:

    Oh Peanut,

    Let me tell you a little story of my buddy on study section who saw me tank twice at the hands of someone I was pretty sure was evil. She had been cut throat in the past and when my ‘buddy’ invited me for a talk at his institution he took ‘confidentiality’ to mean ‘tell you everything about this woman the entire field has decided to screw based other personal vendettas and general assholery’, yes, she never gave ANYONE a good score, and it was a topic of discussion at coffee breaks and whispered to friends to exclude her.

    I got the whispering too late, but he thought he’d make it up to me with his advice and tight relationship with the program officer. I guess I will figure out if this worked when my panel meets, but I know clearly he will want favors in the future for this. The question is….will I wittingly or unwittingly be influenced by these exchanges.

    Like

  13. Dave Says:

    @Grumble: yeh my boss has that policy too as he thinks it is unfair. Unfortunately seems to be a bit of an old-school mindset. I have some colleagues who will review it again and again and again, and they take a lot of enjoyment out of rejecting it every time.

    Like

  14. DrugMonkey Says:

    I would take a retread for review but indicate to Ed that I’d seen it before.

    Like

  15. DrugMonkey Says:

    Bam,
    Suspicious person that I am, I take communications like these with a grain of salt. Especially when ppl claim they were really pulling for my proposal. Maybe they were and maybe they weren’t.

    Like

  16. Dave Says:

    The converse is Mutually Assured Gratification: This time it’s your turn, next time it’s my turn. And screw everyone who is not already in our little circle jerk.

    NIH should use more ad hoc review and panels should be completely anonymous.

    Like


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: