Reviewing NIH Grants: Innocent until proven guilty?

February 24, 2012

Commenter Physician Scientist notes on a prior post that an individual scientist under suspicion for several dubious papers has retained his NIH funding.

OMG

This grant was RENEWED!!!!

Not quite. Or maybe the timeline is not quite what it seems. This would appear to be the most recent competing award in 2009. The budget is listed as ending in 2010 but then it continues onward under the “7R01” code in the next year (indicating a change of University) until 2014.

So the objection may not be all that direct if the news of these alleged frauds, misdeeds and/or actual retractions and corrections hadn’t been known when the grant was reviewed.

However, davebridges continues from my more general query about whether PIs should be viewed as innocent* until proven guilty of fraud.

innocent until proven guilty, but im sure reviewers sure can take into account historical accuracy of a lab. Better to renew a grant of a good lab with an instance sloppy record keeping (if thats the case) than a non-retraction lab whose data is never reproducible

This brings up a related, and more pernicious, issue. In my limited experience, “lab whose data is never reproducible” tends to be the stuff of rumor. Word around the campfire. Suspicion. Widespread far beyond those who might actually have tried seriously to replicate said data.

Correct or false, rarely is it a matter of well-explicated, scientific lack-of-evidence. Which, in itself, would still be problematic. There are many Nobel prizes and other fantastic scientific discoveries with a back story of “nobody believes his data”. At least at first, but that could have continued for years or decades. But if it is only suspicion? even if there are a couple of retracted papers….

Should the grant reviewers bust on an application on this basis? If there was one retracted paper would you refuse to issue a fundable score, even if the application had little to do with the topic of the retracted work?

What if you’ve read some internet clown detailing the “obvious” duplications of figures in papers but they’ve yet to be retracted or corrected? Would you mark the present application from that PI downward?

The flip side is that nobody deserves NIH funding. It is a privilege that is getting rarer all the time, going by the success rates. Seemingly the proposals that make it over the bar are held to the highest standard. As we’ve noted repeatedly, there are LOTS of great applications which are not going to get funded.

So why should we (the system) tolerate even a whiff of impropriety? Why not apply the one-strike and yer out principle?

As you know, we had one major ass retraction in the substance abuse fields in recent memory. Major because of the profile and public interest rather than because it had broad influence on the other scientists. I mean sure, maybe people were trying to replicate and follow up but the retraction came out within a year. Not too much damage was done**.

As far as I can tell Ricaurte kept his grants and kept getting more of them. Never paid any obvious price. Was this right? Should he have been busted out of the business for something over which we still do not know, and will never know, the extent of culpability. Should the reviewers simply moved on to a less tainted individual?

I don’t know. All I know is what I would do as a reviewer which is to try to be as fair as possible and to rely on my fellow panel members to reach consensus over how retractions or more suspicions should be viewed.

What would you do?

__
*from what I can tell in the chatter, this Chu case is limited to suspicion and a few retractions so far?

**yes, if you were the one wasting a year of work it sucked.

Advertisements

No Responses Yet to “Reviewing NIH Grants: Innocent until proven guilty?”

  1. zb Says:

    The innocent until proven guilty is a very very high standard that’s nearly impossible for a scientific fraud case to meet. I suspect, if the fraud case stays out of the criminal domain, that it’s not the standard that’s applied (though fraudulently obtaining federal funds is a federal crime, and would, thus, be in the criminal standard).

    Since the standard is nearly impossible, though, mostly, the legal gibberydook tries to make it go away by limiting the scope (i.e. the lowest level person gets blamed) and then goes away some how (leaves the country, disappears from science, switches fields, . . . .)

    Like

  2. anon Says:

    You say you would be “as fair as possible”. I would, too, but wouldn’t this mean taking the project proposal at face value? Forget about who the author is – does preliminary data, cited references (that are not retractions, but respected papers, even by the author of the proposal), and the logic in the proposal all hold up to some presumably awesome project? As a reviewer on a study section, I would focus on the proposal and review it as I would any other proposal. It’s up to the NIH as to whether the application should be reviewed at all (by being assigned a PO, it should have already passed through some initial evaluation), or whether to continue funding a grant based on some accusations. If I am asked to evaluate whether the applicant has cheated, then I would assume I would need to evaluate whether all applicants have cheated. Just to be fair.

    Like

  3. zb Says:

    PS: I don’t know what the standard should be for blaming the person, but I do think that vast areas of science are not living up to the ideal of replicable science.

    Like

  4. Physician Scientist Says:

    For reference, receptors for intracellular bacterial DNA were generally unknown at the time of the retracted Cell paper. This is why showing it was DNA-PKcs (a well-known DNA repair enzyme which requires the DNA-end binding proteins Ku70 and Ku80 to function) was a big deal. This turned out to be bs however as real DNA receptors like TLR9 and AIM2 came to be known.

    The retracted paper is flat-out wrong and has been retracted. The grant appears to come directly from that faulty, retracted finding. I’m sorry but I’m not on board with giving a grant directly from this type of finding and I blame the study section for not recognizing this.

    Like

  5. drugmonkey Says:

    Was this known with confidence when the grant was reviewed?

    Like

  6. DJMH Says:

    Wonder if it was in the applicant’s CV. Do you think you put a retracted paper on your CV? With a special kind of asterisk? Or leave it off entirely because, duh, it was retracted? But then wouldn’t that seem skanky to people who know that you had a paper retracted?

    Like

  7. Physician Scientist Says:

    very well known. This is not a separate grant on a separate topic. This is a grant that comes directly from the faulty finding.

    Like

  8. drugmonkey Says:

    PS-

    “directly from”?

    the Chu Cell retraction was available online 5 Feb 2009. The competitive renewal was started April 2009. So yes, the NIH could possibly have known of the retraction just prior to issuing the award. The review panel most likely met in Oct-Nov of 2008 however. That would be *prior* to the retraction.

    However, the retracted paper was published in 2000 so while it may have been a major part of getting the *original* award funded it would be unusual for the renewal, five years later, to still be standing on that one paper as exclusive support.

    So then you’d have to get into whether additional work since the 2000 paper was likewise based on “the faulty finding” and was likewise fatally compromised. Was it?

    Like

  9. drugmonkey Says:

    Do you think you put a retracted paper on your CV? With a special kind of asterisk? Or leave it off entirely because, duh, it was retracted?

    Interesting. I bet people leave it off… đŸ™‚

    Like


  10. Forget about who the author is – does preliminary data, cited references (that are not retractions, but respected papers, even by the author of the proposal), and the logic in the proposal all hold up to some presumably awesome project?

    Sorry, but that is not how NIH peer review is mandated to function. You are just pulling review criteria out of your asse. The mandated NIH peer review criteria are Significance, Innovation, INVESTIGATOR, Approach, Environment. NIH peer reviewers are not allowed to “forget about who the author is”.

    Like

  11. Dave Bridges Says:

    I agree with zb here, due to the difficulty in publishing negative results, the amorphous “cant reproduce” is almost never proven. This might be a question for Retraction Watch but arent most retractions for copy-paste type easily provable errors and not for bad science (Chronic Fatigue Syndrome/XMRV notwithstanding). The larger point, is should these “cant reproduce” findings be more public and if so how?

    Like

  12. drugmonkey Says:

    It’s simple. You figure out how to publish a valuable explication indicating what is not likely to be true…or you STFU about how you don’t “trust” some lab or claim they are “never replicable”

    Like


  13. You figure out how to publish a valuable explication indicating what is not likely to be true.

    PLoS ONE is an excellent venue for subjecting such results to rigorous peer review and publishing them.

    Like

  14. slartibartfast Says:

    “PLoS ONE is an excellent venue for rigorous peer review [sic]”.

    BWAAAAA-HAHAHAHAHAHAHAAAAA!!!!!

    Like

  15. Physician Scientist Says:

    Check it out… http://www.science-fraud.org …. more evidence that this guy is shady. The study section did a real disservice in renewing this grant application.

    Like


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: