Innovation in NIH grants is not "hard to define"
August 10, 2010
Comrade PhysioProf recently noted another post on grant review scoring data from the NIGMS Director, Jeremy Berg. One of the comments over there from anon reviewer speculated that the Innovation criterion score is only poorly associated with the Overall Impact score because of reviewer confusion. The comment suggested reviewers struggle to define Innovation- as if they do not know what that means.
Nonsense. I replied as follows:
AR, I would suggest the “difficulty” reviewers have with the Innovation criterion is not confusion over what it really means. Rather it is *resistance* to the notion that Innovation should be more important than Approach and Significance. They just are not on board with this top-down emphasis of the NIH. So they strive to djinn up Innovation compliments for apps that are obviously lacking innovation because they like the approach and/or significance.
Right? Reviewers are not idiots. When you see some gibberish in the Innovation section of the critique written by your fellow reviewers you do not conclude they are fools. You conclude, quite rightly, that the reviewer liked an application that lacks any sign of innovation for other reasons.
The idea that NIH funded science should be all Innovation, all the time is idiocy. In the extreme. We’d never get anywhere without people doing the unglamorous work to follow up, verify, utilize, translate, generalize, extend and connect with the most innovative science.
Reviewers know this, which is precisely why the NIH obsession with innovation fails to translate to study section reviewing behavior.
__
added: I urge my readers to go over to the NIGMS Feedback Loop blog and comment. If you want to see more of these type of data out of other NIH Institutes and Centers, it seems obvious to me that a show of interest on the part of the NIH funded extramural research force (not just PIs, everyone) would be a good thing.
DrugMonkey 
August 10, 2010 at 10:48 am
“The idea that NIH funded science should be all Innovation, all the time is idiocy. In the extreme. We’d never get anywhere without people doing the unglamorous work to follow up, verify, utilize, translate, generalize, extend and connect with the most innovative science.”
What’s worse is when applications get downgraded for lack of innovation because the PI is taking logical next steps using appropriate but standard techniques to answer the question at hand. No innovation in the idea? No innovation in the techniques? That often slips your proposal below the payline, even though the experimental question may be important and informative.
LikeLike
August 10, 2010 at 11:02 am
One of my biggest concerns is what can be considered innovation. Is it acceptable to think that a new paradigm in a field, tested with established techniques, is still innovative? It seems there might be quite a bit of variety on how different reviewers perceive different types of “innovation”.
LikeLike
August 10, 2010 at 11:46 am
Of course there is variation, Dr. O. Why is this a ‘concern’?
Look, there is no instruction in how to write a grant application that will absolutely guarantee funding. It is a feature, not a bug, in the system that peer reviewers are variable, real world judges and not computer programs with a single set of well defined criteria.
Good, excellent or merely strongly contributing science comes in many guises.
Your job as a grant writer is to offer up your best analysis of why your project innovates- in the new format you have an entire section to talk about your innovation.
Perhaps the reviewers will buy your argument, perhaps not. This is not different from you having to make an argument for why your project is significant or has an excellent approach to the problem. If you find yourself getting repeatedly hammered on innovation that you either need a new study section or need to consider that you’ve fallen too far off the tail of what is acceptable.
LikeLike
August 10, 2010 at 12:05 pm
Hmm. Of course, your explanation does not exclude the other. Don’t study sections vary a good deal in how much reviewers agree about which criteria are important? If we looked at ONLY study sections where there is a general consensus that innovation is particularly important (or reviews for grants responding to RFAs that really go above and beyond in demanding innovation), would we find a tight correlation between innovation score and final score? Or would disagreements about what constitutes innovation introduce too much noise in the system for that?
LikeLike
August 10, 2010 at 1:01 pm
Without direct evidence (my experience are quite limited in the context of all NIH study sections) I would speculate that there are sections of low internal variance and those of high internal variance when it comes to innovation. Across this broad distribution, it is very likely that some sections have a central tendency that ignores innovation and some that revere it.
This is one of the many reasons I advise newer faculty to keep sending proposals to the most relevant study section (or three) so as to learn about the expectations of that study section when it comes to a proposal. Talk to colleagues that have also sent proposals reviewed in that section. Every standing study section is going to have a set of cultural tendencies that evolve only slowly. It is possible to get a read on where their biases lay. Again, no guarantee. But the applicant can nudge the odds ever toward her favor by adapting to the study section culture.
LikeLike
August 10, 2010 at 5:31 pm
One of the things that I have found interesting so far in reviewing the new format applications and having my own reviewed is that the applications that do best on innovation only have one or, at most, two paragraphs in the innovation section. And these paragraphs comprise simple declarative sentences that recite the features of the proposed studies that are innovative and what makes them innovative.
It appears that if it takes a long-winded complex argument to make the case for innovation, then either the proposed studies aren’t innovative or the applicant is incapable of making the case effectively.
LikeLike
August 10, 2010 at 6:46 pm
Wouldn’t it be interesting if that applied to more than the innovation section?
LikeLike
September 10, 2010 at 4:22 am
[…] post: Innovation in NIH grants is not "hard to define" | DrugMonkey Tagged with: absolutely-guarantee • answer-the-question • get-downgraded • […]
LikeLike
October 14, 2013 at 1:04 am
[…] I leave you with the wisdom of Drug Monkey from three years […]
LikeLike