NIGMS score data: All about the approach. Still.

July 15, 2010

NIGMS blogger (oh, and yeah, the Director) Jeremy Berg has posted a very interesting set of data on the review of grants.
Director Berg examined the scoring for the 360 R01 applications assigned to his Institute for the October 2009 Council round. This, you will recall, was the first one to use the current scoring scheme . So in some senses this should be regarded as the baseline value.
The analysis Director Berg shows in the graph is the correlation between the “Significance” score and the Overall Impact Score. If you will recall, there has been a bit of grumbling on the part of reviewers and applicants alike about the weird disconnect of the new system.

Each of the five criteria (Significance, Investigator(s), Innovation, Approach and Environment) are to receive an individual score from the ~three assigned reviewers. They are also to assign a preliminary Overall Impact Score prior to the study section meeting. This Overall Impact Score then becomes the post-discussion score at the meeting and the range defined by the assigned reviewers is generally the space in which the entire panel votes. The weird part I alluded to is that there is not supposed to be any explicit numerical connection between the criterion scores and the Overall Impact. Maddening.
Plot of significance and overall impact scores in a sample
of 360 NIGMS R01 applications reviewed during the October 2009 Council round. [source]

What Director Berg has done is to ask about the relationship between these component scores and the Overall Impact in the actual scoring. Now one thing that is unclear to me is whether he’s plotting the eventual voted Overall Impact score or the average of the individual reviewers’ pre-discussion scores. I think the latter [update: I was wrong, Director Berg confirmed via email that it was the eventual voted score.] because the graph does not show the clustering around the integer scores that was predictable and indeed turned up, e.g., in this NIAID graph from a single study section.

As anticipated, the scores are reasonably strongly correlated, with a Pearson correlation coefficient of 0.63. Similar comparisons with the other peer review criteria revealed correlation coefficients of 0.74 for approach, 0.54 for innovation, 0.49 for investigator and 0.37 for environment.

Makes you salivate for similar analyses from all the other ICs doesn’t it? And for individual study sections perhaps? It would be fascinating to see if behavior was more or less consistent across the CSR review sections, whether some prioritized innovation or others were GoodOldBoyGirl sections who prioritized the investigator. I can dream, can’t I?
Anyway the take home from this particular data set seems to be that Approach and Significance are still the most consistent predictors of Overall Impact Score.
[h/t: PhysioProf]

7 Responses to “NIGMS score data: All about the approach. Still.”

  1. juniorprof Says:

    If you look at what looks to be the median (around 3) you see that the overall score spread is massive. I would love to know if the spread around the median shows a high variance when you look at the approach score. This would jive well with all the scores I’ve seen from colleagues (and my own grants). Even if the significance score is good, a bad score on approach causes an absolute dive on overall score.


  2. Not a huge fan of this because all it really says is that getting a good significance score (say, 3 or under) is a necessary but not sufficient condition to be funded. Which shouldn’t be a huge shocker.
    There is still a huge range of impact scores for “high significance proposals”, and the one PI who nailed significance with a 1 probably won’t get funded with his priority score of 28.
    I would be particularly interested in seeing a correlation between final impact score and the average of the 5 sub-scores.


  3. Neuro-conservative Says:

    While I applaud Director Berg for his transparency with presenting these data, I agree with Candid Engineer — linear correlations are not as relevant as identifying necessary and sufficient conditions to get a final impact score in the teens. One would assume this means getting 1’s & 2’s in all domains, but the chart shows otherwise. Several “winners” scored between 2 and 3, and someone got a 5 on significance but still managed to get a final impact score of 19. What’s up with that?


  4. Namnezia Says:

    It would be nice if he had color-coded the ones that got funded. Even better use a multidimensional plot with some type of cluster analysis to see if a clear set emerges that got funded.
    I think it’s the outliers that lead to those WTF moments, as in: “I got a really high significance score but overall impact just plain sucked, while Jimmy-in-the-next lab got a decent impact score while the significance of his proposal was crap!”
    I also want to know what the grant who got a 1-1 was about!


  5. Arlenna Says:

    I recently spent some time on a study section that is all about significance and impact, mandated by its associated RFA. Even there, regardless of the significance of your work (even when very well-scored), if your approach sucked or was poorly constructed you would get a bad overall priority score.


  6. Dr. Feelgood Says:

    The problem with significance, is that reviewers have widely disparate views on its meaning. Reviewers often confuse technical innovation with significance. I have observed poorer significance scores many times when a reviewer misses the idea of, if a grant was successful, how it would push the field forward. Set shifting hypotheses are often missed for their significance if it does not include some techy whiz-bang flash. It’s really too bad..


  7. […] Council round. Based on the interest in this analysis reflected here and on other blogs, including DrugMonkey and Medical Writing, Editing & Grantsmanship , I want to provide some additional aspects of […]


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: