Lake Wobegon effect in NIH grant review

All the Investigators are strong….and the Environments are above-average.
The “Investigator” and “Environment” criteria have been an explicit part of NIH grant review since forever, and have been given approximately equal weight with Approach, Significance and Innovation.
The blurbs in the official NIH notice on the current scheme read:

Investigator(s). Are the PD/PIs, collaborators, and other researchers well suited to the project? If Early Stage Investigators or New Investigators, do they have appropriate experience and training? If established, have they demonstrated an ongoing record of accomplishments that have advanced their field(s)? If the project is collaborative or multi-PD/PI, do the investigators have complementary and integrated expertise; are their leadership approach, governance and organizational structure appropriate for the project?
…
Environment. Will the scientific environment in which the work will be done contribute to the probability of success? Are the institutional support, equipment and other physical resources available to the investigators adequate for the project proposed? Will the project benefit from unique features of the scientific environment, subject populations, or collaborative arrangements?

I always had the distinct impression these were essentially throwaway criteria because they were almost always rated very highly. Sometimes the “Investigator” criterion would be a place to cap on the more-junior career status or lack of productivity but for the most part it was treated very politely.
Sally Rockey has recently posted the verification of this impression on the OER blog.

source

the data presented represents 54,727 research grant applications submitted for funding in fiscal year 2010. Of these, 32,546 applications were discussed and received final overall impact scores.

So for those grants that survive triage and are discussed, almost all of them are given a 3 or better for Investigator and all are given a 3 or better for Environment. Not a lot of range there. Remember the study section is supposed to be clearly telling the Program Staff why the proposal is strong or weak on these different criteria. Telling them “yeah, good” for these two allegedly major criteria doesn’t seem to be that helpful.
Do note, I’m not one that thinks this is incorrect. For the most part, Investigators are well qualified. And the Environments are generally supportive of the work. If there is a serious problem with these…well, it is not atypical that the rest of the application has even more serious problems. So perhaps all the moderately questionable PIs and Environments are linked to really, really bad proposals and are thus triaged? (Hmm, I’d like to see these for the triaged grants, the alternate hypothesis is that even for the lesser proposals the Investigator and Environment scores still see little variation?)
The analysis of correlations between criterion scores and the voted overall-impact scores is the main point of the post, however. In this case the analysis conducted for all research grants reviewed for FY2010 funding echos the results of the prior analysis of criterion and overall impact scores published by Director Berg for his NIGMS applications in FY2010. Approach is still king.
Small differences emerge, however. The Rockey dataset shows that the correlation of Innovation and Significance criterion scores with overall impact scores are 0.62 and 0.69 respectively. In the Berg/NIGMS dataset it was 0.59 and 0.70. Same relationship, but converging.
This is why I’m interested in both differences between institutes and changes over time. Also, between different grant award mechanisms. The NIGMS post makes it clear they are analyzing scores for 654 R01 applications that were discussed and received a voted score. Rockey’s post just says 32,546 discussed “research grant applications” so this could be a big mixture of mechanisms.
But recall that the big push in the years leading up to these review outcomes was for Innovation. Reviewers are supposed to be prioritizing this criterion above “Approach”. Prioritizing “Significance” as well. So the really interesting question is whether this relationship budges in the next several FY’s of grant reviewing. From all the bleating from the direction of NIH/CSR about how we “should” be prioritizing grant proposals, it would be nothing less than a major failure if the “Approach” criterion does not quickly fall behind “Innovation” and, especially, “Significance” in terms of correlating with overall impact score. Right?