Interesting slides from the CSR on NIH grant review

Comrade PhysioProf alerted me to a couple of powerpoint presentations (converted to pdf in the links) of possible interest to the NIH grant geeks in the audience.

First one seems to be from Toni Scarpa to the CSR Advisory Council on May 2.
Lots of interesting data on each slide but I’ll pick out a few things I noticed.

-Slide 18, the number of PIs submitting grants and the number of applications per PI over the past decade. I’ve always thought the NIH paid too much attention to per-application success rate and not enough to per-PI success rate. Nice to see. 1.3 grants per PI to 1.6 Research Project grants per PI submitted each year is the range. Shows how skewed some of our experiences are but fits with the data on the Rockey blog about how most PIs only carry 1 or 2 grants.

-Slides 21,22 show a faster relative increase in the number of R21s submitted over the past 10 years compared with R01s. I’m sure we all know the reasons but interesting to see.

-Slide 23 (and Slide 5) testifies to Scarpa’s crusade to get more grants reviewed with fewer reviewers. I happen to disagree with this (I saw a little bit of this trend during my term of service) but no doubt the cost savings are tremendous.

-Slide 25 continues the cost-savings theme. In particular it is interesting to think about the savings associated with having more online, electronic reviews versus decreasing the number of actual reviewers. I’m not a big fan of the online, asynchronous review but then I’m not a fan of losing specific expertise either. These are not easy tradeoffs to make, clearly. Slide 41 seems to indicate the cost per application might be cut to a fifth by using online instead of face-to-face review mechanisms.

-Slide 33 has an interesting note about 1.7 million views, presumably for the study section description he was using as an example? That would be an interesting way to track the relative interest/load/etc for a given study section.

-Slide 35 shows trends of turning the A1 back in at the very next study section round but it would be a lot more useful as a percentage of all A1s that were put in…

-Slide 40 testifying on how reviewers like the online review methods is nice but is sure as hell needs a lot more context. Should be broken down by those who have served (or are willing to serve) on regular study section panels versus those that are not. And to delve into the reasons for being happy with online formats, etc. Not to mention do some additional queries on how the reviewers feel the outcome is for applications. For example, there are definitely times when I would say “no way, no how” to a study section that required me to visit Bethesda but might take on a phone or online review duty. That gets my expertise (such as it is) engaged where it otherwise would not have been involved. However, I also think applicants are not well served by the online review, all else equal.

-Slide 27 reiterates the accusation that it is a common complaint that there are not enough senior/experience reviewers. I’d still like to see some expansion on who is making this complaint, on what basis and how it is verified in fact. In contrast, the complaints about speed, burden on reviewers and favoring predictable research over innovation seem a lot more based on things that can be quantified and reasonably well described.

-Slide 44 continues this theme because of the heading “Recruiting the best reviewers” on a slide which reports the number of Full, Associate and Assistant rank reviewers over 98-08. You can just see the start of the great Scarpa purge of Asst Profs ( I do wonder why this slide is not updated to 2010). Again, no apparent explanation as to their justification for conflating “best” with seniority. Slide 45 has ways in which they have been convincing more reviewers to serve but again is pretty light on showing where this means they get more of the “best” reviewers. Somehow I feel confident Scarpa didn’t really expand on this in his presentation….

Slide 54- I really like the percentage of New Investigators tracked since 1962! w00t! all of their trends should go back that far.

Okay, that’s enough for you to chew over for now….