The problem is not with review…

A News Focus in the 20 Apr issue of Science outlines some common criticisms of the NIH study section. Once past a very brief touch on the real problem (i.e., the fact that a lot of very good proposals and slim funding lines means that ranking proposals precisely near the funding line is impossible) the authors get to a common refrain:

the pool of reviewers has nearly doubled in the past few years, and “the quality is not always there,” concedes Antonio Scarpa, director of NIH’s Center for Scientific Review (CSR), which coordinates the grant review process. Moreover, many top scientists, under pressure to keep their own labs productive, decline to serve on study sections.

…

The lack of experienced scientists on review panels is a long-standing issue that’s been accentuated by the current funding squeeze, says Edward Kravitz, a neuroscientist at Harvard Medical School in Boston. He says he’s the recent victim of what he claims was a poorly informed review. Although his group has published four papers in high-profile journals since November on the genetic basis of aggressive behavior in fruit flies, this month a proposal to continue this work was “triaged”–rejected without being discussed by the full study section.

The discussion that follows a post at Respectful Insolence illustrates more “there is something wrong with review” rationale. In this case from some individuals who feel that NIH study sections have not been “getting” the brilliance of their work/theory/approach. The argument is the same in both cases, i.e., that there is something “wrong” with review. These sorts of comments make anyone who is actually serving on study section in these tight times bristle. Admittedly, there are limitations to the way that grant review is conducted. The problem is not, however, that the people doing the reviewing are somehow unable to discern all the brilliant applications. And complaints from people who are having to compete just like everyone else sound a lot like petulant whining. Look, each and every PI who submits a proposal thinks their work is worthy of funding! I’ve seen applications from subfield luminaries (20yr plus competing renewals) and from Science/Nature/high impact luminaries get hammered in the study section on which I serve. I’m sure that many if not all of them were crying up a storm about how the review was bad. But here’s the thing. The application was worse than the ones that did get funded. Period. They did not get screwed or get victimized by bad review. Nobody was “out to get them”. My response is, look, I don’t care if you’ve had your grant continued 5 times in a row. I don’t care how many papers you’ve published, nor where. We don’t award scores based on these accomplishments (although they certainly contribute) we award scores based on the proposal. You wrote a bad proposal, someone else wrote a better one and, sorry, you will be invited to “revise and resubmit”.

The reviewers are trying to get to the core of what is the best possible science to support. Unfortunately, what is the “best science” is not always easy to distinguish because just about every research program has pluses and minuses. Is this so hard to understand? Evaluation of science cannot be made quantitative. And, like it or not, reviewers review the proposal and not the CV. Sure, perhaps Dr. Kravitz did publish a bunch of great papers, no doubt in the same area as the proposal. He’s not alone. Everyone has stories like this. What he and his fellow complainers fail to realize is that the competition is immense, there is a huge pile of “good” and “great” grants coming through each and every round. They can’t all get funded…