Peer Review: Opinions from our Elders

February 23, 2007

One of the most critical hurdles for the young scientist is getting a “fundable score” on a grant proposal from a study section, the first and most important link in the peer review process. If you receive a fantastic score from a study section (currently about the upper 7-10%ile), you will almost certainly get funded; receive a bad score and there is no chance of funding. Unfortunately the deck is stacked against the young investigator in many ways. One of the biggest problems, and something that can be changed, is the representation of young and transitioning scientists on panels. In short, they are not represented. The official NIH guidelines for selecting peer reviewers, indicate that one must already have been awarded a grant to serve. That means that scientists who are attempting to transition to independent work, and likely those in the first 2-3 years are unrepresented. There is also an unstated criterion that participation from those scientists who have not yet acquired tenure/associate professor rank is to be minimized. Interestingly, the official guidelines also state that “There must be diversity with respect to the geographic distribution, gender, race and ethnicity of the membership.” So why should there be an attempt for “peer review” to be explicitly unrepresentative of career status?

A 2006 commentary on NIH peer review illustrates one of the problems. The author asserts that grant review has decreased in quality of late and that this problem may be attributed to the participation of too many junior scientists on panels. This is a not uncommon perspective of older scientists. The director of the NIH Center for Scientific Review (CSR) comments in his Sept 2005 report to the Peer Review Advisory Committee that assistant professor participation was up in 2005 over 1998. Then the CSR attempted to limit assistant professor participation to 10% of reviews in the fall rounds of 2006.

Here’s the question. Where are the data? What evidence is there that assistant professors or younger investigators give consistently lower quality reviews? Especially when you take out the previous-experience-reviewing factor, which is obviously circular. Where is the evidence that the more-junior scientists that do participate on study sections have any functional impact? One reviewer out of the three tasked to review a given proposal does not have a categorical impact. If the assistant professors are limited to 10% or less, this means that on average one out of every three proposals will have one out of three reviewers of junior status. It is absurd that the NIH brays about the need for helping the next generation of scientists while ginning the system against them in the place where it most counts.

About these ads

5 Responses to “Peer Review: Opinions from our Elders”

  1. GrantSlave Says:

    Things get tough with low fund lines and people go looking for someone to blame. It couldn’t possibly be that they write bad grants, no. It couldn’t be that there is a huge pile of great grants coming in, no. Must be something wrong with the review…

    Like this

  2. Cynic Says:

    It’s a legitimate question whether reviewers of a given stage of their careers are biased in their review of grants due to established personalor scientific relations with applicants or because of certain habits of thought and technique. This applies both to young and senior investigators, but to the degree that senior investigators are disproportionately represented on review panels, it disadvantages junior investigators. Regarding the subconscious (or otherwise) influence of personal or scientific relations on review judgements, I think it’s a shame that NIH (and journals) do not attempt double-blind review (at least with respect to research plans).

    Like this

  3. PiledHigher, Deeper Says:

    In fact, investigators of lower rank tend to be more qualified, not less. It takes a longer time of “training” to get an independent appointment. It is more competitive to get a grant, too. Modern science is more complex and takes a greater range of techniques as the “elder” are only too eager to claim when explaining the justness of unending post-doctoral work. The outcome is that today’s assistant professor is a considerably more accomplished scientist than in previous generations. They have to be just to “make it”.

    Like this


  4. [...] essentially took over for the R29 in my view. You can no doubt see that this heads us back to my usual ranting about how study section behavior is biased against the New Investigator. In this case, any [...]

    Like this


  5. [...] have also ranted at some length about efforts in prior years to decrease the number of Assistant Professor rank participants on study sections. Going by the CSR data presentations [...]

    Like this


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 39 other followers

%d bloggers like this: