Peer Review: Opinions from our Elders

One of the most critical hurdles for the young scientist is getting a “fundable score” on a grant proposal from a study section, the first and most important link in the peer review process. If you receive a fantastic score from a study section (currently about the upper 7-10%ile), you will almost certainly get funded; receive a bad score and there is no chance of funding. Unfortunately the deck is stacked against the young investigator in many ways. One of the biggest problems, and something that can be changed, is the representation of young and transitioning scientists on panels. In short, they are not represented. The official NIH guidelines for selecting peer reviewers, indicate that one must already have been awarded a grant to serve. That means that scientists who are attempting to transition to independent work, and likely those in the first 2-3 years are unrepresented. There is also an unstated criterion that participation from those scientists who have not yet acquired tenure/associate professor rank is to be minimized. Interestingly, the official guidelines also state that “There must be diversity with respect to the geographic distribution, gender, race and ethnicity of the membership.” So why should there be an attempt for “peer review” to be explicitly unrepresentative of career status?

A 2006 commentary on NIH peer review illustrates one of the problems. The author asserts that grant review has decreased in quality of late and that this problem may be attributed to the participation of too many junior scientists on panels. This is a not uncommon perspective of older scientists. The director of the NIH Center for Scientific Review (CSR) comments in his Sept 2005 report to the Peer Review Advisory Committee that assistant professor participation was up in 2005 over 1998. Then the CSR attempted to limit assistant professor participation to 10% of reviews in the fall rounds of 2006.

Here’s the question. Where are the data? What evidence is there that assistant professors or younger investigators give consistently lower quality reviews? Especially when you take out the previous-experience-reviewing factor, which is obviously circular. Where is the evidence that the more-junior scientists that do participate on study sections have any functional impact? One reviewer out of the three tasked to review a given proposal does not have a categorical impact. If the assistant professors are limited to 10% or less, this means that on average one out of every three proposals will have one out of three reviewers of junior status. It is absurd that the NIH brays about the need for helping the next generation of scientists while ginning the system against them in the place where it most counts.

Wikipedia on MDMA

The Wikipedia entry on MDMA is a classic example of the limitations of this public encyclopedia approach. The “winner” in this case is the viewpoint of those with the will and spare time to take over the entry and constantly edit the content. Read over the talk page and see if this seems like a reasonable process to generate an encyclopedia.

Should scientists care? Certainly. We should not abdicate our authoritative role in the public discussion of scientific topics. MDMA and delta9-tetrahydrocannabinol may be somewhat unique in the drug-of-abuse world since the advocates are so fervent. But similar situations can be found with other public health and ecological science areas. Stem cells, autism, climate change- ring any bells? All areas in which the actual science has been obscured by political agenda, commercial agenda, (well-meaning?) lay public advocacy and the like. It is particularly annoying and insulting that in many areas the motives of scientists are questioned to advance the agenda, thereby creating unwarranted public suspicion of the scientific enterprise.

Lose public support, lose Congressional support, lose public funding of biomedical science.

Slate on MDMA Clinical Trials

The recent Slate article on the clinical trials which are trying to establish 3,4-methylenedioxymethamphetamine (MDMA for short, otherwise known as Ecstasy) as an FDA approved pharmaceutical is the usual media puff. Lately the media has been all on the side of the folks at MAPS who are convinced that MDMA is perfectly benign. What is fascinating is how intellectually similar the MAPS arguments are to those of the tobacco industry and the Bush administration on climate change. All three take advantage of the inherent uncertainty contained in the scientific process to misrepresent the available evidence. It is stone simple to argue that a given study has not “proved” a scientific point. Very few, if any, single studies can do so. So to isolate individual examples of supposed experimental flaws in a few key papers as evidence that an entire body of work is irrelevant to decide likely risks of MDMA (or smoking) to public health is intellectually dishonest. Advocates for the legalization of psychotropic drugs, whether from the personal use or clinical utility, are fans of the libertarian perspective of “let individuals make their own decisions” which is a nice principle. I’m a fan. But decisions should be made on the basis of the best possible information which, frankly, comes from the scientific community in this case.

Mozart would have been dead for 7 years…

Biomedical research scientists in the US (and worldwide) are bright, highly educated and creative folks. Most are dedicated to the public good, undergoing years of low pay while fueling the greatest research apparatus ever built- the NIH-funded behemoth that is American health science. Yet they persist in various types of employment stress and uncertainty for years, with minimal confidence of ever attaining a “real job”. It is dismaying to realize that by the time he received his first R01 (the major NIH research grant) Mozart would have been dead for 7 years (tipohat to Tom Lehrer). The official noises coming from the National Institutes of Health, and even some individual institutes such as the National Institute on Drug Abuse (scroll for comments on the young investigator) are positive, sure. We’ve heard such sentiments before, however, and most objective measures show long, uninterrupted dismal trends for the young and developing scientist.