April 30, 2007
A News Focus in the 20 Apr issue of Science outlines some common criticisms of the NIH study section. Once past a very brief touch on the real problem (i.e., the fact that a lot of very good proposals and slim funding lines means that ranking proposals precisely near the funding line is impossible) the authors get to a common refrain:
the pool of reviewers has nearly doubled in the past few years, and “the quality is not always there,” concedes Antonio Scarpa, director of NIH’s Center for Scientific Review (CSR), which coordinates the grant review process. Moreover, many top scientists, under pressure to keep their own labs productive, decline to serve on study sections.
The lack of experienced scientists on review panels is a long-standing issue that’s been accentuated by the current funding squeeze, says Edward Kravitz, a neuroscientist at Harvard Medical School in Boston. He says he’s the recent victim of what he claims was a poorly informed review. Although his group has published four papers in high-profile journals since November on the genetic basis of aggressive behavior in fruit flies, this month a proposal to continue this work was “triaged”–rejected without being discussed by the full study section.
The discussion that follows a post at Respectful Insolence illustrates more “there is something wrong with review” rationale. In this case from some individuals who feel that NIH study sections have not been “getting” the brilliance of their work/theory/approach. The argument is the same in both cases, i.e., that there is something “wrong” with review. These sorts of comments make anyone who is actually serving on study section in these tight times bristle. Admittedly, there are limitations to the way that grant review is conducted. The problem is not, however, that the people doing the reviewing are somehow unable to discern all the brilliant applications. And complaints from people who are having to compete just like everyone else sound a lot like petulant whining. Look, each and every PI who submits a proposal thinks their work is worthy of funding! I’ve seen applications from subfield luminaries (20yr plus competing renewals) and from Science/Nature/high impact luminaries get hammered in the study section on which I serve. I’m sure that many if not all of them were crying up a storm about how the review was bad. But here’s the thing. The application was worse than the ones that did get funded. Period. They did not get screwed or get victimized by bad review. Nobody was “out to get them”. My response is, look, I don’t care if you’ve had your grant continued 5 times in a row. I don’t care how many papers you’ve published, nor where. We don’t award scores based on these accomplishments (although they certainly contribute) we award scores based on the proposal. You wrote a bad proposal, someone else wrote a better one and, sorry, you will be invited to “revise and resubmit”.
The reviewers are trying to get to the core of what is the best possible science to support. Unfortunately, what is the “best science” is not always easy to distinguish because just about every research program has pluses and minuses. Is this so hard to understand? Evaluation of science cannot be made quantitative. And, like it or not, reviewers review the proposal and not the CV. Sure, perhaps Dr. Kravitz did publish a bunch of great papers, no doubt in the same area as the proposal. He’s not alone. Everyone has stories like this. What he and his fellow complainers fail to realize is that the competition is immense, there is a huge pile of “good” and “great” grants coming through each and every round. They can’t all get funded…
April 27, 2007
Time magazine is the latest to feel that recent efforts to study MDMA for possible psychotherapeutic use are worth a mention. Naturally everything has to be in the context of Timothy Leary so the baby boomers can understand. I digress. One interesting comment from the article:
Antidrug warriors may argue that the research will lend the drugs an aura of respectability, prompting a new round of recreational use
As I suggested in a previous post, the issue is not “respectability” but rather perception of risk. Particularly with MDMA in the last 5 years or so, the perception of danger associated with “Ecstasy” appears to go in close relationship to the incidence of use. Risk perception for a recreational drug is undoubtedly affected by many things including whether or not the drug is available as an approved medication. What bothers me most about media attempts to communicate science is that they botch it so badly by writing articles uncontaminated by any previous data as each new single-paper result grabs their fancy. There’s nothing like institutional memory or incremental reporting based on prior publication to be found. I suppose all they are really after is the controversy. Nevertheless it is irritating. While science may be complex in some areas, issues of drug toxicity just aren’t that complicated to communicate. Neither, in my view, is presenting the nuanced view of things that constitutes real science. This insult to the intelligence of the average Time reader has broader implications because it leads to a wide distrust of science. I have one friend (Ph.D. in the physical sciences) who delights in discounting each health recommendation that hits the popular media because as he says “Wait a year and the advice will change”. Think about the recommendations about alcohol (“No drinking!”, “Err, one a day for your heart!”, etc) and you’ll see where he’s coming from. Translate this in a broader context and you’ll start to see why “oh that’s just a theory” gets applied in a perjorative way to things that are in fact well supported scientific findings.
today we have no priestly Leary figure spewing vertiginous pro-drug proclamations. Instead we have a Leary for a less naive age: Richard Doblin. Also a Harvard guy–his Ph.D. is in public policy–Doblin founded the Multidisciplinary Association for Psychedelic Studies (MAPS) in 1986 to help scientists get funding and approval to study the drugs. … Doblin has painstakingly worked with intensely skeptical federal authorities to win necessary permissions. MAPS helped launch all four of the current Ecstasy studies, a process that took two decades. It’s the antithesis of Leary’s approach.
The notion that Doblin is a noble advocate for an orphan drug is…interesting. His opinions on the recreational use of cannabis and MDMA are readily apparent if you care to read through everything that is available at MAPS.
The connecting point, however, is this. The MDMA-advocacy position is reflexively anti-science which is amazing given they they are obsessed about analyzing the science and appear reasonably informed. No finding in animals studies can possibly apply to the human condition. Each “flaw” they find in a paper questions the whole field. Every scientist that finds anything that may be risky about MDMA is just a pawn of the right-wing anti-drug political position as far as they are concerned.
I understand that legalization advocates and popular journalists have an agenda. Don’t we all? But their way, the political/media/legal way, should be abhorant to scientists. Ferocious defense of a position whether the evidence is with or against it. Our way, the scientific way, is defense of the evidence first and our interpretation of it second. Whether the evidence supports our closely held theory is a distant third. Or at least it should be.
April 26, 2007
A recent paper by Donny and Dierker entitled “The absence of DSM-IV nicotine dependence in moderate-to-heavy daily smokers” and the recent update of the Monitoring the Future (MtF) survey of drug use and attitudes seemingly have little in common. Stick with me.
The essential point of Donny and Dierker is captured by this from the abstract:
RESULTS: Approximately 39.4% of daily smokers never reached nicotine dependence. While the probability of remaining non-dependent decreased with smoking quantity and duration since the onset of daily smoking, a substantial portion of individuals (37.7%) who reported smoking >/=10 cigarettes per day and began smoking daily >/=10 years prior, remained never nicotine dependent.
The overwhelming majority of animal research in the area is based on the self-administration and drug-discrimination paradigms- each of which focus on the animal equivalent of “how (good) does this drug make me feel right after I take it?”. On a design basis, the vast, vast majority of studies are predicated on the notion that what is important is the group result derived from the mean of the whole subject population. I’ve often wondered how can this be relevant to issues of drug dependence? The people smoking 10 cigs a day in the Donny sample are certainly getting some acute reinforcement from the nicotine yet this “drug liking” doesn’t really explain dependence. This paper shows that what is important is distinguishing individual traits or liabilities that dictate which of the population exposed to a given drug will go on to develop dependence.
This also shows what is so hard about convincing people that addictive drugs are going to suck them into a cycle of destructive use…because they are generally NOT going to do so. I.e., for the majority of people who get “acute reinforcement” (aka, pleasure) from using a given drug. (Not to mention their college chums “My friend Joe used to snort cocaine every day in college but got a job and never used again after graduating…” . Everyone has similar anecdotes.) Wouldn’t we do a lot better at convincing people to be wary of drug use if we pursued models that addressed questions related to who will become addicted rather than questions related to “how good does this drug make me feel”?
Ah, says the scientist. I don’t have to worry myself with some sort of PR mission to convince people. I’ll just publish my great papers and eventually this will solve the drug use problem. Ha. The single biggest contributor to population drug exposure is…..PR. (Loathe as I am to acknowledge this.)
From the recent MtF 2006 overview section on Ecstasy:
The figures on the facing page show little change in 12th graders’ perceived risk of ecstasy until
2001, when it jumped by eight percentage points…
…Significant increases in perceived risk occurred again in 2003 for all three grades. This very sharp rise likely explains both the deceleration and the turnaround in use, as we had predicted it would. In 2004 and 2005, perceived risk continued to increase among 12th graders, though at a much decelerated rate. The 10th grade has shown a leveling in perceived risk, while 8th grade has shown a slight decline—perhaps an early sign of generational forgetting. In 2006 the 8th graders’ perceived risk of using ecstasy dropped sharply (down seven percentage points).
Is attitude everything? Does a “scared straight” approach really work best? Examine the data in the MtF 2005 full report and you will find that, in general, attitudes toward health risks of a given recreational drug are remarkably stable, even across decades. (You will also find that drug use follows broad trends unlikely to be explained by specific events. This is a caveat to my main thesis.) There is one other notable case in which the perceived risk for a drug (e.g., for “trying it once or twice”) was initially low in comparison with most other drugs and underwent a rapid increase in the proportion of people thinking it “risky”. For cocaine between 1986 and 1987. Unsurprisingly, the shift in attitudes led to a drop from about 20% of 20-somethings endorsing annual use in the early-mid eighties to about 7-8% in the early nineties. The “why” is material. Two words, Len Bias. It also seems very likely that MSM publicity surrounding the entirety of The Ricaurte Affair explains the Ecstasy use data. I would suggest that in each case the existing scientific evidence played a minor role, that is, the totality of evidence before and after Len Bias’ death and before and after the publication/retraction of Ricaurte’s Science paper was essentially unchanged. The scientific interpretation of risk should have been essentially unchanged. What differed, was the scope of reach of the information. I would so argue anyway.
Is there a point here for drug abuse science? That depends. Do you really want to have a “translational impact”? Do you really want to serve the public health mission by helping to avoid or ameliorate the detrimental impact drugs of abuse have on the public? Then perhaps scientific paradigms that are relevant and understandable to the public should be prioritized. Stop asking which drugs feel good and how good they feel. We know this. Ask rather, what is special about the circumstances of that subset of your sample that is most likely to develop compulsive drug use…
To recap, Shelley Batts blogged on a recent scientific paper, including figures from said paper. Some annoyances from the publisher, Wiley, with respect to use of copyrighted material resulted. John Pieret links to a series of blogo-sponses on the subject.
There are a couple of points of interest to me. First, the blogger Batts did not go through the usual process to request permissions to use published material in advance. It is not clear that had this been done that she would have been refused and indeed permission may eventually be extended. Certainly whenever I have used the publishers’ procedures to request permission to re-publish figures in subsequent papers, I have received prompt permission. In essence by return mail in the pre-Internet days and within a day or two more recently. Calls for Wiley boycotts seem premature.
I had a recent experience in publishing a paper that was unusual and I realize I don’t really understand the ins and outs of copyright permissions in scientific publishing. My prior understanding was that it is basically the finished product, so to speak, that was under copyright. The exact figures / text / layouts as published. I figured that there had to be some rational thought here as well so that one couldn’t simply take a figure and reproduce it almost exactly except for cosmetic changes such as shading or graphical format. On the other hand I also thought that if one wanted to take some portion of the data and re-present it in the context of an entirely new comparison that was OK. Particularly when it is your own work, of course, but presumably OK anytime the original authors are OK with the use (yes I realize the authors have no legal position on this but science IS a community business). In my recent situation I was including the underlying data set (with appropriate citation) from a previously published study in entirely new statistical analyses and data presentation. Obviously there were new data involved. Nevertheless I received a note from the (academic) editor that I was to obtain permission from the original publisher. It was no big deal since it took about 10 minutes to submit the request and permission was granted before I finished revising the manuscript. But I don’t like the precedent one bit. It suggests an assertion of ownership over my research data to an unbelievable degree. Now, I think it likely this was a misunderstanding. The request was from an academic editor, not someone who necessarily is tied into the intricacies of the Elsevier (oh, yeah it was Elsevier) legal department. It may have been that the publisher would have never extended such a request. In the meantime I’m polling my colleagues to see if they’ve experienced anything similar…
UPDATE: Of course, once the appropriately senior person at Wiley was involved, the situation was resolved. This is not a “win”. This is a “loss” in which the blogos look like emotional nutcases willing to go ballistic before all the facts are in and/or considered rationally. That may be okay for the political ranters but surely scientific bloggers can do a bit better?
April 24, 2007
A News Focus in the 20 April issue of Science continues the drumbeat on the dismal NIH funding situation. Although depressing this is good because it is just possible that through much discussion the essential problems will be made clear to those in a position to do something useful.
A strain on the young
Nowhere does the funding gap seem wider than when looked at through the lens of age. “It’s just about inconceivable for a brand-new investigator to get an NIH grant funded on their first submission these days,” says David Sweatt, chair of the neurobiology department at the University of Alabama, Birmingham. Sweatt has hired three young scientists in the past year and worries about their future. “I see it as this dark shadow hanging over people who are just starting out their labs,” he says. “They’re having to spend so much time being anxious over funding, to the detriment of having time to think creatively about their research.”
But NIH officials are increasingly worried about mid-career scientists, those seeking to renew their first or second grant. “That’s probably where the pain is maximal,” says Zerhouni.
BINGO! Perhaps Zerhouni is really grasping that “transition to independence” takes more than one grant (the main problem with the otherwise admirable KangaR00). So here is a simple suggestion. Use the MERIT program (descriptions from NIDA, NCI, NIAID) a little differently. As currently used, the MERIT is a reward for fairly established investigators who acquire a good score on their competing continuation and get on well with Program staff. It converts a standard 5 yr noncompeting interval into a 10 yr noncompeting interval. This makes little sense because it goes to investigators who are fully capable of popping out fundable proposal after fundable proposal and who enjoy every benefit of a close relationship with Program staff. Why not extend this program to the (rare) New Investigator who manages to get a fundable score? S/he is very likely to be the genuine goods after getting through the process. So why not give them the help to succeed? Heck, they should even consider boosting this rare individuals’ budget!
April 18, 2007
The disgruntled postdoc/grad student is a familiar figure in biomedical science. A couple of examples in the blogosphere include Shelley Batts who bemoans the state of NIH funding,
I’ve come to the conclusion recently that the amount and types of grants out there for young scientists is really poor. By the time you begin to write an NRSA (which takes a few months) and go through the obligatory 2 revision periods (another year), even if you receive a fundable score, you’re funding won’t likely begin until another 6 months. By this time, 2 years has easily passed. As these grants, like all grants, rely heavily on preliminary data, many graduate students are not even in a position to apply for an NRSA until they have generated a year’s worth of data (or more). By the end of the process, assuming you are successful (in that lucky 6-9%!), you may have already graduated from school and been awarded your PhD. Does this sound like an efficient and beneficial program?
and Jake Young who wonders
The question for me is whether the institutions of large university hospitals change be modified to make it possible for people to do adequate research without fearing for their academic souls, selling their personal lives down the river, or completely ignoring the benefits of good teaching and competent clinical work. Maybe these are things are all antithetical…
They have plenty of company and in many cases for good reasons. However, as an ex-disgruntled grad student / postdoc I have a few thoughts.
Number 1: This is a good gig, no matter the bad advisor or dismal funding picture. You are in a career that let’s you do essentially what you wish to from day to day, let’s you pursue that which interests you for the most part and in most cases provides you with one of the most informal work environments possible.
Number 2: Nobody is going to hand you a life-long career so grow up, appreciate the helping hands you’ve received and go market yourself or sell that used car. The acquisition of funds to support your ability to pursue your science is part of the job so you might as well start figuring out how to work the system instead of cursing the darkness.
Number 3: Cursing the darkness has a place as long as you curse to the right people. Trainees tend to feel powerless but you can have more impact than you think. Try communicating in the “official” way- write letters to Science and Nature in response to Zerhouni / Scarpa / etc editorials. Talk to program officials (use facts, don’t whine, don’t personalize) at meetings. Write your Congressional representative!
Number 4: Think about this one long and hard. Somebody has to have that job/career/grant. Why should it not be you? What are you doing to make sure it IS you?