How to Fix the NIH
July 9, 2007
Readers will note I do a fair amount of talkin’ about the ways the NIH-funded research game might be fixed. The NIH has recently posted a request for information on they way they do things. I can’t overemphasize how important it is that you comment. Each and every one of you. Whether you are “a reviewer, applicant, or member of the public,” as the RFI puts it. If you are “only” a US taxpayer, well you are funding this stuff and will likely be a consumer of the medical advances that result. If you are nonUS, well, you just may be a consumer of medical advances resulting from NIH activities too. At best, this RFI is a straight game and you can have an influence on changing NIH funded science for the better. At worst, this is a rigged deal in which they are looking for confirmation but the only way to combat this is with overwhelming numbers! Either way, you should get involved. Above all, go to the NIH site and make your comments by August 17!
Writedit tagged me and then floated a set of comments. YHN took an initial shot based on some ideas that were motivated by something else originally. I’ll be working on my more-specific version for the actual NIH comment.
I’ll pass along writedits’ meme and call out a few blogs to participate. Ground rules are
I. Include the links to the NIH RFI and the comments page.
II. Post the following RFI queries
- Challenges of NIH System of Research Support
Please describe any specific challenges presented by NIH’s support of biomedical and behavioral research such as the current array of grant mechanisms, number of grants awarded per investigator, and the duration of grants.- Challenges of NIH Peer Review Process
Please describe any specific challenges presented by the current peer review process at NIH.- Solutions to Challenges
Please concisely describe specific approaches or concepts that would address any of the above challenges, even if it involves a radical change to the current approach.- Core Values of NIH Peer Review Process
Please describe the core values of NIH peer review that must be maintained or enhanced.- Peer Review Criteria and Scoring
Are the appropriate criteria (http://grants.nih.gov/grants/guide/notice-files/NOT-OD-05-002.html) and scoring procedures (http://cms.csr.nih.gov/NR/rdonlyres/B2CFE17E-AA1C-46E5-BADB-FDBF2FBBEE80/11892/CSRScoringProcedure090706.pdf) being used by NIH to evaluate applications during peer review? If not, are there changes in either that you would recommend?- Career Pathways
Is the current peer review process for investigators at specific stages in their career appropriate? If not, what changes would you recommend?
III. Address at least one of the NIH queries in a post.
IV. Reach out and tag 7 someones, be they biomedical scientists, physical scientists, taxpayers or medical consumers.
I’m tagging:
- thomas at Hope for Pandora
- orac at Respectful Insolence
- bill at Open Reading Frame
- Dr. Shellie
- bdf at Aequanimitas
- YoungFemaleScientist
- razib at GeneExpression
Also, if you are a non-blogging reader (Physioprof I’m lookin’ at you!) consider yourself tagged to forward the RFI to 7 colleagues via email.
—
Update: Related posts from Hope for Pandora, Respectful Insolence, YoungFemaleScientist, The Daily Transcript, Pimm and Retrospectacle so far. Also a pickup from Genome Technology Online. Update 2: found another one here, a site which directs to a fascinating letter to Cell entitled “American Idol and NIH Grant Review“.
DrugMonkey 
July 9, 2007 at 12:42 pm
Oooh – My first game of tag. Thanks!
I will get to this tonight or tomorrow morning.
LikeLike
July 9, 2007 at 1:22 pm
Sweet. I’ll get on this asap.
LikeLike
July 9, 2007 at 2:17 pm
To be honest, I think the NIH system is designed about as well as possible, and doesn’t need “fixing”. What does need fixing is either giving the NIH enough funds to function properly, or scaling back on the overall size of the US biomedical research enterprise.
LikeLike
July 9, 2007 at 2:18 pm
I forgot to say, that I will suggest to as many of my colleagues as I can that they should respond to the RFI if they think they have something to add.
LikeLike
July 9, 2007 at 2:27 pm
Physioprof, to reply to your first comment, I’d submit to you that if this is what you believe than you should supply this comment to the NIH. It really smells these days as though they are bound and determined to make some distinct changes. Particularly in the peer-review process going by the things Zerhouni and Scarpa talk about. If you (and other readers) like it as is, you should tell them that!
LikeLike
July 9, 2007 at 7:06 pm
“It really smells these days as though they are bound and determined to make some distinct changes.”
I think they are bound and determined to talk a bunch of shit to take the heat off themselves, since a lot of people are blaming NIH for their troubles. Those that are having troubles should be blaming the Congress and executive, not the NIH.
LikeLike
July 9, 2007 at 7:11 pm
I feel oddly unclean. What the hell is a meme, and how did I tag you with one? Clearly I don’t spend enough time in the blogosphere. But – anything that gets more folks responding to this RFI cannot be bad, so more power to you, Drugmonkey – and have fun with this, all.
LikeLike
July 9, 2007 at 8:03 pm
Physioprof: The thing is that I can point to some direct changes resulting from prior or ongoing schmack talking from the powers that be. Oct06 round there was a directive to drop asst profs from panels, coming reasonably soon after Lenard’s letter to science complaining about asst profs. this one I disagree with. The talk about young/new investigators produced the KangaR00. A GoodThing and a major shift in paradigm for NIH if you ask me. The pilot of fast turn around of summary statements, ditto. there’s a move afoot (pilot program) to shift to online and conference reviewing, forgoing the study section meeting-this one bad, IMO. they did a little two-round data collection on preliminary scores versus outcome in an attempt to get data to support the initiative- if it supported the pre-determined goal we’ll hear about it, if not, it’ll get buried (my prediction anyway). the powers that be are not just talking, i think they have a series of agendas. we might as well get in our two cents.
writedit: i know you weren’t intending to meme the blogosphere, i’m just twitting you. it is a good idea, nevertheless, and you did start it by calling me out!
LikeLike
July 10, 2007 at 6:55 am
“The talk about young/new investigators produced the KangaR00. A GoodThing and a major shift in paradigm for NIH if you ask me.”
So far, the numbers of these things being awarded is so trivially small as to be irrelevant. Accordingly, the competition for them is so keen that I am sure that the only people getting them would be thriving even in their absence.
LikeLike
July 10, 2007 at 7:19 am
Well, not “irrelevant” if you have one! I know two awardees (both women!) already. But yes, I agree that there are not enough (yet) and indeed one of my GoBig suggestions is to drop all F32s and just go straight to the K99/R00 mechanism. The “/” part, meaning “get a job somewhere” keeps this reasonable. And yes, the awardees are probably those who are “in” already, those that are competitive for jobs and for K01s and the like. Sure. The next part is to expand the use of this mechanism beyond the favored few. And most important of all is the principle at stake, that finally NIH is getting real about career development. The prior system was like the old cartoon “and then a miracle occurs” as the bubble between “fellowship” and “research award”.
LikeLike
July 10, 2007 at 7:59 am
Hey, I think your RSS feed is borked. (Am updating blogroll.)
LikeLike
July 10, 2007 at 8:16 am
“But yes, I agree that there are not enough (yet) and indeed one of my GoBig suggestions is to drop all F32s and just go straight to the K99/R00 mechanism.”
Well, that is just robbing Peter to pay Paul, where Peter is early-stage post-docs and Paul is late-stage. I think it all comes down to money. If there were enough money in the system to support the current scale of the biomedical research enterprise, everything would be fine. Fiddling around with funding mechanisms (and peer review) is just rearranging the deck chairs on the Titanic. (Thank you for indulging my multiple egregious metaphors.)
LikeLike
July 10, 2007 at 8:17 am
hmm, never looked into the whole RSS feed feature of wordpress. i’ll take a look at it.
LikeLike
July 10, 2007 at 9:18 am
To be honest, I think the NIH system is designed about as well as possible, and doesn’t need “fixing”. What does need fixing is…
…IMHO, the systematic overproduction of PhDs and postdocs. Until the current pyramid scheme is replaced with a career ladder, the grant process is always going to serve as the limiting factor to continued growth. I’d disagree with the claim that there’s no room for improvement, but ultimately, as PhysioProf says, it’s not the real problem.
Notes:
1) I’m new here and have no idea if you’ve discussed this issue before.
2) I did drop a note to the NIH about this after Zerhouni’s “The NIH budget was doubled, the number of applicants doubled and now we’re back where we started. This is completely inexplicable, so you’d better double our funding again!” article in Science.
LikeLike
July 10, 2007 at 9:28 am
Physioprof, the eligibility for K99/R00 is no more than five years out from Ph.D. on funding so we’re not really talking early/late career. Assuming at least one revision will be necessary, submitting the first proposal by the end of year 3 is the way to go. Since a lot of F32s are going to revision now, it is much rarer that a postdoc can round up NRSA funding by the start of the second year so again, most NRSAs are “late” postdocs! Apart from this current reality, I think there is nothing at all wrong with rolling back the number of years eventual faculty spend in postdoc appointments. Let’s get the good ones out, employed and competing for funding within 3 years, say I.
With respect to “just give us the proper budget”, well, sure I agree that our public dollars are better spent on the NIH than elsewhere. But we also have to face the fact that Congress is not likely to heed the scientists on this anytime soon. So why not use this time of pressure and hand wringing to advance some agenda? After all, when times are good, nobody wants to change the system either. Take the current worry about young investigators. The doubling worsened the situation because most of the gains went to expanding the operations of established investigators, permitted universities to shirk their responsibilities in paying salaries and some infrastructure costs, etc.
LikeLike
July 10, 2007 at 10:13 am
JSinger, the pyramid structure of science is not at all different from many industries in my view. not too far down this road is critique of capitalism and evoking marxist theory, putting me way out of my element frankly.
However, yes, a big lab is like a company in that the head gets disproportional credit and the worker bees get little. It is a debate as to whether there is an exploitative bait and switch. Are technician salaries our benchmark, at least for grad students? Well, grad students get paid less, generally, bought off with the promise of a career. For some fraction this is a scam or, at least, a willing gamble. Post-doctoral workers are a trickier case because of the intellectual input more similar to the PI level. Middle management gets paid less than the CEO, I fear, even if the CEO spends his time on the golf course. PIs flit around the country and world and spend the rest of the time on the phone-does this really contribute as much as the scientists back home to the eventual product? likely not.
what’s the solution, assuming this is a problem? returning to one PI and one trainee laboring side by side at the bench? not a big fan myself, but then I don’t have any good criteria for how big is too big and how exploitative is too exploitative.
as a caveat, I trained mostly in very small labs and never really felt exploited scientifically, intellectually or whatever. I was grumpy about the uncertain career prospects like anyone. I had some significant setbacks, sure. But I was never really in one of these giant depersonalized labs so YMMV.
LikeLike
July 10, 2007 at 3:19 pm
[…] 10th, 2007 Instead of only addressing the core values of the peer review system (that must be retained or enhanced), as requested in the […]
LikeLike
July 11, 2007 at 4:51 pm
Hi drugmonkey,
Thanks for the tag. I don’t know much about NIH funding. But maybe I’ll pick up on the general theme of going after grants and write a post soon.
LikeLike
July 12, 2007 at 11:35 am
“I don’t know much about NIH funding”
For any readers who feel that they are not sufficiently “up” on NIH funding issues…please, please do not let this dissuade you from offering opinion, should you have it. At the very least you are a consumer of the fruits of the NIH mission, if in the US a taxpayer funding it. And didn’t you read “The rats of NIMH” at least once??? 🙂
The NIH request for information means you too…
And for grad students / postdocs and recently transitioned faculty, well, it is YOUR voice that is most missing from the process. Is it any wonder your grievances are so poorly addressed?
LikeLike
August 24, 2007 at 12:11 pm
The NIH is the primary funding agency for biomedical research in the United States and has funded outstanding work by many scientists who contributed to our health and economy. Since health care is the largest category of the GDP, it seems wise to for the US to see that the NIH is provided with proper funding. For us to lobby Congress for increased funding, we need to have the data in hand showing that increased funding increases productivity. Since the NIH budget doubled in a short period of time (1999-2003), there should be a corresponding jump in productivity, and the simplest measure of that productivity is the number of publications. While the number of biomedical publications from US labs increased from 1999-2004, so did the number of publications from labs outside the US where the research budget did not double. Is this lack of correlation between funding and the publication rate an accident of choosing the wrong keyword? “RNA”, “DNA”, “disease”, etc produced similar results.
Apparently the increase in NIH funding had no effect on the rate of publishing new biomedical data.
The lack of correlation between the publication rate and the budget is further supported by the fact that after 2003, when the budget flattened, the publication rate showed no inflection, positive or negative.
This analysis is not inherently insensitive, since over the same period of time there was an enormous increase in publications originating from China. The trend to publish more articles over time seems to reflect the activity of the journal publishers rather than the science, since the number of indexed journals increased from 4500 to 6500 between 1994 and 2006.
Perhaps publication rate is not a good index of productivity and papers now contain twice the information as they did in 1996. That is hard to judge, but analysis of the literature suggest otherwise [1-7]. Discussion of the apparent lack of correlation between funding and productivity with NIH representatives led to the suggestion that there is a significant latency between funding and productivity. If this latency is on the order of ten years, then it seems inappropriate to fund grants for only three.
The best data on US research quality is available at the NIH where the scientific quality of grant applications is judged to three significant figures more than 50,000 times a year. I encourage the NIH to make that data available.
The lack of correlation between funding and publication rate does not suggest that the NIH is funding bad science, but that NIH is not spending its money efficiently. We need to encourage more students to enter science as a career, if for no other reason than young faculty must publish to survive. While senior scientists may do the most competent and voluminous work, revolutionary ideas tend to come from the young [8]. Currently there is little incentive for American students to enter a scientific career. The average age of an applicant to obtain their first independent NIH grant is 43, and there is currently > 85% chance of rejection for each application. According to the director of NIH, doubling of the NIH budget led to a paradoxical decrease in the grant funding rate [8]. The difficulty of getting funded is particularly frustrating for young faculty in research universities where tenure is decided primarily by grant funding.
What did happen to all the extra money that flowed into the NIH [8]? Was it used for clinical trials [3,9]? Was it absorbed by inflation? [8]. Wherever the funds went, they left no clear scientific record.
Reference List
1. Ioannidis J.P.A. (2005). Why most published research findings are false. Plos Medicine 2: 696-701.
2. Ioannidis J.P.A. (2005). Contradictions in highly cited clinical research – Reply. Jama-Journal of the American Medical Association 294: 2696.
3. Marks A.R. (2006). Rescuing the NIH before it is too late. J Clin Invest 116: 844.
4. Butler D. and Hogan J. (2007). Modellers seek reason for low retraction rates. Nature 447: 236-237.
5. [Anon] (2006). Correction or retraction? Nature 444: 123-124.
6. Abbott A. and Schwarz J. (2002). Dubious data remain in print two years after misconduct inquiry. Nature 418: 113.
7. Lawrence P.A. (2007). The mismeasurement of science. Current Biology 17: R583-R585.
8. Zerhouni E.A. (2006). Research funding – NIH in the post-doubling era: Realities and strategies. Science 314: 1088-1090.
9. Marks A.R. (2006). Rescuing the NIH: the response. J Clin Invest 116: 1460-1461.
LikeLike
September 7, 2007 at 8:09 am
[…] The long version including links to other blogging is here. […]
LikeLike
November 30, 2007 at 12:07 pm
[…] Young Female Scientist reminded me of the suggestion in a Nature editorial that this summer’s RFI from the NIH to solicit input on the peer review and grant funding process drummed up about 2,000 […]
LikeLike
August 19, 2011 at 1:56 am
[…] encourage you in the strongest possible terms to comment and let your opinion be heard. The last time there was a similar “how are we doing” type of RFI from the NIH that I got blogwood […]
LikeLike