The NIH has shifted from being an investor in research to a consumer of research
September 21, 2016
WOW. This comment from dsks absolutely nails it to the wall.
The NIH is supposed to be taking on a major component of the risk in scientific research by playing the role of investor; instead, it seems to operates more as a consumer, treating projects like products to be purchased only when complete and deemed sufficiently impactful. In addition to implicitly encouraging investigators to flout rules like that above, this shifts most of the risk onto the shoulders of investigator, who must use her existing funds to spin the roulette wheel and hope that the projects her lab is engaged in will be both successful and yield interesting answers. If she strikes it lucky, there’s a chances of recouping the cost from the NIH. However, if the project is unsuccessful, or successful but produces one of the many not-so-pizzazz-wow answers, the PI’s investment is lost, and at a potentially considerable cost to her career if she’s a new investigator.
Of course one might lessen the charge slightly by observing that it is really the University that is somehow investing in the exploratory work that may eventually become of interest to the buyer. Whether the University then shifts the risk onto the lowly PI is a huge concern, but not inevitable. They could continue to provide seed money, salary, etc to a professor who does not manage to write a funded grant application.
Nevertheless, this is absolutely the right way to look at the ever growing obligation for highly specific Preliminary Data to support any successful grant application. Also the way to look at a study section culture that is motivated in large part by perceived “riskiness” (which underlies a large part of the failure to reward untried investigators from unknown Universities compared with established PIs from coastal elite institutions).
NIH isn’t investing in risky science. It is purchasing science once it looks like most of the real risk has been avoided.
I have never seen this so clearly, so thanks to dsks for expressing it.
DrugMonkey 
September 21, 2016 at 12:45 pm
Have previously commented on “unfunded mandates” from grant review. This is exactly that. In severe funding environment, this is inevitable, unless specific mechs are reserved that explicitly allow risk. They have always had a hard time consistently reviewing this type of mech.
LikeLike
September 21, 2016 at 12:57 pm
I also think dsks hit the nail on the head. And I note that it was not always this way. You used to be able to PROPOSE interesting work, rather than have it more than half done. This is progressive mission creep on the part of CSR- and it is really bad for the scientific enterprise in a million different ways.
I will also note that it is very dangerous to essentially mandate that a specific hypothesis is likely true (using massive preliminary data). This forces people to simplify complex ideas and to bias their findings. You used to be able to say, if I find this, then I conclude that; on the other hand, if I find the opposite, then I will conclude this- and *both kinds of results were okay*. Now you seem to have to solidify a specific hypothesis with specific results all kinds of ways prior to submitting.
No risk that way.
LikeLike
September 21, 2016 at 1:44 pm
“I will also note that it is very dangerous to essentially mandate that a specific hypothesis is likely true (using massive preliminary data). This forces people to simplify complex ideas and to bias their findings.”
Like I said, the Titanic is headed for the iceberg.
LikeLike
September 21, 2016 at 2:22 pm
To note how far we’ve fallen, I was reading some history of science stuff. After WWII, through the 1960’s the US government was primarily concerned about having a stable supply of scientists.
The government would award large block grants to universities (through military organizations like the Office of Naval Research). The only requirement for these grants was to advance knowledge, and do it with students. They were investing in a highly trained population. They thought it was essential for our national security to have highly trained people available.
This model of science funding came crashing down in 1969 with the Mansfield Amendment. Mostly due to opposition of the Vietnam War, Senator Mike Mansfield tacked an amendment onto the military authorization bill that forbid the Dept of Defense from spending money on research unless it had a direct relationship to the military.
The point is that funding models can and do change over decade timescales, and over an entire career one can expect significant and drastic changes in funding models.
LikeLike
September 21, 2016 at 5:13 pm
There is little to no stability in todays science (partly a reflection of the increase in general employment precariousness) but while an individual might be able to hop around, unless there’s a safety net provided by the institution, it can be a fast downward trajectory. This is surely why researchers are applying for more grants. It used to be having one or two was fine because if things went reasonably well, your chance of renewal was well above 50%. Now, every grant submission spins the wheel so layering grants reduces the risk of losing it all. This drives up grant applications, which drives down success rates. All of this and no one feels confident.
And we wonder why research has become more superficial and short term?
LikeLike
September 21, 2016 at 6:03 pm
“Whether the University then shifts the risk onto the lowly PI is a huge concern, but not inevitable. They could continue to provide seed money, salary, etc to a professor who does not manage to write a funded grant application.”
To which University (or other institutions) are you referring? Lemme know so I can get a job there!
But, yes, yes, and more yes to dsks. I am tired of the panic and self doubt associated with playing roulette with my funds. I am also tired of hearing “but, but it might not work – you haven’t convinced me this is absolutely going to cure cancer (or whatever)”.
REAL science is discovery coupled to clear ideas of how *different* outcomes will support different mechanisms or hypotheses. Good scientists – scientists that should be funded – are ones that are good at designing experiments to give the best next step forward on a mechanism or hypotheses. Building collective understanding. Testing hypotheses with uncertain outcome. A hypothesis, by definition, has not been tested yet. Isn’t that what we’re supposed to be doing?! It’s not what my grant reviewers want
#BringBackRealScience
LikeLike
September 21, 2016 at 6:48 pm
AAP, I agree, and that’s probably how most of the labs function. The problem is that study sections (not necessarily the NIH) want to see that you have already tested your hypothesis and know the outcomes. Based on these outcomes one would generate new hypotheses to move things forward and would be tempted to propose them in a grant. I suspect that the new exciting and untested hypothesis would kill a grant that otherwise would be funded with the pretty much proven hypothesis that you are leaving behind.
LikeLike
September 22, 2016 at 1:17 am
If enough of you on a study section felt this way you could get together and change all that.
But you don’t…sounds like we are projecting our own risk aversion onto “nih”
LikeLike
September 22, 2016 at 2:19 am
Is it possible that the increased demands on preliminary data and pre-proven hypotheses are also because of the push for scientists to also be entrepreneurs? As people exaggerate claims, reviewers get more suspicious.
LikeLike
September 22, 2016 at 4:25 am
Is it possible that the increased demands on preliminary data and pre-proven hypotheses are also because of the push for scientists to also be entrepreneurs? As people exaggerate claims, reviewers get more suspicious.
Ironically, start up investment seems to be one of the few fields of endeavour left where investment in risky ideas, with the expectation that most of your portfolio will fail, is accepted.
LikeLike
September 22, 2016 at 8:33 am
Well, the NIH is not a hedge- or angel-fund. It’s a gubmint agency gambling with taxpayer dollars. Don’t you think there’s an obligation for them to only make “safe” bets? If you want funding for risky avant-garde hypotheses, look elsewhere than gubmint (or pitch your ideas to a branch of gubmint that cares less about risk, such as DoD).
LikeLike
September 22, 2016 at 9:07 am
“Don’t you think there’s an obligation for them to only make “safe” bets?”
But that is the whole scam, there are not many “safe” bets in science!
Then it would be more honest to just not fund basic research anymore.
This is a discussion we can have, but to pretend that we can significantly advance our understanding of nature without investing in many ideas that probably don’t work out as proposed is just dishonest or ignorant.
LikeLike
September 22, 2016 at 9:35 am
Grumpy- Who do you think I am talking to?
LikeLike
September 22, 2016 at 10:24 am
Arguably, the libertarian angle is that if a topic is compellingly practically applicable (i.e. medically relevant) and also a short-term near sure bet, the gubmint has no business in funding it anyway.
Though I have my issues with that philosophy, the *long term investment value* of a particular avenue seems key to deciding how federal funds are spent. How to institutionalize that value for NIH study sections is a whole other kettle of fish.
LikeLike
September 22, 2016 at 12:07 pm
Ola – This is exactly backwards. The whole point of government investment in science is that it is risky and the return is far in the future. If these bets were easy and the return near-term, then industry would invest in it. They don’t because some other company will come in and eat their lunch. (The classic example to compare is Xerox Parc.) But these investments are absolutely critical. Every technology that has improved our lives (whether it be medicine, economic, etc.) is due to science done on the edge of risk about 30 years previously. Whether it be digital cameras, the internet, or the medical breakthroughs that mean my friend with cancer is still alive years later, it was highly risky government investment that made the difference.
We need government investment to take the risks that industry can’t.
LikeLike
September 22, 2016 at 12:13 pm
@Grumpy: “If enough of you on a study section felt this way you could get together and change all that.”
So here’s the thing. Let’s say I’m reviewing a grant that proposes a very exciting hypothesis, but with scant preliminary data. I’m confident enough in the PI’s track record to believe that she can pull these experiments off. I give the grant a good score, but my colleagues on the study section point out the “obvious flaw” — too little preliminary data to be assured of success.
How am I supposed to convince them that it’s not our job to “assure success”? Yes, if everyone listened to you, DM and me, then this would be less of a problem. But also note that if you’re evaluating grants on how exciting the hypothesis is and the ability of the PI to test it, and explicitly discounting the details of the approach’s feasibility, you’re moving towards a people-not-projects system (because how else are you supposed to evaluate a risky project except by the PI’s track record of successful risky projects?) The easiest way to accomplish that would be by NIH fiat, not by scolding grant reviewers to allow more risky projects through.
Course change, not flower arranging.
LikeLike
September 22, 2016 at 12:51 pm
Grumble – the study section problem is deeper than this. It’s more like the class and grit problem I’ve brought up before.
What everyone has to realize is that the reason great grants with no preliminary data don’t get funded is not because crappy grants with preliminary data are getting funded, it’s because great grants with preliminary data are getting funded. The problem at study section is that you are going to have to argue that a great grant with no preliminary data is a *better* bet than a great grant with lots of preliminary data. (Hint: it’s not.)
The only way to solve this is to improve the funding line that both great grants with preliminary data AND great grants with no preliminary data can be funded.
LikeLike
September 22, 2016 at 1:56 pm
I think nih’s risk transfer was partially fueled by soft money and the transfer of risk to the nih and individual investigator. In the olden days, the investigator who missed a grant cycle could at least be employed and have another shot. And with hard tenure, the university couldn’t easily replace him (yes, in the olden days he was usually a him), and so had more of an incentive to provide support through the downturn.
Everybody has shifted risk to the lower guy in the hierarchy.
LikeLike
September 22, 2016 at 2:16 pm
@Grumble- “I give the grant a good score, but my colleagues on the study section point out the “obvious flaw” — too little preliminary data to be assured of success.”
Isn’t this where the “alternate approaches and hypotheses” come into play? Could you not argue that the applicant has a good backup plan and clearly spells out what she’ll do with the money if the experiments don’t work?
LikeLike
September 22, 2016 at 3:04 pm
My problem is still with the hypocrisy in the timing. I had a great and long-standing grant submitted with terrific preliminary data (the data in the upcoming paper). It scored in the high teens – not fundable. Meanwhile the postdoc finished and wanted to leave with a paper. I published the paper, so I could not propose these experiments, but without any continued funding for new personnel, I now cannot generate new “preliminary” data for resubmission(s). (BTW productivity counted not a bit in this initial review as it was my most productive cycle ever. Projects, not people!).
Grumble, I think it was you that said you were submitting a new grant with 5 years of data from your grad student=their soon-to-be-submitted paper. What if your student needs this paper published to fill a graduation requirement (and find a postdoc), but your grant needs to be resubmitted- maybe even more than once? If you publish without these “preliminary” data, you risk not getting paid for the work you already did. If you hold onto the paper, your student does not graduate.
Either way, someone loses big-time. Time for a revolution.
LikeLike
September 22, 2016 at 3:22 pm
Baltogirl, I feel you!! God help you if your paper comes out describing results for Aim 4C subaim 5J! Because then nothing in your grant is novel or exciting any more! This has happened to me twice, where papers actually got accepted quicker than expected and study section didn’t like that certain results were already published. Now I’m just not submitting any papers until I get a grant. If students need a paper to graduate, too bad, they have to wait.
LikeLike
September 22, 2016 at 6:48 pm
@Newbie(ish) PI – published, or not published, when you’re an untested PI, they can get you either way. Even when prelim data was present, I had two study sections declare that the lack of a pub from the prelim data suggested an inability to “seal the deal”. Previously strong publication record – irrelevant! Now I just publish freely because I refuse to have my papers held hostage for a system that can reject my proposals for 1,000 different reasons.
LikeLike
September 22, 2016 at 6:49 pm
Grumble- you could evaluate on Significance and Approach.
qaz- your point still values preliminary data. I say you should shed that. Free your mind.
LikeLike
September 22, 2016 at 6:53 pm
I sense lots of conflicting advice re: the advantages of providing preliminary data, and whether it’s helpful or harmful if preliminary data is already published….
LikeLike
September 22, 2016 at 8:25 pm
It’s not conflicting, it depends on the scenario. Published work great for feasibility. Great for methods. Great for general support. But if it is highly specific to proposed experiments, obvious problem. Why pay for what has already been done?
LikeLike
September 23, 2016 at 1:26 pm
“What if your student needs this paper published to fill a graduation requirement (and find a postdoc), but your grant needs to be resubmitted- maybe even more than once? If you publish without these “preliminary” data, you risk not getting paid for the work you already did.”
Yeah then I’m screwed. Except that the grant is not just to cover the experiments in his paper, it’s for further experiments that build on that work, too. So I’ll have to scramble to get preliminary data for another aim, then resubmit.
(By the way, those 5 years of grad student work have most certainly been paid for. If I fail to get a grant based on those results, it would not be a failure to get paid, but a failure to fully capitalize on an investment of time and effort into the student and his project.)
LikeLike
September 23, 2016 at 1:27 pm
“you could evaluate on Significance and Approach”
Lack of preliminary data usually results in a low Approach score.
LikeLike
September 23, 2016 at 1:34 pm
@eeke: “Could you not argue that the applicant [with no prelim data but otherwise great grant] has a good backup plan and clearly spells out what she’ll do with the money if the experiments don’t work?”
Yes, but as qaz points out, the other reviewers will have a Favorite Grant in mind that has everything my Favorite Grant has AND solid preliminary data – so the other reviewers will see no reason to improve their scores for mine. You can argue until you are blue in the face and still rarely convince anyone.
As for qaz’s point that more money would fix this problem – of course it would. But what annoys me so much about the NIH is their nearly complete failure to adapt to the new paradigm of high applicant to $ ratio. They continue to pretend that everything is A-OK and that the quality of science is not affected by the way their system is dysfunctional when there isn’t enough money to go around.
LikeLike
September 23, 2016 at 2:27 pm
Grumble – It’s an interesting question of whether the quality of science has suffered or merely the quality of scientist’s lives. The NIH money is going to quality projects, it’s just that we as individuals are depending on winning the lottery and hoping our investment will pay off. I’m reminded of a local restaurant that doesn’t take reservations. It’s always full and there is often a multi-hour line. As individuals, waiting in that line sucks; it would be much better to be able to make a reservation. But the restaurant would be less full if it took reservations. Yes, it would be much better for us as scientists to have a system that didn’t depend on preliminary data, but would it be better from the funding agency’s point of view?
LikeLike
September 23, 2016 at 2:28 pm
BTW, when I review grants, I want preliminary data proving feasibility, but not necessarily answers. My favorite grants are questions where either answer is interesting.
LikeLike
September 23, 2016 at 4:37 pm
Does NIH policy affect the quality of science? I’ll repost what baltogirl said:
“I will also note that it is very dangerous to essentially mandate that a specific hypothesis is likely true (using massive preliminary data). This forces people to simplify complex ideas and to bias their findings.”
A related issue is that if you play the grant game well despite doing crappy science, you still get funded.
LikeLike
September 24, 2016 at 12:48 am
Asking as a total newb, but what evidence is out there that the amount of preliminary data required has grown?
LikeLike
September 24, 2016 at 2:11 pm
“Now I’m just not submitting any papers until I get a grant. If students need a paper to graduate, too bad, they have to wait.”
I can’t tell if you meant this to be serious or at least somewhat funny/sarcastic, but if the former, I’m curious about why the costs of this solution don’t outweigh the benefits. Making someone wait to graduate keeps them in a job where they are making almost no money when they could potentially be in a job that pays reasonable money. It keeps them in a position where they are likely not able to save for retirement, and they can’t recoup those lost savings, both because there are yearly limits on how much you can contribute to retirement accounts and because retirement money is (at least theoretically) supposed to grow with every passing year. This solution also potentially keeps the grad student on the market for longer, if there was a job that would have hired them in the year they were waiting to graduate, but that job or a similar job are not there anymore when you are ready to let the student apply for positions. Forcing someone to pay these costs seems … unkind.
LikeLike
September 25, 2016 at 10:57 pm
Couldn’t NIH just essentially say “No preliminary data allowed except to prove researchers are capable of this kind of work, and publications demonstrating said expertise are PREFERRED over mere preliminary data since that saves time and space in the grant?”
It should put a stop to the “sitting on ‘preliminary’ data else I be punished for publishing it” stupidity that some of you are describing, no…?
LikeLike
September 27, 2016 at 9:56 am
Thanks for the shout out. I’ve been following the conversation intently, but haven’t had much to add, especially in the area of substantive solutions. Qaz nails the essential problem that reviewers have little rational reason to choose a risk over certainty.
Now, I had been managing to attain a certain zen and philosophical attitude to all this when I honestly believed that it was simply a matter of insufficient funds to go around. But DM’s recent post about the way the doubling was parceled out, and Berg’s bit about NIH folk balking at giving money to those bums coming in with scores in the 30s and 40s… that sticks in the craw a bit. There does seem to be a bit of artificial scarcity in how the NIH manages its funds, imho. And we have to wonder to what degree future budget increases will be used to ease the bottleneck in traditional investigator initiated mechanisms versus acquiescing to lobbying and cronyism with additional handouts to super-wow boondoggles. Boondoggles that always seem to have such plastic and downwardly evolving criteria for “success” (“We’re gonna find the keys to disease, man!” to “We facilitated significant technological developments that will allow future scientists to find the keys to disease, man!”).
LikeLike
September 27, 2016 at 10:56 am
Whenever I need money from NIH I just start blathering about “Reactive Oxygen Species.”
Always works.
LikeLike
September 27, 2016 at 12:14 pm
Happy Balloon – I guess I was being somewhat sarcastic about holding up my students from graduating. My students have publications already, and they are not being held back by my current refusal to publish. They might graduate and not look as good on paper as they could if they had another paper, but I think they need to learn about the realities of a career in science, i.e., that there needs to be some strategy involved in the timing of when to submit papers. It’s not only the question of having preliminary data published or not, but also if you wait to publish until you have the grant, then those papers will count as productivity for the renewal application.
LikeLike
September 27, 2016 at 4:11 pm
I think the point about the transfer of risk is well taken. It used to be, for example, that you could propose a mouse model (as a smallish part of a larger grant). Now you have to HAVE the mouse and to show that it actually does what you expected. If it does not, the PI eats the costs. And the time and the money involved to get these (possible) preliminary data-! Not every experiment is successful- should we receive funding only for those that are? who will do the risky, exciting stuff then?
With regard to several different outcomes of a given hypothesis being acceptable, qaz- that would be the place where a reviewer would say, hey, you don’t actually know what is going on here. You could be wandering in the desert forever. Ambiguity. Risk.
I agree that the problem is not all due to not enough money in the pot. Yes, there are too few dollars generally- but there needs to be more in the R01 pot, not these giant $$ initiatives designed to cure something or make rich grantees richer with technology development awards. You can’t legislate cures.
And basic science is still important, even though it receives only lip service these days (I have had reviewers tell me my molecule seems to be a poor drug target, so why study it?)
Alright. I’ll stop here. Oct 5 submission.
LikeLike