Collins proposes HHMI style "people, not projects" awards. Big deal.

December 12, 2013

From the description in Nature.

On 5 December, agency director Francis Collins told an advisory committee that the NIH should consider supporting more individual researchers, as opposed to research proposals as it does now — an idea inspired in part by the success of the high-stakes Pioneer awards handed out by the NIH’s Common Fund.

Pioneer awards are described as follows:

The NIH Pioneer Award initiative complements NIH’s traditional, investigator-initiated grant programs by supporting individual scientists of exceptional creativity who propose pioneering and possibly transforming approaches to addressing major biomedical or behavioral challenges that have the potential to produce an unusually high impact on a broad area of biomedical or behavioral research. To be considered pioneering, the proposed research must reflect substantially different scientific directions from those already being pursued in the investigator’s research program or elsewhere.

Another report I saw on this quoted Francis Collins as referring to “superstars”.

I’m unimpressed by this whole business. By referring to “superstars”, the HHMI approach and the Pioneers program NIH Director Collins makes it clear that he is talking about picking a very limited number of winners. At best each IC will get one? Maybe? So this will not do very much to help with the large bulk of NIH supported (and those desiring future support) investigators who feel that the job of securing grant money is taking away from their ability to do great science. This will not be some wholesale conversion of the NIH from project-based proposals to person/lab support. That’s my prediction anyway.

And as such, this reflects no real change. The primary concern of those opposed to this would be that it cordons off a part of the NIH pot in a place that they cannot try to reach it. If these selected superstars have the money based on their genius, then your project cannot be funded by those dollars.

Moving slightly down the road, the selection of superstars also means that the vast majority of us know that we have no shot at those funds in any case.

But here’s the thing that leaves me unimpressed.

This whole line of attack is nothing but a recognition that the superstars have to grub for grant money in the trenches now, but that they never had to do so in the past.

The NIH system has been a hybrid system that incorporates both project-based and people-based approaches. The latter is not formal, but it is reality. Once upon a time if you had a fairly healthy scientific pulse, you could renew your core grant (which rapidly evolved into a lab-based funding reality, no matter what was on the page every 5 years for competing renewal) for 25+ years. “I just applied for money when I needed it” said a colleague to me within the last two years. These people could also pull in additional grants for just about whatever half-decent additional project struck their fancy. In nearly all ways that count, many, many of our respective subfield luminaries (not superstars, I’m talking the top 20-30%) in the past three decades enjoyed defacto person-based funding.

Because of this, there was a pool of money the rest of the plebes, and the noobs, could not realistically access. In theory, sure. But in practice, no.

The current Collins trial-balloon will very likely only turn back the clock a tiny bit. It will be incredibly unfair on paper, but in reality it is no less fair than what was going on during the 80s and 90s and yes, well into the 00s.

The sad part is that it is unlikely to work. The genius superstars are still doing okay when it comes to funding. And of them, there will be many who fail to produce the genius, superstar, pioneering breakthrough innovations that Francis Collins is intimating they will all produce. There will be many of them that, without Collins’ intervention, will indeed make amazing breakthroughs. Many of both categories that might perhaps be awarded grants under this new expansion of the Pioneers program would still manage to win an equivalent amount of project-based funding in the absence of Collins’ plan going through.

I’m just not a big believer in making bets on who is going to revolutionize science and give them all the grant money. I believe a more distributed, less directed, individual investigator initiated approach is the demonstrated success model. When we try to pick a few winners we do less well at creating innovation.

So my suggestion is to figure out a way to relieve far more of the extramural research team from the current tyranny of the grant game. Not just a handpicked few but many. 30%? 50%? More.

All of us are spending far too much time on grants. Spending far too much time on creative thinking about data and what-ifs for yet another application, instead of following up on those great ideas. Many, many of us just-folks in the system would do a lot better if we were able to “just apply for a new grant when we needed it”. The scientific product would be much better and the cost-ratio would be improved.

Streamline the process for more of the NIH extramural force and guess what? The “superstars” will also be relieved! They will likewise get to spend more time thinking about innovation and, since they are superstars (right?) their innovation will be amazing.

My best proposal for how they should do this is easy because it uses an existing mechanism. They could start this process….tomorrow.

My proposal for making the system more person-based and less subject to the vagaries of review is to expand the R37/MERIT program. This is the program that awards an occasional highly-meritorious competing award an extended non-competing interval. So instead of having to think about renewal in 5 yrs, you have 10. There is still noncompeting review and rumor has it that some ICs have been willing to cancel R37s midstream for lack of production. Rumor also has it that many ICs take an extra hard look during year 5. But regardless, the structure is there.

A five year proposal that is now given 10 years? That should make almost any PI feel a lot more free to pursue blind alleys and risky new directions.

Advertisements

38 Responses to “Collins proposes HHMI style "people, not projects" awards. Big deal.”

  1. Ola Says:

    Yeah, tell me about it. My most recent comp renewal would have been eligible for MERIT, if NHLBI hadn’t abolished the program 6 months earlier!

    I will say though, the thought of having “genius” attached to one’s name has never appealed. So much pressure to actually do genius-y stuff.

    Like

  2. drugmonkey Says:

    Here’s a thought. Maybe Collins is dragging up any idea that will get tons of press attention but that he knows won’t change the finances around very much? This could be a perfect example.

    Like

  3. The Other Dave Says:

    Reading proposals and evaluating projects is a pain in the ass. Much easier to simply give money to your buddies, and maybe some buddies of buddies.

    NIH is less and less of a meritocracy, and more and more about protecting scientific royalty.

    Like

  4. Dave Says:

    Fully agree with everything written here. I don’t get Collins at all. I really don’t. What problem does this solve?

    Like

  5. bsci Says:

    As a hypothetical, what if getting one of these grants meant someone was not allowed to apply for other NIH grants? This could be a way to tell a PI, we’ll give you $1million without asking many questions, but don’t swamp the system with grants asking for another million. If every PI grant turned a $2million lab into a $1million lab, that does add up to a few more R01s for everyone else.

    It’s also interesting to note that the Nature piece said 5% of the current NIH budget goes to such PIs, but intramural research is also PI, not project grants and makes up a bit less than other 10% of the NIH budget last time I looked (though a good bit of that money goes to infrastructure, such as running a free hospital, rather than PIs grants).

    Like

  6. joatmon Says:

    I assume NIH’s attempt to do this is to secure some high impact research. I don’t agree this is the right strategy but instead they should try their best to spread the money across. That said, we should we should weed out investigators who routinely publish something with IF <4.

    Like

  7. old grantee Says:

    It isn’t that Collins instituted that idea. That started during Zerhouni’s tenure aiming at propelling big ideas with potential for invaluable biomedical breakthroughs. Have they taken place since then?. Is such a policy, given the results so far, on the verge of producing any?.

    Maybe what Collins should do I to launch a discussion on the appropriateness of keeping such a policy given the prospects of (likely) lasting harsh financial times for the future of biomedical research.

    It is in the country’s best interest that the scientific community has a decisive word on it.

    Like

  8. CD0 Says:

    Ridiculous. I want to see data demonstrating that pioneer awards promote higher productivity than equivalent funds through the RO1 mechanism. Because the evaluations of the program that I read did not support that.
    The biggest difference between the funding system in the US and elsewhere is that here you are as good as your next project. You need to be sharp and earn the funds to support your research until the end of your career.
    In Europe, for instance, your name (and in many cases the name of your family) matters much more than your new ideas. If this is the pathway we are following, I hope that Collins will be removed soon from his position and somebody with common sense will replace him.

    Like

  9. Ola Says:

    @Joatmon That said, we should we should weed out investigators who routinely publish something with IF <4.

    Assuming this is not just trolling, such a filter would catch just about every society journal out there plus (given the trend) JBC a couple of years from now. It also includes most of the newer publishing enterprises trying to shake things up a bit (like PeerJ) and don’t even have IF’s yet.

    Like many junior investigators, I had numerous papers in IF10 journals in the early part of my career, and eventually found a home somewhere lower down the ladder. Years later those papers are highly cited and respected, and the ideas have been “further developed” by BSDs and published in higher ranking journals. Despite what the journals would have you believe, the ability to publish at high IF has very little to do with the actual science in the paper and a lot to do with whether the lead investigator has “broken in” to the particular field. As a newly minted PI, unless you come from a BSD lab with friends on the inside, it is virtually impossible to publish high IF right off the bat. As such, applying your IF<4 filter would essentially defund all new investigators.

    Like

  10. Ola Says:

    edit – “as a junior PI I had numerous papers SUBMITTED TO IF10 journals […] eventually found a home down the ladder”

    Like

  11. Grumble Says:

    Let’s suppose you have three post-docs. One of them is incredibly smart, always comes up with great ideas, and does her work extraordinarily well. The other two just kind of plug along; you have to guide them constantly to the next experiment, they never seem to come up with anything original on their own, and their experimental work gets done at a reasonable pace, but slower than the first post-doc’s. Then your budget is cut, so you have to get rid of two post-docs. Which one will you keep?

    What Collins is doing is, in his view (I’m guessing), no more than you would do: the NIH has dire money problems, so he wants to keep the best and cut the rest.

    On one hand, what this strategy completely fails to realize is that advances in science do not just come from the flashiest, high-profile brand-name scientists. There is no such thing as “the rest”; if you cut the rest, you risk cutting off someone who might one day make an incredibly useful contribution.

    On the other hand, you were pretty convinced about which post-doc was the best and should be kept, right?

    Like

  12. DJMH Says:

    But grumble, that’s often not the dynamic. You have one postdoc with great ideas and capacity for synthesis, but who is slow to finish experiments; another workhorse who has to be guided but gets a lot done; and a third who is invents awesome new techniques but is an asshole. Who do you keep?

    That’s the problem with people not projects…people are very complicated to “score.”

    Like

  13. MorganPhD Says:

    We already partially score projects by who proposes them.

    That’s why we include a Biosketch and have scoring criteria like “Environment” and “Investigator”.

    It doesn’t matter how good of an idea “Dave Smith” from “Southwest North Dakota A&M University” has, it will be more difficult for him than the guy/gal from Harvard or Berkeley.

    Like

  14. miko Says:

    This train has left the station in Canada. CIHR funding is being split into “project” grants (the traditional mech, which is like an R01’s undersized but snarky Canadian cousin), and a new “program”-based mech that goes up to 7 years and budgets up to $1.5M/year (unless you’re a new investigator, then only 5).

    They like to pretend that these are two “styles” of doing science and you should pick the “style” that’s right for you and your work, but most seem to assume it is an attempt to formalize the BSD/grocer distinction. However, they are doing 2 pilot rounds in which new investigators get a small advantage by being up against a smaller pool, as those with ongoing operating grants can’t apply in the pilot rounds.

    http://www.cihr-irsc.gc.ca/e/46791.html

    Anyway, if you’re curious to see how it goes, look up in a couple years.

    Like

  15. NIH sufferer Says:

    Dear MorgnaPhD,
    that is exactly why aft least the criteria “environment” should be removed from the scoring system. It is bias, has nothing to do with the quality of the project, or any other valid scientific criteria.
    These days a score of 3 in “environment”, even if you get 1 and 2 in everything else, will get you out of the funded pool. It is sad for the investigator and it is even worst for the sensible/ productive investment of federal funds. We should be evaluating the IDEA and not the location where the project is going to be performed!

    Another interesting bias that are doing a lot of damage:
    1)The Program Officer that refers to certain applicants as “my PIs” should be moved to a different institute to avoid excessive “coziness” with some PIs regardless of the merit of their project….

    2)I do not care how well you application fits in the “special panel of the chair of your department”. If the chair of your department is in the study section, your project should not be allowed to go to that study section. I know that if this happens they removed themselves from the room. If I was 5 years old, I will believe in fairies and also that removing your “Boss” from the room is an effective way of removing the positive bias towards your application 🙂

    Like

  16. drugmonkey Says:

    What problem does this solve?

    The PR blackeye of every Nobel winner going on and on about how their work “wouldn’t have been possible under the current NIH funding system” is his target, I would suggest. Also the whinging about not getting a grant every time from the people who are so awesome that FC actually listens to them. The circle closes…

    we should we should weed out investigators who routinely publish something with IF <4.

    You mean the papers themselves, not the JIF, I assume? :-p But at any rate….you don’t see anything wrong with this plan? if your goal is to get really out-there thinking would not a demand that it hit certain JIF rank standards tend to decrease risk? There is a certain boring “me-to” ness to the CNS game you know.

    That started during Zerhouni’s tenure aiming at propelling big ideas with potential for invaluable biomedical breakthroughs. Have they taken place since then?.

    IGNORE THE LITTLE MAN BEHIND THE CURTAIN!!!

    It is in the country’s best interest that the scientific community has a decisive word on it.,

    I don’t know about this. If we do it by some-are-more-equal as we are now, the BSD’s get what they want and everyone else can go fish. do it by gutter democracy and we end up with one tiny grant for everyone who has a hard money job teaching undergraduates in the academic side of campus and all the Med Schools, research institutes, intramural and the like can sod off. Also there will only be awards to obscure Universities that haven’t managed to secure a decent IDC rate.

    I like neither extreme. I am, I suppose, a moderate who wishes to retain the current spectrum, while making it easier for people across this diversity to survive. I land upon “fewer of everyone” as my only workable and obvious solution.

    On the other hand, you were pretty convinced about which post-doc was the best and should be kept, right?

    Your analogy is incomplete. It assumes the creative postdoc actually gets stuff done and generates publication quality results. Instead of spending three years pursuing something that fails while never keeping the lights on with the pedestrian stuff. If you have 20 postdocs like this, no problem, you can play the odds. But if you have a three postdoc lab, this is going to be a difficult call. As long as I’ve set the laboratory on a path to things that are generally important, then a competent slogger should be able to arrive at *something* good. A safer bet.

    Like I said above, on a large scale, we need all approaches.

    people are very complicated to “score.”

    Preach it Sister! Especially when your scoring is really about a priori “predicting” and not post hoc “scoring”.

    Like

  17. MorganPhD Says:

    Environment shouldn’t be scored for an R01. It should be like the Data Sharing Plan. “Is the paperwork in order? Is it filled out? Good, now we can move along to the good stuff”

    Like

  18. drugmonkey Says:

    that is exactly why aft least the criteria “environment” should be removed from the scoring system. It is bias, has nothing to do with the quality of the project, or any other valid scientific criteria.

    Within my limited study section experience I cannot ever recall the Environment criterion being used to discriminate big Us from smaller Us. It barely seems to be considered in most cases (backed up by Berg’s and Rockey’s data showing the correlation of overall impact score with the five criteria scores). Exceptions include when there is some rather unusual bit of kit or unique resource necessary as well as a place to complain about the job category and apparent “deal” being extended to a junior/transitioning investigator. In these latter cases, a not insubstantial fraction of reviewers imagine they are helping the PI by giving her some leverage with the local Chair or Dean to cough up a better appointment or more space.

    Like

  19. drugmonkey Says:

    removing your “Boss” from the room is an effective way of removing the positive bias towards your application

    I have never experienced a study section in which the members were in any way under sway of the Chair of the section. He or she is not exceptional. Social power dynamics on study section, in my experience, accord to the people’s scientific record and standing outside of the study section, not status within it. The only internal bit of social standing/power structure comes from the invididuals’ reviewing style and competence, not from being the Chair.

    Like

  20. Dr. Noncoding Arenay Says:

    Environment should be scored but not nitpicked upon. If it meets the requirements for the project, then just give it a full score. Sometimes, the perceived awesomeness of an institute’s environment may not translate to actual awesomeness.

    Like

  21. Joe Says:

    I see mostly 1’s and 2’s for environment. 1’s for Harvard, Stanford, etc. and also for Univ of ___ often. 2’s for Univ of ___ and the next tier. There’s an occasion 3 for a place where you wonder if they really do have all that would help with getting the project done and the core facilities are not great if they exist at all. 1’s and 2’s are not going to hurt. I do think though that where the applicant is can seep into the investigator score. If she’s at East Bumble Univ, then she may lose a point or two on the investigator score, which could be a killer.

    Like

  22. drugmonkey Says:

    Especially if one assumes that the awesome stuff is actually extended to the PI in question. This may not be true.

    Keep in mind, as well, that reviewers sometimes bust on a PI for not accomplishing as much as “should” have been accomplished based on local alternate resources. If those are not actually made available to the PI (politics or whatever), he or she can be treated unfairly.

    It happens.

    Like

  23. MorganPhD Says:

    drugmonkey,
    I recall those analyses from Berg and Rockey and think they did a good job analyzing the data they had in-hand.

    Unfortunately, it’s one thing to do a retrospective analysis of grant funding and another to do a randomized, double-blind trial (ie give reviewers the approach and significance sections and score those VS scoring the entire application).

    Like


  24. Environment – is usually used to assess whether younger folks have what they need to do the work. I have not seen 1.0 for the BSD universities and 2.0 for the not-so-big-and-not-so-swinging-dicks.

    Like

  25. MorganPhD Says:

    Potnia,
    If it’s really only use for younger PI’s, then it’s a crap criteria. It should apply to all or none. Not “sometimes new PI’s, and then sometimes for older PI’s at smaller schools, etc…”

    Like

  26. MorganPhD Says:

    There are only 131 Pioneer Awards. 631 R37/Merit Awards. And approximately 23,000 R01’s. Thinking about transitioning to a more “Pioneer or Merit” award system is much different than doing it.

    Like

  27. drugmonkey Says:

    It should apply to all or none.

    It does, it is just that it has severe dynamic range restriction. Brought about in part by the fact that people tend to propose work for which they are clearly competent to conduct and have the essential resources to complete.

    Like

  28. MorganPhD Says:

    Drugmonkey,
    Potnia Theron’s comment seemed to suggest that it’s usually used to assess younger PIs, which would lead me to believe that reviewers assume if you’re already established (ie older), that obviously you have what you need.

    Like

  29. Evelyn Says:

    I don’t know, I just can’t get worked up about this. Or any of the proposed “solutions.” As long as the ridiculous political mantra “more STEM graduates!” persists, we are just pushing dirt around. I went to a PhD defense today and had to worry about what this talented and smart woman will be doing in 5 years. The chances are it will probably not be science.

    Like


  30. My response was perhaps to terse. DM is correct about dynamic range. MorganPhD, if an established PI proposes something new, it is important to show that they have what they need to do the work. Thus environment is not just “is this a mighty fine place”, but “do you have the equipment to do what you say you will do”. I have never seen a big place/small place distinction on this. This assessment is applied to old and young alike. But as DM says, most Oldies are doing something for which they can say demonstrate a fine environment based on publications, which is not true for younger scientists.

    Like

  31. Grumble Says:

    “There are only 131 Pioneer Awards. 631 R37/Merit Awards. And approximately 23,000 R01’s. Thinking about transitioning to a more “Pioneer or Merit” award system is much different than doing it.”

    Right. In order to really make a difference, several thousand (if not 10,000+) of those R01s would need to become R37s or Pioneers.

    This is clearly not what Collins has in mind – he just wants to carve out a “NIHMI” (an NIH-funded version of the HHMI) out of the NIH’s ass. Which will only make things worse for the rest of us. And which will, in my view, be bad for scientific advancement in the long run.

    Like

  32. Grumble Says:

    ” I went to a PhD defense today and had to worry about what this talented and smart woman will be doing in 5 years. The chances are it will probably not be science.”

    And I went to a talk by a talented, established professor today, and had to wonder what he’ll be doing in 5 years. The chances are it will probably not be science.

    Like

  33. Joe Says:

    @ Potnia Theron. Thus environment is not just “is this a mighty fine place”, but “do you have the equipment to do what you say you will do”.
    Environment sometimes is also judged by colleagues and reputation. If that institute has a terrific group of scientists in that particular field, the environment score will be better.

    Like

  34. BrainyOne Says:

    I have a few honest questions about all the “sky-is-falling” rhetoric.

    Everyone complains about low success rates, spending so much time writing grants, etc. So WHY NOT just give the money to the “superstars”, and let everyone else worry about doing good science and writing good papers. The superstars will be judged, in the end, on the productivity that comes out of their labs. To get funding for my own project, I simply only have to convince my superstar boss that it is a great idea, rather than a few reviewers on an NIH study section who may or may not be reading my proposal for the first time on the plane ride to DC, and have much less general familiarity with the subject matter. (And please: no outraged assertions that this NEVER happens; I’ve served on enough study sections to know it happens A LOT. Or, in the immortal words of Judge Judy, don’t pee on my leg and tell me it’s raining.) And, furthermore, this is more or less how the superstar labs ALREADY operate.

    Of course this goes against the general idea that the sine qua non of scientific success is obtaining an R01 as PI. Maybe it’s high past time this sacred cow got slaughtered.

    As for the idea that there isn’t enough money or grants to fund all the scientists these days, has anyone actually done a breakdown of NOT R01s funded per faculty-level scientists, but rather total NIH funding divided by a reasonable estimate of a faculty-level salary, divided by a reasonable factor to take into account grants also pay for equipment, post-docs, graduate students, etc. I’m not saying I know what the answer is. I don’t. But I don’t see where anyone has even attempted to make a guess at it. The point is that the typical R01 does not ONLY fund the salary of the PIs, but several faculty-level collaborators as well. The idea that the PIs are writing grants and post-docs are doing ALL the work, is, at least in my field, bollocks. Post-docs do NOT have the necessary expertise to do all the work.

    Like

  35. drugmonkey Says:

    Huh?

    Like

  36. Nobrainer Says:

    Yeah, the “superstars” (not being the scientific minds) but the grants’ operators who use their “faculty salary level” scientists to direct the work, mentor postdocs etc… Yes, of course this has been happening. But it doesn’t mean that it is good for science in the short or the long run. Grants operators, in the majority of cases and there are also exceptions, are SHARKS utilizing all kind of strategies to survive in their “scientific” grandeur and “ingenuity to make a difference in the lives of people with disease”. In reality, they are just grant administrators eating up a huge chunk in salary and supporting University administrators to support them and viceversa. A real waste!. Yes, many publications in glamour journals, and so what?. Is the increasing the volumes of the National Library helping anyone in cancer, neurodegenerative disease, rare disease etc? Are those glamour journals helping in the future of science?

    “the sine qua non of scientific success is obtaining an R01 as PI”. You are right that this is not true. But the reality is that many excellent scientists have been forced to leave science and many to come will not be allowed to succeed due to the excesses of “superstars” and the likes.

    Like

  37. coldhot3 Says:

    Collins will get things fXXked up big this time. Biology is a ponzi scheme, with too many ppl trying to get a shot at it while too few actually get anything meaningful out. If he went out to promote such superstar programs, the majority of postdoc will quickly realize that they will not make it and drop out of postdoc pool.

    Sound nice, huh? Here’s the bitter part: witnessed the horrible fate of older postdocs, the PhD student will demand training in tranferable skills to survive in other jobs instead doing slave labor for their masters. THAT will hurt productivity. If their masters refuse provide such education, the smart and motivated ones just don’t go for PhD any more.

    Plus, @DJMH
    “invents awesome new techniques but is an asshole”. I disagree with this one. The reason we develop new techniques is because old ones do not meet our demand. Pointing out the fact some one is using bullsXit old method and generate BS data is a offensive, and make other ppls think one as a annoying asshole. But it is required to let them understand advantage of new methods and moves to new ones. A lot people in my place think I am annoying because I repeated point out the devastating loopholes in their experimental protocols and tried to push better experimental standards. But I think it is the way to go. General public support my work through donation and grant is because they want data reflect real world disease progression, not to let me pamper my collegues.

    Like

  38. Grumble Says:

    “If their masters refuse provide such education, the smart and motivated ones just don’t go for PhD any more.”

    As long as graduate biomedical education is free — actually, better than free, because most colleges guarantee students a stipend — students will continue to go to grad school.

    That doesn’t mean there won’t be fewer PhDs graduating from biomedial programs soon. There will be, not because students aren’t applying, but because colleges are cutting admissions due to reductions in the ability of PIs to get grants to pay the stipends. Some colleges (perhaps most now) won’t let underfunded PIs accept graduate students, so they have no choice but to cut admissions – there’s nowhere for students to do their thesis work once admitted.

    Like


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: