NINDS issues a fascinating Notice on R21s
May 14, 2010
From NOT-NS-10-017:
The NINDS will continue to accept applications for investigator-initiated exploratory/developmental projects (R21s) for all program areas supported by the Institute. Previous NINDS language stated that R21 proposals were “limited to those with the potential for truly ground-breaking impact”. We would like to emphasize that such impact, as described in the trans-NIH parent R21 announcement (http://grants.nih.gov/grants/guide/pa-files/PA-10-069.html), can be achieved in many different ways. For example, projects can assess the feasibility of a novel area of investigation, develop new techniques or models, apply existing methodologies to a new scientific area, etc. (see parent announcement for additional examples).
Umm, what? Is what they are trying to say here is that they are no longer insisting on R21’s meeting the criteria of “truly ground-breaking”? Couldn’t they just say that?
Or are they saying that their scare language has dissuaded anyone from bothering to submit an R21 when they could just write a small R01? Or go to some other IC for funding?
Or perhaps their applicants are getting so beat up in study sections which are trying to apply their previous language that they never get anything with a fundable score?
To me this move points the finger square at the problem with the R21 reviews, regardless of the qualifying language used in a Program Announcement. There are multiple explicit and implicit goals for the R21. They are poorly specified, and some are in conflict with each other. Therefore the criteria become a point of much contention in review discussions.
Apparently NINDS attempted to cut through that fog with additional specification about “ground-breaking” impact. A decent idea. Too bad it didn’t work out for them.
There was also this bit at the end which was interesting, and pertinent for many of our readers:
It is important to note that analyses of new investigator applications to NINDS indicate that the success rate for R21 applicants is lower than for R01 applicants (FY 2009 success rates for NINDS R21 New Investigators: 11% vs. NINDS R01 New Investigators: 19%). Given the current policy of the NIH to support New Investigator R01s at success rates equivalent to those of established investigators submitting new (Type 1) applications (http://grants.nih.gov/grants/guide/notice-files/not-od-09-013.html), the NINDS encourages New Investigators, and in particular Early Stage Investigators (http://grants.nih.gov/grants/guide/notice-files/NOT-OD-08-121.html), to apply for R01 grants when seeking first-time funding from the NIH.
Great first step. I’d like to see an argument that advances the cause with study section members who are still operating on the “starter grant” mentality. I’d like to see some NINDS data on productivity of R01 awards to n00bs vs. experienced investigators with 0, 1, 2… other awards.
DrugMonkey 
May 14, 2010 at 8:39 pm
I think that this forward-thinking regarding new investigators is more or less unique to NINDS and due in large part to Story Landis, who I think has done a great job setting the direction of the institute. She was also discouraging new investigators from applying for ARRA funding since, if they got it, it would invalidate their new investigator status.
LikeLike
May 14, 2010 at 10:14 pm
It is important to note that analyses of new investigator applications to NINDS indicate that the success rate for R21 applicants is lower than for R01 applicants (FY 2009 success rates for NINDS R21 New Investigators: 11% vs. NINDS R01 New Investigators: 19%).
What are they talking about? Is there some rule that forces the Institutes to use the same payline for R21s as R01s? Otherwise, if the success rates are different it’s because NINDS wants it to be that way.
LikeLike
May 15, 2010 at 1:50 pm
The low R21 payline for new investigators, by the way, illustrates the systemic problem with NIH. R21s from new investigators are THE source of ground breaking ideas. Younger people just don’t know that “something is not supposed to work” and tend to think out of the box. Most of the time they are wrong, but whatever truly ground breaking is conceived, it usually comes from people in their early 40s at most. Now, if established investigators (who, as a rule, tend to capitalize on earlier achievements and ideas) have a higher success rates on R21s than new investigators, that simply means that NIH institutional culture stifles innovative ideas.
LikeLike
May 15, 2010 at 8:57 pm
Gummi Bear, I think you are missing the point here. What NINDS is saying is that it is to the advantage of a new investigator to spend their valuable time writing R01 applications and getting more money for a longer time, rather than wasting your time with piddly R21s. With the extra money and time an R01 offers you, you have a lot more leeway to explore and bring to fruition your novel, groundbreaking ideas. If you get an R21, as soon as you get it you basically have to start applying for more funding, because an R21 is certainly NOT going to get your lab off the ground. Realizing this, NINDS has made it much easier for new investigators to get R01s, even if they are scored substantially below the payline (last year they were funding grants from new investigators at the 25th percentile).
LikeLike
May 15, 2010 at 11:50 pm
Yes, Nemnezia, this is precisely what they are doing, but I will rephrase it a little: they are saying that their own R21 Program Announcement is a pile of crap. That new investigators, with “groundbreaking ideas” shouldn’t, for their own good, take it too seriously.
“If you get an R21, as soon as you get it you basically have to start applying for more funding”
And this is precisely how it should work for “ground breaking” ideas: get an R21, test the idea, and if it works, apply for more money. The rationale for such a milestone-based approach is that only a small percentage of new ideas are actually viable. Some initially seem viable, but later it becomes apparent that the thing is not going to work. The R21 mechanism was evidently invented by someone at NIH who realized this, and wanted to make the utilization of the available funds more rational (for the price of one R01, you can have several R21s, test several ideas instead of just one, generate preliminary results and fund only viable ideas with subsequent R01s).
“because an R21 is certainly NOT going to get your lab off the ground”
This should be only up to me to decide. The reviewers and the NIH staff should be bound by the official Program Announcement and the philosophy behind it, instead of tinkering with the system as they please.
By the way, I applied for an R21 only once. Haven’t even resubmitted, after being told quite clearly that I don’t belong to the club. What is typical to the corrupt culture of peer review at NIH, is amplified severalfold at the panels dealing with R21s. This is my personal, biased opinion, of course.
LikeLike
May 16, 2010 at 3:32 am
“This should be only up to me to decide”,
Of course it is only up to you to decide. And I don’t think that NIH would ever make a decision for you. But NIH has also a moral obligation to alert and advice new investigators on strategies that work to build a research career. NIH has or should have a general view, based on their experience on grantees all over the country, on what is working or not working and for who…..
What I see is that NINDS has taken very seriously their responsibility on managing public funds to serve the interest and future of science for scientists and the country.
LikeLike
May 16, 2010 at 3:22 pm
“And I don’t think that NIH would ever make a decision for you.”
By “NIH” I mean the whole system, together with the review panels. These panels often run amok and do what they please, instead of adhering to the principles behind particular funding mechanisms, programmatic priorities and even scientific integrity. And the NIH staff only offers some lame “advices” to the PIs, instead of disciplining these panels. Such infuriating advices are equivalent to posting signs “Warning, damaged road” by the local Department of Transportation, and doing precisely nothing FOR YEARS to fix it.
LikeLike
May 16, 2010 at 6:42 pm
I’m with Gummi on this one. The NIH is owned by the review panels, rather than the other way around. Look at the NIH’s ultra-weak grant review “expectation”: the NIH should simply say grant review is absolutely *required*, no exceptions. I have no idea why the NIH cares about the opinions of its grant recipients, particularly where these conflict with the wants of the NIH’s primary paying customer, the American taxpayer. Why does the NIH keep moaning about the lack of translational research? Just keep on cutting non-translational funding until they get what they want.
LikeLike
May 16, 2010 at 8:00 pm
I’m sorry but two years is not nearly enough to know if your “groundbreaking idea” is going to work, particularly when you are first setting up your lab and your time is split between getting more funding, training people (if you are fortunate enough to have any), figuring out your teaching duties, etc. It might be enough time if you have a fully running lab equipped with competent students and postdocs.
Don’t you see that NINDS is trying to help new investigators here? I strongly disagree that all they do is offer “lame advices” to the PIs. When I was first trying to get a grant and failing miserably I kept thinking the same thing, that why bother talking to the PO since I knew what boilerplate advice he would offer. However in desperastion I went to introduce myself during the Society for Neuroscience meeting and he basically told me “if you want your grant funded, do exactly what I tell you, and call me at every step along the way”. I did so, and it got funded. Certainly not lame advice.
There is no secret “R21 club”. NINDS is just offering advice based on their experience on what works for new investigators, and you would be wise to listen to them.
LikeLike
May 16, 2010 at 10:28 pm
1. I am sorry, but this depends on the nature of the research. For me, the statutory R21 time span and budget limit would work just fine.
2. Don’t you see that the Department of Transportation is just trying to help drivers with these “bridge broken” signs? So, don’t be a pest, don’t call these signs “lame”, and don’t keep saying “fix it, fuckers”. 🙂
The analogy is direct. The R21 mechanism is broken. BROKEN, and the only thing NIH is doing, is letting people know that it’s broken. Thank you, thank you, thank you, NIH, your concern is appreciated!
3. In my desperation, rather than kowtowing to the Supreme Beings at NIH, I finally submitted my stuff to NSF, received scientifically sound critiques (something I couldn’t get from the rotten NIH for YEARS), and got funded on the very first try.
4. You are NOT SUPPOSED to contact POs, “do exactly what they say” and so on. You are, pardon me, supposed to read the program announcement, construct your application accordingly, and leave it at the mercy of reviewers (who, in turn, should maintain appropriate professional integrity while evaluating it). Now, we know that at NIH this is different. The PAs, RFAs and the publicly disclosed procedures are just Potemkin villages, a lip service for fools (and Congress critters) to behold, while the actual funding decisions are made differently. And here we have a fundamental disagreement: you are suggesting that this is just how things are, and we, the small folks, should adjust and play along. I, on the other hand, am saying that this is rotten, and this is not how it is supposed to be.
LikeLike
May 17, 2010 at 12:36 am
“You are, pardon me, supposed to read the program announcement, construct your application accordingly”
Yes Gummibears, I agree with you that’s the predictable way to respond to PA, RFAs etc.
“and leave it at the mercy of reviewers (who, in turn, should maintain appropriate professional integrity while evaluating it)”.
No Gummibears, I believe it is more productive
Not to be negatively prejudiced on the professional integrity of reviewers, unless there are signs suggesting so. When this happens one has several options:
a) What you said you did: Send your grant to a different agency hoping that they don’t have human beings in their Review Panels but Extraterrestrials Agents who always see excellent science where excellent science is and always give fundable scores to the right applications. It worked for you, so those agencies must be around.
OR
b) Share your concerns on the evaluation of your grant (including Summary Statements) with colleagues who you trust to make sure your assessment is not off-base or over-biased. If your concerns are also seen by others, then write to your Program Director and send a copy to your SRO so that everything is in the open and you’re in charge of your own case.
People make mistakes. IMO mistakes can be minimized and, hopefully, corrected when views and experiences are discussed at different and appropriate levels.
LikeLike
May 17, 2010 at 7:04 am
This gummibears clown is a fucking loon. Listen to his or her paranoid ravings at your peril.
LikeLike
May 17, 2010 at 7:59 am
One thing I think is underappreciated in these “Go for the R01” recommendations, whether from trusted bloggers or from an Institute itself, is preliminary data. Or should I say lack thereof. We new investigators are often a little short on it, and it is that, not a lack of “truly transformative” ideas or even the old-boy network, that I feel prevents me from applying for an R01. It doesn’t help me to say, “You should go for the R01” when I have what I think are great ideas, prelim data to show the ideas are feasible, but not enough to justify an R01 or defend against “fishing expedition” criticisms. Not to mention a publication track record. I don’t resent this — if I was giving a PI $500K I’d give it to the PI who has years of great publications and boatloads of preliminary data too.
But this is why I (in a basic science dept) am primarily applying for NSF and R21s for now, hopefully in 1-3 years I’ll have the data for the R01. Is this wrong? Should I be floating out an R01 just in case I get the review panel that decides my project is terrific despite my few-pubs-as-an-independent-PI-and-bare-bones-prelim-data?
LikeLike
May 17, 2010 at 8:31 am
The bar for preliminary data on a R01 appears to be much lower. I just had my 1st R01 funded, and while I had a nice amount of data, it was nothing compared to more senior colleaugues who were submitting renewals, or even brand new R01s. Standard caveat applies: different study sections treat new applicants differently.
LikeLike
May 17, 2010 at 8:32 am
Should I be floating out an R01 just in case I get the review panel that decides my project is terrific despite my few-pubs-as-an-independent-PI-and-bare-bones-prelim-data?
Yes.
LikeLike
May 17, 2010 at 8:50 am
@Mbench, it seems that there are Institutes that do not promote R21s anymore but instead they have the Eureka (4-5 yrs) mechanism for trying out-of-the-box ideas and you don’t have to have/show preliminary data.
I mean, this is kind of R01s (solid funding in time and space) without having to show that, you are a miracle-producer of loads of data deserving an R01.
I heard it from an Institute Director on one of those NIH videocasts.
LikeLike
May 17, 2010 at 10:08 am
“This gummibears clown is a fucking loon.”
Indeed, because I disagree with “redefining peer review” perpetrated by you, fucking charlatans on NIH panels.
LikeLike
May 17, 2010 at 11:03 am
FYI, prior thoughts on R01 versus R21 from PhysioProf are here:
http://scienceblogs.com/drugmonkey/2008/01/r01s_versus_r21s_for_new_inves.php
http://scienceblogs.com/drugmonkey/2008/02/hey_new_pi_submit_that_r01_now.php
http://scienceblogs.com/drugmonkey/2009/09/why_r21s_stink_a_lot.php
Hopefully in the comments it comes across that I am more positive about R21s than is PhysioProf. I look at it as another opportunity to take a slightly different whack at the piñata, rather than viewing it inevitably as a waste of time better spent writing R01s.
mbench@#13- what whimple said. Absolutely DO NOT get paralyzed by thinking you need to have Aim I pretty much wrapped up in the preliminary data. I have had many applications favorably reviewed sans the expected StockCritique (and some even funded) with only *feasibility* type of preliminary data. Study sections will vary and reviewer assignments within study sections will vary in terms of how they view preliminary data. If you don’t send in an otherwise fantastic application, you are pre-judging your application for the reviewers. Don’t do that.
LikeLike
May 17, 2010 at 11:35 am
Hmmm. Thanks for the targeted advice. After the summer NSF deadline, maybe I will put together an R01 for fall NIH. I do have projects that would fall under the Eureka program (ie “potentially transformative”), but are paylines much lower for Eureka’s than for other R01’s?
LikeLike
May 17, 2010 at 12:27 pm
…maybe I will put together an R01 for fall NIH
What do you mean “maybe”? Submitting R01s is your JOB now. Better do it.
LikeLike
May 17, 2010 at 5:58 pm
I’m on a roll posting on the DM blog, so I might as well point out that I interviewed (by e-mail) Bob Finkelstein, NINDS’s director of extramural research, about the new NINDS announcement. A little bit of light was shed. I posted some of his comments on the Science Careers blog.
Finkelstein on NOT-NS-10-017
BTW, an absolutely essential point that some commenters here are missing is that NINDS has done a superb job funding new and early-stage investigators at above-average rates. While the preliminary-data question is worth discussing, the statistics show that they’re getting it done. [This year, NINDS is funding new PI R01s to the 20th %ile in most study sections, and most “early stage investigators” (i.e. those within 10 years of terminal degree) to the 25th %ile.] In any situation where people fail 80% of the time — and it would be good if that number were a good bit smaller — a lot of people are going to have a negative opinion of the process and the fairness of the competition.
Jim Austin, Editor
Science Careers
LikeLike
May 17, 2010 at 9:05 pm
Okay, okay, whimple, point taken. I have a lot of irons in the fire at diverse agencies and institutes so it becomes more an issue of opportunity cost, a calculation of time invested vs. probability of success, that I have to weigh. But I get it.
LikeLike
May 18, 2010 at 1:48 pm
NINDS has done a superb job funding new and early-stage investigators at above-average rates. While the preliminary-data question is worth discussing, the statistics show that they’re getting it done
My continued point is that programmatic pickups are a BandAid adopted in lieu of fixing the real problem(s). My preference is for more systematic fixes that head off this problem now and also impact other problems.
LikeLike
May 18, 2010 at 4:26 pm
“more systematic fixes”
1. Blind reviews. This is not always possible with mechanisms, where preliminary data are expected, but at least for R21s it can be done. Reviewers should NOT be given any information about the PI and his/her affiliation. Only the PO should have the authority to access and use this information while making the final funding decision.
2. Every application should be discussed during panel meetings. This would eliminate the most blatant nonsense in the critiques (the “shame factor”). Blatantly nonsensical proposals should be eliminated prior to the external review stage by the SRA, to reduce the panel’s workload. NSF does something quite similar, NIH can too.
3. No -A1 resubmissions (also to reduce the workload). Instead, PIs should be allowed to submit 2-page rebuttals to the initial critiques, to be presented during panel meeting, when the application is discussed.
4. Hiring active, established scientists for temporary stints as POs. This would strengthen the position of the PO, compared to the current POs, who are mostly career bureaucrats with quite meaningless doctorates.
5. Introducing an efficient mechanism for detecting and eliminating reviewers lacking appropriate expertise/ethical standards. Substandard reviews should be treated as scientific misconduct. On the other hand, a carrot should also be offered: the panel duty should be handsomely paid.
LikeLike
May 18, 2010 at 8:27 pm
re: gummibears ideas for ‘making things better’
1) Having a good idea is not enough in science. Do you have the means to get it done? Historically, institutionally, etc. To go blind would ignore this. This are important aspects of review, and to leave this to program only would put a great deal more power in program hands. Would that really benefit everybody equally, or just well known labs? something to ponder.
2) There is not enough time to discuss everything. Perhaps if they drafted many more reviewers, reaching down in the rank, they might be able to handle this, but there are a great deal of scientists, submitting a great deal of proposals. too many for the current system.
3) That is basically what a A1 is…no?
4) The PO’s I have interacted with have been working scientists who stepped away from running their labs, or after having successful post-docs. To claim they all have ‘meaningless doctorates’ is utter nonsense. You are a scientist, stop arguing based on anecdotal evidence.
5) This would be great. Not sure how to do it, but it would be great to have more expertise on study section.
LikeLike
May 18, 2010 at 9:52 pm
Having a good idea is not enough in science.
In fact, having a good idea is irrelevant to getting NIH funding. What you need is an idea that study section reviewers *think* is a good idea.
LikeLike
May 18, 2010 at 9:58 pm
Pinus:
1. I am not suggesting totally removing the “environment” and “investigator” factors from the review process. I am advocating, however, removing them from the list of factors evaluated by the review panels. The panels should just utilize their expertise to evaluate (as blindly, as possible) the idea and approach, nothing else. Then, the final judging if the PI has the means to conduct the research would be up to the PO (a binary decision: yes or no).
2. The current crappy peer review process at NIH inflates the number of resubmissions and overloads the review panels. When PIs can’t get consistent, quality reviews (even NEGATIVE, but scientifically sound reviews), they keep resubmitting the same stuff over and over. So, improving the quality of peer review will IMO substantially reduce the number of unnecessary resubmissions. NSF does this – they discuss every proposal, and their critiques are on average of much better quality.
3. No. A rebuttal would only give an opportunity to dispute the argumentation used in critiques. Amending the application (as in -A1) would not be permitted. Write the proposal well in the first place! 🙂
4. If they stepped down, they are emeriti. If they have “completed successful postdocs”, they are NOT YET active, independent, experienced scientists. I am advocating placing NIH representatives on the panels, who would be competent and not easily intimidated peers to the panel members, not afraid to say “what fucking crap is this” when presented with a particularly lunatic critique. The current NIH POs do not have the standing on a par with panel members, and this shows.
LikeLike
May 18, 2010 at 10:08 pm
“In fact, having a good idea is irrelevant to getting NIH funding. What you need is an idea that study section reviewers *think* is a good idea.”
Absolutely correct. However, we have the right to expect that the THINKING of the panel members will more or less follow the paths of objective truth, established knowledge and elementary logic.
LikeLike
May 18, 2010 at 11:47 pm
Just because study sections think your proposals are shit, doesn’t mean they are failing to follow “the paths of objective truth, established knowledge and elementary logic”. Maybe your proposals really are shit.
LikeLike
May 19, 2010 at 11:20 am
“Maybe your proposals really are shit. ”
After I lost patience with the rotten NIH, I submitted the “shit” to NSF, for two different mechanisms, with two different objectives. Two sets of good reviews, one award. So stick your theories you know where. NSF actually has lower success rates, but the POs and reviewers there do what they are supposed to do: they evaluate the scientific merits, rather than play an academic equivalent of corporate power games, masked as “scientific peer review process”.
LikeLike
May 19, 2010 at 11:36 am
NIH and NSF have quite different statutory funding mandates. Your applications were probably shit in relation to the mandate of NIH, and not shit in relation to the mandate of NSF. I have multiple NIH awards, some of which are based on ideas I have run by NSF program officers who have told me in no uncertain terms not to bother submitting to NSF. Does that mean these NSF POs “play an academic equivalent of corporate power games, masked as ‘scientific peer review process'”?
Face it holmes, you are a loon who can’t grasp the fact that *everyone* thinks their grants are the bestest most awesomest science in the history of forever. The difference between you and most people is that youa re a paranoid freak who thinks that anyone who disagrees must be failing to follow “the paths of objective truth, established knowledge and elementary logic”. Grow the fuck up.
LikeLike
May 19, 2010 at 11:56 am
NIH has clearly defined and publicly announced programmatic areas, and the reason I bothered with these imbeciles for so long is that my stuff matches these areas. The problem is that the review panels don’t give a damn about the NIH intended mission and research directions. They want to redefine them as they please. If I am a loon by noticing this, then a friendly NIH PO, whining to me about the systemic suppression of certain whole scientific fields (systems biology, to be specific, which is BTW not my area of interest), must be a loon too.
Having my own experience with these review panels, I dare to formulate the following theory: they don’t suppress because it doesn’t fit NIH. They suppress, because the system of reviewer selection at NIH promotes ignorant, dumb fuckers, who are UNABLE to understand a little more complex subjects, and instead promote the same old, familiar shit they feel comfortable with (in addition to acting on biases and promoting friends, naturally). Face it, Holmes, you may be one of them. As for growing up, if it means that I will become one of them too, then thank you, I prefer not to grow up.
LikeLike