A Bold Proposal to Fix the NIH
January 6, 2014
Our longtime blog commenter dsks is always insightful. This time, the proposal is such a doozy that it is worth dragging up as a new post.
… just make it official and block all triaged applications from subsequent resubmission. Maybe then use the extra reviewer time and money to bring back the A2, perhaps restricting it to A1 proposals that come in under ~30%ile or something.
Hell, I think any proposal that consistently scores better than 20%ile should be allowed to be resubmitted ad infinitum until it gets funded. Having to completely restructure a proposal because it couldn’t quite make the last yard over what is accepted to be a rather arbitrary pay-line is insane.
On first blush that first one sounds pretty good. Not so sure about the endless queuing of an above payline, below 20%ile grant, personally. (I mean, isn’t this where Program steps in and just picks it up already?)
This reminds me of something, though. Unlike in times past, the applicant now has some information on just how strong the rejection really was because of the criterion scores. This gives some specific quantification in contrast to only being able to parse the language of the review.
One would hope that there would be some correlation between the criterion scores and the choice of the PI to resubmit. As in, if you get 4s and 5s on Approach or Significance, maybe it is worth it. 7s and 8s mean you really better not bother.
DrugMonkey 
January 6, 2014 at 3:57 pm
I’ll play, DM. From my recently unscored A0:
Reviewer 1:
Significance: 5
Investigators: 3
Innovation: 6
Approach: 6
Environment: 2
Reviewer 2:
Significance: 2
Investigators: 2
Innovation: 4
Approach: 4
Environment: 1
Reviewer 3:
Significance: 3
Investigators: 3
Innovation: 2
Approach: 2
Environment: 2
On the numbers, what does your gut say vis a vis revise and resubmit?
LikeLike
January 6, 2014 at 4:45 pm
@Cassius King: Go for it, but that intro page and the matching content had better be tight butthole WRT reviewer 1’s comments on approach!
LikeLike
January 6, 2014 at 4:45 pm
I say get rid of all “resubmissions”. Every grant that is submitted is submitted as a new grant.
No response to reviewer comments, no having to rewrite sections that were fine the first time. If someone wants to keep sending in the same ND crap, the reviewers can keep sending back the same critiques.
No more having CSR spend time determining if a grant is a virtual A2.
No more appeals (not sure about grievances). This would save Program and Council some time.
Some folks would keep submitting hoping to hit the lottery but at some point I would think that schools would get embarrassed and stop letting them.
This would cause some Review workload problems in the beginning but I would guess that after a few rounds the workload wouldn’t be too much greater. Especially once reviewers and SROs start realizing who the trouble makers are and place those grants in a special pile.
In the olden days there were no A1s,2s or 3s. In fact I think that this was considered during enhancing peer review but the idea tabled.
Stork
LikeLike
January 6, 2014 at 5:20 pm
CK-
Yeah, revise and resubmit and hope #2 and #3 talk #1 into it.
LikeLike
January 6, 2014 at 5:43 pm
I agree with Stork. The NSF already does this – you can submit as many times as you like. I don’t think it adds any more burden on reviewers, and is more likely to result in reduced effort (on the part of CSR) overall.
LikeLike
January 6, 2014 at 6:47 pm
NSF does allow you to resubmit an application as many times as you like, but there is cap on the number of applications that you can submit per cycle as PI or co-PI. Presumably this helps reign in the number applications that need to be reviewed.
LikeLike
January 6, 2014 at 8:09 pm
DM, what about an A0 from an ESI that was a 19 percentile but not funded and then triaged as an A1? What if the reviews were completely stocked with Stock Critiques™?
LikeLike
January 7, 2014 at 5:09 am
What about it? You have to write a new one anyway.
LikeLike
January 7, 2014 at 5:24 am
With regard to Stork’s comment, this is what was proposed in the original NIH Enhancing Peer Review report (http://enhancing-peer-review.nih.gov/meetings/NIHPeerReviewReportFINALDRAFT.pdf , see page 33). The report also proposed including the possibility of study sections assigning a Not Recommended for Resubmission check box that would provide a relatively small number of applicants with feedback that the idea is not likely to go anywhere even with lots of word-smithing or even a change of approach. These ideas were dropped based on feedback from the scientific community including FASEB (see http://www.faseb.org/portals/2/PDFs/opa/2008/NIHPeerReviewSelfStudy.pdf ). The “No A2” policy was developed as a compromise that many (including me see http://www.asbmb.org/asbmbtoday/asbmbtoday_article.aspx?id=32362 ) feel has many additional undesired consequences.
LikeLike
January 7, 2014 at 5:43 am
@Ola, @DM Thanks for weighing in – I have decided to resubmit, can’t win the lottery unless you buy a ticket.
LikeLike
January 7, 2014 at 5:56 am
Cassius, if one of my PI’s came in with those scores, the first thing I would ask them is if they have anything else cooking for that round. If so, I would tell them to focus on the new grant. My biggest concern with those scores is the lack of 1’s on the Significance, Innovation, and Investigators (I don’t pay attention to the Environment, where we are, our Environment always scores top grades). When you read the reviews, did reviewer #2 or #3 get excited about the idea? Had a recent R21 where one reviewer graded 2-4 but wrote almost 2 pages summary on why the project was needed and why more investigators should be working in this direction. If so, then I think you have a chance to turn it around. But watch for the review summaries that leave you feeling like they shrugged when they finished. In that case, I think a complete makeover is needed. Good luck!
LikeLike
January 7, 2014 at 8:34 am
“Not so sure about the endless queuing of an above payline, below 20%ile grant, personally. (I mean, isn’t this where Program steps in and just picks it up already?)”
Yes, I was probably getting overexcited there, particularly given that in reality its unlikely that a proposal would be consistently scored in such a narrow window more than a few times (if even more than once). Pinko Punko’s trip about the 19%ile A0 triaged in A1 is not an uncommon anecdote*, and recalling That Terrible Graph charting the face-palmingly flat relationship between funded proposal score and impact of the subsequent results from a while back, one can see why.
* Is there a chart plotting A0 vs A1 (and even A2s for when they were applicable), just to see if, as one would expect/hope, there is a genuine trend towards improved scores with each submission or whether, as some have suggested, each submission is its own lottery ticket. If the latter, then it certainly wouldn’t be such a bad idea to just go the way of the NSF (not including the recent move to one submission per proposal per year, which stinks).
LikeLike
January 8, 2014 at 5:04 am
The other great advantage of the “Everything is New” grant submission policy is that it greatly simplifies everything.
The success rate = payline.
No more trying to parse the “Culture of the Study Section” to decide if they like your work and just want you to wait your turn of they think your ideas are mehhhhh.
No more “what if my grant got XXXXX then XXXX” Just keep sending it in.
I am not sure if I like the idea of capping the number of apps/PI/round. That complicates things and NIH already has a rule about not reviewing the same science at the same time. If a PI has enough distinct ideas to send in 6 grants a year go for it. Let’s admit it. If your percentile is between a 15-30 then it is more like a lottery whether your get a “fundable score” or not anyway. So, let the folks keep buying tickets.
LikeLike
January 8, 2014 at 6:06 am
Stork: The success rate and the payline are related but are not the same thing. The success rate is almost always higher than the payline. See http://nexus.od.nih.gov/all/2011/02/15/paylines-percentiles-success-rates/ for an explanation.
LikeLike
January 8, 2014 at 7:00 am
Dr. Berg – I appreciate your comments and the document links that you provided, thank you. I read the FASEB response, and although they do not support the “every application is new” concept, they did not provide any reason for their lack of support or what they think the negative consequences, if any, would be. The NSF model allows for responses to reviewers in grant re-submissions, but it must be included within the page limits. Furthermore, they also have a “not competitive” ranking, which discourages re-tries, but doesn’t necessarily disqualify resubmissions.
I think the behavior of applicants to both agencies is the same. At the NIH, applicants can easily get around this “A0” “A1” business by re-crafting or re-wording the same ideas in disguise of a new application. If this is what’s really happening, why go through all this trouble to enforce something that everyone has a “work-around” for anyway? It seems to be a wasted effort.
LikeLike
January 8, 2014 at 7:04 am
JB – I think what Stork was saying is that by eliminating A1 and making every application “new” the success rate wouldn’t be skewed by the fudging of the denominator by counting A0/A1 as a single submission that the NIH currently performs. This is ridiculous on its face and it makes no sense (for PR or for information for investigators) to present success rate in such a fudged fashion.
LikeLike
January 8, 2014 at 8:44 am
JB,
meshugena313 just answered for me. With no A1s then the Success Rate has to equal the payline…more or less. GM has a fuzzy payline so the correlation is approixmate. But, with AI’s hard (or harder) payline the SR=PL.
LikeLike
January 8, 2014 at 9:09 am
Thanks for the clarification. I am sorry that I misunderstood the statement. However, the payline and success rate will still not necessarily be the same. It will depend on, for example, the mix of new (Type 1) and competing renewal (Type 2) applications. Type 2 applications general score better than Type 1 applications (so that more than 10% of the type 2 applications score better than the 10th percentile). For example, in fiscal year 2012 for NIGMS, the apparent payline (midpoint on the funding curve) was around the 20th percentile and the success rate was 25%. This included funding 441 out of 2418 new (Type 1) applications (18%) and 411 out 987 (42%) renewal (Type 2) applications. More than 85% of these applications scored at the 20th percentile or better. The correction for A0 and A1 applications that are reviewed in the same year is generally a smaller effect, at least at NIGMS.
LikeLike
January 8, 2014 at 9:20 am
eeke: My recollection of the reaction to the “treat all applications as new” policy boils down to a fundamental split in intended role of peer review. Some felt that the NIH system which explicitly allows the applicant to respond to the previous review (with the expectation of having a clear path to be better score) was a strength of the system. Others felt that this “deal making” between the applicant and the study section was problematic in that the assumption that the score would (or should) go up if previous concerns were addressed is inappropriate because it assumes that the previous review noted all significant flaws and that the same reviewers would get the resubmission (which is often not true or possible). In the end, the NIH leadership tried to listen to the concerns of the community with the “no A2” compromise. This policy may have worked to discourage the downscoring of strong A0 applications because that have another chance but, in large part because of the subsequent drops in theNIH appropriation, have led to other problems.
LikeLike
January 8, 2014 at 9:52 am
“…because of the subsequent drops in theNIH appropriation, have led to other problems.”
That’s precisely it.
When 1/3 of applications get funded the first time, then it makes sense to discourage more than 1 resubmission: if it’s too crappy to make it the first and second time, it’s probably too crappy to make it the third time.
But when paylines are 6%, and it’s clearly not the case that 94% of applications are crappy, then it is very harmful to deny applications a second, third, fourth and ad-nauseam chance. All that does is force PIs to spin their wheels, trying to find new made-up angles on their preliminary data to spin into new grants. It’s waste of time and effort.
I’d much rather tweak a grant with an idea I really love and think has merit – and submit it 6 times to get it funded – than come up with 3 ideas, two of which I’m not as thrilled with as the first, and write whole new applications around each of them. Not only does that save me time *even if my overall success rate doesn’t increase* (although I expect it would), but it’s better for science if scientists write grants about ideas they love than about something they’re less convinced is worth doing.
LikeLike
January 8, 2014 at 11:31 am
Disagree. This makes you think more creatively about all the implications and applications o your data.
LikeLike
January 8, 2014 at 12:33 pm
Grumble: I took another approach at the effects of the limitation on submissions in semi-quantitative terms in my essay “On deck chairs and lifeboats” (http://www.asbmb.org/asbmbtoday/asbmbtoday_article.aspx?id=32362 ). The bottom line is that with reasonable estimations of the uncertainty in scoring, the percentage of the best (assuming that there is a true ranking even if it is not readily measured) that do not receive fundable scores goes up to significant levels when paylines are as low as they are now.
LikeLike
January 8, 2014 at 12:52 pm
Jeremy said,
“Some felt that the NIH system which explicitly allows the applicant to respond to the previous review (with the expectation of having a clear path to be better score) was a strength of the system. Others felt that this “deal making” between the applicant and the study section was problematic in that the assumption that the score would (or should) go up if previous concerns were addressed is inappropriate because it assumes that the previous review noted all significant flaws and that the same reviewers would get the resubmission (which is often not true or possible).”
Jeremy, in re my question earlier on and relevant to your last post, as part of the discussion that led to dropping the A2, did anyone present a chart showing the relationship between A0, A1, and A2 scores to see if there is a general increase in score or not?
LikeLike
January 8, 2014 at 12:58 pm
“This makes you think more creatively about all the implications and applications of your data”
There is a difference between being creative and being forced to be creative. The first is inspiration and productivity. The second is busywork.
LikeLike
January 8, 2014 at 1:10 pm
DM,
“Disagree. This makes you think more creatively about all the implications and applications o your data.”
I don’t disagree with this, but there is a caveat to reshuffling your central hypothesis and specific aims for a proposal, particularly given the requirement for large amounts of preliminary data that, whether anyone admits it or not, is provided to reassure reviewers that the project is already well underway. Where preliminary data is presented with the chief purpose of showing actual work-in-progress towards a particular specific aim, it’s an ethical gray area to encourage PI’s to start switching up hypotheses (due to the pitfalls of post hoc theorizing etc).
OTOH, it’s of course perfectly legitimate to present preliminary data with the intent only to prove the feasibility of some proposed experiments (“See, I can do this” data); one can even refer to published data for this purpose.
LikeLike
January 8, 2014 at 1:15 pm
dsks: I have been searching for this. These data were collected and discussed during the “Enhancing Peer Review” process internally. My recollection is that the average increase was about 30 points (old system, e.g. 240 to 210) but that there was, of course, considerable scatter with some scores getting worse. I am checking with sources with NIH to see if any of this was formalized or made public.
LikeLike
January 8, 2014 at 3:08 pm
, did anyone present a chart showing the relationship between A0, A1, and A2 scores to see if there is a general increase in score or not?
Even asking this question is playing the stupid game* when it comes to grants at the NIH. We get right back to my criticisms of the situation when there was still an A2 and the (unsolicited) A0s had terrible to no chance of funding.
I dispute the assumption that the critique/revision process so much as alters the resulting science, never mind whether it improves it or not. Sure the text on the page changes. Sure, with an additional 9 months time, maybe the field has changed a bit, maybe new preliminary data change directions. But the bottom line for investigators, once funded, is still the same.
Do the best science in the highest priority that they can think of at the current moment. Follow the field. Get advice from colleagues. Collaborate with other scientists, including the lab’s trainees. Navigating this, and the manuscript review process is what dictates what science is accomplished in the end. Comparatively, the comments of the grant reviewers and the alterations that were made to the grant text have next to zero significant influence.
This is my assertion.
I now look forward to all the testimonials from PIs about how some comment or demand on the part of a grant reviewer made a significant difference that would never have been discovered in the doing of the eventual science or the attempt to publish it.
*I was a fan of the limit to A1 for the promise it contained to get reviewers to stop nitpicking and to identify the good bones and hand out great scores the first time. This hasn’t happened from what I’m seeing.
LikeLike
January 9, 2014 at 4:31 am
The problem with every application is new is that it moves us even farther away from stability. There was a time when you got in line – the A0 had a low chance of funding, but the A1 had a better chance, and by the time you included the A2, you had a pretty good chance, even with funding rates relatively low. What this meant was that you had to plan three years ahead to get funding – BUT YOU COULD PLAN! You knew if you were making progress towards funding. With every grant being new, you are playing a lottery. You have absolutely no idea whether you have any chance at all of getting funded. This is a disaster for planning. It means you have to submit lots of grants to have a chance at one, and sometimes you get none (which means you’re out of money), sometimes you get too many (which means you’re overworked and less efficient), and sometimes what gets funded is not really the thing you wanted to do.
I have worked with both NSF and NIH grants and the every-grant-is-new NSF situation meant that you were always chasing a new goal post. With the resubmitability at NIH, you had the ability to answer reviewer comments. (Yes, I know, in practice, you can get new reviewers at NIH too, but it was still better than NSF.)
I also think we need to get over the idea that writing a grant helps your thinking. Writing a grant is a waste of time for science. Many faculty (including me) like to tell their students that “nothing exists until it’s published”. The proof that grants are a waste for the scientific enterprise is that they are confidential and not published. If you need help in organizing your thinking, write a review paper or a book. Grants are a means to an end – they get you funding so you can do your real science. Saying that writing a grant helps your thinking is silly. Even if it’s true for you, it’s not the purpose of grants, and there are better ways to help your thinking process, ways that will help science in general, not just get you funded… if you’re lucky.
LikeLike
January 9, 2014 at 5:59 am
“With every grant being new, you are playing a lottery”
qaz – The application process is a lottery anyway. You can submit an application of staggering genius that everybody seems to gush over, and still not get funded. What you describe is what’s going on anyway. Since everyone works around this two-time limit (see commentary on DM’s previous post), it seems a wasted effort to enforce this useless rule. It’s a fucking masquerade. With the shrinking and restricted budget that the NIH has to deal with, waste needs to be cut wherever possible.
LikeLike
January 9, 2014 at 7:55 am
qaz – the increased uncertainty is a feature of the budget woes, not of the rules for revising.
LikeLike
January 9, 2014 at 9:42 am
eeke – The application process didn’t use to be a lottery. This is a major change in how science is done from the not too distant past. (I’m old enough to have seen it when I was a young student.) Faculty could reliably know that their grant would have a very high likelihood of being funded, which meant that they could actually survive on a single R01, and didn’t waste their time writing a dozen applications to get one funded.
DM – Certainly, the budget makes things worse, but I don’t agree that the lottery effect is due to the budget problem. In fact, the system got worse as the budget supposedly doubled. Now that the budget has crashed, it’s even worse that it was. I saw the A0 to A2 rules and (at least in the study sections my immediate colleagues went to), the air-traffic-control sequence often meant that you knew you had a damn good shot at funding eventually, so you could survive if you planned well enough.
I don’t want to go back to the A2. I want a system that doesn’t have me wasting time writing grants that will never be funded and never contribute to science.
LikeLike
January 9, 2014 at 12:16 pm
you could always publish each grant proposal that goes unfunded on a blog or website qaz.
LikeLike
January 9, 2014 at 5:57 pm
And would people cite it?
LikeLike
January 10, 2014 at 5:59 pm
Just a datapoint; my first R01 (back when the A2 existed) was triaged twice, and scored 3% on the third submission. There was very little difference between the A1 and A2, I just got different reviewers.
Just because a grant is triaged doesn’t mean its a bad grant.
LikeLike
January 13, 2014 at 5:03 am
I have reviewed a lot for NSF, where they do things exactly as proposed by dsks (and stork): Every proposal is new; revise and submit it as many times as you like (although NSF also does actually have a ‘not recommended for resubmission’ category).
I agree that the NIH system is nice in that it gives you a formal chance to reply to reviews. Although I wonder whether that leads to more back and forth arguments rather than actual proposal improvement.
Either way, the problem really has nothing to do with stuff like that. The problem is the high number of excellent proposals relative to the number of proposals that can be funded.
And that is a *serious* problem. It used to be that being funded or unfunded depended on the project, the investigator, grantsmanship, etc. — stuff that SHOULD matter. And maybe a little bit on politics. Now, perfection is pretty much the necessary baseline state in all categories that should matter, and funding success depends almost entirely on politics. You’ve got to have a great project, AND be a great schmoozer. All of my success lately I can attribute directly to sucking up to funding agency personnel or potential reviewers. The problem with politics determining funding success is that this will exacerbate existing problems regarding sex/race/age/etc-related discrimination. It will also reinforce existing lines of thought, because projects that threaten the intellectual status quo have no chance. It doesn’t matter whether you’re a great scientist. You also have to be what people ‘like’. Scientific diversity is suffering. More money is being concentrated in the hands of a few — the club.
LikeLike