Crocodile tears from experienced NIH investigators over the discontinued A2 revision

February 15, 2011

Ahh, just when the thoughtful, completely self-interest free prescriptions for saving the NIH extramural grant system from D. Noonan and Cedric Wesley were losing steam (see Odyssey for more), we have a new entry. This one is via an email chain letter from one Robert Benezra (PubMed; Institutional website; Research Crossroads). Unfortunately RePORTER doesn’t seem to generate a good link on a PI search but homeboy’s been very successful at the NIH game. He had a one year F32 in 89-90 and then a 7-8 year stretch to his first R01 but he’s been a busy beaver ever since. I note only three -01A1s on his list, one of which is competing renewal for a Program Project for which he’s only a component head. Benezra seems to have had multiple R01-level awards (some are P01 components) continually and he even has a mainline grant that survived a switch from NIGMS to NCI. The website seems to confirm that he runs an ~18-member lab operation.
In short, a guy who is as inside as you could wish (I mean I saw Harold Varmus listed as a component head of one of the Program Projects this guy is on) now has some complaints.
[Before we move on to my ramblings, you may wish to read a bit of pro/con debate from DrDra of BlueLabCoats and Comrade PhysioProf as a warmup. Additional from Prof-like Substance and Genomic Repairman.]

The letter runs as follows:

Dear Colleagues,
I am writing to solicit your help in changing a new NIH policy that I
believe will have an enormous negative impact on our field. As most of
you know, a recently adopted rule states that if a grant proposal is not
funded on the first submission, only one revision can be submitted with
the same specific aims. If that revision is not funded, the proposal
must be “substantially” changed. As far as I understand, the rule was
adopted to discourage “serial resubmitters”. While such a policy could
make sense in an era of reasonable paylines, with the projected budgets
rumored to be funding at the 7th percentile in some sections, this could
have a dramatic and I would argue devastating effect on the research
efforts in this country. Consider the following:
The rule will have a disproportionately negative impact on young
investigators with early stage and therefore less diverse programs, or
more senior investigators who also have more narrowly focused programs.
How can a young investigator, for example, who is just starting
“substantially” change their aims when they have to focus their efforts
on a very limited number of projects undertaken with limited funds and
staff. These people are often hired by senior faculty on the basis of
their first projects and to be told they must change on the basis of
applications that might fail despite being ranked better than 90% of
grants submitted, seems patently absurd. And worse, it is likely to be
profoundly discouraging and destructive.
All of us who have sat on study section know that we cannot distinguish
a 15th percentile grant from a 5th percentile grant. It is simply beyond
the resolution of the process. Therefore, this new rule will have the
consequence of redirecting the science of many of our very best
scientists on the basis of what will essentially be an arbitrary
The meaning of “substantially changed” has not been clearly defined.
Program Officers themselves are not sure what this term means and are
not being given adequate guidance. I have heard things from “51%
different”, change the tissue or cell type you are working on, any aim
included in either the first application or revision cannot be included,
etc. We need clear and unequivocal guidance on this point, and I would
argue we need it immediately as “new” applications are being prepared by
a large number of investigators at this time.
The alternative that I advocate would be to go back to a system where at
least 2 revisions of the same application would be allowed. While we
will still obviously lose some superb applications if the pay line stays
where it is, I think this would provide a much fairer assessment of the
research proposals received by the NIH.
My intention is to let the feelings of a large number of scientists on
this subject be known. If you are willing to sign an email that will be
sent to both Francis Collins and Tony Scarpa (Director of Center for
Scientific Review) that raises these points, please let me know by
simply responding to this email and (if possible) forwarding it to 10
people who you know (not on the current recipient list) that might also
want to sign. If I can accumulate a large enough number of signatures
(100-500, say) I will draft a letter and send it first to all who have
expressed interest in signing to get feedback.
I must say, I am not generally prone to such activism but I think things
have just gotten to the tipping point.
I look forward to your responses.
With kind regards,
Robert Benezra

Now far be it from me to jump the gun here. Okay, I will…
I am reflexively suspicious of extremely well established investigators who have now, oh-so-conveniently, taken up the plight of the early stage and transitioning newly independent investigator. Could be that this guy has been on this tack all along, sure. But as we see this email get circulated virally, somehow I suspect that many of those established types that are enthusiastic about this letter are a tad, disingenuous shall we say.
Where the hell were you?
No, really. Where were you from the mid-90s onward as many of the study section cultural memes and habits were being laid down in stone? Where were you when the “expectation” for preliminary data kept ramping up? When the R29/FIRST was the expectation (that or some other lesser award) prior to R01? Where were you when the new investigators were taking it on the chin just because they were, um new, instead of inferior? Where were you when the average age of first award was increasing? Were you on the barricades against the ever increasing trend for “necessary” 5, 6 + years of postdoc experience?
Where were you on the multiple awards to exceptionally well funded faculty issue*?
You refer to study section service, Professor Benezra. Yes, we agree about the resolution of the judging when we are at the 10-20%ile line. I wonder, however, if you were the burr under the saddle of study section culture, constantly advocating for the ESIs, debating the merits of “Well Prof BlueHair is so damn productive that we gotta give this clusterbork of a proposal a good score anyway”, dismantling the expectation for yet more preliminary data and prioritizing giant, GlamourMag chasing labs over those of newbs. Were you? Were you on the barricades over putting junior folks (who could ill-afford it) into the endless revision holding pattern?

Whew, this is getting long. How bout some data to break it up?

When the NIH started making noises about special paylines for ESI investigators….did you endorse them enthusiastically? Did you use these moves during discussion at Study Section to further emphasize your position vis a vis helping the newbies? Did you rebut your peers who were grumbling about affirmative action for newly transitioned investigators? And how about the great Assistant Professor Purge from study sections? Did you cheer Scarpa along or did you oppose that?
Where were you on new hires, for that matter, if you have been in a position to affect such matters? Were you one who opposed trends for more and more postdoctoral training? Or did you slot right in a nod sagely with the wagging gray beards (From the Googs, actually I think Prof Benezra came along far too late to have a significant effect on the *start* of this process)?
To be clear, my queries go well beyond the personal situation of the petition writer himself. I generalize this to all of the experienced and comfortably funded (until recently) PIs who are signing the petition. Even if Professor Benezra has been a bleeding heart n00b PI supporter his whole career I think it safe to conclude that many of the petition signers have not.
My take in all of this is that I find the timing to be highly suspicious. I suspect that this crying over the fate of ESI under the A1-and-out rules to be disingenuous. I suspect quite strongly that what is really motivating people is the fear about their own grant sinecures. Their own expectation of continual competing renewals of their core projects. Their outrage that finally, finally, at long last, they themselves are facing the uncertainties and fears of losing grants that many folks of my approximate transition generation and below have had as a reality for our entire careers. My conclusion arises from talking to more than one established, continually well funded investigator much like Professor Benezra, not just random internet clownery, I will note.
Now, since I may perhaps have a few new readers due to this kerfuffle, I’ll repost a proposal that I made in the past. I do have a favored solution to the endless-revision-churning problem.

The ICs should set a “desired” funding target consistent with their historical performance, say 24% of applications, for each Council round. When they do not have enough budget to cover this many applications in a given round, they should roll the applications that missed the cut into the next round. Then starting the next Council round they should apportion some fraction of their grant pickups to the applications from the prior rounds that were sufficiently meritorious from a historical perspective. Perhaps half roll-over and half from the current round of submissions. That way, there would still be some room for really outstanding -01 apps to shoulder their way into funding
The great part is that essentially nothing would change. The A2 app that is funded is not going to result in scientific conduct that differs in any substantial way from the science that would have resulted from the A1/15%ile app being funded. New apps will not be any more disadvantaged by sharing the funding pie with prior rounds than they currently are facing revision-status-bias at the point of study section review.

What do you say? Shall we have a petition for this?
*I will admit to be a scallywag on this one, sorry peeps.

18 Responses to “Crocodile tears from experienced NIH investigators over the discontinued A2 revision”

  1. neurowoman Says:

    simple solution if the concern is for ESI/NI: bring back the A2 for only the newbies. That should separate the established PI woe-is-me vote from the genuine concern vote for ESIs.


  2. becca Says:

    By your own accounting, in the mid nineties this dude was between his F32 and his first R01. Is it too much to hope that maybe, just maybe, perhaps, this is the extraordinarily rare bird… a successful established person… who has actual memories of what it was really like when he was not successful?!!!! Who was not co-opted by the powers that be; converted by Stockholm syndrome style cognitive dissonance into believing a system in which oneself has succeeded must be a valid system; or corrupted by financial incentives capable of e.g. converting even the best-meaning public health advocate into a defense attorney for the tobacco companies via the almighty compelling power of the mortgage?
    Ok, so I feel like I just yelled “ZOMGUNICORN!” and now everyone is going to laugh at me. Hey, I’ve gotta have some hope for humanity every once in a while. You know I don’t suffer from an excess of optimism.
    But seriously DM. You do better when you propose actual change yourself rather than cynically cut down anyone else who suggests something about NIH as simply ‘bitter about their own funding prospects’. I just don’t see how this proposal helps this guy. So your suspicions of some sinister selfish motive for this guy seems… to not make any sense. Like CPP said, you can’t change the arithmetic.


  3. DrugMonkey Says:

    I just don’t see how this proposal helps this guy.
    Type2 grants, the competing continuation application, do tremendously better than new proposals. Even those from the self-same category of investigator. They are the very embodiment of the program-funding sinecure aspects of the NIH system. The definition of funding as a reward for past performance rather than the prospect of what is being proposed.
    [see graph at NIGMS blog]
    This guy has lived on at least three different projects that have been competitively renewed.
    But you have managed, somehow, to miss the part where these comments are not specifically about Benezra, he stood up as a proxy representative for all the PIs similar to himself who are doing this same whining about the A2. I talk to these types IRL with some frequency. Perhaps you should chat up your local successful midcareer peeps about their take on the A2 and special ESI paylines and see what they have to say.


  4. Lorax Says:

    Personally, I do not see how A2s help as CPP or DM noted on DrDra’s post. However, the difference between a funded grant and triaged grant is currently subjective (not all triaged proposals are just one tick from funding and would be still be triaged with paylines at the 30%). However, the current rule that an unfunded A1 can not be resubmitted is devastating to the entire enterprise.
    My anecdote: My first R01, funded in the mid 2000s, went from scored new submission, better scored A1, triaged A2 (program officer put it up for a merit award because they thought I got screwed in the review). Regardless, I submitted it as a new proposal again to the same study section and it was scored at the 4th %. This fundamentally the same proposal with some cosmetic work, but it went from unfunded to 4th %.
    Under the current rules, I could not have resubmitted this unfunded proposal and would now likely be asking DM if he wants fries with that when he stops by for a quick evening dinner.


  5. DrugMonkey Says:

    Regardless, I submitted it as a new proposal again to the same study section and it was scored at the 4th %. This fundamentally the same proposal with some cosmetic work, but it went from unfunded to 4th %.
    Um, you do realize that nothing has changed in this regard, right? You were also “not supposed” to submit a thinly disguised A3 as a new A0 back when A2 was permitted.
    This is why I have said on multiple occasions on the blogs that A1-and-out is not the end of the line for a given line of research. Just like when someone at the study section table would say “This is the last chance for this one..” in an attempt to sway the panel over the line, I would roll my eyes or frequently comment that that was wrong.
    Look, you may need to take a slightly different tack but you should be able to do this easily.


  6. becca Says:

    Yes, established programs-type2 grants- do better than new programs.
    (although that alone is not sufficient evidence that study sections are basing their decisions on past performance- though I’m sure that does occur. However, it is entirely possible that there is an *objective* loss of average productivity when moving to a new area. It’s also possible the opposite is the case and the whole system is fundamentally flawed with respect to supporting innovation. I’m certainly open to both hypotheses- though I suspect we’d need to agree on metrics of successful science before having a meaningful conversation about this)
    However, this has *no impact* on whether new programs face more or fewer obstacles than type2 grants under the old system vs. the new system…
    (type 2 success rate- new grant success rate) with the A2
    is either less than, equal to OR more than
    (type 2 success rate- new grant success rate) without the A2.
    It’ll be answerable with data in a little while. In the meantime, is your argument really that the type 2 success rate will actually *decrease* without the A2, and so this guy is in favor of that to protect his own type 2 submissions, all the while pretending he wants the opposite so that new investigators have a chance?


  7. DrugMonkey Says:

    there are Type 2s that are out in year 21, 26. multiple successful competing renewals. If you get busted out on the Yr -06, it kinda prevents any subsequent competing renewals of that project….
    This is what has these folks so scared.


  8. saban_lab Says:

    Nice commentary, and totaly agree w/ your view re: Sr PIs wanting to revive the A2 to help their own grants make it(particulalrly since competitive renewals are said to be even more difficult to get funded recently)
    That being said, and you have heard this from me before DM, it is very difficulty for n00b PI to write up a new R01 should their original A1 not be funded. It’s not a lack of “creativity” (as some have implied) which makes this difficult, but rather a lack of resources!
    I for example (a card-carrying n00b)only recently (Dec 2010) obtained my first postdoc and prior to that I was relying on data generated by medical students stints in my lab (or should i say, small lab space cut out from my boss’s lab). I am also at the lab bench, not mention writing up papers, new cmte admin work, and the list goes on. And this is just one example (personnel) of limited resources effect, I dont even want to mentions supplies (which including mouse purchasing, per diem, and husbandry) is pricy!!
    Im not complaining, b/c I love it. BUT, you caould easily imagine how it would be difficult fro me to generate a new R01 should my prev one not get funded!
    What if we give only ESI/NIs the option of A2, with the understang that they should not be “held” in queue?


  9. DrugMonkey Says:

    while I sort of understand what you are saying saban_lab, my experience has been that it is *possible* to write multiple distinct proposals that are supported by feasibility type preliminary data rather than specific type data. I’m polling over at Scientopia to try to figure out just how unique my experience has been.


  10. whimple Says:

    2R01s (renewals) score better than 1R01s (new proposals), nevertheless, the NIH funds twice as many 1R01s as it does 2R01s.
    I also have no idea why you felt it necessary to cast ad hominem aspersions at Benezra and to question his motives. If you disagree with Benezra, refute his arguments.
    As a data point, it is my impression that CSR is getting much stricter about administratively culling insufficiently new proposals after the -A1 goes unfunded. There’s nothing wrong with that: there’s no money to fund these proposals anyway.


  11. Pinko Punko Says:

    If proposal is better than say 25-30th percentile, I favor allowing an A2. The difference between 25 and 20 or 20 to 15 might be one vote on the panel. Why screw what is clearly a quality proposal with have to perform cosmetic surgery to change 40% or whatever the number is for the language of the proposal so you are in bounds on what a new proposal should be. Admit that there are quality proposals out there that can’t be funded if the paylines are at 12%. Perhaps this is similar to DM’s plan. Whimple is correct, your proposal goes through a software check to see how similar it is. It is more realistic to admit that top 25% proposals are likely worthy of consideration and an extra shot at funding.


  12. Neuro-conservative Says:

    Am I missing something?
    If I am reading these slides correctly (particularly the second slide), the total pool of investigators applying for R01’s increased by 50% since 1998, while the number of awardees has gone up by only 12.5% in the same period.
    That means we would need to attrit nearly a quarter of the current pool of applicants out of PI-dom, in order to return to the success rates seen in the pre-doubling era. (Assuming, of course, that increasing the total pie is off the table in the current climate).
    In the face of this kind of massive supply-demand imbalance, it seems like all of this other stuff about A2’s, study section behavior, etc etc, is entirely besides the point.


  13. CashMoney Says:

    Why is nobody proposing to drop the salary cap from $190K to, say, $110K? You would add one assistant prof slot for every senior PI currently on the books. And you wouldn’t really touch too many Associate level folks so they won’t whinge too much…


  14. Dude, you’re a paranoid wacko.


  15. arrzey Says:

    The cap won’t change because 1) Asst Prof MD’s make more than $110K (in some specialties, they are making more than than $200K). 2) There is no way any clinical dept will do cost sharing on salary to free up folks to do research. 3) Lowering the cap will drive MD’s out of the research game.
    We can debate the wisdom or rationality of MD’s making that much money, but the reality is that Uni/Research salaries for MD’s are still significantly less than private practice. Medical schools have a hard time getting young physicians to give up the $$ to do research. They will not let (to the best of their ability) NIH make it harder for them.


  16. Cashmoney Says:

    It isn’t just the MD scientists arrzey…but yeah, screw ’em. If MDs don’t want to do research that’s just fine by me.


  17. DrugMonkey Says:

    PinkoPunko- I like the idea of only some applications being permitted revisions, based on initial score. Tentatively anyway. Agreed it has a functional effect similar to what I am proposing so that’s good by me. I do wonder if a Jeremy Berg style data mining mission could do this *voluntarily*. The NIH occasionally generates the “eventual fate” funding success data from A0 to A2..something like this could be tweaked to show people that if you get triaged your odds of eventually winning funding are low. I think they retain prelim scores for triaged apps, or they should anyway. Be interesting to see the chances of funding and the chances of the PI resubmitting as shaped by the priority score of the A0.
    Neuro-con- yeah. thanks for the link. I’ve often wondered about these data but never run across them for some reason. The total funded pool of investigators is critical. And wow, what a steep rise in experienced investigators during the doubling. A steep rise that crept slowly back as the budgets got tighter. Damn, if I look at these graphs much longer I’m going to start down the CPP path of advocating that we need to shell out a large number of currently funded investigators to get back on the pre-doubling trendline.


  18. Neuro-conservative Says:

    DM — You and I have only just discovered these data, but NIH has obviously known about it for some time. Occam’s razor suggests that CPP is correct and that these considerations have been guiding NIH decision-making for several years.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: