Today in NIHGrant Special Flower Pleading
October 24, 2016
It started off with a tweet suggesting the NIH game is rigged (bigly) against a “solo theoretician”…
https://twitter.com/cryptogenomicon/status/789118235182526465
interesting. Then there was a perfectly valid observation about the way “productivity” is assessed without the all-important denominators of either people or grant funding:
https://twitter.com/cryptogenomicon/status/789124227706281984
good point. Then there was the reveal:
https://twitter.com/cryptogenomicon/status/789177870027329536
“It’s her first NIH application”.
HAHHAHHHAAA. AYFK? Are you new here? Yes. Noobs get hammered occasionally. They even get hammered with stock critique type of comments. But for goodness sake we cannot possible draw conclusions about whether “NIH grant review can handle a solo theoretician” from one bloody review!
This guy doubled down:
https://twitter.com/cryptogenomicon/status/789196033653780481
Right? A disappointing first grant review is going to “drive a talented theoretical physicist out of biology”. You can’t make this stuff up if you tried.
and tripled down:
https://twitter.com/cryptogenomicon/status/789208927904862209
See, it’s really, really special, this flower. And a given “line of critique” (aka, StockCritique of subfieldX or situationY) is totes only a problem in this one situation.
News friggin flash. The NIH grant getting game is not for the dilettante or the faint of heart. It takes work and it takes stamina. It takes a thick hide.
If you happen to get lucky with your first proposal, or if you bat higher than average in success rate, hey, bully for you. But this is not the average expected value across the breadth of the NIH.
And going around acting like you (or your buddies or mentees or departmentmates or collaborators) are special, and acting as though is a particular outrage and evidence of a broken system if you are not immediately awarded a grant on first try, well……it is kind of dickish.
There is a more important issue here and it is the mentoring of people that you wish to help become successful at winning NIH grant support. Especially when you know that what they do is perhaps a little outside of the mainstream for a given IC or any IC. Or for any study section that you are aware of.
In my opinion it is mentoring malpractice to stomp about agreeing that this shows the system is awful and that it will never fund them. Such a response actually encourages them to drop out because it makes the future seem hopeless. My opinion is that proper mentoring involves giving the noobs a realistic view of the system and a realistic view of how hard it is going to be to secure funding. And my view is that proper mentoring is encouraging them to take the right steps forward to enhance their chances. Read between the summary statement lines. Don’t get distracted with the StockCritiques that so infuriate you. Don’t use this one exemplar to go all nonlinear about the ErrorZ OF FACT and INCompETENtz reviewers and whatnot. Show the newcomer how to search RePORTER to find the closest funded stuff. Talk about study sections and FOA and Program Officers. Work the dang steps!
Potnia Theron was a lot nicer about this than I was.
That post also got me wandering back to an older post by boehninglab about being a Working Class Scientist. Which is an excellent read.
DrugMonkey 
October 24, 2016 at 6:50 pm
Girl Theoretical fizzycyst “I’d like to do this bunny hopping research using mathematical theory”
NIH reviewers: “AAHAHAHAHAHAHA NOPEITY NOPE NOPE. You are not productive, you’re all alone, and theory SUCKS”
Boy failed experimental fizzycyst “I’d like to do show WHAT MAKES LIFE using mathematical theory”
MRC reviewers “… uhm? you sure?”
Boy: “Naw, it’s cool. I know a guy. I’ma steal data from a girl”
MRC reviewers “OH WE’RE SO SORRY DR. CRICK” *THROWS MONEY AND NOBEL AT HIM*
#NotAllPhysicists
LikeLike
October 24, 2016 at 6:57 pm
DB’s post was/is excellent. That is all I have to say here.
LikeLike
October 24, 2016 at 7:23 pm
“Drive a talented theoretical physicist out of biology”?
To whom should I address the thank you note?
LikeLike
October 24, 2016 at 9:17 pm
I thought like this after my first few reviews, but it was my mentors that helped put things in perspective (DM’s point!). I quickly figured out that the grant game was extremely difficult and not for the faint of heart. Do I still bitch and moan when I get a review I do not agree with? Heck yeah! But, I have slowly come to understand, especially when getting stock critiques, that the reviewers were not that excited about the science to begin with which led them to be critical in other areas. In contrast, I have also come to understand that if reviewers really like your idea that they often let small things slide (limited publications for example).
LikeLike
October 24, 2016 at 10:48 pm
The seven stages of 1st grant grief
Shock – “What the f**k does “not discussed mean!?”
Denial – “But my grant was perfect, my grad students didn’t find any typos! This must be a mistake”
Bargaining – “I’m going to call the SRO and see if I can get one of those R56 my BSD post-doc mentor always seemed to be able to get.”
Guilt – “Am I really meant to be here if I can’t get a grant? I only have 4 years left until I’m up for tenure, which and if I have to bust my ass writing 1 grant a year I’ll never see my children.”
Anger – “Reviewer #3 has it out for me. Based on the study section roster, I bet it’s that guy Physioprof…. I’m going to find where he lives and TP his house”
Depression – “I wonder if McDonald’s is hiring…”
Acceptance – “OK, 8 months is long enough to mope. The next deadline is in two weeks, that’s plenty of time to revise and resubmit.”
LikeLike
October 24, 2016 at 10:54 pm
Yes, the game is rigged against a solo theoretician, but this is unavoidable: a paradigm change ALWAYS goes against the established views of the majority. By definition. Get used to this and to critiques that read like Alice in the Wonderland. N= over 50 in my case, over the last 10 years. Funding from NSF and a foreign source, and only some truly hilarious reading from NIH reviewers who need to write something, but simply can’t. That’s fine, I wouldn’t mind re-submitting and addressing legitimate issues, but here lies the problem: the garbage emerging from study section is too often not legitimate, can’t be addressed in the available space, and NIH lacks any viable appellate process.
LikeLike
October 25, 2016 at 12:07 am
I quickly figured out that the grant game was extremely difficult and not for the faint of heart.
This is true, but the question remains: is this really what we want to select for? I have a student right now who is a veritable genius and a kind person to boot. He started off wanting to be faculty, but, after seeing me do this job for several years, he now says he doesn’t want to be a professor, because the job is too hard; he doesn’t think he can take incessant grant writing and all the rejections and wants to get a job in industry. So this brilliant person will not remain in science. I am pretty sure he’s much more intelligent and talented than me, and if someone else were there to provide him with the infrastructure, I bet he could make great contributions to the field. Yet I am here and doing this job, and he won’t be. Does science really benefit more from my tenacity and grantsmanship (or, as a colleague would put it, the skill in turd polishing) than it would from the technical work of a very talented person? It is sickening that grants seem more and more like an end unto themselves, rather than a means to do science.
LikeLike
October 25, 2016 at 12:27 am
Uh, speaking as a theorist… maybe these are actually pretty relevant critiques? Theory, especially physics-originating theory, without strong coupling to biology can go off in not particularly productive directions. The theorists I know with NIH grants either have strong collaborations with experiment – i.e. established evidence that someone is going to commit lab time to testing that theory – or are working in molecular dynamics, with very specific technical goals motivated by experiment. This doesn’t seem too crazy, either – having to convince experimentalists that your theory is worth testing seems like a reasonable bar for NIH funding.
With somewhat less confidence, the publication stats also seem like a potentially fair issue. Usual caveats apply about counting papers and people in different fields/positions (and different quality papers!) – but I’m a physicist working on biology and 0.5 papers/year does not strike me as a good number for a theorist. When data acquisition is not the limiting factor, I see 2 papers/year for good established grads/PDs.
Maybe all of these concerns are wrong – maybe the grantwriter has a strong established collaboration, and their publication record is genuinely strong. But I could imagine writing that critique for some proposals, with good reason.
LikeLike
October 25, 2016 at 2:17 am
“Does science really benefit more from my tenacity and . . .”
Yes. Tenacity counts. Brilliance isn’t enough.
LikeLike
October 25, 2016 at 9:30 am
@Emad: “But, I have slowly come to understand, especially when getting stock critiques, that the reviewers were not that excited about the science to begin with which led them to be critical in other areas. In contrast, I have also come to understand that if reviewers really like your idea that they often let small things slide (limited publications for example).”
This is EXACTLY correct. Stock critiques never mean exactly what they say. They mean, “I wasn’t as excited by your grant as I was by the one I read just before it, but that’s not enough of a criticism to sink it, so instead I’m going to point out your average productivity, your lack of preliminary data for Aim 3, Experiment 4, Part C, and the fact that Experiment 4C is ever so slightly dependent on Experiment 2B.”
LikeLike
October 25, 2016 at 9:31 am
@xyk: “Does science really benefit more from my tenacity and grantsmanship (or, as a colleague would put it, the skill in turd polishing) than it would from the technical work of a very talented person?”
Do, it does not, and that is why the system needs to be changed in a radical way.
LikeLike
October 25, 2016 at 9:50 am
My mentors have always commiserated with me when I’ve received unreasonable critiques. But they would have laughed me out of the room if I told them I was thinking about changing careers after one failed attempt.
What I think is pervasive in NIH study sections is the “holding pattern” mentality. I’ve had people from my study section makes comments to me like, “Oh, you’ve been a PI for 3 years now. You’re almost ready for an R01.” I’ve also seen my colleagues have funding dry spells for a year or two, and then have the same old grant funded that wasn’t discussed in the past. For newbie PIs, unless you get really lucky, my (anecdotal) experience suggests that the most likely scenario for a first R01 is that you need to submit a credible series of applications so the study section gets to know you, but that you won’t be funded until you have a combination of both a really good application and a perceived need for funding (or looming tenure deadline).
LikeLike
October 25, 2016 at 10:16 am
From one my favorite (self) posts. The four questions at Passover: https://drtheron.wordpress.com/2014/02/26/triaged-grants-and-a-passover-lesson/
The wise child asks: “What is the meaning of … the rules and guidelines, written and unwritten, the things we should do to get funded?”
The wicked or selfish child asks: “These guidelines are fine for you, but they don’t apply to me”.
The simple child asks: “What should I do?”
This final child is incapable of asking a question.
The standard answers to the questions are also instructive:
To the wise child: We should instruct this child in all the laws and customs of Passover grant writing.
To the wicked/selfish child: It is obvious that this child does not want to be included in the celebration or the community, so we answer harshly, “We offer help in writing grants because someone once helped us and we pay it forward. If you had been in Egypt, you would not have been thought worthy to be funded.” [note: this is possibly one of my favorite responses, simply because I would love to say this to various trainees. But you don’t have to. They won’t get funded.]
We answer the simple trainee with: Here is how you start….
Because the fourth child doesn’t have the capacity to ask a question, we must explain that writing a grant is a difficult process, and here is how it works.
LikeLike
October 25, 2016 at 10:57 am
I agree wholeheartedly that the system is flawed and changes are desperately needed. Please, people who have more influence than I… please, please advocate with us for updating the process!!
BUT.
For those of us who have to work within the system that we have now on our way to tenure, @drugmonkey is 100% right. AYFK? The subject in question might “leave biology” for good based on a FIRST review?!? (My indignant snort should be implied at this point).
In my field, I do not write NIH grants. I do, however, write for NSF, USDA, and EPA. I am expected to write a minimum of 3 – 5 grant proposals per year during my pre-tenure term. I am expected to interact with program directors at all of these organizations, attend workshops at my institution dedicated to improving my grant writing ability, and I am expected to take advantage of any opportunity to learn something new about grant writing and apply what I’ve learned to the next proposal. Everyone involved with the grant writing process at my institution knows that it is a constant struggle for improvement. Just because we know it’s terrible, that doesn’t mean that we don’t still fight the battle.
It is TOUGH to become consistently successful at grant writing. It takes YEARS of trial and error. ONE REJECTION might send this theoretical investigator out of the field???
Bye, then.
LikeLike
October 25, 2016 at 12:42 pm
If ONE rejection is going to drive this person out, they’re in the wrong field. (If you can’t take the heat….get out of the furnace room.) Tenacity is absolutely crucial in science. What are you going to do if your reactions don’t work the way they are “supposed” to? Quit? Some projects take years of banging your head against the wall and your hammer against equipment before they start to work. Tenacity is something you’re supposed to learn in grad school. (This points out one disadvantage of being in a BSD group and getting handed a hot project that’s had most of the kinks worked out so glamour pubs rain from the sky. You never have to learn how to deal with failure and rejection.)
LikeLike
October 25, 2016 at 2:05 pm
@Grumble and Emad:
Isn’t there a “significance” or “innovation” section on which reviewers log a score? If the experimental aspect of the project is fine, but the significance of the expected findings is ~meh, don’t you think that alone is sufficient to sink it? Why bother inserting bullshit and petty remarks that the applicant could waste energy agonizing about instead of just getting to the point and letting them know that the expected findings of the project aren’t worth the trouble? Go back and try again with something different, Holmes. Otherwise, the reviewers are likely to see the same old shit re-worked based on the petty stock, meaningless critiques. They still won’t like it and they will have wasted a year or more of the applicant’s life, not to mention their own time. It baffles me that reviewers just can’t get to the point and say specifically why they don’t like the project. It’s something I try to do when I write reviews for these things (I’ve reviewed for the NSF).
LikeLike
October 25, 2016 at 3:36 pm
“If the experimental aspect of the project is fine, but the significance of the expected findings is ~meh, don’t you think that alone is sufficient to sink it?”
Put yourself in the reviewer’s shoes.
Let’s say you think it’s actually a pretty reasonable project, but of the 10 you reviewed, it comes in at around #3. You’re really excited by 1 and 2, and you know that only the top 10-15% are going to get funded. You also know that you have very little direct impact on the scores of any grant other than those you are assigned to review. How do you make sure that 1 and 2 get the best scores possible?
Hint: it involves making sure that #3 and worse die and stay dead.
LikeLike
October 25, 2016 at 4:33 pm
@ xykademiqz
I think that is unfortunate, but is is the reality (and happens in other fields). People that seemed destine to thrive in such an environment take a different path. I have a friend that was a natural, the first person I have ever met where I said, “100% you are going to be a successful PI.” But, he left his post-doc after one year and now works in the clinical research arm of a large hospital chain. And he loves it and is good at it. So, that superior intellect can be used in other areas to thrive as well. He could have cared less about impact factor and writing grants over and over.
I hope all universities are coming around to this thinking that becoming a professor is not the only path. I have noticed over the last few years that the graduate and post-doc support offices are offering a lot more sessions/workshops/lectures on careers outside of academia. That being said, we will be losing some fantastic people, but there are also plenty still waiting that are struggling and maybe will not make it either.
@ eeke
I also had to figure that out the hard way. I thought I was so clever at addressing every little stock critique and small thing that they picked on, but upon resubmission, my score did not improve substantially. I was left dumbfounded and pissed off. Maybe reviewers should be more honest, but the brutality of their comments in the current system is already tough to swallow!
LikeLike
October 25, 2016 at 10:11 pm
@ xykademiqz
My guess is that most people in science would rather just do science than “be resilient”. But that’s not really an option in life. We all want someone else to do the infrastructure so we can do the special thing we can do.
“It is sickening that grants seem more and more like an end unto themselves, rather than a means to do science.”
Grants are not an end unto themselves. We all know that. Just like money doesn’t buy happiness. But, rich or poor, it’s always better to have money. Now, kidding aside, grants give you options. Grants let you do things that are nice, like paying the salary and have the infrastructure for someone who is brilliant and working on a cool project.
xykademiqz, your grants are also ways to support brilliant students. If that’s what you are good at, why not do that?
LikeLike
October 26, 2016 at 11:18 am
@xyk: “Does science really benefit more from my tenacity and grantsmanship (or, as a colleague would put it, the skill in turd polishing) than it would from the technical work of a very talented person?”
I think you are selling those skills (tenacity and grantsmanship) way short here. They are reflective of a lot of the mindset and knowledge needed to be a successful scientist, even independent of the grant game. I think we all know people that are much smarter than us that we run circles around scientifically (although, maybe it’s just me and I am a dull-witted imposter!).
I think back to an interview with the poker player David Sklansky where he went on about how he’s so brilliant that, if he was in physics or such, he’d probably have a Nobel Prize. He is making the common mistake of thinking that the limiting factor in scientific progress is intelligence. It’s so much more than that. A certain level of intelligence is a threshold competency, but so much more is needed.
To quote the late James Gandolfini in Zero Dark Thirty: “We’re all smart.”
I’m rambling… back to the grant!!! 🙂
LikeLike
October 26, 2016 at 11:27 am
Sam-
A refusal to understand that “we are all smart” creeps out of the special flower syndrome to my ear. People don’t seem to understand we have 3-4 times more excellent proposals than can be funded. They seem to feel it obvious that the app they like surely must be in the top 10%, and it is only error in the system that this hasn’t been recognized. I don’t see this as being true, personally.
LikeLike
October 26, 2016 at 12:15 pm
@Grumble – the funding decision is ultimately NOT yours. It is that of the program officers or other officials (this is definitely the case at NSF). There are grant applications that you don’t even see that also get good scores that could end up flushing even your most favorite application. If the significance of a project is so-so, my point was to sink it for that reason (even if it is your 3rd favorite) and to steer away from the petty bullshit comments that torture applicants. These days, it seems easier to sink applications than to save them. I’ve seen on review panels that it’s quite a skill to pull an application out of the trash can and place them within a respectable range of fund-ishness.
Let’s say that the applicant doesn’t agree that the significance is so-so. If they can figure out that’s what you didn’t like about it, it’s on them to improve the significance statement and make a stronger case for why the project is important. I’d prefer that line of attack than distracting them with irrelevant reviewer statements.
LikeLike
October 26, 2016 at 2:33 pm
I am a new commenter, but have followed this blog for years. It might come as a surprise to some that all of the problems and difficulties that are discussed here, are agonized over at CSR as well. Solutions, however, are not easy to find. Reviewers are human, paylines are low, outstanding grants are not all funded, and brilliant geniuses get frustrated by the reviews of those they consider inferior. I once had an applicant explain to me that there was no one in the world who was competent to review his proposal, because it was so innovative. I asked what he suggest we do, and he said, “Well I could review it myself. I am the only one competent to do so”. I politely declined his suggestion.
What people should know, especially DM and those who comment here, is that the folks at NIH are not deaf to the community. New ideas and approaches and suggestions are welcome, and are sometimes tried and tested. Everyone wants the same thing, a vibrant, creative and successful research enterprise.
LikeLike
October 28, 2016 at 9:12 am
A refusal to understand that “we are all smart” creeps out of the special flower syndrome to my ear.
I agree 100%. Special snowflakes almost always completely ignore any evidence that would make them less special. It is the root of their problem. I definitely know at least one theoretical physicist who is NIH funded.
LikeLike
October 28, 2016 at 10:33 am
@eeke – I know all that. I’m talking about psychology. Although I framed it as a rational choice (“I’m not excited by this grant, therefore I will load my review with niggling Stock Criticisms that are irrelevant to the main problem in order to make sure the score is unfundable no matter what”), in fact I think that logical process is mostly subconscious. You read an application that just doesn’t float your boat, so you say bad things about it. What could be more natural and effortless than that?
Of course, your point is that good reviewers see through their own psychology and make sure to provide applicants with a valid critique. However, when I have a stack of 10 grants to read, I find it very difficult to be a Good Reviewer on all of them. And from my experience, I’m not the only one in this boat.
LikeLike
October 28, 2016 at 11:01 am
XCSR-
Some people in CSR, I assume, are good people.
LikeLike
October 28, 2016 at 3:16 pm
I thought NIH had a mechanism for “talented theoretical physicists” something like this: http://grants.nih.gov/grants/guide/pa-files/PA-16-194.html
I sometimes think half the battle for new investigators is finding the right study section/mechanism. I even know BSDs who got their app thrown into the wrong study section and get triaged. It’s just they know how to redirect the app to the correct study section on resubmission.
LikeLike
October 28, 2016 at 3:25 pm
I think Grumble nailed an important point: “You read an application that just doesn’t float your boat, so…”
We’re all most excited about whatever we’re doing, and next most excited about whatever is most like whatever we’re doing. If it’s not like what we’re doing or counters assumptions that we rely on or even just looks like it might divert resources from things that we like… then we don’t like it.
This is a problem when panel subject areas are narrowly focused, because novelty will be penalized. Panel review reinforces the status quo. Same old same old. The rich get richer. Etc. But narrowly-focused panels also maximize expertise needed to judge feasibility. Obviously we want proposals judged by experts.
So primary reviewers should be a mix of expert and non-expert? But non-experts fail when it comes to the discussion; expert opinions prevail, and everyone votes for the status quo.
Should NIH have two completely independent levels of review? One level for ‘excitement’ and another for feasibility? Or should program officers be even more encouraged to go outside the panel rankings to ensure that novelty gets funded?
We keep assuming that there is a perfect way to review proposals. But figuring out the perfect way requires that we measure whether a proposed project was successful. And those experiments are flawed, because we’d need to compare crappy proposals and good proposals, and crappy proposals don’t get funded (and funding rates are so tight lately that we can’t even distinguish between excellent proposals and merely very good proposals). And what are the best measures of scientific success anyway? Citations per project? Citations per dollar? Trainees per project? Patents? I’m not convinced that we even agree on what we’re arguing about.
LikeLike
October 28, 2016 at 4:21 pm
@ UCProf
The K25 may be a great option for that person–it is a career development award for someone with a physical sciences/engineering background to get into biomedical research (I had one that just ended last year). But even if this person is eligible and would get it, the challenge is the next step–where do you send your R?
I have had similar problems with the interdisciplinary work that I do (cell/molecular biology and engineering), and I have talked to others with similar experiences. There are not many standing study sections dealing with bioengineering. And in my limited experience with one, I felt as though some of the biological novelty of the proposal was overlooked (but was discussed and scored twice). Similarly, when I sent the grant in as a new R01 and picked a biological/disease-focused study section, I felt that some of the engineering novelty was overlooked (not discussed or scored either time). Regardless, it takes a lot of careful writing to get your point across and make it understandable. I do feel that there would be some benefit to additional study sections that are similar to NSF ones in topic.
That being said, the original point of this post was the reaction to the reaction of a new investigator’s summary statement. We have all been there. The reasons may be different, but the outcome is shared across all disciplines–why didn’t they see the brilliance of my proposal? The PI is just going to have to learn the hard way as all of us did. After all, my grant was eventually funded by a disease-focused society (and reviewed by a bunch of cell biologists). So maybe I ended up learning something from the multiple federal and non-federal submissions I put in over several years . . .
LikeLike
October 30, 2016 at 5:13 pm
How do we know that American science is failing? Because most scientists are trying to figure out how to get funded rather than how to do great science.
LikeLike