GrantRant VI
January 8, 2013
If you had to put an explicit weighting on the relative influence of Investigator/Environment/TrackRecord versus Idea/Plan/Significance/Innovation in evaluating grant proposals, what would it be?
50/50?
20/80?
95/5?
-feel free to elaborate on your career stage as you stake your flag. 🙂
DrugMonkey 
January 8, 2013 at 8:33 am
Based on scores I received, 95% approach, 5% everything else. How else to explain scores of 4 for approach plus 1 & 2 for everything else, but an overall score close to 5? Still, at least the approach is fixable. There ain’t much to be done in the space of 3 months if’n you score a 7 for investigator.
LikeLike
January 8, 2013 at 8:54 am
I would say 95% approach 5% everything else. Otherwise people should just quit science if they ever hit a bump in their career or personal life. I think that history is being used to trash potentially great scientists all the time.
Plus on the flip side – just because you had one great idea doesn’t mean they are all gems 😉
I recently left academia for biotech after completing a 2 year postdoc. I compared my accomplishments to those of other around me an figured I had zero hope of grant funding in this climate, so I changed careers. No regrets so far.
LikeLike
January 8, 2013 at 9:11 am
“How else to explain scores of 4 for approach plus 1 & 2 for everything else, but an overall score close to 5? ”
That’s the answer to DM’s question – add up the scores!
More seriously, the answer depends on how much I trust the investigator to be productive. People rarely give “Investigator” scores greater than 3, because come on, that’s just insulting. That means that the Nobel laureate with 50 papers per year gets the same score (1) as the reasonably productive assistant professor. But I might still weight that score of 1 a lot more in my final score for the Nobelist than the assistant prof.
LikeLike
January 8, 2013 at 9:21 am
I feel like Investigator/Environment can pretty much only work for you, not against you. In other words, I think it would be pretty unusual for an excellent Research Plan to go unfunded because a researcher has less of a track record or is at a non-R1 Uni, but a shmancy pants PI may see a bump of good faith on a less than stellar proposal.
LikeLike
January 8, 2013 at 9:38 am
For most R01s, I would go 20/80. For large project grants, 50/50.
Mid-career, not on tenure track, relatively stable funding.
LikeLike
January 8, 2013 at 10:11 am
The summary statement words imply what subscore is driving the overall score, though some scores will be dragged down for another low score. Anyhow, Approach is dominant, but in the era of death by thousands of cuts, if your approach is a great score and investigator is low, I would guess that investigator would for that grant be driving the overall score more than is usual. I think Investigator counts more for competing renewals than new grants, though it always counts. I think a lot of reviewers put everything into approach because they don’t want to give a low investigator score. I tend to start at a grant with the following: Approach followed closely by Significance then Investigator. Bonus for really innovative, but this really would matter most with a good approach (for example- if ideas are cool and the approach is a mess, then that will have to be dominant). It is a multidimensional puzzle. Environment is weird. I think this is more likely to matter for younger investigators- how well they make a case for maximizing their environment. With funding as tight as it is, I am not going to give too much credit for a poor or less good grant from a killer lab. They can make that decision in council that it is worth keeping in the portfolio or not. If grants are funded like legacy applicants to selective colleges, there will eventually be no spots for anyone else, given that dollars are shrinking. Obviously, it has to be taken into account, but I really try to focus on what is in the document. In a system that is becoming somewhat arbitrary, this is one place to try to be as fair as possible. However, the argument for the other side is that funding decisions are bets, and some labs are likely safe bets, no matter how lazy they are in warmups, but this isn’t really scientific review of the document.
LikeLike
January 8, 2013 at 10:38 am
I think it varies WIDELY depending on the person and the project. I’ve seen R01 renewals skate by with 1’s because the PI has a fabulous track record and published 30 papers in CNS in the previous funding period. On the other hand, I’ve seen R15s from small labs get 1’s because they had a really great research plan. And I’ve seen pretty intriguing research plans score low because it wasn’t clear that the PI had access to the facilities/expertise necessary to do the work. I don’t think it’s possible to devise a set breakdown.
LikeLike
January 8, 2013 at 10:47 am
40/60 PI/plan
If the PI does great stuff, I’m not so worried about minor problems with the approach. She/he will do great stuff with the funds.
If the topic is insignificant, me-too science, or just a bad idea, I’m not scoring it well no matter who proposed it.
-tenured assoc prof with 2 r01s and an r21.
LikeLike
January 8, 2013 at 12:44 pm
What if it is a brand new researcher, a fresh hire at the state school, with a brilliant idea and the means (equipment, experience in proposed techniques, ect.) of conducting the proposed experiments. Should the “Investigator/Environment/TrackRecord” play a role in the funding decision? Investigator: fresh = 2 or 3? Environment = 3 just because it is not UCLA or Duke? TrackRecord = 2 or 3 cause its not a 100 pubs yet? But does that mean that the proposed experiments cannot be properly executed at the state school lab of a young investigator with great understanding of the problem and a creative solution. Does that mean that this young investigator cannot greatly contribute to the science in a way that will potentially benefit the taxpayer/society/youpickthebeneficiary.
0/100% – just because the way “Investigator/Environment/TrackRecord” is currently scored (e.g., state school/small lab = 3 [why?]) just does not seem fair
LikeLike
January 8, 2013 at 3:42 pm
I think overall I view the approach as key (and it seems others do too), with the other factors being operators/modifiers/sensitizers/multipliers, choose your term. Thus, a crappy project with a crappy PI suffers a multiplier effect. The same project from a good PI will do better. As has been said far more elegantly here than I can express, there is no fixed scale, and the weighting will shift depending on the modifiers.
LikeLike
January 8, 2013 at 3:43 pm
I have absolutely seen (and given) Investigator scores of 5, 6, and even 7.
LikeLike
January 8, 2013 at 4:27 pm
Well, CPP, it’s well known that you’re a dicke.
LikeLike
January 8, 2013 at 5:37 pm
small_fish, a three could be given for environment and have no meaning on the overall score. You can’t worry about that, well you worry about it by larding up with supporting letters about all the things you are doing to try to guarantee success, so you get mitigating language. “Institution is OK, but PI has critical collaborators in place to overcome perceived weaknesses, environment is adequate to good”= even though those words could equal a 3 on environment, it just won’t matter if everything else is in place.
LikeLike
January 8, 2013 at 5:49 pm
[…] to an exchange with PhysioProf after this comment, I dug up the summary statement for the first R01 proposal I ever submitted to the NIH as a PI. I […]
LikeLike
January 9, 2013 at 7:03 am
I just dug up my first R01 summary statement. One of the comments in the “Investigator” section stated that since most of the tools were developed in the post-doc lab I should have a letter from post-doc mentor showing support and non-overlap of the studies. Thus, I am with Dr. Becca on this one! Another thing: I have had a few good summary statements and more than a few train wrecks. Of the particularly bad ones the worst investigator score I have gotten is a “3”.
LikeLike
January 9, 2013 at 10:38 am
I heard of someone at Harvard Medical School getting a 2 on environment. Basically, they hadn’t taken the Facilities and Other Resources page seriously at all.
LikeLike
January 10, 2013 at 2:29 pm
I have seen the description of the environment as outstanding, everything there what is needed – and then a score of 3… Little disconnect.
LikeLike
January 10, 2013 at 5:35 pm
I have seen pink sheets from grants written by colleagues at “1” environments that got 1s and 3s for environment in the same review. The reviews giving the “3” were clearly not rating environment per se, but lack of approval of the application for other reasons. Which I guess is fine, if a 3 on environment serves as nature’s warning sign for “disgruntled”.
LikeLike
January 10, 2013 at 6:06 pm
Why do you think every reviewer sees the environment for each PI and each proposal the same way? I could easily propose a project for which my environment score should be north of 5…..
LikeLike
January 10, 2013 at 7:06 pm
I think people are misunderstanding what environment means. Just because you are at Harvard Medical School doesn’t mean you should automatically get a 1 on environment. Environment has little to do with the prestige of your institution, and much more to do with the availability to you of the facilities, resources, and colleagues required specifically to perform the studies you have proposed.
Suppose you are proposing research on arctic tern toenail regeneration as a model for male pattern baldness, but terns can only live in the arctic and can only be kept in captivity in outside cages in the arctic: University of Alaska = Environment 1. Harvard = Environment 9.
And the idea that it is some kind of shocking flaw in peer review for one reviewer to give a 1 for environment (or any other criterion) and another to give a 3 is gobsmackingly absurd. I’ve heard a lot of wacky shitte ideas about study section from naifs before, but this is a new one.
LikeLike