There is no secret information that your NIH Program Officer is withholding from you
July 18, 2012
Chill.
Really, my friends, you are going to drive yourselves completely insane if you don’t step back and wait.
The examples can be found over on writedit’s mondo thread on paylines, scores and whatnot for NIH grants. Start here and page up.
I find it alternately saddening, maddening and hilarious to read these comments. In the end, I do sympathize. After all, I’m in the same boat as all of the rest of you seekers of NIH largesse.
It boils down to this. Nobody knows with any precision if your grant will get funded, unless it is well within any conceivable payline. Right now, I’d say you can have confidence with 3%ile. The POs don’t know how the funds will be spent around the margins. If it is for the Dec 1 funding date, well, we’ve talked before about how this depends on what Congress does and the relative conservativeness of a given IC in the face of a continuing resolution budget.
So believe your POs. When they give you a space of probability to work with, they are not lying to you or hiding some sekrit information just to mess with little old you. You are NOT this important. More critically, some random Internet yahoo like YHN or writedit cannot improve on your PO’s information. We deal in generalities and those ICs of our closest experience. With random Internet anecdata. These sources of information are far less specific than your PO.
So do yourself a favor and stop trying to game out your chances. Just get back to writing your next application. It will serve you better in the long run.
DrugMonkey 
July 18, 2012 at 8:22 am
You had me right up until “Just get back to writing your next application. It will serve you better in the long run”
I’m not sure that last sentence is true any more! Of course there’s the approach which says “the only grant you’re guaranteed not to get is the one you don’t apply for”, but even so I’m not convinced writing more grants necessarily leads to more hits, or that grant writing in-and-of-itself is a particularly worthwhile pass time which serves you well in the long run (i.e., over and above it being a necessity). In-fact, the overall NIH figures show exactly the opposite: more applications=lower success rate.
Sure, if its your only RO1, get right back and start drafting the next iteration, but if it’s the 8th grant you’ve written this year and none of them got funded so far, going back to writing is probably a bad idea – you’re doing something wrong. Similarly if it’s your 3rd RO1 and the other 2 both got funded, give someone else a chance you greedy fuck.
But yeah, I agree, the PO’s are clueless, not even worth the ‘phone call.
LikeLike
July 18, 2012 at 8:55 am
if it’s the 8th grant you’ve written this year and none of them got funded so far, going back to writing is probably a bad idea – you’re doing something wrong.
If the consensus payline is 10%ile (and under) then why would anyone start with the assumption that they needed to submit anything fewer than 10 applications to have a shot at getting one?
This is basically saying that you are relying on some factor that makes you better than everyone else. Are you?
LikeLike
July 18, 2012 at 8:57 am
the PO’s are clueless, not even worth the ‘phone call.
This is not my message at all and this is a really stupid way to view the system. They have information that the PI does not have and can transmit it to the PI in a way that can at times be helpful.
The fact that they do not know rock-solid certainties about whether your grant will be funded the second your priority score emerges from the study section doesn’t change this.
LikeLike
July 18, 2012 at 9:00 am
Similarly if it’s your 3rd RO1 and the other 2 both got funded, give someone else a chance you greedy fuck.
Why do you say this? Why is it any “greedier” for an excellent lab doing top notch work to seek a third grant than it is for a crappy lab doing pedestrian work to seek their first grant?
Or for a lab to seek a third grant doing new stuff not otherwise funded by the NIH versus a lab seeking to do stuff that falls into an already generous portfolio of the IC in question?
How do you come up with your internal calculus for who “deserves” a grant and who is “greedy”?
LikeLike
July 18, 2012 at 9:00 am
The funny thing is that you’d think of all people, scientists would be the best able to deal with uncertainty and the unknown, given that their entire job is to explore the unknown and uncertain.
LikeLike
July 18, 2012 at 9:03 am
Scientists are people too and I’m fairly certain that people dont like uncertainty and the unknown when it comes to their future paychecks. But maybe thats just me.
LikeLike
July 18, 2012 at 9:04 am
How do you come up with your internal calculus for who “deserves” a grant and who is “greedy”?
Dude, how fucken stupid are you? *I* deserve a grant, and those other motherfuckers are greedy.
LikeLike
July 18, 2012 at 9:05 am
The funny thing is that you’d think of all people, scientists would be the best able to deal with uncertainty and the unknown, given that their entire job is to explore the unknown and uncertain.
Dear Sweet Jesus on a Dinosaur, this.
LikeLike
July 18, 2012 at 9:08 am
One of my friends wrote a shit-ton of grants and they all got funded. He’s PI on five RO-1s now. He seems a little freaked out about it.
LikeLike
July 18, 2012 at 9:09 am
CPP- their entire job is to make the unknown and uncertain MORE known and more certain. Not to ’embrace the mystery’ like some spiritual kook.
Also, what PinkGlitteryBrain said.
LikeLike
July 18, 2012 at 10:17 am
becca and Glittery…
Are you suggesting it is good advice to just follow whatever impulses are prompted by our anxieties?
LikeLike
July 18, 2012 at 10:47 am
DM- I am less effective as a scientist because I am *distracted* by not knowing if my kid is going to be covered by health insurance in a month’s time. If you aren’t in a similar situation, perhaps you ought not judge my reactions to my anxieties.
LikeLike
July 18, 2012 at 10:51 am
And if you are not responsible for the employment and health insurance of many people, including their children, perhaps you should not jump so quickly upon your high horse, becca.
LikeLike
July 18, 2012 at 10:54 am
Getting back to DickeMonkey’s post, the point isn’t that people shouldn’t *feel* anxious about their grants. It’s what they should *do* in the face of that anxiety.
LikeLike
July 18, 2012 at 11:17 am
“Getting back to DickeMonkey’s post, the point isn’t that people shouldn’t *feel* anxious about their grants. It’s what they should *do* in the face of that anxiety.”
The way I see it, you have a few options:
1) Shoot yourself in the face
2) Forget about the grant game for a while and try and generate more data and, of course, more papers
3) Revise and re-submit immediately anticipating rejection
4) Try and get money from elsewhere, if possible (i.e. pharma)
For us younger guys, the balance between (3) and (2) is critical.
LikeLike
July 18, 2012 at 12:06 pm
@DM, regarding the “crap shoot” approach of writing grants (i.e., write 10 = get one), I have just never been a big fan of this approach. Sure, there are some who enjoy staying at work until midnight for large chunks of the year. For me, I can do that for a few weeks before a grant deadline maybe about 3 times a year. Being married to another scientist, the toll taken on family life by doing this a combined 6 times a year is pretty heavy. I dread to think what would happen to our kids if we upped that to 20!?
Given that I’m only going to submit 3 grants, and the payout is (optimistically) 1 in 10, yes I’m assuming (hoping) that my grants are better than everyone else’s. I thought that was a given for everyone in this game – aren’t we all from Lake Wobegone, where all the children are above average? If you’re coming from a starting point of knowing that probably 8 out of those 10 grants you submitted last year were complete crap anyway, how do you justify all the effort? I’m sorry, but “that’s what’s needed to be in the game” is not a convincing enough argument for me at least.
LikeLike
July 18, 2012 at 12:17 pm
If you’re coming from a starting point of knowing that probably 8 out of those 10 grants you submitted last year were complete crap anyway,
When did I ever say that?
Thinking that the 80% of grant applications that you are up against are “complete crap” may be comforting but it is entirely discordant with the grant review experiences that I have had. It is my belief that operating with a false understanding of reality is an inferior way of succeeding in the real world in which we operate. I could be wrong on that though…..
LikeLike
July 18, 2012 at 3:44 pm
What’s this up until midnight writing for 3 straight weeks? Here’s my schedule…kids to bed at 830pm, write for an hour, personal time before bed at 11pm. Every night. 6-9 grants in per year – most are done early and 1-2 get funded per year.
LikeLike
July 18, 2012 at 5:01 pm
OK DM, so now I get it… this is a dick swinging competition. You’re seriously of the opinion that every single last detail of the 10 grants you submit, is on a par with a grant from someone like me who writes on a 3x slower timetable? Wow, those are some real chestnuts! I guess some people really are just fundamentally 3x more productive than others. Who knew?
Putting a nautical viewpoint on things… There’s an unwritten rule in sailing – when going upwind, you have a choice to tack a lot, which is a pain in the ass but gets you there eventually, or you can go out on a long reach and only tack once, which is less effort and faster, but then you run the risk the wind changes direction while you’re way out, and you get screwed. I was always of the “go out there and risk it persuasion”, so I guess I take that attitude to grant writing too.
I can see how people are attracted to the other model too (both in sailing and science), but it just doesn’t appeal to me. Does it make me less of a real scientist? Are my opinions somehow less valid because I choose to produce 3 Ferraris a year instead of 10 Kias? I get it, there are people out there who claim to be capable of producing 10 Ferraris. Hats off to you, captain!
LikeLike
July 18, 2012 at 5:33 pm
Being new at all of this I have been fretting endlessly about the debt ceiling, budget, etc. and based on “water cooler” talk feeling out of the loop relative to the older folks around. I really appreciate the advice to concentrate on what we can control (get to work on something else once I’ve done what I can with my PO) – guess I’ve been putting a lot of energy into figuring it out just because I didn’t want to have my head in the sand – but hearing people say there’s no real special info out there and just move on to more science is very heartening! That’s what I want to do anyway! So thanks!
LikeLike
July 18, 2012 at 6:38 pm
You’re seriously of the opinion that every single last detail of the 10 grants you submit, is on a par with a grant from someone like me who writes on a 3x slower timetable?
No. I am of the opinion that I have never, ever submitted a grant application that is “complete crap”. Nor do I ever intend to do so.
I am also of the opinion that, roughly speaking, about half to 60% of applications are pretty damn good and have at least something about them that fully justifies a fundable score. Consequently, the things that make the difference between funded and not funded rarely come down to meticulous preparation wrt “ever single last detail”.
Does it make me less of a real scientist? Are my opinions somehow less valid because I choose to produce 3 Ferraris a year instead of 10 Kias?
Not at all. You are the one that brought up the dick-swinging bit. What I am expressing is my opinion about the way to succeed in the NIH grant game given my understanding of the quality of the applications and the relative value of meticulous preparation in driving any given application over the line. You are welcome to your opinion and you are hardly alone in advising that the way to succeed is to “write better grants”. I just happen to think that is insanity. I have a few personal examples of proposals that received awesome percentiles (and were funded) and ones that were not even close (but not triaged) where an honest viewer would be hard pressed to show the awesome ones were in fact better applications. Perhaps people who really do submit “complete crap” applications would have a different experience…
LikeLike
July 18, 2012 at 6:42 pm
guess I’ve been putting a lot of energy into figuring it out just because I didn’t want to have my head in the sand
I did not mean to imply that the noobs shouldn’t inform themselves about how the system works. not at all. this should be obvious from my rather extended history of blogging about this career path over the years.
My thoughts here arise out of a frustration that even when people are told the fuzzy limits, they somehow don’t get that this is as good of a prediction as they are going to extract out of the system and it is a waste of time to expect anything else.
Sometimes, I will admit, it sounds to me as if people are understanding the imprecision but somehow expect that they are so special that they deserve more. You have made me realize that from the noob point of view, it may sound as if the more experienced types have some special, extra knowledge that you just haven’t figured out how to access yet.
LikeLike
July 19, 2012 at 2:57 am
If the consensus payline is 10%ile (and under) then why would anyone start with the assumption that they needed to submit anything fewer than 10 applications to have a shot at getting one?
Um.. because study section can reliably differentiate between the top X% and the rest? Whether we feel X=25 or X=50, this means that you need to get into that top X% and then write enough to hit the payline. So, with payline=10% and X=30%, you need to write three grants. If you’re writing 8 grants and none of them are getting funded, the most likely reason is that you are doing something wrong and are in the not-X section. If all your grants are getting triaged, then it would probably be a better use of your time to figure out what you are doing wrong and get your grants into that upper section. On the other hand, if you are getting 30% scores (with some distribution) and just haven’t gotten lucky yet, then maybe you need to write more.
We’re scientists. We work with distributions. You know you have to get into that 10% tail of the distribution. If you are at 5% +/- 1%, then write one proposal. If you are at 30% +/-10%, you need three or four. If you are at 60% +/-10%, you better figure out how to change your mean.
LikeLike
July 19, 2012 at 3:44 am
“And if you are not responsible for the employment and health insurance of many people, including their children, perhaps you should not jump so quickly upon your high horse, becca.”
Ah yes, the noblesse oblige. With great power comes great responsibility. Heavy the head that… fuck it.
LikeLike
July 19, 2012 at 5:00 am
qaz-
And if you are ~evenly distributed with triages, 30-40%ile and <30%ile scores? What do you conclude from that?
LikeLike
July 19, 2012 at 5:49 am
We’re scientists. We work with distributions. You know you have to get into that 10% tail of the distribution. If you are at 5% +/- 1%, then write one proposal. If you are at 30% +/-10%, you need three or four. If you are at 60% +/-10%, you better figure out how to change your mean.
The realistic a priori distribution for a decent grant writer is more like 40% +/- 40%.
LikeLike
July 19, 2012 at 6:54 am
Not to cap on qaz and Virgil here but I grow increasingly fascinated with our (and yes, my views also qualify) apparent need to find agency in our grant writing. To imagine we’ve found a way to beat the system, tilt the odds. It’s delusional but a very common feature of human brains, no?
LikeLike
July 19, 2012 at 6:59 am
Not everyone cares about their employees miko, see Romney. But some of us actually give a shit that in addition to doing all the science, we actually create and sustain jobs.
Sure, NIH funds are zero sum and if I didn’t get a grant, some PI halfway across the country would do so. But you can say that about a construction company too- if ProffeBilt, Inc didn’t exist, PIbuilt, Inc would take the house contract. Still, we credit the small business owner who succeeds.
LikeLike
July 19, 2012 at 7:05 am
It’s delusional but a very common feature of human brains, no?
There’s a reason casinos are a multi-billion dollar industry.
LikeLike
July 19, 2012 at 7:06 am
OK, what I thought was going to be a pretty straightforward discussion has taken a turn for the fascinating.
DM – do you assume no ‘agency’ in grant writing? Do you not imagine that the content of your grant at all affects its eventual score?
The only way your comments make sense to me is if you believe writing grants is strictly a game of chance (ie. the probability of grant getting funded is independent of stuff in grant).
I’ve been lurking here for a while, so I’m aware that there sometimes just a bit of cynicism in the air, but still…
LikeLike
July 19, 2012 at 7:11 am
Yeah, I know my PI honestly cares about my career, and I certainly believe you do for your trainees based on your interests. More a comment about the structural differences and worrying about your own job/career existing at all not really being a remotely comparable burden to being in a “safe” (relatively, of course) management position.
LikeLike
July 19, 2012 at 7:25 am
poke-
PP’s example is apt. If you don’t know how to play blackjack well, you are going to lose your money the fastest. But even if you are an expert, you are still subject to the chance of the shuffle/deal.
LikeLike
July 19, 2012 at 7:26 am
did you miss the part where I am in a soft money job category miko?
LikeLike
July 19, 2012 at 7:54 am
The casino example is only apt to a point.
Certainly the vagaries of the grant review process introduce a ‘noise source’ that decouples objective grant quality (whatever that is…) and grant score to some extent.
But as others have pointed out, reviewer-introduced noise just results in variance around the mean. Acknowledging that source of variance is different than assuming because payline = 10%, if I submit 10 randomly generated grant proposals I should expect one of them to be funded.
LikeLike
July 19, 2012 at 7:58 am
Who said anything about “randomly generated”? You are making the same mistake Virgil is making….of assuming that people who generate more applications than you currently imagine are possible for yourself are throwing up “complete crap”.
LikeLike
July 19, 2012 at 8:02 am
This is really not that fucken complicated. The suggestion has not been to submit randomly generated grant proposals. The suggestion is that you should submit as many well-prepared grant applications as possible, and that for statistical reasons, it is better to spend the same amount of time and effort to submit ten pretty good ones than three supposedly outstanding ones.
LikeLike
July 19, 2012 at 8:06 am
If you’ve sat on study section, you realize that alot of the grants are pretty damn good and that there’s a large discrepancy in how each reviewer scores. I realized that there’s really not that much difference between a 4oth percentile and a 10th percentile and that some of this range is due to reviewer differences in scoring. As long as you are able to write a good grant that is likely in that range, you then know that you need to be submitting more grants. Study section has enlightened me to this fact. Drug Monkey is 100% right.
LikeLike
July 19, 2012 at 8:06 am
The content of the grant is related to it’s probability of funding at the bad end of the scale but not so much at the good end.
Complete crap is pretty sure not to be funded, but the converse (excellent grant = sure to be funded) isn’t true.
I look at one of my proposals that was triaged and another that was funded at the 12th percentile and they just aren’t all that different in terms of quality.
I’m not sure that means that spamming the NIH with a dozen applications a year is the way to go, but you do have to take the stochastic nature of the process into account.
LikeLike
July 19, 2012 at 8:30 am
“did you miss the part where I am in a soft money job category miko?”
I did. Apologies.
LikeLike
July 19, 2012 at 9:39 am
DM,
I think that some in your audience are responding negatively to your advice to submit as many good grants as one can, because they know people who do this, but do it poorly. Do you know this phenotype? A guy* has a significant research topic, writes ~10 applications a year, and they all get either triaged or scored nowhere near fundable. Sometimes he submits 2 or 3 applications for the same cycle. He complains constantly about the funding problem and that politics plays too big a role in science. He’s doing it wrong.
Clearly, you are not that guy, but I think of him, when I hear advice to submit more applications.
*always a male in my experience
LikeLike
July 19, 2012 at 9:43 am
Physician Scientist: If you’ve sat on study section, you realize that alot of the grants are pretty damn good and that there’s a large discrepancy in how each reviewer scores.
In the study sections I was accustomed to, the three reviewers were pretty close in their scores (3 point range) 75% of the time. They could differentiate between the best 30% and the worst 30%, but they have trouble with the edges that overlap the middle 40%. Differentiating between the best 10% and the second best 10% is damn-near impossible. It’s a really, really hard job and the reviewers took it very, very seriously.
Comrade PhysioProffe: The suggestion is that you should submit as many well-prepared grant applications as possible…
The problem is that none of the serial applicants believe that his or her application is poorly prepared until he/she gets the summary statement. If I had a dollar for each time I heard “My Chair/colleague/mentor read it and said it was good…” I think there are a lot of junior PIs getting bad advice because their mentors believe that 10 randomly generated applications will yield one grant.
LikeLike
July 19, 2012 at 10:03 am
I think there are a lot of junior PIs getting bad advice because their mentors believe that 10 randomly generated applications will yield one grant.
and I think there are a lot of PIs getting bad advice to “write better grants” because Program Officers are so blinded by their (necessary) belief in the validity of the review order. The question is not what the summary statement says. For POs the question is, have they ever done the experiment where they read and rank the proposals that are assigned to them prior to review and then see where the scores fall? How many times would a PO say “it was good” and then have to eat crow when the summary statement appeared? I’d say…a lot.
Yes, reviewers often agree with each other. But they often do not and there is no way to compare the three assigned this time, in this section with a different set of three in another section. Nor to assess the degree to which the three in agreement are simply expressing a common bias that may not obtain with a different section or different set of reviewers.
but in any case, you apparently agree that within the pool of damn good proposals (20% in your description) there is very little difference between those that will be funded and those that will not. if so, what is your strategy to maximize falling into the funded population? What did you advise applicants to do?
or are you suggesting that reviewers actually succeed at this “very hard job” and come up with the “correct” rankings most of the time?
LikeLike
July 19, 2012 at 10:07 am
Clearly, you are not that guy,
How do you know I am not that guy?
they know people who do this, but do it poorly
What does that even mean? Is it impossible in your view that someone could be doing everything right and still come up short in the grant process?
And are the “Ferrari” types that don’t happen to get a grant with a fundable score doing it “poorly” as well?
LikeLike
July 19, 2012 at 10:38 am
“How do you know I am not that guy?”
Your writings here suggest that you care a lot about writing a good application. The serial submitters I know do not. They think their brilliance or past accomplishments are sufficient, or they think that their applications are really good and that the reviewers are too petty or uninformed to appreciate the brilliance.
“Is it impossible in your view that someone could be doing everything right and still come up short in the grant process? ”
Of course this is possible and is happening now to an increasing number of applicants. Are you suggesting that it would not help these people to spend more time making 5 Buicks rather than continuing to submit 10 Kias?
LikeLike
July 19, 2012 at 10:56 am
Are you suggesting that it would not help these people to spend more time making 5 Buicks rather than continuing to submit 10 Kias?
If someone is writing good applications already than I see no reason why writing “perfect” applications is the best use of their time. Variables that are much more important include finding an idea that will resonate with three assigned reviewers, matching an application to a study section culture and finding an idea that gives a specific PO a big ol’ dopamine rush. Also taking advantage of the marginal gains that are possible when responding to RFAs, revising near-miss scores, using select-audience mechanisms like R15, having a study section “get to know you” and distributing your attempts across risky and wheelhouse type proposals.
Many of these things require trying it to find out. I.e., submitting proposals to study sections and seeing what they say about it, y0ur approach, your style and then how excited the PO gets.
LikeLike
July 19, 2012 at 1:03 pm
Not to cap on qaz and Virgil here but I grow increasingly fascinated with our (and yes, my views also qualify) apparent need to find agency in our grant writing. To imagine we’ve found a way to beat the system, tilt the odds. It’s delusional but a very common feature of human brains, no?
Yes, but data-wise, there is a clear learning trajectory to grant-writing. (Isn’t that called “grantsmanship”?) When I started, my grants were in the triage +/- who-gives-a-****-because-its-triaged category. Now my grants are “competitive”. In my experience as a grant-writer, as an observer of my colleagues, and as a reviewer, I would definitely say that there is agency in our grant writing. I believe we tilt the odds. Beat the system, no. But improve the odds? Hell, yeah.
I don’t think we’re disagreeing here. Everyone agrees that one should submit as many great grant proposals as one can and everyone agrees that one should not submit random strings of letters. The deeper question (and the more interesting one IMHO) is when it is better to spend time on a grant and when it is better to spend time on a paper or on other work. (Because we are only human and do not have magic clocks that stop time – at least I don’t!)
LikeLike
July 19, 2012 at 1:22 pm
“The deeper question (and the more interesting one IMHO) is when it is better to spend time on a grant and when it is better to spend time on a paper or on other work. (Because we are only human and do not have magic clocks that stop time – at least I don’t!)”
I’m hoping we get more discussion/advice on this question.
LikeLike
July 19, 2012 at 1:57 pm
“… when it is better to spend time on a grant and when it is better to spend time on a paper…”
The advice I was given as a postdoc was to go for the papers first and the funding will follow. All things being equal, I follow that strategy as a PI. Getting grants (and institutional support) is generally easier when high quality papers are being produced.
LikeLike
July 19, 2012 at 2:31 pm
When I started my lab, I had two R01s awarded before we had published a single paper.
LikeLike
July 19, 2012 at 7:36 pm
CPP, that’s the strategy my most successful newby prof peers have followed. Getting the money is the #1 priority. In the lab, focus on generating preliminary data.
LikeLike
July 20, 2012 at 1:41 am
just to be clear, whe we say wrte 10 grants a year, what is the distribution of R01 types vs R21 types/foundation money vs R03 types/pocket money? because the thought of writing 10 R01 a year makes me nauseous.
LikeLike
July 20, 2012 at 4:09 am
thought of writing 10 R01 a year makes me nauseous.
Keep in mind that submitting X RO1 proposals doesn’t necessarily require having X full blown free standing projects.
“Overlap” only becomes an issue once something is funded. Up to that point the same preliminary data can fuel multiple proposals that are spun to appeal to the tastes of different study sections.
That said, I agree that, for me, 10 full RO1 proposals a year would be excessive.
LikeLike
July 20, 2012 at 5:21 am
Just to be clear, because I too received my first R01 before having a publication, “papers first” refers to publishing being foremost in importance when weighing priorities throughout your career as a PI.
LikeLike
July 20, 2012 at 5:58 am
@CPP: I wonder if you wouldn’t mind elaborating just slightly. For example, what was your situation like going into your first position? Did you have the grants in hand already or did you focus on getting the R01s in the first couple of years of the TT process? If it was the latter, I’m guessing you had some start-up funds that paid for techs/post-docs and they were busy generating preliminary data etc while you were preparing your applications.
LikeLike
July 20, 2012 at 7:40 am
Did you have the grants in hand already or did you focus on getting the R01s in the first couple of years of the TT process? If it was the latter, I’m guessing you had some start-up funds that paid for techs/post-docs and they were busy generating preliminary data etc while you were preparing your applications.
It was the latter: I had no grant support at all when I started my lab, but I had a substantial start-up budget.
LikeLike
July 20, 2012 at 5:20 pm
The question is not what the summary statement says. For POs the question is, have they ever done the experiment where they read and rank the proposals that are assigned to them prior to review and then see where the scores fall?
That’s not their job. Some POs have very diverse grant portfolios and cannot be expected to be experts on everything in them. Besides, if one of your points is that three reviewers may have vastly different opinions of an application, adding the PO’s opinion as a “4th reviewer” doesn’t prove anything.
…there is no way to compare the three assigned this time, in this section with a different set of three in another section.
Actually there is – percentiling. However, as you know, it’s most useful for R01s in chartered study sections.
Nor to assess the degree to which the three in agreement are simply expressing a common bias that may not obtain with a different section or different set of reviewers.
So the choice is this bias with these reviewers or that bias with those reviewers?
or are you suggesting that reviewers actually succeed at this “very hard job” and come up with the “correct” rankings most of the time?
I’m not sure how you define success of the review. They are asked to decide if the application is likely to have a big impact on the field and put that in numerical form. No one has done a retrospective analysis to correlate scores with impact. Anyway, the sample would be limited to funded applications and there would be too many confounders.
What did you advise applicants to do?
There’s isn’t really a lot to say when an application, with few weaknesses, doesn’t make the payline because there were others in that study section that the reviewers were more excited about for whatever reasons. That’s the reality of flat budgets and shrinking paylines.
LikeLike