Summary Statements: Reading Between The Lines
October 25, 2008
Applicants are starting to receive their Summary Statements from grant reviews that took place in September/October NIH study sections. The Summary Statement is the written record of the review of your grant.
It contains the written reviews of the assigned reviewers (usually two, three, or four of them), as well as the Resume and Summary of Discussion, which is a summary written by the Scientific Review Officer of the discussion of your grant by the study section. This latter section is only there if your grant was discussed–i.e., not triaged without discussion.
A colleague of mine from another institution was kind enough to share with me the Summary Statement she just received, as she found it confusing and wanted to get my take on it.
I opened the Summary Statement PDF she forwarded to me and started, as I always do, with the Resume and Summary of Discussion:
…very good proposal…broad significance…PI is very creative and productive…abundant preliminary data…considerable enthusiasm…despite numerous strengths, concerns were raised that limit enthusiasm
The concerns that were raised were solely technical issues, and the reviewers all considered those technical issues to be addressable, and they gave good suggestions for how to address them. So that sounds like the grant should receive a good score, near, but probably not within, the fundable range, right?
HAHAHAHAH! The grant received a priority score–the average of the numerical scores from 100-500 assigned by each member of the review panel to the grant–of ~200, which for this particular study section corresponded to a percentile of ~40. WTF!? Based on the Resume and Summary of Discussion, this grant should have ended up in the ~20 percentile. So what happened?
Well, as I read the written reviews of the assigned reviewers, I noticed something very interesting about one of the Investigator sections–where the reviewer is supposed to provide her opinion of the qualifications of the PI to carry out the proposed research:
Dr. So-and-So is an outstanding young investigator who has been very successful in establishing herself as an independent investigator. An important indication of her success is her high level of funding.
Ruh, roh!! High level of funding was the death knell for this application. It is nearly certain that this phrase was intentionally inserted in there by this particular assigned reviewer specifically so that all of the rest of the members of the review panel–none of whom are likely to have looked at any of the application other than the Specific Aims–would perceive that this PI does not need to have this grant funded in order for her lab to have enough funding to exist. And this perception led the entire panel to substantially downgrade the scores.
Some other colleagues have disputed this interpretation, claiming that the reviewer was just being very laudatory and complimentary to the PI in pointing out her high level of funding. I call bullshit on that.
First, this reviewer did not point out any of the other aspects of this PI’s success–including, most importantly, the substantial number of published manuscripts from her lab in high-impact-factor journals. This reviewer just happened to choose to highlight out of the various outstanding accomplishments of this PI solely the fact of her high level of funding.
Second, in assigning a score to a grant, it is de jure impermissible to take into account per se the level of funding of the PI. If the level of funding arose during discussion, the SRO and/or chair of the study section would–unless they were asleep at the wheel–immediately and forcefully stop the discussion and remind the panel that this is not a permissibile consideration. The reviewer knew this, and snuck the information in as a supposed signifier of the strength of the PI’s success, knowing full well that it would have exactly the opposite effect.
Having said all of this, it is important to recognize that this kind of reviewer behavior does not represent some kind of terrible personal ethical lapse on the part of this reviewer or the study section members who responded to this intentional signal by downgrading the scores. It is understandable behavior that is ethical and rational in the current funding environment if one is convinced that keeping as many labs as possible funded sufficiently to continue to exist in some form is a more important good than single-mindedly funding the best possible research.
DrugMonkey 
October 25, 2008 at 4:29 pm
CPP, you are right on the money. I have been rejected numerous times for postdoc grants because my PI has “3 R01s.” They figure we are rich enough that he will pay for me no matter what. No one bothers to see that the funding is all running out at the same time. As far as I’m concerned, I am out of a job in a few months and have been told so by the PI himself.
LikeLike
October 25, 2008 at 5:03 pm
PP,
So what is your position on that issue? Fund other labs that are not well funded or fund the well funded PI because her proposal is excellent and will advance our knowledge.
My own take on the issue is that if the NIH requires applicants to list their funded grants as part of their application, it is because such funding will be considered when the new application is reviewed. If one’s present funding is not to be considered as part of the review process then, applicants should not be required to list their funded grants. Alternatively, maybe the NIH should limit grant applications to PIs who are unfunded or who are funded up to a given dollar amount and not above it.
LikeLike
October 25, 2008 at 5:14 pm
Well, so are you saying this gives my no funding beside small foundationish stuff ass a better shot. Makes me feel better I guess, I’ll only vomit once every time I think about getting my next set of summary statements.
BTW, all the older cats call ’em pink sheets, I think we should go back to that, it has a gentle ring to it that softens the blow.
LikeLike
October 25, 2008 at 5:27 pm
if the NIH requires applicants to list their funded grants as part of their application, it is because such funding will be considered when the new application is reviewed. If one’s present funding is not to be considered as part of the review process then, applicants should not be required to list their funded grants.
This is a critical issue of grantsmanship. My reading of the instructions is that one is not required to list grant support on the biosketch. That section is NOT for the purpose of anyone deciding if you have too much money. That is the role of the Other Support page that you submit to the IC just prior to funding.
The biosketch listing is to demonstrate your competence to run projects of the proposed size and scope and other things that burnish your credentials as far as the conduct of the present proposed studies are concerned. period.
In short I concur with PP. This is not supposed to be explicitly considered by reviewers, nor is the institutional overhead rate, for that matter.
LikeLike
October 25, 2008 at 6:07 pm
Thanks for the info CPP. I’ll put that in the “review-dog-whistle” folder!
LikeLike
October 25, 2008 at 6:07 pm
What DoucheMonkey said.
LikeLike
October 25, 2008 at 6:21 pm
The problem is the cognitive dissonance arising from the notion that considering existing support is inconsistent with the official NIH policy that projects, not people, are what get funded. As we all know, this is a complete, total, blatant lie. For evaluation of an R01 (or R21, or R03) the calculus is not “should this project get funded?”, it is “does this person merit additional resources?” An excellent proposal can help support the argument that additional resources are merited, but the identity of the applicant is always the major consideration, regardless of what the NIH says. If you don’t believe this, try sending an application to a study section where you don’t know any of the members. I don’t care how great the proposal is, and how perfectly it fits the described nature of the study section’s mandate, this proposal is DOA.
If the level of funding arose during discussion, the SRO and/or chair of the study section would–unless they were asleep at the wheel–immediately and forcefully stop the discussion and remind the panel that this is not a permissibile consideration.
Stop, please, my laughter is killing me with this one…
LikeLike
October 25, 2008 at 6:56 pm
Dude, if a study section got into a discussion about “this PI is very well-funded and doesn’t need this grant”, any awake SRO would immediately stop that line of discussion. Failing to do so would be a serious failure on their part, and this is the kind of shit they and their supervisors take seriously. I am surprised that someone who acts as savvy as you do is not aware of this. This is exactly why this kind of thing only goes on in the wink-wink-nod-nod fashion I described in the post.
LikeLike
October 25, 2008 at 7:08 pm
While we’re sharing pink sheet stories, I’ve just received a novel criticism in the review of a HBCU partnership/program development grant from the DoD’s breast cancer initiative. A little background may be in order for DrugMonkey readers.
The goal of the funding mechanism is for the HBCU researchers to partner with senior breast cancer investigators at a predominantly white institution (PWI) in order to 1) support the career development of at least two HBCU faculty members and 2) establish a sustainable breast cancer research program at the HBCU.
I’ve been working primarily in prostate cancer and health disparities, and have been trying to expand my program into breast cancer. My luck hasn’t been too good in securing independent breast cancer funding in part because DoD reviewers have stated that I lack training or experience in breast cancer. I thought that perhaps this partnership mechanism would provide me and my small laboratory with the opportunity to develop this street cred.
Our partner was a highly-regarded (and highly-funded) breast cancer physician-scientist at a nearby PWI who has a record of training graduate students from our HBCU. Moreover, she had some great preliminary data on clinical specimens from patients who have a type of breast cancer that disproportionately affects young (<35-40) African American women. She already was a friend and was pretty stoked when I was recently hired at this HBCU – it was to have been win-win for us both and, IMHO, the patients she treats in our community.
While there were some legitimate scientific and training criticisms, I found this one to be unique:
There is no opportunity to resubmit so we can only put in a new app if the program is renewed for FY09. But I’m not sure how I’ll head off this particular criticism, anticipating that another year at the HBCU will still be inconsistent with the “slow pace” at which “trust evolves.”
LikeLike
October 26, 2008 at 12:25 am
PP: This is exactly why this kind of thing only goes on in the wink-wink-nod-nod fashion I described in the post.
You’re absolutely right. What’s funny is pretending that the SROs can effectively control the behavior of the study sections, when in fact the NIH is completely the bitch of the study sections.
My favorite completely ignored NIH instructions to study sections (in very rapidly decreasing order of funniness):
1) “No preliminary data is required but may be included if available.”
2) the NIH funds projects, not people
3) “The reviewers have been given strict instructions not to communicate with applicants and to inform me [the SRO] if they are contacted by applicants.”
LikeLike
October 26, 2008 at 12:38 am
What douchewhimple said.
BTW, as I have mentioned on this blog before, the reputation of a PI when submitting a new app plays an important role in determining whether or not the app is funded. Thus, although in a way it works against the idea that a well funded PI (albeit, with less reputation) may face difficulties in getting a new app funded, a well known PI may get her sub-par app funded.
LikeLike
October 26, 2008 at 2:50 am
Dude, try to read more carefully. No one said that SROs can control the behavior of study sections. What they can control, however, is the explicit content of what is said in study section. There’s a big fucking difference.
LikeLike
October 26, 2008 at 9:23 am
Dude, if a study section got into a discussion about “this PI is very well-funded and doesn’t need this grant”, any awake SRO would immediately stop that line of discussion.
No kidding, PhysioBro. My current SRO, and the two previous ones, would have disemboweled any of us for such a discussion. It happened one time recently and I expected something like the baseball bat killing at the dinner table in GoodFellas.
No doubt that the case you describe is indeed indicative of the funding times but, by protocol, should not have been allowed in writing and certainly forbidden from discussion.
LikeLike
October 26, 2008 at 10:18 am
That is interesting. You don’t think that couching it in terms of the “success” of the PI allowed it to squeak within the rules?
I am very curious to hear what my colleague has to say about her conversation with her Program Officer about this. I will, of course, report back when I hear from her.
LikeLike
October 26, 2008 at 10:49 am
Part of the problem here is that we have two different ways of asking whether someone should be funded or not. On the one hand, we have the strategic question of maintaining a strong and diverse population of science labs. On the other hand, we have the tactical question of which project should be funded.
NIH has explicitly rejected any strategizing and has explicitly pushed this onto the tactical question addressed by study section. A perfect example of this is that the way NIH treats new investigators is to give them a check-box and then NIH asks study section to “be nice”. (Whether study section is nice or not in response to the checked box is debated, but my point here is that this is forcing study section to change its scores based on the strategic new vs. established investigator question.) This is really a question that should be addressed at the NIH program level, not at the study section level.
It is true that programs do try to help struggling investigators by reaching a little below the line for them, but in practice, most programs are completely subject to the tyranny of the percentile score. This means that no one is really looking out for the strategic question of our scientific stable. Some reviewers take it on themselves to try to do that. Whether they should or not is debatable, but no one else is.
That being said, I do have to agree with DM – I have never heard anyone say “this PI is well-funded”. I think both my SRO and the head of the study section would explode if someone tried. But I do know of many cases where scores got better because people knew someone was hurting.
LikeLike
October 26, 2008 at 10:52 am
Imma —
I’ve been on an NRSA study section for a number of years now. And this is one exception where study sections *are* explicitly told to take funding into account. However, what the study section is looking for is to make sure that there is sufficient other money in the lab to cover the project. This is because the NRSA only funds the salary of the student or post-doc.
Some projects don’t require extra money (like computational projects), but most neuroscience projects require a lot of additional money to run beyond the salary of the student (think animal care, chemical reagents, technical support, etc.)
Are you sure the “3 R01s” line is bad in your reviews? Or is it just that it was mentioned?
LikeLike
October 26, 2008 at 11:07 am
most programs are completely subject to the tyranny of the percentile score.
This is what is so fascinating and I long for data on IC behavior with respect to the strict percentile order. My most-familiar Institutes sure as hell go outside of the order….a LOT. Just about everyone I know (to the degree we talk about such things) has received at least one pickup, so far as what they can tell from their score, other (unfunded) scores and from what POs say to them.
I wouldn’t be surprised if other ICs adhere more closely to the percentile order. I would tend to doubt it, but I wouldn’t be totally shocked if someone had a big sample and never heard of Institute of X doing any shifting of ranks…
LikeLike
October 26, 2008 at 11:34 am
I have been told by more than one Program Officer that NINDS funds *all* grants within their payline. They do reserve funds every fiscal year to go below the payline, but don’t shitcan grants inside the payline for any reason. This is part of the reason why the NINDS payline always look substantially tighter than those of most other ICs, which supposedly do decline to fund some grants inside their nominal paylines.
LikeLike
October 26, 2008 at 12:18 pm
DM – You are certainly right that different institutes treat paylines differently. The three I am familiar with all act very differently. NINDS claims to have a very strict payline. They very very rarely reach beyond it. NIDA claims to have a broad payline, but I don’t think they guarantee funding within that payline. They try to fund X% of all grants, but they’ll probably be in the 1.5*X to 2*X payline range. I believe that NIMH funds everything within a payline, but reserves a subset of its budget for special cases. Note that all three of these are talking the possibility of funding a grant 5% worse than the payline, not 20%.
But in all of these cases, the decisions are made on an individual basis with no official strategizing. At least there’s no publicly available information about how they are doing the strategy. (This is why its very worth talking to your program officers. They can sometimes find ways to reach down and help you up.)
LikeLike
October 26, 2008 at 12:38 pm
That is interesting. You don’t think that couching it in terms of the “success” of the PI allowed it to squeak within the rules?
Indeed, the context and intent of the reviewer is certainly a factor in determining the appropriateness of the statement. But I share your concern that the intent was for the reviewer to telegraph a message to your colleagues’ other reviewers. I’ll be interested what she hears back on this particular point.
However, now that critiques are posted in advance (as compared with when I chiseled mine on a stone tablet when first serving ad hoc as a young pharmboy) I have known at least one well-prepared SRO who searches the PDFs in advance for “fund,” “funding,” or “funded.”
LikeLike
October 29, 2008 at 6:57 am
I think the whole assumption of this line of discussion is incorrect. It is VERY common in the current funding climate to get a 200 with a reasonably good grant. The problem is score compression. As funding levels decline, reviewers tended to, over the past 3 years or so, push the scores of grants they wanted funded (even though they should only examine scientific merit, not funding issues). This causes HUGE score compression below 150. The study section i recently rotated from (CNNT) has this chronic problem. 150 is like 25% percentile, while 130 is 10th percentile. This make a 130 a great score, and a 135 a total loser score. Of course this causes 200 to be about 47% percentile as mentioned by the initial post. The problem is you have veteran reviewers on the section with a real bad habit of compressing scores, and new people coming in going on their idea of scoring by scientific merit. When the two collide, you get an OK score with a ludicrous percentile. This is also complicated by the fact that your percentile is calculated based upon the last 3 rounds of scoring in your study section, not the current meeting only. This makes you suffer from the previous rounds score compression problems as well. A good SRO will give reviewers guidelines of where certain scores fall at the beginning of the meeting to help titrate their scores. (e.g. 1.2=8%, 1.5 =15%, 2.0= 37% etc)
They need to “reset” the scoring on these sections and get people scoring on merit rather than on feeling they need to fund a grant because they have seen it 2 or 3 times and feel bad. I know SROs that have inherited sections and begged to reset, and they always get a big fat NO. We all suffer from reviewers in the past feeling bad and compressing scores….its the biggest problem as I see it in the screwed up scoring system we have at the current time.
LikeLike