NIH Regional Consultation Meeting on Peer Review
October 10, 2007
I attended the NIH Regional Consultation Meeting on Peer Review held this Monday in New York City. The meeting was led by Lawrence Tabak and Keith Yamamoto, co-chairs of the Working Group of the Advisory Committee to the NIH Director on NIH Peer Review. The purpose of the meeting was for the advisory committee to solicit the views of the extramural community and other interested stakeholders (damn, I hate that word). Here are my impressions of the meeting.
Not surprisingly, no one showed up at this meeting to assert that the current peer review system is just peachy. Pretty much everyone who spoke felt that their grant applications were reviewed unfairly, and that because of structural problems in the system, the true brilliance of their research was not recognized. Several people complained that junior investigators were in over their heads at study section, and should be barred from service. Others complained that their multidisciplinary research could not get a fair hearing, because no study section members had the requisite combination of disciplinary expertise. There was also the usual complaining that the current peer review system selects against truly groundbreaking research.
In terms of proposed solutions, there were three that I found interesting:
One was the suggestion that investigators be given the opportunity to view the reviews before study section and provide a rebuttal that could be seen by the members of the study section. This seems like a pretty good idea, as it would allow an investigator to correct simple factual errors or clarify things that were stated unclearly in the application and led to misunderstandings by reviewers.
The second was to incorporate a “review of reviewers” system to impose quality control on peer review. The motivation for this was mostly to “weed out the incompetent reviewers who unfairly failed to recognize the brilliance of my application”. While this motivation is clearly self-serving, the idea has some merit.
An interesting hybrid of these first two ideas is the system that the Medical Research Council of the UK currently uses: Written reviews from the first level of peer review are provided to the applicant, who writes a rebuttal. Then the rebuttal and the first level reviews are reviewed together by a second level review panel that does not contain any of the people who participate in first-level review.
The third was as follows: The peer review system simply can’t distinguish between the 5-10 percentile range and the 10-15 percentile range. Since paylines currently sit right in between these ranges, this is a big problem. One suggestion was that half of the grants in a percentile range of 8 points or so that straddles the payline should be selected randomly for funding. Nobody laughed.
As far as the demographic of the commenters, let’s just say that I was the youngest person there by far.
DrugMonkey 
October 10, 2007 at 12:34 pm
The second was to incorporate a “review of reviewers” system to impose quality control on peer review.
1) Is there a supply of people interested in reviewing reviewers on top of their existing workload of reviewing and being reviewed?
2) Given your point three, unless truly magnificent proposals are being shot down (which is possible, but if true it has alarming implications that go well beyond grant applications) doesn’t that suggest that good proposals are being shot down in favor of equally good proposals and that this problem isn’t worth solving?
LikeLike
October 10, 2007 at 2:30 pm
“As far as the demographic of the commenters, let’s just say that I was the youngest person there by far.”
and did they ask for a show of hands for those who are currently appointed on panels and those that have ad hoc’d more than once in the past three years?
LikeLike
October 10, 2007 at 7:34 pm
I understand why the system needs fixing in regards to review cycle times and shortening of the grant application length, but is it unfair? I do not buy the argument that junior faculty are over their heads or that they specifically try to limit the funding of well established faculty. If that was the case, why do many prominent senior investigators carry multiple grants even though their best work is years behind them.
LikeLike
October 11, 2007 at 5:35 am
I think JSinger makes a good point; adding more levels of review that require additional reviewers (and additional time) may push the system beyond its capacity. However, I wonder if SROs or study section chairs could provide some quality control. I think as reviews are being discussed, they could get a sense of whether or not a particular reviewer is adequately supporting their criticisms or not. Any reviewer that had a record of 3 “dings” for unsupported or destructive criticism wouldn’t get asked to review in future. However, this should really only be focused on weeding out truly unfair and unconstructive reviewers; there obviously should be room for differences of opinion (distinct from “unfair”) that are supported by fact and logic. Journal editors should also consider applying this rating more systematically, although I’m sure they do so to some extent now.
Other have made the suggestion that investigators who receive NIH funding should be required to serve on study section if asked. Perhaps to address the problem of getting more senior people on study section, senior (i.e. tenured) scientists could be asked to serve 1 yr of consecutive study section/R01 at any time during the grant period. Those that receive multiple R01s would be expected to serve more, as seems appropriate. Junior people should participate, but perhaps could be limited to 2 ad hoc study section/grant period. This would allow junior faculty to become familiar with the review system and would ensure that senior faculty were well represented.
LikeLike
October 11, 2007 at 6:43 am
[…] the meantime, the intrepid PhysioProf has blogged his observations on the Oct 8 NYC meeting over in DrugMonkey’s […]
LikeLike
October 11, 2007 at 11:44 am
tva says: “I do not buy the argument that junior faculty are over their heads or that they specifically try to limit the funding of well established faculty.”
[sigh]. I also continue to hear this argument in just about every venue of discussion. Still waiting on the data to show that these opinions are based on anything other than biased suspicion. Still waiting to hear how we know, even if a young reviewer DOES rip a senior-investigator grant apart when a senior reviewer does not, that the “bias” is on the part of the younger rather than the older reviewer. all I can say is whenever you hear these opinions advanced, challenge them…
JSinger says: “doesn’t that suggest that good proposals are being shot down in favor of equally good proposals and that this problem isn’t worth solving?”
no, no, of course not. if my proposal isn’t doing well it simply must be because there is a problem with the review process. LOL
BugDoc says: “Any reviewer that had a record of 3 “dings” for unsupported or destructive criticism wouldn’t get asked to review in future. “
I’m hearing a LOT lately about the mythical unscrupulous bastige, incompetent, destructive reviewer. I dunno. Maybe I’m in an isolated backwater of nicey-nice. Maybe I am this reviewer. But through the lens of my appreciation of what the reviewer is really doing when writing a summary statement (communicating to other study section members and program in addition to the PI), well, I don’t see reviews that meet these accusations very often. And seeing a collaborator go all crazy about a review that I thought was quite understandable, well, I question the perspective of people making such comments…
any chance you have direct experiences and can elaborate, BugDoc?
LikeLike
October 11, 2007 at 12:43 pm
I was in NYC on the 8th as well, and believe it or not, the vibe was far, far less of the “But my brilliant work should be funded!” than I expected. Seriously. I’ve seen amazing whiners at NIH regionals.
The truth of the matter I don’t mind a grant going down with a fair review. I do mind a grant going down because the reviewers threw NIH’s stated review criteria out the window (yes, that was me speaking up about that problem, and the problem is genuine in my world).
Unfair reviews I can quote? “There is no international component.” (Reviewer didn’t read 1.5 pages of the application.) “This is irrelevant because it isn’t a mammal.” (There’s no way to do the experiments in a mammal, thanks to the anatomy, but the field has established that the conservation of the circuits across species is very good.) “I see no reason to include [X], and this dilutes enthusiasm for the proposed program.” (X was there because the RFA reqired it be there.)
Like you, I’ve seen faculty get tense about what I thought were legitimate comments, but I have seen hypercritical comments that were nearly stated as personal attacks on the PI. I generally attribute this to the anonymity of the review, and assume the person might have been more tactful in person or personal email. But in my mind, the “broke” part of the system is that it was designed for far, far fewer applications, and it’s impossible for overloaded reviewers to do the kind of job the applicant would like. I don’t mean call it excellent and fund it, but give it a thoughtful and fair commentary.
The suggestion I liked best was jokingly called the “ugly baby” approach, where a grant was scored, and also given the best possible score the reviwer thought the grant could reach. For example 180/180 would tell you that it just wasn’t interesting/significant enough, or was too deeply flawed, to be fixed by addressing a few comments. Many of us know the frustration of three submissions where the score essentially doesn’t change, or goes down. If you knew from the outset that you had an ugly baby, you could save a lot of effort (and reviewer time) not trying to dress up a piglet in a bonnet. 180/130 would tell you it was fixable, and you’d know to keep trying.
LikeLike
October 12, 2007 at 6:59 am
“I was in NYC on the 8th as well, and believe it or not, the vibe was far, far less of the ‘But my brilliant work should be funded!’ than I expected.”
Well, there was Fernando Nottebohm’s “pile of shit” comment.
LikeLike
October 12, 2007 at 8:37 am
Two themes that I’m hearing in local versions of this type of forum that are of interest:
1) The Stick. People are all over the “mandatory service” or “jury selection” approach, even when the “carrot” approaches are part of the discussion. Funny thing is that these self-same people are also highly concerned with the “right” or “best” scientists reviewing (explicit subtext, we currently have incompetent reviewers screwing up their proposals.)
2) Revisiting scores at the end of the meeting. Letting the panel see the ranked scores and somehow re-voting if desired. (At present, while the panel knows the post-discussion score range recommendations, one never knows the resulting full panel scores.) This intrigues me. Among other things it would facilitate examination of micro-biases. Such as “geez, we didn’t score a single R03 or R21 higher than a 180.” or “wow, how come no new investigator broke 200?”.
LikeLike
October 12, 2007 at 8:42 am
oh, and I should mention that we were piloting simultaneous online scoring this round. previously it was paper which would introduce a delay while some poor schmuck inputted the scores manually. (‘course I guess this is going to put a couple of clerical jobs at risk)
I’m not sure how widespread this is and whether we are the leading or trailing edge with respect to study sections. The point being that once this gets put in full operation the above mentioned review of scores at the end of the meeting would be technically trivial. As would the process of revising one’s scores.
LikeLike
October 12, 2007 at 10:03 am
iGrrrl:
ok, with respect to your “unfair review” specifics… the thing is that in my experience as a reviewer, the things that are flat out mistakes are rarely the deciding factor. as an applicant, I have the same level of outrage as anyone “GEEZ, I SAID right freakin’ there on page 48….”. but i find that in discussion it is rare that 3 reviewers all base their essential judgments on a mistake. now the relevance of your nonmammalian system…i’m intrigued and trying to understand what this means. you may be running into a basic-science vs. translational application thing where it is not an “error” so much as a difference of opinion on the larger issue. It is not an “error” for a reviewer to want some evidence that a project might possibly apply to human health. nor is it an “error” for a reviewer to think that “chances of getting into C/N/S” is a metric of quality. it is a difference of opinion.
the “ugly baby” split score thing is a very interesting suggestion although perhaps we should recognize that we already have the essentially never-used NRF for this purpose. another thing it possibly suggests is providing the pre-discussion range average to the applicant. my section, for example, has reviewers assign prelim scores even to triaged apps. ave of 4.5 might mean something different than ave of 2.6 even though each would be “triaged”. but getting at what you really mean, I think the predictions of whether a 180 will ever go higher or not is a bad business. i doubt we could predict very well in this range for one thing. and it sort of leverages the opinion of the first three reviewers into later reviews. one of the advantages of re-review is that there can be some additional people looking at an app.
LikeLike
October 12, 2007 at 10:49 am
I just got an A1 back unscored after, according to the 3 reviewers on this latest panel of the study section, doing an outstanding job addressing essentially all of the initial 3 reviewers’ concerns. It was quite apparent that I had 3 new reviewers from the same study section this time around for the A1 and they all raised new issues while ironically complimenting me for how well I addressed the critique of the initial submission. I understand that this is in some ways unavoidable, but for me as a new PI, I see turnover on study sections as too high these days and it is a major issue.
Imagine having your manuscript submitted to a journal be reviewed, working hard to address all the concerns of the reviewers, and then when you send it back to the journal they have potentially 2 or 3 totally new reviewers read it and your response to the original review. Isn’t that nuts? So why isn’t it nuts to do that for grants?
So now when I send back for an A2, again doing my best to respond to these study section members’ new comments, who is to say they will even be present on the section at the next meeting?
LikeLike
October 12, 2007 at 2:46 pm
gnupi, this is something you for sure will want to take up with the SRO once your A2 is assigned to study section. Call him or her up and make exactly these points. Politely.
with my reviewer hat on, think about the big picture here. the thing is that with a triage you don’t know for sure if your app went from a 4.8 to a 2.6 or from a 2.8 to a 2.6. on a very bad app, the reviewers simply do not have time or inclination to hit on every single issue. so it could be that what happened is “fair” in the sense that the remaining problems really were legit problems.
with respect to the new-reviewers issue, it could very well be that the SRO thought you were not getting a fair shake from the first set of reviewers and was trying to help you. this is why it may not be “nuts”.
strong statements from the new reviewers about the quality of your reply to critique could, likewise, be their recognizing that you were being unfairly dinged and trying to pre-empt that line of attack. after all, they may be assuming that there will be repeat reviewers as well (they don’t have access to who the previous reviewers were, unless they were at the prior review.)
LikeLike
October 12, 2007 at 3:30 pm
drugmonkey,
Thanks for the input. I appreciate your perspective on things because some possible scenarios are hard for us new PIs to imagine.
I wish I did know if my application was an “ugly baby” as mentioned earlier or just plain ugly. The mentors I have helping me by reading it, etc. are very positive overall and helpful with their critiques before submission.
I think upon first submission my application indeed did not get a fair shake. This time the 3 reviewers were surprisingly positive overall, especially considering they triaged the grant, and all 3 focused mainly on one particular experimental issue they felt was problematic that I believe can be addressed with new strong preliminary data already in hand so maybe there is hope when I resubmit.
I’m leaning towards not doing the expedited resubmission for new PIs as I need to put my best foot forward for the final (gasp) A2 submission and may get additional data, etc in the mean time. What do you think?
In the mean time I have already submitted a 2nd independent R01 on a different topic. Crazy, huh? In the past would a PI in their first year or so have 2 submitted R01s along with numerous foundation grant applications pending?
LikeLike
October 12, 2007 at 3:54 pm
drugmonkey: …the thing is that in my experience as a reviewer, the things that are flat out mistakes are rarely the deciding factor
I actually agree with this statement, but the problem is that in the minds of the reviewed, gross errors in reading in one area throw the rest of the review into doubt. Yes, a smart person can usually tease out the substance, but for a lot of people, it’s hard to get over the gut-level emotional response of “S/H’es an idiot.”
I see your point on the ugly baby proposal, but kind of disagree. I went through the effort of supporting three submissions of a G20, where I knew there were substantial improvements in terms of RFA requirements and response to reviewer comments, only to watch the initally high score go down by small increments. Conversations with the program officer – which were in fact trying to get at the “Do we have an ugly baby?” question – weren’t helpful, because it is not in NIH’s interest to have fewer submissions of applications for grants. (“Look, Congress! We have so many excellent submissions we can’t fund! Please give us more money!”) On the investigator-initiated applications, being told the reviewers think an application is unlikely to ever reach fundable levels may make the PI sit back and re-think the approach in a productive way.
My bottom line philosophy is that biomedical research is the luxury of a wealthy society, and no one is owed funding. Whining about peer review “injustices” won’t make more money appear. The two-year-old in me comes out when I’ve been told the goal post is in one place, but am then required to score based on the goal post being somewhere else. I would mind far less being told that our approach to X sucked, than to be told the RFA-required section X shouldn’t even be there.
LikeLike
October 12, 2007 at 4:00 pm
Oh, and on the mammal thing: I didn’t think the reviewer was in “error” so much as to be so entirely biased about the lack of fur that the application wasn’t even read with any attention. The end of that story was that the program officer agreed with the PI, and the grant was funded on appeal. Rare.
LikeLike
October 12, 2007 at 4:19 pm
“In the mean time I have already submitted a 2nd independent R01 on a different topic. Crazy, huh?”
No, that’s exactly what you should be doing. My first R01 that got funded was my “2nd independent R01 on a different topic” that I submitted. This 2nd application was on based on a kind of crazy idea that came out of some surprising preliminary data that had nothing whatsoever to do with any research I had ever done before. The study section loved it, and it was funded as an A0.
My first R01 submission based on the research that flowed out of my post-doctoral research, is supposedly the “heart” of my research program, and was assigned to the study section that contains all my “buddies” from one of the subfields I inhabit only got funded as an A2.
“In the past would a PI in their first year or so have 2 submitted R01s along with numerous foundation grant applications pending?”
Absolutely, if they were ambitious and wanted to grow their lab as rapidly as possible.
LikeLike
October 12, 2007 at 4:22 pm
Oh, BTW, I hope you made sure that your second R01 submission is not going to be assigned to the same study section as your first, and that you keep your damn mouth shut about the fact that you submitted two R01 applications.
LikeLike
October 12, 2007 at 4:35 pm
yep. I did keep the old mouth shut and I requested a totally different study section and it is already assigned to that one, thank god.
It’s a jungle out there!
Thanks physio for the support and advice
LikeLike
October 13, 2007 at 12:06 pm
won’t NIH be able to figure out I have 2 R01s submitted?
LikeLike
October 13, 2007 at 12:50 pm
“[W]on’t NIH be able to figure out I have 2 R01s submitted?”
Of course. The point is that you don’t want study section members knowing you have more than one in play. The reasons for this should be obvious.
LikeLike
October 15, 2007 at 8:34 pm
“I’m hearing a LOT lately about the mythical unscrupulous bastige, incompetent, destructive reviewer. I dunno…[]any chance you have direct experiences and can elaborate, BugDoc?”
The study section I ad hoc on is actually very fair on the whole. However, I do recall clearly a few discussions where one reviewer out of the 3 seemed to feel extremely negative on a point that appeared to the rest of the panel (according to the discussion) to be rather minor, and easily addressed. The vehemence of the one reviewer (the anti-advocate!) tended to pull these applications out of the fundable range…yes, I know we are not supposed to think about that. But who really doesn’t? Actually,the “destructive” reviewer may be a bigger problem in manuscript review, IMO, since they get to be anonymous instead of having to defend their viewpoint in front of 30 people.
LikeLike