NIH Peer Review Outcome: A Reader Poll
February 9, 2010
Okay DearReader. Believe the NIH grant review process is irretrievably broken? Now’s your chance.
If you have ever submitted a research grant proposal to the NIH, please estimate the percentage of your grant submissions (include each revision as an independent submission) that have received reviews with serious flaws.
The approximately _____ of my grant applications that have received seriously flawed reviews.(trends)
Feel better? Okay, now tell us:
Approximately how many research grant proposals have you submitted(polls)
I see, I see. Okay, how about your study section experience as a reviewer?
Have you ever served on a study section reviewing NIH research proposals?(opinion)
If you feel strongly enough to comment on these matters, perhaps give us some idea of about how many proposals you’ve submitted to reach your conclusion of major problem / that’s life / works perfectly or whatever. Also whether you have served on study section.
Obviously, I am curious as to whether the number of summary statements received and actual review service modulates one’s view about the flaws in the system. It is my hypothesis that more experience gives you a more complete view of the strengths and weakness of the system and permits a more nuanced viewpoint. This is just one of the many, many reasons I think that CSR should be getting people closer to the beginning of their independent career involved with review as soon as possible.
DrugMonkey 
February 9, 2010 at 12:43 pm
I’m in the 5-9 range on submitted proposals (not all to NIH). I’ve been extremely satisfied with the reviews I’ve gotten – and no, they were not all positive! In fact, some of my early rejects provided exactly the kick in the rear end I needed to think about how I was presenting my work. Particularly the reviewer who told me that there was a lot of good science in one of my proposals, but they had to dig way too hard to get it out and that I needed to be nicer to my reviewers!
I haven’t served on an NIH study section – but I did serve on a review panel for another outside funding agency. It was an extremely valuable and eye-opening experience. Until I served on a study section, I think I pictured a whole room full of people with expertise in my particular area of research; I didn’t appreciate the breadth and wealth of knowledge in other fields, and the breadth of proposals that come in to a review panel for a funding mechanism. I also didn’t realize how much of a difference the presentation of the research makes. Sure, I’d been told over and over that you need to walk the reviewer through your proposal holding their hand – but I didn’t really understand until I read a series of proposals from scientists who did or didn’t do exactly that.
The best advice I’ve gotten (other than the aforementioned reviewer) was from a program officer who told me that most of the comments reflected a new proposal writer who needed to get a lot more input from senior colleagues on my proposals before I submitted them. I took her advice – and sure enough, it works. It doesn’t matter how good your science may be if you can’t explain it to your readers.
LikeLike
February 9, 2010 at 2:17 pm
I am in the low range for having seriously flawed reviews. Mainly, I think I have been pretty lucky…there is the usual bullshit (like reviewers saying I have no experience with a technique that I used in 4 papers)…but I see that as my failure to sell myself and convince them I was good. so I put in some more controls and let them know about the papers.
LikeLike
February 9, 2010 at 4:50 pm
The results so far are making me wonder what the average number of summary statements received is by the time a reviewer enters her first study section assignment…
LikeLike
February 9, 2010 at 5:03 pm
EVERY review I have received has been flawed, just like every application I submit has flaws. Sometimes the flaw reflects something I can fix (like #2 above), and sometimes the reviewer just doesn’t like what I am doing and cannot be swayed (I don’t care if 3 papers support the role of X in this process and 1 paper shows no role of Y, you should be studying Y not X!). A friend responded to a review questioning one technique by taking out the entire specific aim using it, but was labelled as “unresponsive” to that part of the first review!
Problems such as #2 will become more problematic as the application is shortened. I hope the NIH can make the process more conceptual rather than detail oriented, but I am not overly optimistic. I will let you know after my grant gets reviewed this summer.
LikeLike
February 9, 2010 at 6:48 pm
DM – one more, please:
Approximately _____ of my journal submissions have received seriously flawed reviews.
LikeLike
February 9, 2010 at 7:03 pm
My complaint is with the philosophy of the review, rather than the content. There is a very high intolerance of risk, lack of support for new methodological development, and little or no value attributed (sometimes the opposite in fact) to proposals involving living breathing humans. All of these reviewer attitudes contravene the stated goals of the NIH, but there you are, the reviewers call all the shots.
LikeLike
February 9, 2010 at 10:30 pm
Reviews of my grants where the review had serious flaws. 100% of the ones that were not funded! Reviews that I’ve done as a member of study section that have serious flaws. 0%
Actually, one of the surprises I had when I got onto study section was how often my reviews matched my colleagues. Particularly surprising was how often they matched even when I was sure they wouldn’t. (Like the time I wanted to defend this risky grant [yes, whimple #6, sometimes risk is supported – especially if well-defended] and like the time I hated this flawed-by-everybody-is-ok-with-it method.)
As one of my senior colleagues tried to teach me – when the review has a deep misunderstanding of your grant, you didn’t communicate it successfully.
LikeLike
February 9, 2010 at 10:45 pm
The issue that I have in my limited number of submissions, mostly to NSF and a couple to NIH, is the brevity of the reviews. Sometimes these are terse to the point of being useless when it comes to revising. The comprehensive ones on the other hand have been quite thorough in picking apart the strengths and weaknesses. I know that the reviewers are an overworked lot, but what about the folks submitting the proposals?
The few times that I have served on panels made me realize how capricious the whole enterprise can be. I ended up going to the mat for a few proposals, which made all the difference in their being funded.
LikeLike
February 9, 2010 at 10:52 pm
I have submitted about twenty grants, and have served about ten times as ad hoc on study section. I have never had a grant review that I felt had serious flaws to an extent that appeared to me that it made the difference between being funded or not.
LikeLike
February 9, 2010 at 11:35 pm
Submitted 3 NIH grant proposals as a PI (1 triaged, 1 resubmitted and funded in 2nd round, 1 funded first round) – hence received 4 sets of reviews.
I’ve had minor disagreements with reviews, especially when dinged for being junior/inadequate dept support, but never what I would call a major flaw of review or a completely unreasonable criticism/demand.
Served on 3 study sections ad hoc, and once as a mail-in.
LikeLike
February 9, 2010 at 11:47 pm
I have received six reviews on NIH grants (three grants plus revisions). I felt that all but one round of reviews were legitimate in general; the round that was flawed, in my view, was a special ad-hoc panel that had a particular agenda about how diversity issues needed to be addressed that was not evidence-based nor warranted within the context of our aims.
I haven’t been on a panel or even an ad-hoc reviewer (yet), but no review process will ever be ideal.
LikeLike
February 10, 2010 at 7:21 am
This is just one of the many, many reasons I think that CSR should be getting people closer to the beginning of their independent career involved with review as soon as possible.
Oh, and I forgot to say I agree with this. Another reason is that is reduces the risk that some delusional fuck-up like Fucklin will end up being allowed to serve.
LikeLike
February 10, 2010 at 5:45 pm
If summary statement is the unit, then I got about 90% of faulty reviews. If individual critiques are counted separately, this brings the number down to about 30% of serious faults, 10-15% being wild nonsense. Twice an interesting scenario occurred: (1) Application was first triaged, then (without any substantial changes) scored close to the payline, and then triaged again on the final resubmission. (2) In the two round system: first scored, with reviewers praising the methodology, but demanding experimental details that were playing only a supporting role (subcontracted, routine, published and commercialized assaying protocol). These details were provided in the resubmission, the application was triaged.
WTF? (this is a rhetorical question, of course; I think I know well enough what the f… is going on)
LikeLike
February 25, 2010 at 9:07 am
How can a proposal get a decent score on first review and then after all comments are addressed not be scored on the second review? This makes no sense and only confirms that the NIH peer review process is broken or worse. Clearly no one is supervising the process. Ultimately this will kill innovation which is already on life support in the US.
LikeLike
February 25, 2010 at 9:30 am
How can a proposal get a decent score on first review and then after all comments are addressed not be scored on the second review? This makes no sense and only confirms that the NIH peer review process is broken or worse.
The system is overtly set up so that scores for revised applications are not anchored to the score of the prior version. Scoring is supposed to be within-round *only* from the perspective of the reviewer. The fact that reviewers are human and have a distinct tendency to anchor scores to the prior one is a bug, not a feature, from the CSR/NIH perspective.
You could argue with the NIH rationale for wanting this to be the case but if reviewers do what they are asked this is no evidence for a broken system or lack of supervision of the process. One might observe it is the opposite.
A second consideration is that your proposal may simply have gotten worse. Not all revisions are improvements. Or, a different set of reviewers may simply reflect natural variance in a tricky judgement- perhaps your original score was at the top end of that variability for your “true” score.
LikeLike
February 25, 2010 at 10:27 am
Or more likely, there’s no such thing as a “true” score and all such scoring attempts are inherently specious.
I love the argument that the NIH has to fund all kinds of human-irrelevant “basic science” because we just can’t predict what kind of science will lead to the next big breakthrough, but then the NIH sets up this elaborate scoring system to figure out what to fund on the exact opposite predicate, that it is possible to identify today the proposals that will produce the best science tomorrow.
LikeLike
February 25, 2010 at 10:54 am
“but then the NIH sets up this elaborate scoring system to figure out what to fund on the exact opposite predicate”.
Wimple,
The goal has been “to fund the best science with the less administrative burden possible”.
To fund the best science. Obviously, not all best science will lead to big breakthrough, some will. This outcome is not unexpected according to historic records.
LikeLike
February 25, 2010 at 11:30 am
whimple. homes. you’ve been slightly grumpier than usual lately. did you get a non-fundable grant score? I feel your pain. Group hug?
LikeLike
February 25, 2010 at 12:15 pm
Paylines are
LikeLike
February 25, 2010 at 12:17 pm
Comment got chopped. With paylines under 10%, it’s a pretty big group needing a hug. 🙂
LikeLike
February 25, 2010 at 12:43 pm
Payline under 10% is insane and I agree that there must be a rather sizeable group needing a hug. That payline needs to go higher, at least up to 20%. So, one thing NIH could do is to have a better look at multigrant holders and find out if there is justification ( in terms of science and clinical goals) for that ‘out of ordinary’ situation. We are at an “extraordinary time” and “extraordinary measures” might be needed for the best science to be funded.
LikeLike
May 18, 2010 at 3:32 pm
Who the heck serves more than one freaking appointed term? Cripes.
LikeLike