In NIH Grant Review, "Help" For the Applicant is Incidental
April 3, 2012
This is my understanding, anyway. The job of the reviewer of the NIH grant application is to discuss the merits and weaknesses for the audience of the Program Staff within the ICs. One is supposed to be helping them to make funding decisions with respect to the current pool of applications.
The job is not to help the applicant improve her chances the next time. Nor is it to help the application be more attractive for the PO/IC.
This is my understanding, anyway.
See ProfLikeSubstance’s comment.
DrugMonkey 
April 3, 2012 at 10:38 am
I think this is technically true for NSF as well. My point was in response to Whoosh’s comment above mine, that ad hoc reviews do not get the same consideration as the panel ones, IME. Therefore the actual function of the ad hocs (barring a major difference from the panelists that was overlooked or an ad hoc chorus of dissent) serves the PI more than it does the panel. Perhaps this is why there is so much effort right now in the BIO directorate to reduce the load on ad hocs.
LikeLike
April 3, 2012 at 3:35 pm
This is demonstrably false, as all of the tweaking of the new critique format and all of the reviewer exhortations have been with the explicit goal of helping the applicant (and not just program or council) to understand how the overall impact score was arrived at. They changes from bulleted Overall Impact section to a narrative paragraph format because *applicants* went shittenuttes, not program or council. And note that the critique template contains a section called “Additional Comments to the Applicant”. This implies that the rest of the critique contains what are “comments to the applicant”; otherwise it would make no sense to refer in the title of that section to “additional” comments to the applicant.
One major purpose of the critique and summary statement is to help the applicant understand why she received the score she did.
LikeLike
April 3, 2012 at 9:16 pm
I am entirely with CPP on this. The planets have aligned!
LikeLike
April 4, 2012 at 3:56 am
On the contrary PP! The “Additional Comments for the Applicant” places the emphasis on the three final words. It implies the foregoing critique is not “for the Applicant” but rather directed at someone else.
LikeLike
April 4, 2012 at 3:59 am
You are also incorrect, btw, about the justification language that was used to defend the change to bullet points. It talked mostly about making the points really clear for *Program* and in fact applicants have been going “shitnuttes” about how the bullets provide too little interpretive context.
LikeLike
April 4, 2012 at 6:25 am
No, you are wrong, confused about the history, making shitte uppe, and exhibiting poor reading comprehension.
Slow down and read more carefully: What I was referring to was the change from bullet points in the Overall Impact section back to a narrative paragraph, which was done because *applicants* went particularly shittenuttes about the bullet points in the Overall Impact not sufficiently helping them to understand why their score sucked.
As far as “Additional Comments for the Applicant” and your claim that it “places the emphasis on the final three words”, you are just pulling that out of your asse and have no evidence for your contention. The plain English of the phrase implies that there were other comments for the applicant, and this section is providing additional ones. If the rest of the critique was supposed to be directed not to the applicant, but rather to program and council, then this section would be entitled “Comments for the Applicant”, to distinguish it from the rest of the critique.
So, in summary, you are totally full of shitte. Anyway, all you are doing is lamely trolling disgruntled self-absorbed delusional applicants who think the world revolves around them and that the reviewers and study section owe it to them to make them feel good and to fund their science. You can do better than this, holmes.
LikeLike
April 4, 2012 at 9:00 am
PP,
someone is sure as hell pulling his opinion out of his ass but it ain’t me.
I suggest you go back to the NIH notices and documentation
http://grants.nih.gov/grants/guide/notice-files/NOT-OD-09-024.html
http://enhancing-peer-review.nih.gov/guidance_reviewers.html
http://enhancing-peer-review.nih.gov/implementation_goals.html
are all excellent places to start but make sure you drill down to the PDF docs on review as well.
Click to access Talking_Points_for_SROs.pdf
http://grants.nih.gov/grants/peer/reviewer_guidelines.htm
are very helpful.
What you will find is that the emphasis is on making the reason for the present impact score very clear. Also for making the relative merits and weaknesses clear. What you will not find one bit of is making it clear to the applicant how she could improve her proposal in revision.
Now, let us turn to the 2008 report on the peer review of NIH grants that is (allegedly, allegedly) the touchstone or bible for all these new changes.
The second goal listed in the Executive Summary makes it clear “Goal: To focus on the merit of the science presented in the application and not the potential improvements that may be realized following additional rounds of review”
This should be clear enough even for PP to understand.
p11, overview of peer review: The objective of the first stage of peer review is to evaluate and rate the scientific and technical merit of proposed research or research training.
Yep, “evaluate and rate”. Not a word to be said in this section about helping the applicant to improve for next time.
p15, overview of CSR goals…again, the identification of highly meritorious proposals is noted, no sign of any effort to improve proposals for the next time around.
p16-20, “consulatation” with interested parties reveals a lot of interest in clear, unambiguous feedback, sure, but nothing about helping applicants to improve. I.e. the reason for the ranking is the goal, not the way to improve. So this goal is no criticism of my position either. clear writing and justification for the score is what Program is after. The communication with the applicant is incidental.
p.22: “lack of useful feedback”….aha, the first part that reflects PP’s position, but of course this is an applicant whine….make sure to keep your eyes out in the subsequent policy documents on review as to whether this “useful” business ever is used. if not, why not eh?
p23, “core values” working groups: decoupling “guidance” from reviewers by restricting direct responses of applicants and reviewers to any previous reviews;
hmm, seems pretty clear to me that review is not about helping applicants but about identifying the merits and demerits of the proposal under review.
p.24 The use of multiple-criteria scoring, to explicitly measure multiple dimensions of an application, could help ICs to make funding decisions using the information provided during review.
Nothing in this section about helping the applicant….
p25: Providing feedback to all applicants is an important part of ensuring review quality and fairness. However, there is strong consensus that it is not the job of a reviewer to guide applicants by suggesting methodological fixes to a proposal that is inherently flawed and non-competitive with respect to potential impact or other key issues.
again, the same refrain….
P30-33: discussion of the ways review should be used to guide applicant responses seems to address PP’s position. But the emphasis, you must admit, seems to be on how to persuade applicants to give up on hopeless cases. and the point about making all applications new, without any chance for revision seems to dovetail with the prior comments about not fixing applications for the applicant. Thereby undercutting any suggestion that this is about writing clearly for the applicant. After all, the simplest way to communicate “don’t resubmit this dog” would be to actually say so. Although I will agree that NIH doesn’t appear to have adopted the most-effective of these suggestions, i.e. to expand the use of the dreaded NRF.
p. 41: This enriched information enables the NIH to consider public health and scientific need when making funding decisions, which helps the agency carefully steward its resources.
ok….the report goes on to cover some other topics and you should all definitely read it. With an eye to what has emerged in policy changes and what has not emerged….some parts were clearly rejected or put off for the future. That in and of itself is meaningful action in my view.
go back to the first few links and read over anything that is a notice and you will start seeing what was used to guide current policy and what was not.
Read all this over and you will be very hard pressed to support a position anything other than mine. I.e., that the job of the peer reviewer is to identify the merits and weaknesses of a current application so that Program can decide what to fund. Period. That is the audience. The stuff directed at the applicant is all leaning toward allowing her to understand the reasons for the score but nothing more. The applicant is in no way a primary audience.
LikeLike
April 4, 2012 at 10:20 am
It makes me sad when Mommy and Daddy fight.
LikeLike
April 4, 2012 at 3:29 pm
It makes me sadde when mommy is so fucken longwinded thatte I need to stop and go take a fucken whiz halfway through her comment.
LikeLike
April 5, 2012 at 6:15 am
At last study section, we were explicitly told “Do not make suggestions on how to improve the grant”. “Discuss only the merits & weaknesses of the document in front of you”. And, (off the record) “the words ‘should consider doing…’ do not belong in an NIH review”.
When someone commented that they wanted to help the applicant, Staff shook their head and said “not your job”.
The final section “Additional Comments to the Applicant” is for grantsmanship issues (or so we were told).
LikeLike
April 5, 2012 at 7:00 am
Shorter arzey: ” DM is totes right and CPP is a big ol bag of wrong”
LikeLike
April 5, 2012 at 9:29 am
Defining explicitly and clearly the strengths and weaknesses of the proposal DOES help applicants (AND Program), even if that is not the purpose of the review. Sometimes people seem to confuse the idea of providing a rationale for your evaluation of flaws in the application with improving or rewriting the grant for the applicant. If I as a reviewer don’t feel that a certain approach is optimal, there is no need to suggest new methods or approaches, but I have always tried to specify why that method may be problematic.
LikeLike
April 5, 2012 at 10:26 am
You are conflating “help the applicant understand why the impact score was what it was” with “help the applicant improve their grant”. I have only been talking about the former.
LikeLike
April 5, 2012 at 11:21 am
What part of
in the original post was so mysterious to you, PP?
LikeLike