Your Grant in Review: Appealing the review outcome

A recent Notice from the NIH (NOT-OD-11-064) indicates that there is a need to standardize and refine the appeal process.
Here’s what struck me on seeing this Notice pop up: I bet there has been a massive uptick in the rate of appeals since the sunsetting of the A2 and the threats to rigorously weed out thinly concealed revisions as “new” submissions.
One viewpoint on the wisdom of appealing the scoring of your grant proposal that is very common is captured in this comment over at the NIGMS blog:

Based on everything I have read about the appeals process on various Web pages of the NIH and Institute Web sites, it seems like you’d have to be extremely foolish and poorly informed to bother appealing.

NIGMS Director Berg responded:

First, applicants are strongly encouraged to discuss issues related to their summary statements with their program directors. In most cases, program staff do advise applicants that a formal appeal is unlikely to be fruitful.
That said, let me provide some data. Over the past 5 years, the NIGMS Advisory Council has acted on 177 appeals. In approximately 2/3rds of these cases, the Council concurred with the Initial Review Group. Outcomes for the remaining cases included deferral for re-review, designation as high program priority, recommendation for consideration for possible funding, recommendation for interim funding, and restoration of funds, depending on the nature of the appeal.

“deferral for re-review” Keep that in your mind for a minute.
Back to the Notice:

Following discussion of concerns with the PO, if the PD/PI and/or an official of the applicant organization wishes to appeal the outcome of the initial peer review process, an appeal letter must be submitted, either in hard copy or electronically, to the PO. The appeal letter must display concurrence from the AOR of the applicant organization for the application. Although the content of the appeal letter may originate from the PD/PI, Contact PD/PI for multiple PD/PI applications, or an organizational official(s) (not necessarily the AOR), the AOR must send the letter directly to the PO, or must send his/her concurrence to the PD/PI who will forward the materials and AOR concurrence to the PO. A communication from the PD/PI or official of the applicant organization (other than the AOR) only or with a “cc” to the AOR will not be accepted.

Emphasis added. This should do quite a bit to cut down on the impulsive whining if PIs have to get their Office of Grants/Sponsored Programs/Extramural Funding to concur with their complaints, eh?

An appeal letter will be accepted only if the letter 1) describes the flaws in the review process for the application in question, 2) explains the reasons for the appeal, and 3) is based on one or more of the following issues related to the process of the initial peer review:
Evidence of bias on the part of one or more peer reviewers.
Conflict of interest, as specified in regulation at 42 CFR 52h.5.”Scientific Peer Review of Research Grant Applications and Research and Development Contract Projects”, on the part of one or more peer reviewers.
Lack of appropriate expertise within the SRG.
Factual error(s) made by one or more reviewers that could have altered the outcome of review substantially.
Appeal letters based solely on differences of scientific opinion will not be accepted.

Emphasis added. Heh. Still, you have FACTUAL ERRORS to hang your hat on. I can assume it will be fascinating to read some of those letters which try to show that a criticism derived from a FactualErrorTM rather than a difference of scientific opinion!
The “evidence of bias” and “Conflict of interest” part is a disaster. There are 20-30 people on the review roster. Three of them reviewed the grant application in depth. And if the grant was triaged (which category are highly likely to be appealed, right?) then only these three saw the grant. The applicant writes up a scathing appeal based on the evidence from the summary statement and the SRO response is “Yeah, um, that reviewer didn’t write that comment, she wasn’t even assigned to the proposal!”. Oh glory. The hilarity.
Or even if the grant being appealed is one that was discussed and juuuuuust missed the payline (another highly likely situation, in my prediction). How the hell is the applicant (who was not in the room for the discussion) to make a case for bias or conflict that materially affected the outcome of the review? How the stones is this supposed to work? There could be a REAL conflict and the appeal could focus on the wrong reviewer! So presumably the appeal would fail, right?
Ok, so you submit your appeal. Then what?

NIH staff will consider the basis for the appeal letter, and evaluate the merit of the appeal. If both review staff and program staff support an appeal, then the original application, without additional materials or modifications, will be re-reviewed by the same or a different SRG. In this case, only the results of the re-review, and not the first review, are made available to Council, and information about the appeal is not made available to Council.
If review staff and program staff do not support the appeal, or do not agree on its merit, the PD/PI and/or an institutional official (not necessarily the AOR) may elect to withdraw the appeal letter….
If review staff and program staff do not support the appeal, or do not agree on its merit, and the appeal letter is not withdrawn, the appeal letter will be made available to Council. The IC may not deny the PD/PI or applicant organization the opportunity to have an appeal letter made available to Council…The Council may concur with the appeal, and recommend that the application be re-reviewed…The Council may concur with the SRG’s recommendation and deny the appeal.
The recommendation of Council concerning resolution of an appeal is final and will not be considered again by the NIH through this or another process.

I think the calculation has changed on the wisdom of the appeal. In prior eras, I totally endorsed PP’s position. It seemed like a tremendous waste of time to appeal when you were 1) going to be denied in the vast majority of cases and 2) if your objection was sustained, all you got out of it was re-review of the same application. It seemed much wiser to me to just revise the damn thing and send it in again, trying to use your arguments about bias or unfairness or FACTUAL ERRORS to guide what study section it went to or which reviewers it did NOT go to.
But now we are in this situation of sharply limited reviews and the possibility that you cannot submit anything similar for three years.
All of a sudden the prospects of inserting a third round of review of a given proposal into the process seems like a decent idea, doesn’t it?