On SROs and child soccer coaches

As someone who once spent much of his life watching kid soccer games, from the herding cats stage onward, I have many anecdotes. Anecdotes about why I should never coach kids sports, and about why the people who coach kids are saints who deserve many thanks from the parents of their players. But one of my favorite vignettes was only fully fleshed out in retrospect. The coach of one of my kids’ teams would often exhort the players to “anticipate!”. This was somewhere around the late elementary school, perhaps early middle school, age of the players.

And indeed, as a spectator it was indeed frustrating that some of the players, perhaps most, did not seem to be able to read the flow of the game. This is something critical in passing sports such as soccer. Running to where the ball should be, instead of reacting to a pass after it was kicked, was apparently the main goal of the exhortation.

“Anticipate!“, the coach would shout.

Several years later I was chatting with one of the other parents and she said her player went for a good year and a half on this team before finally telling this parent that they did not know what “anticipate” means! The coach was not a dumb man, in fact quite the contrary. But he never stopped to think perhaps he needed to be very specific about what his pleas for anticipation meant and coach the little minions in how to do the thing he expected.

I think the Scientific Review Officers of the Center for Scientific Review of the National Institutes of Health could perhaps learn something from this.

I located this CSR Pilot study when hunting for an illustration of the way that “consensus” on study sections can play havoc with percentile ranks. It led with an unmistakable depiction of score clustering, in this case induced by the whole integer scoring system and the cultural pressure of study sections to “reach consensus”. Reviewers can only use whole integers (lower is better) and the mean is then multipled by ten to get the voted overall impact score. Reviewers typically vote within the range recommended by the three assigned reviewers after the discussion and thus if they “reach consensus” on 2s, 3s or 4s, there end up being a lot of 20, 30 and 40 voted scores. As depicted here in three rounds from 2016.

I cannot remember ever having been on a NIH study section convened by the CSR that did not at one point or another experience the SRO urging reviewers to spread scores. It started with my very first ad hoc invitation where the SRA (the job title at the time was Administrator, now it is Officer) circulated the score distributions from the past dozen or so rounds with a set of lengthy comments about why clustering scores around the perceived payline was bad. In those days the scoring was 1.0-5.0 (lower was better), and the voted average would be multiplied by 100 (in contrast with the current 1-9, multiplied by 10 scheme). We’d be told that the ten 20 point bins from 100 to 299 should have an “ideal” distribution of equal numbers of proposals in each bin. Of course there were too few in the first two bins and 2-3 times the “ideal” number in the 160-179 and 180-199 bins, or something like that, in the SRAs data report. And we’d be told how a score that would be a 20%ile in a flat distribution would creep up to a 25%ile due to score clustering/compression. We were shown that if one plotted the voted scores against the resulting percentile, the slope diverged from ideal. The good grants got worse scores than they should have with none being awarded perfect scores of 100 (remember this is a percentile base, i.e. all three rounds). Somewhere above 22%ile, scores were better than they should have been, and too many proposals were stacking up in that 180-190 zone.

I have never received such detailed instruction about score compression from any other SRO and I have no idea to what extent this was the SRO’s personal approach or general to the times. I was new.

Still, the SRO encouragement to spread scores is invariant up to this day.

I am wondering today if this consistent trend is in part because nobody tells reviewers how to spread scores. Perhaps we reviewers would like to do what we are asked but we just don’t know how to do it.

There are a lot of internal tendencies and assumptions about scoring. There are psychological factors about scoring, particularly acute because reviewers are recipients of reviews. It’s tough to receive a very bad score and tough to hand out worse* ones. There are cultural factors in the discussions of study sections that shape behavior**. And the above graphs have more features that tell the tale.

Reviewers tend to cling to “perfect” scores as if they are precious jewels they personally own and refuse to award them to proposals. Sure, we see the tied-1s uptick in the first graph but the infrequency of anything below a 15 or so, maybe below a 20, is striking. Just like my depiction in the second graph of that older state of affairs. There is a huge drop off in the first graph after voted scores of 50 and it is clear that review panels agreeing on 7s, 8s or 9s does not happen. While those terrible initial scores might be used, but tossed into the triage bin (spoiler, the 9 is almost never seen and the 8 is quite rare), there is no reason a study section cannot decide to use the full range after triage. No reason they couldn’t have their 50%ile grants garnering 90s. But this would require some very explicit instruction. Anticipate!

I may have over-interpreted what the SRA was telling me in the lead up to my first meeting. I decided that if I put the best grant in my pile at 1.0, the worst at 5.0 and used even intervals for the rest of them in-between this would be a way to spread the scores properly.

I came to believe*** everyone should adopt this strategy.

Use the full range for best and worst, and apply even intervals. As a scoring baseline. Then, if you really convinced yourself the best wasn’t really deserving, or the worst was better than garbage, adjust. Or if some of the in-between ones needed to be moved closer together, fine, do that. But the key was to start off highly spread and closely consider violations of spreading on…well, the merits.

I concluded my scheme was unlikely to give out that many 100s if everyone did it, since how often would three assigned reviewers agree it was the best in their pile through discussion and everyone else concurred? Once or twice a year for the entire panel, I bet. And if reviewers did agree, what was the harm in using the full extent of the range by handing out “perfect” scores once or twice a year? It isn’t as if NIH ICs decide to give the applicant extra money just because they received a perfect score, instead of the same percentile with a sub-perfect score. The R37 MERIT extension may be reserved for low single digit percentiles but I’ve never heard it required a perfect score at any IC.

The proposals in the middle of one person’s pile inevitably would have some variability across reviewers, but that was the fodder for discussion and panel voting. Just like usual. It just seemed as though that right hand part of the Actual line above would be pulled straighter, into better agreement with the flat distribution by radical score interval spreading.

Maybe there are other concrete strategies that would help review panels to spread scores. My approach is but one suggestion. But this does require thinking about how/why panels devolve into certain patterns and how to help reviewers overcome these motivations. They need permission to hand out perfect scores (when bizarrely CSR does the opposite, telling us to reserve the 1 for a bestest-lifetime-ever grant). They need permission and emotional support for handing out 7s, 8s and 9s to voted, non-ND, proposals. They need permission to do some sort of forced choice ranking within their piles.

And above all else, they need feedback, using the actual voting data of the panel to show them what is actually being done with respect to scoring.

---

*Back in those days there were no criterion scores. A ND grant did not get any indication of whether it was a just-missed 289 or a 499 disaster. Still, the other members of the panel would know if some reviewers was giving out 5.0s.

**I would hear literal gasps around the table when I said my pre-discussion score was a 1.0.

***I gave up this approach after I left empaneled service. Because, say it with me now, peer review of grants is a communication. And when you are ad hoc, there is minimal time for a substantial number of people on the panel to understand the communication if you use an unfamiliar dialect.

NIH announces 16% more proposals will be considered for funding!

Dudes! It’s good news.

LOL.

A recent NIH Notice (NOT-OD-26-012) includes the first formal announcement of an expanded triage rate that will be used for the next two rounds (at least) of study sections. This broke as a rumor on the socials, but now we have confirmation.

The percent of applications discussed in most meetings will be reduced to 30-35%, instead of the current ~50%.

Of course, this was met with dismay then and we are seeing some additional kvetching today. Obviously getting triaged in study section (aka “Not Discussed” or ND) is not good for your grant. I’m a bit fuzzy on this, but I have always assumed it takes some very heavy lifting to fund a ND grant and this may even be impossible in practice if not in law. A proposal that gets discussed and then gets a 50%ile-plus ranking is more likely (not likely, likely) to be funded than one with essentially identical criterion scores, preliminary overall impact scores and level of criticism/enthusiasm that happens to be ND.

After that we get into the nebulous value of being discussed / scored versus ND. One thought is that the resume of discussion is able to better guide the applicant towards an amended version (or A2asA0) that will be funded. Another is that the applicant isn’t as depressed. Another is that the score benchmarking that is not supposed to guide reviewers of follow-up proposals (amended or A2asA0) still works here and there so this must surely be an advantage. There is reference to internal University or Department policies where being discussed may extend bridge funding that is not accessed with a ND.

I dunno. As good old Report 302 reminds us, proposals that are in the 30th percentile and higher are not that likely to fund. Of the 16,357 R01eqv apps submitted for FY2024, 30.6% (5,001) were funded. Only 0.27% (44) were funded at ranks of 30%-ile or worse. This represents only 0.88% of the funded R01eqv getting to the promised land with a percentile rank within this new triage window. (Another 0.38% (63) of proposals were funded as R56 awards. )

That databook report also shows quite clearly that the rough overall payline was 10%ile. The 50/50 crossover point was around 14-15%ile. Some 4.1% of grants scored at 30%ile were funded but only 2.4% at a 31%ile…and it was all downward from there. We are pretty sure that the multi-year funding will push those numbers downward for FY2025 despite NIH spending out the same budget level. And we surely have to anticipate that FY2026 will be no better in overall budget, may see a reduction and will continue with the multi-year funding plan.

So even if we were facing a permanent change to the triage line, I’m not sure this has much functional impact on us as a whole. But the Notice makes it clear this is just to catchup for the fact that:

The shutdown required that NIH cancel over 370 peer review meetings, impacting the review of over 24,000 applications. The volume of missed meetings complicates NIH efforts to catch up.

My assumption is that this saves time by moving re-scheduled meetings to a single day, likely much easier to reschedule for an entire panel of busy scientists. It also cuts down on the number of resumes of discussion the SRO has to prepare as s/he is deep into the pipeline of preparing for the next round of meetings in Feb/Mar.

The notice even says this move is to help get the proposals into consideration at the scheduled January Advisory Councils.

This seems well worth the rather nebulous “costs” in my view.

Ok, so what about the title of this post, you are now asking yourself. Well, there is an interesting little bone thrown to us.

Applications voted by the committee to be in the middle third will be designated as “competitive but not discussed” and applications in the lowest third will be designated as “not competitive and not discussed”. Applications in the middle third will be considered for funding, along with the discussed applications.

This dovetails with another recently announced policy which is directed at diminishing the relative contribution of percentile rank to the selection for funding. I am just making up this 16% number, but basically this says that the proposals from 33%ile (based on preliminary score, presumably?) to 66%ile ( ok, ok, 66.7%ile) will be in the running for exception pay. Previously, only the 33%ile to 50%ile subset were discussed and in the running for a pickup. Now, those extra 16.7% from 51%ile to 66.7%ile will be similarly considered.

Isn’t this great, guys? Should not we be celebrating, oh ye who were going ballistics about the new triage line?

NIH extends the Oct-Nov deadlines to Dec 8, 2025

The NIH has issued a Notice (NOT-OD-26-012) which expands upon the prior warning (NOT-OD-26-005) about extending the current submission deadlines. It seems to be quite comprehensive.

All grant applications submitted late for due dates between October 1, 2025, and December 5, 2025, will be accepted through 5:00 PM local time December 8, 2025. There is no additional 2-week late window. This notice applies to all relevant Notices of Funding Opportunity (NOFO), including those that indicate no late applications will be accepted. Institutions need not request advance permission to submit late due to the government shutdown and a cover letter providing a justification is not required.

I am still not entirely sure who this is for and what the logic is for extending the deadline. The eRA commons submission system was working as normal during the slowdown for grant submissions. There was no particular reason to miss the normal deadlines as far as submitting the grant goes. The (NOT-OD-26-005) referred cryptically to “access to NIH staff and the help desks as they develop their applications“, but come on. Is this really a lot of proposals?

One suspects the major impact of this will be to allow people who weren’t actually ready to get their proposal submitted to get it in earlier. Or to allow those who were ready to prepare another proposal and submit it.

At my old place of employment, this would be no problem. I could whip out another* grant submission on short notice. At my current bureaucratically enhanced institution, this is not going to happen. Even if I had geared up based on that Nov 14 announcement it would have been a tough an impossible sell to my ridiculously long lead-time grant approval processes.

When the prior warning broke cover there was a little frisson on the twitts about how this was unfair to those of us who made deadline. meh. I am not really all that fussed about it, I don’t think this will amount to very many new proposals coming in. Maybe one per study section? Two? This can’t possibly affect our own chances* very much.

---

*Note, there is nothing in the Notice that says these will be shoehorned into the same study sections in Feb/Mar, but I assume they will be. After all the continuous submission window runs to Dec 15 (I think) so this is not a major new ask.

On the inherent conservatism of NIH grant review

Most of you have by now seen a graph like this one, depicting the most calamitous picture on what is know as the opioid crisis. It is the deaths per 100,000 US citizens each year from 1999 to 2023 by the class of opioid that is involved, as reported by the CDC. Deaths primarily from oxycodone began to rise circa 2000 and hit a sustained level from about 2011 through 2022. Starting right around 2011, there was an increase in deaths from heroin, which reached a peak around 2016 and sustained this through about 2020. Finally, the third wave hit when illicit supplies of fentanyl became available, showing up in increasing numbers of deaths from 2013 onward.

As I say in my grant proposals, deaths are but the tip of the iceberg of the total scope of the opioid problem, since many more people are engaging in problematic use without dying. Yes, we’re going to talk about grants today. In particular, we are going to talk about how NIH funded research responds, or does not respond, to something some of us might describe as an emerging and relatively new health concern.

Now, opioid use for non-medical purposes, leading to addiction, dependence and overdose hazards has been with us for approximately forever, of course. In the most recent handful of decades the impact of heroin, in particular, on various communities within the US has received occasional attention from a political, medical and scientific perspective. The enterprise of science has discovered a ton about how opioids work both medically and not-so-medically. Mostly this has been via research conducted with morphine and heroin, particularly when it comes to animal models.

Most of the opioids of greatest health concern have a similar mechanism of action, but they do differ in some particulars.

In retrospect one of the main drivers of the current opioid crisis was the marketing of oxycodone for pain reduction along with an explicit claim that it had low addiction liability. Low propensity for generating the host of problems associated with non-medical use. As per the above graph, the warning signs that perhaps this was not entirely true grew from 2000 onward and ran for at least a decade as clearly higher risk (from a population numbers perspective) than heroin. Heroin may have particularly nasty effects on the population that was affected….but oxycodone was clearly reaching a lot more people.

You might think that relevant areas of science would leap into high gear to try to determine if the problem was something to do with the neuropharmacological and behavioral properties of oxycodone as it might differ from, say, heroin. Well, here is how my field responded to the growing crisis, represented by published papers identified by the three opioids and “rat self-administration”.

An appreciable literature on heroin just kept chugging away from 2000 to 2010. As you can see, there were comparatively fewer publications on oxycodone even as the oxycodone-related deaths were increasing. There was a minor apparent increase in heroin studies starting around 2012, i.e., just as heroin-related fatalities were on the rise. This also heralded a much-belated appearance of studies on oxycodone, which took until 2019 to get anywhere close to the heroin output.

The field was starting to wake up by the third wave, when illicit fentanyl appeared 2015-2017. It only took until 2019 for my field to start generating more papers on fentanyl self-administration. It still, however, continues to this year to pump out more publications based on heroin self-administration over oxycodone or fentanyl.

As I remarked long ago about sex-differences research on this blog, the funding IS the science.

So what had NIDA chosen to fund across these key intervals of time?

From 2000-2009, there were 18 new R01 funded that hit on the search term “heroin self-administration rat”. Two that hit substituting fentanyl for heroin and one for oxycodone. N.b. some of these searches will be pulling up the same grant, if the application mentioned two or three of these opioids. In this case the one that hit for oxycodone also hit for one of the fentanyl grants and one of the heroin grants.

From 2010-2019 there were 18 new R01 for heroin funded, 3 for fentanyl and 7 for oxycodone. Looking more closely at the latter, this included one new R01 in each of 2014, 2016 and 2017, and two in each of 2018 and 2019. So it was only the latter half of this decade that got things going, grants-wise. Fifteen years after the start of the oxycodone crisis, and maybe 2-3 years after the start of the second wave.

Innovation? Significance?

Look, it would be one thing if NIDA had no interest in opioid research whatsoever but they were funding lots of heroin grants at the same time!

Just to bring us up to date, there were 13 new R01 on oxycodone rat self-administration funded from 2020 to the present. Another 18 on fentanyl and good old heroin is holding steady at 19 new R01 funded.

Of course we do not have access to any data on how many proposals might have been trying to get grants funded on oxycodone or fentanyl self-administration rather than heroin. But it would be very strange if nobody in the opioid fields (or other fields) weren’t thinking about this growing crisis. Very strange if nobody was putting in proposals. I bet some people were. And I bet they were not getting fundable scores due to all sorts of the usual. The people on study section worked on heroin so obviously heroin-related grants were better. The data were mostly in heroin. So making “real progress” was more assured if people proposed using a familiar and well supported model. After all, “everybody knows” that opioids are “all basically the same”.

And…well you can think up all sorts of reasons and justifications for why fentanyl or oxycodone grants just didn’t do very well. And why applicants might have been leery of even submitting them. Or having been kicked in the teeth a few times, why they might have stopped trying.

This is why we need a chance for Program to step in and decide to fund a few of those grants on oycodone or fentanyl instead of the dozen(s) they were funding on heroin. I say chance because in this particular case NIDA was not exactly eager to do this, going by the long delay until some grants got funded. But they could have been. And if they had chosen too, they were well within their rights and usual practices to completely ignore the rankings of peer review (if this is what was happening) to skip over that fifteenth or eighteenth heroin grant to fund an oxycodone proposal.

In which I fruitlessly beg NIH grant-seeking folks to focus on what is actually important.

A new webpage on the NIH site called “Implementing a Unified NIH Funding Strategy to Guide Consistent and Clearer Award Decisions” is causing a small kerfuffle on the socials. As per usual, a number of people are tilting at the wrong target in their ill considered outrage.

There is a minor change that has people all a-tizzy. It has to do with them altering any prior strict payline (quote unquote) ICs decision making on funding grants. Some institutes, this page says “Around half”, previously:

set paylines (akin to a cutoff) based on peer review scores or percentiles as part of their funding decision process. … Applications that fell within the payline would be funded, though not in all instances, and applications could still be funded if they fell outside the payline in special cases.

That also means that around half of the ICs with funding authority do NOT use a strict payline approach. In fact many of these ICs claim* they do not even use a payline. This new policy will apparently move all of the ICs to this approach.

from Kienholz and Berg, 2014

To give you a better picture on what this means at ICs, NINDS has been one of the stricter ones (FY2024 data are messed up, go back to prior years), with a cleaner cutoff for funding. NIDA is one of the other kind. One of the first depictions of the actual funding practices across several ICs was published by Kienholz and Berg in 2014, from FY2012 data they obtained by FOIA request. You can see the relative steepness of the funding cliff at NINDS. The “special cases” making up that above-payline bump are, I have always assumed, from ESI policy. NIDA has comparatively a more gradual slope away from a virtual payline.

Jeremy Berg pioneered this kind of funding transparency when he was Director of NIGMS and eventually NIH as a whole followed suit. By now, you can just go to this page on RePORTER and step through the ICs to see how this works from FY2014 to FY2024.

The new webpage posted today essentially re-states what the other kind of IC has always done, as long as I have been in this business, anyway.

Our funding decisions must balance many competing and dynamic factors when determining the most meritorious research ideas to support. These factors center around peer review, health priorities, scientific opportunities, the workforce, availability of funds, and the wider research portfolio.

This means the stuff I talk about on this blog as “programmatic priorities”. They come in all flavors and are used formally or informally to decide which proposals outside of the de facto payline will get a pickup (formally: exception pay) and which will not.

This is not new. At all.

ICs work up priorities in many formal ways, from long term strategic plans to Advisory Council activities, to Notices, to Director comments at academic meetings, to targeted funding opportunities, etc. There are undoubtedly internal priorities bubbling up from Branches and down from the Director that we may not really hear about in a direct way.

I, for one, endorse the general idea that grant selection should absolutely not be a direct reflection of the percentiles arising from the scores voted at study section.

I beg you, Dear Reader, not to go off in a lather fighting over this thing, which is already a thing at NIH, being extended to all of the ICs. It is a red herring distraction and we don’t want the science friendly media following after a false trail. We don’t want Congress Critters fighting the wrong fights.

The real issue is the nature of these priorities which will be used to decide on grant selection going forward, and the expertise of those making the decisions. And the process for making those decisions. Program officers and Directors were scientists up until now. With considerable expertise and experience within the topic domains of each IC. Advisory Councils were also made up of scientific peers. Policy documents were drafted up by sub-committees of scientists and sometimes policy was arrived upon with RFI input from the broader academic community.

THIS is what we should be monitoring and protecting. The who and the how and the why of applying priorities beyond the study section evaluation.

Not the mere fact of doing so.

---

*This is not really true. It’s a semantic distinction to avoid getting into arguments with whiny PIs who pretend not to understand the multiple layers of decision making. The funding patterns show very clearly that every IC has a sort of virtual payline below which almost everything funds and above which the funding varies ~by percentile rank.

“Nice” NIH study sections screw their applicants because of the way NIH calculates percentile.

I often write blog comments about NIH grant review matters that exist in an uncomfortable tension between what NIH wants us to do on study section and what I see as our professional obligation to try to make NIH do what we want it to do. Part of peer contribution to determining what grants should be funded is inevitably doing things that, you guessed it, determine what grants should be funded.

NIH could not be any clearer that they do not want their peer reviewers determining what grants should be funded on an acute and tactical basis. We evaluate and they decide. I point this out with regularity on this blog. I defend this with regularity on this blog.

But there are many ways in which I think we extramural academic scientists also contribute to what Program should decide. Including by being very intentional about how we evaluate merit of the proposals within NIH’s review structure and desired approach.

As we face yet another contraction of paylines and implicit paylines*, I believe this tension will become increasingly acute over the next few to several rounds of review. NCI famously threatened a 4 percentile payline for the end of FY2025. I don’t know if that was so, but this represents roughly halving their high single-digit paylines of many recent Fiscal Years. Multi-year funding inevitably means success rates (and implicit paylines) are going to be lower in FY2025 and likely for a few more to come, even if the budget remains where it is. As paylines and virtual paylines sink lower into single-digits, the way peer review hands out those paylines gets increasingly important.

A two-percentile shift from 3%ile to 5%ile had no impact at even the stingiest ICs a year or two ago. Now it may mean a clear difference between funded and not-funded.

So what is the problem with nice study sections? Sure, everyone knows that percentiles are there to account for hardass sections that give everything a 30 or worse and easy sections that hand out 20s like tic tacs. But there is a more technical feature of percentile calculation that is not apparent to everyone. and it interacts with the kind of niceness in study sections that reflects being nice to other reviewers and other proposals.

And this is the two sided coin of “consensus”.

When NIH adopted the 1-9 scoring scheme there was a subtext that they actually intended to produce a lot of tied scores at 20, 30 and 40. This was a highly predictable result of study section consensus and data have shown it to have arrived as predicted.

https://public.csr.nih.gov/AboutCSR/HalfPointPilotStudy

[Updated to add 11/25/2025: I located a depiction of the scores here in a NIH/CSR website “A Pilot Study of Half-Point Increments in Scoring“. The takeaway interpretation of their study was that adding the helf-point increments did not affect score spreading. It says absolutely nothing about percentile calculation. Which I chalk up to the usual way NIH goes about evaluating their changes in policy, i.e., misdirection to support their policy moves, rather than seeking true illumination.]

Reviewers nominate pre-Discussion and post-Discussion scores in whole digits. The study section is mostly expected to vote within the range of the post-Discussion scores. Understandably, it is often the case that reviewers “reach consensus” and all give a 2. Or a 3. Or even, less frequently a 4. Even a 1.

So let’s take a theoretical study section that reviews 100 R01 applications in their three rounds of review. Let’s say they “reach consensus” on seven of them at 2 impact scores, which translates into a voted score of 20. I can’t recall a single study section I’ve been on since the new scoring system was put in place that didn’t have at least one application getting a 2-2-2 on post-discussion, and mostly it has been more than one. Let’s also say for didactic purposes that there are three applications in the year that score better than 20, even if only a single person voted a 1**.

NIH calculates percentile as the average rank of tied score applications. So the first three get 1%ile, 2%ile, 3%ile but the seven apps tied at 20 which take up ranks 4-10 all get 7%ile. The 4th ranked grant that should have been a 4%ile has now shot way over the payline. For those ICs that use a soft payline and a robust pickup behavior, there is a problem for the 11th ranked proposal. All of a sudden the 10-11 split is not 10%ile to 11%ile but 7%ile to 11%ile. Forming a perceptual quality distinction that is not at all warranted.

To take another example, suppose only the 4th through 7th proposals are tied at 20? No biggie, the average rank is 5.5, pretty close to the perception of 4%ile, right? Nope. The NIH rounds up. So that now becomes a 6%ile instead of a 4%ile score

It gets worse. Suppose that instead of 100 proposals, the study section has somewhat fewer? Say, 90. Well, now that 5.5 average rank calculates to 6.1%ile so no difference, right? NOPE!

When I say the NIH rounds up, I mean they ROUND UP!

Anything over the whole digit goes UP. A 6.1%ile is rounded up to 7%ile. And honestly I do not know how strict this is but I assume anything over the whole number is rounded up. So 99 apps in the denominator may have this impact.

I remind you this was intentional. At least some of the plan for this scoring system was to try to force Program Officers to stop relying on nonsensically small differences in score/percentile and apply their brain to pick and choose which of tied score grants were going to get funded. Or maybe we should phrase this as allowing them to ignore the tyranny of false distinction.

The result of this is to diminish the impact of study sections in deciding what grants were going to fund and which were not. Study sections which are “nice” by reaching consensus after discussion, and voting in a bunch of tied scores, are diminishing their own impact. Maybe that proposal that should have been the fourth ranked in their section gets a 7%ile instead of a 4%ile, dooming its chances for funding.

Depending on what other study sections are doing, they could even be almost guaranteeing their section results in fewer funded proposals compared with other sections simply because they are “nice”.

Thereby screwing over their own people.

Which isn’t actually very nice.

---

*As always, I remind you to look at the NIH Data Book page on funding by percentiles and start using the dialogs for FY and IC to see what’s what in past funding climates.

**Many years ago when I was appointed on study section a new Chair declared that we “must” reserve our very best scores for superlative applications that were on the order of the best we’d ever reviewed, lifetime. I may have blogged how silly and statistically ignorant it was, I can’t recall. This sentiment, I’m here to tell you, is still with us. So anything below a 20 would be very rare in this approach to review.

Notes on NIH matters as the government opens up again

A tweet from someone apparently positioned to know claimed the damage totals from the government shutdown amounted to 379 study sections postponed, within which 24,380 NIH grant applications were supposed to be reviewed.

That’s an average of 64 proposals per section. If we go by pre-chaos success rates, this is something on the order of 4,876 NIH projects which would eventually be funded.

I was on one of these cancelled study sections and I can report that the SROs have lept into action to try to reschedule the cancelled meetings. Due to the requirement to post meetings for 30 days in advance on the Federal Register, this pushes the first possible dates into mid December, the week before the winter holidays hit. I think it is very unlikely that many of these will be held before January. This means a VERY tight window before the usual late-January scheduling of Advisory Councils that would normally be taking up these now delayed proposals. Maybe NIH ICs can push their Council meetings back by a month? This would seem to be the only way to stay roughly on track for April 1 as the first possible funding date.

Interim good news about the the cancelled study sections. Mine entered read phase during the shutdown, and we were able to upload our critiques. This is slightly annoying now because instead of simply uploading a document template for each review, we had to enter text directly into the web interface. But when the meeting date passed, the study section disappeared from the Internet Assisted Review section of eRA Commons. If anyone had not kept a copy of their review comments, well, there was concern. Luckily, our restored study section has gone back to Submit but all the prior work appears to be intact, at least for my critiques. Phew.

We’ve heard rumor from several directions that the next two rounds of study section will use a 70% triage line. Meaning instead of discussing approximately half of the applications assigned to a given study section, only 30% will be discussed. Referencing the above-mentioned average this takes it from about 32 proposals discussed in a study section to 19 proposals that will be discussed.

Why? Well, this may permit one day meetings in place of two day meetings and it will certainly cut down the amount of time SROs have to devote to writing up the resume of discussion. This latter is not as trivial as it sounds, I estimate. The former may assist with actually scheduling replacement meetings, as it is easier to get 25 busy PIs to coincide on a single free day in their schedule compared with an adjacent pair of days.

People on the socials seem to think this will be bad for applicants. It does suck if you get a ND. From an emotional perspective for sure, but also for a thin advantage you might have for 1) a pickup from program and 2) a better score on the revision. This latter is anecdotal and I don’t think I’ve ever seen data suggesting a discussed grant and a not-discussed grant which have similar criterion and overall impact scores from initial review fare differently in revision.

The NOT-OD-26-005 gives “Interim Guidance on Reopening of NIH Extramural Activities” within which there is one interesting promise.

As of today, we can confirm that we will be rescheduling all October and November grant application submission deadlines (specific dates to be announced in a future Notice).

This sounds like we will be able to submit grant applications to be considered in Cycle III, as if they had been submitted Oct 5-Nov15, etc. This is a bit odd since as far as I know (having submitted a proposal during the shutdown) eRA commons accepted submissions during the shutdown. They haven’t been referred to study section or IC, but they were accepted. So what gives? Why is it important to open up submissions if the usual dates were available?

The rationale for this is given as:

By delaying due dates that occurred both during the lapse in funding and in the week following, applicants will have access to NIH staff and the help desks as they develop their applications.

I dunno that this is convincing, although I’m sure the less research intensive universities and private companies applying for SBIR funds were more likely to have problems. I do wonder what fraction of submissions received by any new (Dec 15?) deadline will be from applicant institutions that genuinely needed NIH staff help on a technical level versus additional proposals from institutions like mine that have no need for such assistance.