On the inherent conservatism of NIH grant review

Most of you have by now seen a graph like this one, depicting the most calamitous picture on what is know as the opioid crisis. It is the deaths per 100,000 US citizens each year from 1999 to 2023 by the class of opioid that is involved, as reported by the CDC. Deaths primarily from oxycodone began to rise circa 2000 and hit a sustained level from about 2011 through 2022. Starting right around 2011, there was an increase in deaths from heroin, which reached a peak around 2016 and sustained this through about 2020. Finally, the third wave hit when illicit supplies of fentanyl became available, showing up in increasing numbers of deaths from 2013 onward.

As I say in my grant proposals, deaths are but the tip of the iceberg of the total scope of the opioid problem, since many more people are engaging in problematic use without dying. Yes, we’re going to talk about grants today. In particular, we are going to talk about how NIH funded research responds, or does not respond, to something some of us might describe as an emerging and relatively new health concern.

Now, opioid use for non-medical purposes, leading to addiction, dependence and overdose hazards has been with us for approximately forever, of course. In the most recent handful of decades the impact of heroin, in particular, on various communities within the US has received occasional attention from a political, medical and scientific perspective. The enterprise of science has discovered a ton about how opioids work both medically and not-so-medically. Mostly this has been via research conducted with morphine and heroin, particularly when it comes to animal models.

Most of the opioids of greatest health concern have a similar mechanism of action, but they do differ in some particulars.

In retrospect one of the main drivers of the current opioid crisis was the marketing of oxycodone for pain reduction along with an explicit claim that it had low addiction liability. Low propensity for generating the host of problems associated with non-medical use. As per the above graph, the warning signs that perhaps this was not entirely true grew from 2000 onward and ran for at least a decade as clearly higher risk (from a population numbers perspective) than heroin. Heroin may have particularly nasty effects on the population that was affected….but oxycodone was clearly reaching a lot more people.

You might think that relevant areas of science would leap into high gear to try to determine if the problem was something to do with the neuropharmacological and behavioral properties of oxycodone as it might differ from, say, heroin. Well, here is how my field responded to the growing crisis, represented by published papers identified by the three opioids and “rat self-administration”.

An appreciable literature on heroin just kept chugging away from 2000 to 2010. As you can see, there were comparatively fewer publications on oxycodone even as the oxycodone-related deaths were increasing. There was a minor apparent increase in heroin studies starting around 2012, i.e., just as heroin-related fatalities were on the rise. This also heralded a much-belated appearance of studies on oxycodone, which took until 2019 to get anywhere close to the heroin output.

The field was starting to wake up by the third wave, when illicit fentanyl appeared 2015-2017. It only took until 2019 for my field to start generating more papers on fentanyl self-administration. It still, however, continues to this year to pump out more publications based on heroin self-administration over oxycodone or fentanyl.

As I remarked long ago about sex-differences research on this blog, the funding IS the science.

So what had NIDA chosen to fund across these key intervals of time?

From 2000-2009, there were 18 new R01 funded that hit on the search term “heroin self-administration rat”. Two that hit substituting fentanyl for heroin and one for oxycodone. N.b. some of these searches will be pulling up the same grant, if the application mentioned two or three of these opioids. In this case the one that hit for oxycodone also hit for one of the fentanyl grants and one of the heroin grants.

From 2010-2019 there were 18 new R01 for heroin funded, 3 for fentanyl and 7 for oxycodone. Looking more closely at the latter, this included one new R01 in each of 2014, 2016 and 2017, and two in each of 2018 and 2019. So it was only the latter half of this decade that got things going, grants-wise. Fifteen years after the start of the oxycodone crisis, and maybe 2-3 years after the start of the second wave.

Innovation? Significance?

Look, it would be one thing if NIDA had no interest in opioid research whatsoever but they were funding lots of heroin grants at the same time!

Just to bring us up to date, there were 13 new R01 on oxycodone rat self-administration funded from 2020 to the present. Another 18 on fentanyl and good old heroin is holding steady at 19 new R01 funded.

Of course we do not have access to any data on how many proposals might have been trying to get grants funded on oxycodone or fentanyl self-administration rather than heroin. But it would be very strange if nobody in the opioid fields (or other fields) weren’t thinking about this growing crisis. Very strange if nobody was putting in proposals. I bet some people were. And I bet they were not getting fundable scores due to all sorts of the usual. The people on study section worked on heroin so obviously heroin-related grants were better. The data were mostly in heroin. So making “real progress” was more assured if people proposed using a familiar and well supported model. After all, “everybody knows” that opioids are “all basically the same”.

And…well you can think up all sorts of reasons and justifications for why fentanyl or oxycodone grants just didn’t do very well. And why applicants might have been leery of even submitting them. Or having been kicked in the teeth a few times, why they might have stopped trying.

This is why we need a chance for Program to step in and decide to fund a few of those grants on oycodone or fentanyl instead of the dozen(s) they were funding on heroin. I say chance because in this particular case NIDA was not exactly eager to do this, going by the long delay until some grants got funded. But they could have been. And if they had chosen too, they were well within their rights and usual practices to completely ignore the rankings of peer review (if this is what was happening) to skip over that fifteenth or eighteenth heroin grant to fund an oxycodone proposal.

In which I fruitlessly beg NIH grant-seeking folks to focus on what is actually important.

A new webpage on the NIH site called “Implementing a Unified NIH Funding Strategy to Guide Consistent and Clearer Award Decisions” is causing a small kerfuffle on the socials. As per usual, a number of people are tilting at the wrong target in their ill considered outrage.

There is a minor change that has people all a-tizzy. It has to do with them altering any prior strict payline (quote unquote) ICs decision making on funding grants. Some institutes, this page says “Around half”, previously:

set paylines (akin to a cutoff) based on peer review scores or percentiles as part of their funding decision process. … Applications that fell within the payline would be funded, though not in all instances, and applications could still be funded if they fell outside the payline in special cases.

That also means that around half of the ICs with funding authority do NOT use a strict payline approach. In fact many of these ICs claim* they do not even use a payline. This new policy will apparently move all of the ICs to this approach.

from Kienholz and Berg, 2014

To give you a better picture on what this means at ICs, NINDS has been one of the stricter ones (FY2024 data are messed up, go back to prior years), with a cleaner cutoff for funding. NIDA is one of the other kind. One of the first depictions of the actual funding practices across several ICs was published by Kienholz and Berg in 2014, from FY2012 data they obtained by FOIA request. You can see the relative steepness of the funding cliff at NINDS. The “special cases” making up that above-payline bump are, I have always assumed, from ESI policy. NIDA has comparatively a more gradual slope away from a virtual payline.

Jeremy Berg pioneered this kind of funding transparency when he was Director of NIGMS and eventually NIH as a whole followed suit. By now, you can just go to this page on RePORTER and step through the ICs to see how this works from FY2014 to FY2024.

The new webpage posted today essentially re-states what the other kind of IC has always done, as long as I have been in this business, anyway.

Our funding decisions must balance many competing and dynamic factors when determining the most meritorious research ideas to support. These factors center around peer review, health priorities, scientific opportunities, the workforce, availability of funds, and the wider research portfolio.

This means the stuff I talk about on this blog as “programmatic priorities”. They come in all flavors and are used formally or informally to decide which proposals outside of the de facto payline will get a pickup (formally: exception pay) and which will not.

This is not new. At all.

ICs work up priorities in many formal ways, from long term strategic plans to Advisory Council activities, to Notices, to Director comments at academic meetings, to targeted funding opportunities, etc. There are undoubtedly internal priorities bubbling up from Branches and down from the Director that we may not really hear about in a direct way.

I, for one, endorse the general idea that grant selection should absolutely not be a direct reflection of the percentiles arising from the scores voted at study section.

I beg you, Dear Reader, not to go off in a lather fighting over this thing, which is already a thing at NIH, being extended to all of the ICs. It is a red herring distraction and we don’t want the science friendly media following after a false trail. We don’t want Congress Critters fighting the wrong fights.

The real issue is the nature of these priorities which will be used to decide on grant selection going forward, and the expertise of those making the decisions. And the process for making those decisions. Program officers and Directors were scientists up until now. With considerable expertise and experience within the topic domains of each IC. Advisory Councils were also made up of scientific peers. Policy documents were drafted up by sub-committees of scientists and sometimes policy was arrived upon with RFI input from the broader academic community.

THIS is what we should be monitoring and protecting. The who and the how and the why of applying priorities beyond the study section evaluation.

Not the mere fact of doing so.

---

*This is not really true. It’s a semantic distinction to avoid getting into arguments with whiny PIs who pretend not to understand the multiple layers of decision making. The funding patterns show very clearly that every IC has a sort of virtual payline below which almost everything funds and above which the funding varies ~by percentile rank.