Corruption of NIH Peer Review

The Office of the Inspector General at the HHS (NIH’s government organization parent) has recently issued a report [PDF] which throws some light on the mutterings that we’ve been hearing. Up to this point it has mostly been veiled “reminders” about the integrity of peer review at NIH and how we’re supposed to be ethical reviewers and what not.

As usual when upstanding citizens such as ourselves hear such things we are curious. As reviewers, we think we are trying our best to review ethically as we have been instructed. As applicants, of course, we are curious about just what manner of screwing we’ve suffered at the hands of NIH’s peer review now. After all, we all know that we’re being screwed, right?

“NIH isn’t funding [X] anymore“, we cry. X can be clinical, translational, model system, basic…. you name it. X can be our specific subarea within our favorite IC. X can be our model system or analytical approach or level of analysis. X can be our home institution’s ZIP code, or prestige, or type within the academic landscape.

And of course, our study section isn’t giving us a good score because of a conspiracy. Against X or against ourselves, specifically. It’s a good old insider club, doncha see, and we are on the outside. They just give good scores to applications of the right X or from the right people who inhabit the club. The boys. The white people. The Ivy League. The R1s. Those who trained with Bobs. Glam labs. Nobel club.

Well, well, well, the OIG / HHS report has verified all of your deepest fears.

NIH Has Acted To Protect Confidential Information Handled by Peer Reviewers, But It Could Do More [OEI-05-19-00240; March 2020; oig.hhs.gov; Susanne Murrin, Deputy Inspector General for Evaluation and Inspections.

Let’s dig right in.

In his August 2018 statement on protecting the integrity of U.S. biomedical research, NIH Director Dr. Francis Collins expressed concern about the inappropriate sharing of confidential information by peer reviewers with others, including foreign entities.5 At the same time, Dr. Collins wrote to NIH grantee institutions to alert them to these foreign threats, noting that “foreign entities have mounted systematic programs to influence NIH researchers and peer reviewers.”6 As an example of these programs, NIH’s Advisory Committee to the Director warned NIH of China’s Thousand Talents plan, which is intended to attract talented scientists while facilitating access to intellectual property

…

Additionally, congressional committees have expressed concerns and requested information about potential threats to the integrity of taxpayer-funded research, including the theft of intellectual property and its diversion to foreign entities.8, 9 In a June 2019 Senate hearing, NIH Principal Deputy Director Dr. Lawrence Tabak testified that NIH was “aware that a few foreign governments have initiated systematic programs to capitalize on the collaborative nature of biomedical research and unduly influence U.S.-based researchers

So the rumors are true. It’s about the Chinese. One of the reasons I’ve been holding off blogging about this during the whispers and hints era was this. This may be why NIH itself has been so circumspect. Nobody wants to conflate what looks like racism along with what appears to be state-sponsored activity to take advantage of our relatively open scientific system. Many academic scientists love to bleat about the wonderful international nature of the scientific endeavor. I like it myself and occasionally reference this. I wish it was not inevitably and ultimately wrapped up in geo-politics and what not. But it is. Science influences economic activity and therefore power.

I am on the record as a protectionist when it comes to academic employment in the public and public-funded sectors. I don’t think we need hard absolute walls but I also think in hard times, we raise serious and very high barriers to funding NIH grants to foreign applicant institutions. I think, of course, that we need to take a harder look at employment politics. Like any other sector, immigrant postdocs and graduate students often devalue the labor market for domestic employees. I’d like to see a little more regulation on that to keep opportunities for US citizens prioritized.

But I also appreciate that we are an immigrant nation founded on the hard work of immigrants who often ARE more eager to work hard than native born folks (of which I am one, people. I’m including myself in the lazy sack category here). Hard. So we need to have some academic science immigration, of course. And I am not that keen on traditional lines of white supremacy dictating who gets to immigrate here to do science.

So, when I started getting the feeling this was directed specifically at the Chinese, let’s just say the hairs on my neck went up.

But, this report makes it pretty clear this is the problem. They are targeting this “Thousand Talents” effort of China very specifically and are going after US-employed scientists who do not report financial conflicts….from China. And other sources, but…the picture in this report is sharp.

I have heard of more than one local investigator who had a Chinese lab or company who was not reporting this appropriately. They also hold NIH funds and so were disciplined. Grants were pulled. At least one person has disappeared back to China. At least one person is apparently under some sort of NIH suspension but the grants are still running out the clock on the current fiscal year so I can’t quite validate the rumors. A multi-year suspension from grant seeking is being whispered around the campfire.

So what about the reviewers? Where does this come in?

As of November 2019, NIH had flagged 77 peer reviewers across both CSR- and IC-organized study sections as Do Not Use because of a breach of peer review confidentiality. A reviewer who is flagged as Do Not Use may not participate in further study section meetings or review future applications until the flag is removed

…Between February 2018 and November 2019, NIH terminated the service of 10 peer reviewers who not only had undisclosed foreign affiliations, but had also disclosed confidential information from grant applications. For example, some of these reviewers shared critiques of grant applications with colleagues or shared their NIH account passwords with colleagues.

There is a bunch of more of this talk in bullet points about reviewers being suspended or under investigation for both violations peer review and undisclosed foreign conflicts of interest. It could be companies or funding, although this is not clearly specified. Then….. the doozy:

As of November 2019, NIH dissolved two study sections because of evidence of systemic collusion among the reviewers in the section. At least one instance involved the disclosure of confidential information. NIH dissolved the first study section in 2017 and the second in 2018. All grant applications that the study sections reviewed were reassessed by different reviewers.

AHA! There IS a conspiracy against your grants. Look, this is bad. I’m trying to maintain some humor here, but the fact is that this would be relatively easy to pull off, so long as the conspirators were all on board and nobody ratted. What would you need? A third of a study section? A quarter? Half? I dunno but it isn’t *that* many people. Some are in on the main conspiracy (puppeted by a foreign government?), some are willing pawns because their own grants do well, some are just plain convinced by their buddies that this is how it actually works here?. And if they are all in contact, how long would it take? five minute phone conversations about how they need to support applications from A, B and C and run down those likely looking top-scoring apps from X, Y and Z?

I don’t know how they caught these conspiracies but there were probably emails to go along with the forensic evidence on their foreign conflicts of employment, affiliation and funding. Oh wait, the report tells us:

One way NIH learns about instances of possible undue foreign influence is through its national security partners. Since 2017, NIH has increasingly worked with the FBI on emerging foreign threats to NIH-funded research. NIH reported that in 2018, the FBI provided it with referrals of researchers—some of whom were also peer reviewers—who had NIH grants and were alleged to have undisclosed foreign affiliations.

It also says that program staff may have noticed papers that cited funding that has not been disclosed properly (on the Other Support that PIs have to file prior to funding, I presume).

As of November 2019, NIH determined that allegations against 207 researchers were potentially substantiated. Of those 207 researchers, NIH determined that 129 had served as peer reviewers in 2018 and/or 2019. NIH designated 47 of these 129 peer reviewers as Do Not Use. When OIG asked NIH about the remaining 82 peer reviewers—i.e., those who had potentially substantiated allegations but who had not been designated as Do Not Use—NIH did not respond.

What the heck? Why not? This is the IG ffs. How do they “not respond”?

Between February 2018 and November 2019, NIH confirmed 10 cases involving peer reviewers who were stealing or disclosing confidential information from grant applications or related materials and who also had undisclosed foreign affiliations. Two of these 10 cases involved peer reviewers who were selected for China’s Thousand Talents program. The breaches of confidentiality included disclosing scoring information, sharing study section critiques, and forwarding grant application information to third parties. In some of these instances, reviewers shared confidential information with foreign entities.…In two cases, NIH dissolved a study section

So the worst of the worst. How long had this been going on? How many proposals were affected? How many ill gotten grant awards aced out more legitimate competitors? Were those PIs made whole (hahaha. Of course not.) For the dissolved study sections, just how bad WAS it?

Look, I’m glad they caught this stuff. But I have no confidence that we are getting anything even remotely like a full picture here. The tone seems to be that this was sparked by some pretty egregious violations of Other Support declarations leading to scrutiny of those PIs who happened to review grants. The NIH then managed to find evidence (confessions?) of violations of peer review rules. The description of the actual peer review violations leans heavily on inappropriate disclosure of confidential information. Showing critiques and grants to people who have no right to see them. Is this all it was? This is what led to a study section dissolution? Or, as I would suspect, a lot more going on with grant-score-deciding behavior? That is what should lead to dissolution of a section but it is a lot harder to prove than “clearly you gave your password to someone who is logging in from half a world away two hours after you logged in from the US”. I want answers to these harder questions- how are these conspiracies and conflicts leading to funding for those inside the conspiracy and the loss of funding for those who are not?

NIH is highly motivated to soft-pedal that part. Because they are really, really, REALLY motivated to pretend their system of grant selection works to fund the most meritorious science. Probing into how easy it would be to suborn this process, as a single rogue reviewer OR as a conspiracy, is likely to lead to very awkward information.

I never feel that NIH takes participant confidence in their system of review and grant award seriously enough. I don’t think they do enough to reassure the rank and file that yes, it IS their intent to be fair and to select grants on merit. Too many participants in extramural grant review, as applicants and as reviewers, continue to talk with great confidence and authority about what a racket it is and how there are all these specific unfairnesses I alluded to above.

Well, what happens if reviewers believe that stuff?

“Everybody is doing it” is the most frequent response when scientists are caught faking data, right? Well….

A loss of confidence in the integrity of NIH review is going to further excuse future misdeeds in the minds of reviewers themselves. If the system is biased against model systems, it’s okay for me Captain Defender of Models System Neuroscience, to give great scores to fly grants, right? I’m just making up for the bias, not introducing one of my own. If the system is clearly biased in favor of those soft money high indirect cost professional grant writers than hey, it is totally fair that I , Professor of Heavy Teaching Load Purity to do down their grants and favor those of people like me, right? It’s just balancing the scales.

Because everyone knows the system is stacked against me.

Do it to Julia, not me, Julia!

I think the NIH needs to do far more than to blame the dissolution of two study sections of foreign influence and call it a day. I think they need to admit to how easy it was for such efforts to corrupt review and to tell us how they can put processes in place to keep review cartel behavior, explicit OR IMPLICIT, from biasing the selection of grants for funding.

They need to restore confidence.