Is your NIH PO a little….grouchy?

Another one, paraphrased from multiple correspondents:

Dear DM: 
is it just me or are all the POs getting increasingly grouchy and unhelpful?

A. Reader

I am not certain, since I hardly have a representative sample. But I’d say no, this is probably just a bad run for you.

When encouraging you to interact with your Program Officer(s) I tend to emphasize the useful interactions that I have experienced. Consequently I may fail to convey that most of the time they are going to be unhelpful and even discouraging.

Try to see it from their position. They hear from dozens of us, all complaining about some dirty review deed that was done to our application and looking for help. Round after round, after round.

They cannot help everyone.

So take it in stride, as best you can, when you get a seemingly dismissive response. This same PO may become your best advocate on the next one*.

__
*and then treat you like effluvium again after that. It’s happpened to me, I can tell you.

What would you ask Sally Rockey?

Apparently Sally Rockey, NIH Deputy Director in charge of the Office of Extramural Research is on some sort of University tour because I have received several queries lately that go something like this:

Dear Drugmonky:
Sally Rockey will be visiting our University soon and I have the opportunity to ask a question or two if I can get a word in edgewise between all our local BigWig voices. Do you or your Readers have any suggestions for me to add to my list of potential things to ask her?
A. Reader

I have my thoughts and suggestions, of course, but mostly my Readers know what those are.

How about you folks in the commentariat? What would you ask Sally Rockey if you had her in a small room with your peers?

Sustaining NIH funding then and now: 58% as many Type 2 awards in FY2014

Datahound has a cool new analysis posted on the distribution of competing continuation R01/R37 awards (Type 2 in NIH grant parlance).

There is one thing that I noticed that makes for a nice simple soundbite to go along with your other explanations to the willfully blind old guard about how much harder the NIH grant game is at the moment.

Datahound reports that in FY 1995 there were 2653 Type 2 competing continuation R01/R37 awards funded by the NIH. In FY 2014 there were only 1532 Type 2 competing continuation R01/R37 grants awarded.

I make this out to be 58% of the 1995 total.

This is a huge reduction. I had no idea that this was the case. I mean sure, I predicted that there would be a big decline in Type 2 following the ban of A2 revisions*. And I would have predicted that the post-Doubling, Undoubling, Defunding dismality would have had an impact on Type 2 awards. And I complained for years that the increasing odds of A0 apps being sent into the traffic holding pattern itself put a kibosh on Type 2 because PIs simply couldn’t assume a competing continuation would be funded in time to avoid a gap. Consequently PIs were strategically putting in closely related but “new” apps in say Year 3 of the original noncompeting interval.

But I think if I had been asked to speculate I would have estimated a much smaller reduction.

__
*I can’t wait until Datahound brackets this interval so we can see if this was the major effect or if the trend has developed more gradually since 1995.

Nature is not at all serious about double blind peer review

The announcement for the policy is here.

Before I get into this, it would be a good thing if the review of scientific manuscripts could be entirely blind. Meaning the authors do not know who is editing or reviewing their paper- the latter is almost always true already – and that the editors and reviewers do not know who the authors are.

The reason is simple. Acceptance of a manuscript for publication should be entirely on the basis of what is contained in that manuscript. It should rely in no way on the identity of the people submitting the manuscript. This is not true at present. The reputation and/or perceived power of the authors is hugely influential on what gets published in which journals. Particularly for what are perceived as the best or most elite journals. This is a fact.

The risk is that inferior science gets accepted for publication because of who the authors are and therefore that more meritorious science does not get accepted. Even more worrisome, science that is significantly flawed or wrong may get published because of author reputation when it would have otherwise been sent back for fixing of the flaws.

We should all be most interested in making science publication as excellent as possible.

Blinding of the peer review process is a decent way to minimize biases based on author identity, so it is a good thing.

My problem is that it cannot work, absent significant changes in the way academic publishing operates. Consequently, any attempts to conduct double-blinded review that does not address these significant issues is doomed to fail. And since anyone with half a brain can see the following concerns, if they argue this Nature initiative is a good idea then I submit to you that they are engaged in a highly cynical effort to direct attention away from certain things. Things that we might describe as the real problem.

Here are the issues I see with the proposed Nature experiment.
1) It doesn’t blind their editors. Nature uses a professional editorial staff who decide whether to send a manuscript out for peer review or just to summarily reject it. They select reviewers, make interim decisions, decide whether to send subsequent revised versions to review, select new or old reviewers and decide, again, whether to accept the manuscript. These editors, being human, are subject to tremendous biases based on author identity. Their role in the process is so tremendously powerful that blinding the reviewers but not the editors to the author identity is likely to have only minimal effect.

2) This policy is opt-in. HA! This is absurd. The people who are powerful and thus expected to benefit from their identity will not opt in. They’d be insane to do so. The people who are not powerful and are, as it happens, just exactly those people who are calling for blinded review so their work will have a fair chance on its own merits will opt-in but will gain no relative advantage by doing so.

3) The scientific manuscript as we currently know it is chock full of clues as to author identity. Even if you rigorously excluded “we previously reported…” statements and manged to even out the self-citations to a nonsuspicious level (no easy task on either account) there is still the issue of scientific interest. No matter what the topic, there is going to be a betting gradient for how likely different labs are to have produced the manuscript.

4) The Nature policy mentions no back checking on whether their blinding actually works. This is key, see above comment about the betting gradient. It is not sufficient to put formal de-identification in place. It is necessary to check with reviewers over the real practice of the policy to determine the extent to which blinding succeeds or fails. And you cannot simply brandish a less than 100% identification rate either. If the reviewer only thinks that the paper was written by Professor Smith, then the system is already lost. Because that reviewer is being affected by the aforementioned issues of reputation and power even if she is wrong about the authors. That’s on the tactical, paper by paper front. In the longer haul, the more reputed labs are generally going to be more actively submitting to a given journal and thus the erroneous assumption will be more likely to accrue to them anyway.

So. We’re left with a policy that can be put in place in a formal sense. Nature can claim that they have conducted “double blind” review of manuscripts.

They will not be able to show that review is truly blinded. More critically they will not able to show that author reputational bias has been significantly decoupled from the entire process, given the huge input from their editorial staff.

So anything that they conclude from this will be baseless. And therefore highly counterproductive to the overall mission.

A lesson on Supplemental Materials

What happens when you bury data in supplemental figures b/c reviewers ask for it? Another lab publishes a primary paper on same topic.

— chemstructbio (@chemstructbio) February 24, 2015

Good! That’s my response. It is fantastic if someone can publish a paper on stuff that was essentially hidden in the Supplementary Materials of some other paper. 

This is great news. 

Brian Williams, Bill O’Reilly and the RealProblemTM with the alleged profession of journalism

Brian Williams’ evolving story..

“We”. “Our”. “in front of us”. “all four of our low-flying Chinook took fire”

Bill O’Reilly’s alleged war journalism story has been covered by David Corn who details how O’Reilly uses terms like “active war zone”, “combat situation” and “I’ve been there”.

What really chaps my hide is not that Brian Williams eventually conflated* all of his reporting in his own mind into it being the helicopter he was riding in that took a hit from a RPG. It is not the fact that eventually, at one point, O’Reilly directly conflates** his presence reporting the Argentine / GB conflict from Buenos Aires with the actual combat operations in the Falklands by saying “a war zone situation, in Argentina, in the Falklands”.

What I deduce from all the he said/ she said is that Williams was indeed flying around in a Chinook when one of them in the group got hit by RPG. This appears to have been miles away from the chopper Williams was in and they were all ordered down to the ground for related or unrelated safety issues. It also seems reasonable that perhaps the chopper Williams was in was hit by the odd AK-47 round.

O’Reilly, it seems, was in Buenos Aires and never in the Falklands, over a thousand miles away. He was probably in a street protest. Probably, there were armed authorities, either police or soldiers present at the street protest. It may or may not have been a threatening and frightening situation to each individual journalist but there is no evidence of authorities firing on civilians to any large extent.

With this understanding of the probable facts, go back and look at how Williams AND O’Reilly carefully parse their words. You can see how carefully they select the words they use to describe things, how tenderly they craft their story to generate a false impression without actually lying. They want you to come away from their reporting with a feeling that they were deep in the danger. In O’Reilly’s case, he seems mostly to deploy this for the purpose of bolstering his war-time correspondent journalism street cred, long after the primary reporting was done.

No matter.

This speaks to how the professional journalist type views the ethics and acceptable behaviors of their profession.

It is perfectly okay, even desirable, to create an entirely false image in the minds of their audiences just so long as they do not directly tell a clear falsehood. That is what their ethics hinges upon….whether it can be proved they told a lie. Creating a lie in the ear of their audience by using words that are not, strictly speaking, false? That’s perfectly okay. Williams and O’Reilly are only being criticized now because they slipped over the line and said something that was directly falsifiable on the face of their words. Not because they carefully selected superficially true statements to create a false narrative in the mind of their audiences.

This is my problem with journalism.

__
*aha! gotcha.

**aha! gotcha.

Patricia Arquette, Privilege and the Oppression Olympics

I was watching the Oscars last night when Patricia Arquette busted out some equal-pay feminism in her acceptance speech.

“To every woman who gave birth, to every taxpayer and citizen of this nation, we have fought for everybody else’s equal rights,” Arquette said, her voice intensifying. “It is our time to have wage equality once and for all and equal rights for women in the United States of America!”

HECK YEAH!!!!!

I was hooting and hollering too much, I assume, because I got shushed. Apparently some other people in the room wanted to hear what else was being said or whatever.

So it was with some confusion that I saw backlash later on the Twitts about her. It seemed to be of the intersectionality sort of criticism. Also known as the Oppression Olympics. Not to make light of it but look, we all come with various attributes that confer privileges upon us in this society we inhabit. Most of us have one or two attributes that confer the opposite. Some unlucky folks have a pretty tough menu of biases slanting against them. So yeah, there seemed to be a drumbeat of Twitterage against Patricia Arquette’s immense privilege of wealth, whiteness and heteronomativity. I thought at first that this was undeserved, based on what she said from the stage…it’s the Oscars for goodness sake, of course they are all white and perfect and immensely rich.

Then today I finally happened upon her expanded backstage comments. From this account:

“The truth is: even though we sort of feel like we have equal rights in America, right under the surface, there are huge issues that are applied that really do affect women,” she mused. “And it’s time for all the women in America and all the men that love women, and all the gay people, and all the people of color that we’ve all fought for to fight for us now.”

Breathtakingly tone-deaf.

Look, I’ve spent a lot of time in my life feeling sorry for myself. I get it. It is really, really easy to focus narrowly on that one aspect, attribute, experience, factor or misfortune that leaves the self at apparent disadvantage. And it is correspondingly easy to forget all about all the other factors and attributes that have conveyed immense privileges upon our lives.

This is not solved by the data, of course. Firstly, because we can all pick and choose which truthy stat we want to brandish. Is it equal pay? Very easy to brandish the generally accepted, broad brush stats for men versus women. And very easy to ignore that women of color are even more screwed than woman not of color. Easy to have no idea whatsoever how well minority men are paid relative to women not of color. Or what being gay confers in terms of salary.

And it is incredibly seductive to argue the anecdote. Well, Oprah! And J.Lo. And Eddie Murphy! And FFS Neil Patrick Harris is the Master of Ceremonies for goodness sake! They are sitting right there, so therefore why would anyone think of how their respective skin tones and desired life-partner would have anything to do with equal pay for women, eh?

Academic science is no different my friends. If this highly public case makes the intersectionality issue clearer to you than it has ever been, do try to turn that inwards.

We run across these examples on the blog all the time, of course. Whether we’re discussing the struggles of women in science, the Ginther report, outing yourself to search committees or thesis advisors, the Baby Boomer hegemony of NIH Grant funding, postdoctoral pay rates or the evils of PIs with too many grants, the issues are the same.

“Sure, sure, there are these other biases in careers. But what is REALLY important is that I, the speaker, haven’t experienced* any of those advantages that adhere to my classes and characteristics. And let me tell you about my specific set of life events that prove that really, I personally have been at huge disadvantage. So it is totally misplaced to talk about the general advantages of my characteristic X because the anecdote of me proves that X is much less important than this totally other thing that I happen to suffer from.”

At this point one or the other of you, DearReader, may suspect I am talking about you in particular. Naturally, I am not. This is a common theme. Very common.

It is something that I have suffered from in my life and continue to do so. I have felt immensely sorry for myself a lot over the years.

Like many of you, I can claim one or two disadvantages within a context of immense privileges when it comes to pursuing the career of academic science. Like many of you, I CANNOT HELP BUT IGNORE MY PRIVILEGES AND PITY MYSELF ABOUT MY HARDSHIPS. Like many of you, I feel compelled to speak out about perceived injustices in the world. Like many of you, some of those injustices I speak about happen to be ones that I think affect me. Like many of you, some of those injustices I speak about do not happen to affect me in any direct way.

And, like many of you and Patricia Arquette, I often speak about injustices in a way that appears to ignore the fact that other people have it a lot worse.

Social media has a way of helping us to remember that other people have it even worse. And that trying to recruit others to help you in your fights, without ever appearing** to be that concerned about their fights comes across as selfish and tone deaf.

__
*of course you have, you just think that this is totally normal and average and deserved, and thus not worthy of inclusion in any discussion.

**For all I know Patricia Arquette is a huge fighter for underrepresented groups, including ethnic minorities and LGBT folks. But her comments certainly didn’t convey that.

The tldr; version of this post:

I love y'all white leftie women academics. I do. But you are still really myopic self-focused motherfuckers. Like #PatriciaArquette

— Drug Monkey (@drugmonkeyblog) February 23, 2015

On my TL, I hear about black issues from black folks, gay issues from gay folk and feminist issues from all of the above. It's a thing.

— Drug Monkey (@drugmonkeyblog) February 23, 2015

NIH clumsily tries to .. [something] … for grant reviewers

I noticed a funny one in the NIH Guide notices today.

NOT-OD-15-035 Reinforcing Service to the Biomedical Research Community

Yes, yes. I see. “Reinforcement” of a behavior like “Service to the Biomedical Research Community” means increasing the strength or probability of the behavior. So yes, that’s good. What are they trying to do here?

Purpose
This Notice gratefully acknowledges, and seeks to reinforce, service to the biomedical research community by recipients of National Institutes of Health (NIH) research funding (see NOT-OD-10-089). Obtaining input from qualified experts across the entire spectrum of the extramural research enterprise furthers diversity of scientific thought, inclusiveness, and breadth of perspectives necessary to evaluate applications in a review process that strives for integrity and fairness. The interdisciplinary, collaborative, and global nature of biomedical research today requires increasingly complex review panels that need both broad and specific expertise in countless topic areas. Thus, the NIH, the biomedical research community, and the general public benefit from the service of NIH-funded investigators and maximize the Nation’s investment in biomedical research.

Yes, yes. Very nice. but what are they actually doing to reinforce the behavior?

Policy
The NIH expects principal investigators of NIH supported grants and contracts to serve on NIH peer review groups, when asked. Therefore, the NIH expects grantee institutions and R&D contract recipients to encourage their NIH-funded investigators to serve on NIH peer review and advisory groups. These groups include Scientific Review Groups (or “study sections”) in the initial peer review of grant applications and technical evaluation of R&D contract proposals, National Advisory Boards or Councils (NACs) for second-level peer review, NIH Boards of Scientific Counselors (BSCs) for intramural programs, and Program Advisory Committees (PACs) for initiative development and concept review.

emphasis added.

Okay, so any University with a pulse is already encouraging their PIs to serve on study section. Right? They know about how this will help their bottom IDC line, yes? And if they are discouraging any subset of investigators from serving I imagine it is the Assistant Professors…who the NIH / CSR isn’t looking to recruit anyway.

Hmm.

I have a suggestion. Two actually. The first one is hey, if you want to reinforce a behavior, why don’t you use the delivery of a rewarding stimulus? I mean sure, you give us reviewers a delay in the submission deadlines, that’s cool and all. But obviously the NIH thinks they need something more. How about protection from budget reductions? A couple of extra percentile points on newly competing awards?

No?

Okay, that costs you money, I realize. How about something very cheap with some motivational value? Journals often publish a list of their reviewers at the end of the calendar year and thank them for their service. It’s nice. But the NIH can do this one better. Set up a website with a list of reviewers and the number of grants they’ve been assigned to review. Maybe do it by year too and provide permalinks.

Trust me, academics will eat this up. They will check out how many reviews their buddies are/are not doing and give them a little hell for not matching up around the conference coffee table. They will start linking to their entry from their websites and bragging about it in their P&T documentation.

I wonder. Really, NIH. Do you have anyone making policy that understands people even the tiniest little bit? I am about the opposite of a people person and it took me like two tweets to think of this.

Poll: graduate student applicant standards

What are your program’s standards for GRE percentiles and GPA?

That is, what would be the minimum score that would be essentially unremarkable, and require no other compensating attributes, to justify an invitation to interview?

Thought of the Day

I think I have received at least four NIH grants on the topic of “how in the heck is there very little science done on this entirely obvious (to me) idea?”.

If you care about the Emeritus NIH Grant…

…you need to take action on the official form.

Click through and get that done.

“We’ve seen it all before, this is just a cyclical downturn for the NIH funding”

Popular thought. But it is nonsense.

A close collaborator was recently experiencing this common denial trope from one of the more established type of scientists. The thinking is that

“…sure, things are tough for younger scientists right now but hey, things have been tough before. It’s all just a cycle and oh, stop complaining kiddos. We had it hard too.”

Here is why it is in error to argue this- the magnitude of the downturn was lesser and it lasted for a shorter duration in those prior “cycles”. Let us refer to the infamous Undoubling graph.

Figure 1. NIH Appropriations (Adjusted for Inflation in Biomedical Research) from 1965 through 2007, the President’s Request for 2008, and Projected Historical Trends through 2010.
All values have been adjusted according to the Biomedical Research and Development Price Index on the basis of a standard set of relevant goods and services (with 1998 as the base year). The trend line indicates average real annual growth between fiscal years 1971 and 1998 (3.34%), with projected growth (dashed line) at the same rate. The red square indicates the president’s proposed NIH budget for fiscal year 2008, also adjusted for inflation in biomedical research.

The previous downturns in the NIH funding (and you can verify the scientist complaining by looking through old Science magazines, btw) occurred approximately in the late 1960s, the early 1980s* and the early 1990s. I happened to join this career path right around the 1990s downturn and I remember the whining about grant funding quite clearly. That 1990s downturn was what led to the infamous NIH Doubling. The late 1960s downturn led to Congressional action as well. In both cases you can see where the lapse in Congressional interest led to the following episode of downturn. It is here that we should also review the subsequent update on the Undoubling graph, the even more sinister Defunding Graph.

Via Michael White, presumably via John F Sargent, Jr.

It should be emphatically clear to even the casual observer that the magnitude of the decline in the NIH budget and the duration of the downturn prior to the next Congressional rescue differs. Dramatically. Make sure you check the corresponding longitudinal trends in grant success rates. In case you are wondering about the most recent numbers, according to Sally Rockey, the overall RPG success rates for FY 2012-2014 are 17.6%, 16.8% and 18.1%, respectively. Things are most emphatically not good for the kids these days.

These are the facts. We can argue until the cows come home over how and why various up and down cycles have occurred. We can dispute whether Congressional appropriations intended to rescue the NIH extramural community do harm, good, a balance of the two and what this means for the future.

It is not optional, however, to act like the present downturn is of the same magnitude or impact as the prior ones.
__
*”I remember multiple study section rounds in which nothing ended up getting funded” –a senior colleague

Generational privilege under the NIH system in one easy video

Another key bit of information to which I frequently refer when describing generational privilege is depicted in this video from the NIH.

Facts matter.

Repost: More data on historical success rates for NIH grants

Our recent discussion of topics related to the Emeritus Award being considered by the NIH powers that be has been robust. I, of course, have been reminding one of the target demographic scientists that she and her generation have had a pretty good run under the NIH system. It seemed like a good moment to remind everyone that there are data upon which to base our understanding of how difficult it has and has not been for various scientific generations. Time to repost an older blog entry.

This was first posted 11 July 2012.

---

Thanks to a query from a reader off the blog and a resulting request from me, our blog-friend microfool pointed us to some data. Since I don’t like Tables, and the figure on the excel file stinks, here is a different graphical depiction:

The red trace depicts success rates from 1962 to 2008 for R01 equivalents (R01, R23, R29, R37). Note that they are not broken down by experienced/new investigators status, nor are new applications distinguished from competing continuation applications. The blue line shows total number of applications reviewed…which may or may not be of interest to you. [update 7/12/12: I forgot to mention that the data in the 60s are listed as “estimated” success rates.]

The bottom line here is that looking at the actual numbers can be handy when playing the latest round of “We had it tougher than you did” at the w(h)ine and cheese hour after departmental seminar. Success rates end at an unusually low point…and these numbers stop in 2008. We’re seeing 15% for R01s (only) in FY2011.

Things are worse than they’ve ever been and these dismal patterns have bee sustained for much longer. If we look at the ~30% success rates that ruled the day from 1980-2003, the divergence from the trend from about 1989 to 1996 was interrupted in the middle and, of course, saw steady improvement in the latter half. The badness that started in FY2004 has been 8 unrelieved Fiscal Years and shows no sign of abatement. Plus, the nadir (to date) is much lower.

Anyone who tries to tell you they had it as hard or harder at any time in the past versus now is high as a kite. Period.

Now, of course, it IS true that someone may have had it more difficult in the past than they do now, simply because it has always been harder for the inexperienced PIs to win their funding.

sourceAs we know from prior posts, career-stage differences matter a LOT. In the 80s when the overall success rate was 30%, you can see that newcomers were at about 20% and established investigators were enjoying at least a 17%age point advantage (I think these data also conflate competing continuation with new applications so there’s another important factor buried in the “Experienced” trace.) Nevertheless, since the Experienced/New gap was similar from 1980 to 2006, we can probably assume it held true prior to that interval as well.

A tiny bias goes a long way when it comes to grant review

From ScienceInsider:

Now, a new computer simulation explores just how sensitive the process might be to bias and randomness. Its answer: very. Small biases can have big consequences, concludes Eugene Day, a health care systems engineer at the Children’s Hospital of Philadelphia, in Research Policy. He found that bias that skews scores by just 3% can result in noticeable disparities in funding rates.

T. E. Day, The big consequences of small biases: A simulation of peer review, 2015, Research Policy [epub ahead of print 28 Jan] [Publisher Site]

from the paper Abstract:

When total review bias exceeds 1.9% of grant score, statistically significant variation in scores between PC and NPC investigators is discernable in a pool of 2000 grant applications. When total review bias exceeds 2.8% of total grant score, statistically significant discrepancies in funding rates between PC and NPC investigators are detectable in a simulation of grant review.

Day generated a Preferred Class of applications and a NonPreferred Class of applications and ran a bunch of 3-reviewer scenarios with and without reviewer bias against the NPC applications. As far as I can tell the takeaway conclusion about funding here refers to a situation in which the effective payline is 10%. You will immediately grasp that NIH grant review was a strong contributor to the model parameters.

I will admit I am only able to grasp the main points here and I am in no way able to evaluate the nitty gritty.

But it appears to have a very strong message. Namely, that our introspections that “well, if there is bias it is very tiny so we don’t have to be worried about it” needs to change.

There is something even scarier in this paper. From the Discussion:

The threshold level of bias in this environment seems to be 2.8% of the total possible score of the grant; this is the level at which the 95% CI of the odds ratio “kisses” 1.00. This represents a single reviewer with a bias of 0.75 points (or three reviewers each with biases of 0.25 points), which is less than half (44.4%) of the standard deviation in a single reviewer’s score. What this suggests is that levels of bias which are sub-noise – that is, that are dramatically less detectable than normal variation in reviewer scores – are sufficient to substantially bias the number of funded applications in favor of preferred investigators.

RIGHT???? The bias can be of smaller effect size than many “normal” sources of variability in scoring that we accept as the resolution of the system. And it still leads to a statistically significant bias in funding outcome.

We are talking in recent days about bias in favor of highly established, older scientists. It has been longer but the Ginther report indicating disparity of grant review outcome for African-American PIs is clearly relevant here.

What this simulation cannot do, of course, is to model the cumulative, iterative effects of review bias. Namely, the way that selection of PC applications for funding has a tendency to increase the bias in the reviewer pool, since those beneficiaries become the next reviewers. Also, the way that over the long haul, disparity of the first award can lead to actual quality differences in the subsequent applications because PI #1 had the money to pursue her science and PI #2 did not have as easy of a time generating data, publishing papers and recruiting postdoc talent.