NIH Program Officers do not understand what happens during review

May 22, 2015

It is one of the most perplexing things of my career and I still don’t completely understand why this is the case. But it is important for PIs, especially those who have not yet experienced study section, to understand a simple fact of life.

The NIH Program Officers do not completely understand what contributes to the review and scoring of your grant application.

My examples are legion and I have mentioned some of them in prior blog posts over the years.

The recent advice from NIAID on how to get your grant to fit within a modular budget limit.

The advice from a PO that PIs (such as myself) just needed to “write better grants” when I was already through a stint on study section and had read many, many crappy and yet funded grants from more established investigators.

The observation that transitioning investigators “shouldn’t take that job” because it was soft money and K grants were figuring heavily in the person’s transition/launch plans.

Apparently honest wonder that reviewers do not read their precious Program Announcements and automatically award excellent scores to applications just because they align with the goals of the PA.

Ignorance of the revision queuing that was particularly endemic during the early part of my career (and pretend? ignorance that limiting applications to one revision round made no functional difference in this).

The “sudden discovery” that all of the New Investigator grants during the checkbox era were going to well-established investigators who simply happened not to have NIH funding before, instead of boosting the young / recently appointed investigators.

An almost comically naive belief that study section outcome for grants really is an unbiased reflection of grant merit.

I could go on.

The reason this is so perplexing to me is that this is their job. POs [eta: used to] sit in on study section meetings or listen in on the phone. At least three times a year but probably more often given various special emphasis panels and the assignment of grants that might be reviewed in any of several study sections. They even take notes and are supposed to give feedback to the applicant with respect to the tenor of the discussion. They read any and all summary statements that they care to. They read (or can read) a nearly dizzying array of successful and unsuccessful applications.

And yet they mostly seem so ignorant of dynamics that were apparent to me after one, two or at the most three study section meetings.

It is weird.

The takeaway message for less NIH-experienced applicants is that the PO doesn’t know everything. I’m not saying they are never helpful….they are. Occasionally very helpful. Difference between funded and not-funded helpful. So I fully endorse the usual advice to talk to your POs early and often.

Do not take the PO word for gospel, however. Take it under advisement and integrate it with all of your other sources of information to try to decide how to advance your funding strategy.

25 Responses to “NIH Program Officers do not understand what happens during review”

  1. physioprof Says:

    I think the bottom line is that actually nowadays, POs never attend study sections in person–except maybe for RFAs where the special study section is reviewing all grants that they are responsible for–and only, if at all, dial in on the phone and only listen to discussion of their assigned grant(s). This makes it impossible to understand the study section review and discussion dynamics that drive scoring.

    Just as one among many examples of this, I’m sure you’ve experienced panel discussion where without any additional words being said, the oxygen just disappears from the room and an application dies. And in order to understand why grant A got the score it did, you also need to hear the discussion of grants B, C, D, E, etc and see the scores that they got.

    It would be interesting to find out the average number of assigned grants handled by each PO, as increased workload would certainly contribute to this situation.

    Like

  2. physioprof Says:

    Oh, and I should say: This complete absence of POs attending study section in person is based on my own service on a very broad cellular and molecular neuroscience panel, where any one PO couldn’t possibly have more than one or two assigned applications under review. Maybe on narrower panels POs are more likely to attend in person, because they are more likely to have a greater number of assigned apps? I haven’t seen a single PO show up in person at my study section in years.

    Like

  3. Emaderton3 Says:

    It would be nice to even be able to come to a conclusion as to their helpfulness if one could get a hold of them! Emails often go unanswered. And the latest trick I picked up on–full voice mailboxes so that you cannot leave a message!

    That being said, I have had a PO that appeared to be in my corner who really liked a grant and tried to get it funded via select pay.

    Like

  4. drugmonkey Says:

    This makes it impossible to understand the study section review and discussion dynamics that drive scoring.

    While I imagine that this is a strong contributor to worsening the situation, I first noticed this back when there were always a bunch of POs in the room for the study section discussion. And many of the POs I have now were seasoned veterans by that point in their careers even back then.

    I haven’t seen a single PO show up in person at my study section in years.

    I assume that when Scarpa put down the rule to review apps in order of initial scores, instead of trying to cluster by PO or IC assignments, this was the admission (or mandate?) that POs attending in person was over.

    Like

  5. drugmonkey Says:

    That being said, I have had a PO that appeared to be in my corner who really liked a grant and tried to get it funded via select pay.

    Yep. This is one of the best things they can do for you. I am very pro-PO even though this post is about warning noobs that their understanding of the process can be limited.

    Like

  6. lurker Says:

    When I tell my senior colleagues how I’ve been batting 0-20 in grants, the knee-jerk question is always, “Well, did you talk to the PO for advice?” No shit, of course I did and it was mostly useless, validated by DM’s post here, most POs are too aloof to have any real insight anymore, especially to noobs, to whom I doubt they have any energy left to give a fcke.

    The phone calls are just lip service and pretend therapy, at best. Sometimes the therapy is good, like two recent chats I had with two POs who were women and on the “younger” side, actually empathetic, made me feel warm and fuzzy and hopeful my resubs would have fighting chance. Who knows if the advice they gave me was real or made up? With the older dudes, it’s a brick wall of “well I wasn’t there, but looking at notes, you should revise accordingly and resubmit”. Thanks for nothing….I could have just watched the youtube animation again for the same advice….

    Like

  7. Brain Says:

    I think there is a lot of variability btw POs and it comes down to luck of the draw. I’ve had one PO give me what, in retrospect, was advice that landed me in the triage pile. I’ve also had a different PO from same institute get an application funded with marginal score. This same PO has also reached out to me, unsolicited, to find out what my plan is for unfunded applications. It seems to good to be true, but I will try and stay on this person’s good side for as long as possible….sometimes it’s better to be lucky than good.

    Like

  8. NewbiePO Says:

    “It would be interesting to find out the average number of assigned grants handled by each PO, as increased workload would certainly contribute to this situation.”

    I think this is certainly one of the big factors. I took a look at the number of applications being handled by my branch that are about to go to review. Each PO has about 24 applications on average, and from what I’ve heard our IC has a lower application/PO ratio than the NIH average.

    Like

  9. Grumble Says:

    My PO attends study section REGULARLY. I know of at least one other one at another institute who does as well. These are excellent POs whom I know to be extremely responsive to PI’s questions (well, at least to mine).

    Sadly, they are getting on in years and I won’t be surprised to hear that they are retiring soon. My impression is that the younger POs don’t care as much and don’t attend grant review sessions as much. I’m guessing those are the same ones who are unresponsive to PI’s e-mails and who generally don’t do their jobs well. It’s sad. NIH should force POs to attend some minimum number of grant review panels, and they should also reinforce minimum standards of good “customer service.”

    Like

  10. drugmonkey Says:

    I think part of PO training should be reviewing grants. Yeah, yeah, I know. Rules. Firewalls. Etc.

    Like

  11. . Says:

    Once my PO suggested that I knew any politicians, maybe I should talk to them about helping to get my NIH grant funded. Being more junior, I took him at his word and talked to my friend, a chief of staff to a majorly influential senator on the appropriations committee. He was always more formal in communications after that, and I learned not to take what POs have to say as the gospel truth.

    Like

  12. Philapodia Says:

    NewbiePO, how is the announcement for study sections that POs can go to handled (just e-mailed out or is there a sign-up list?) and is there any tracking of PO attendance? I’m curious if there are any quotas or if the system is more organic.

    Like

  13. physioprof Says:

    Yep. This is one of the best things they can do for you. I am very pro-PO even though this post is about warning noobs that their understanding of the process can be limited.

    Absolutely. I am also very pro-PO, given what they have the skill-set, requisite information, and power to do. Helping applicants navigate peer review in standing CSR study sections, however, is not really one of those things.

    Like

  14. Dave Says:

    I got some terrible, god awful advice from the PO who was dealing with my K.

    He told me just before re-submitting my app that he had been casually chatting with a few other POs about my app (odd) and that they thought I should change my mentor from a BSD-type to a middling PI who was out of funds, had no mentoring track-record and no experience in my field. This was after the study-section overwhelmingly approved of my mentor and my mentoring plan in the original submission, and gave me 1s and 2s.

    We were totally blown away by it and it threw me for a massive loop just before resubmitting (I’m still upset about it!). I tried to get him to explain it based on the very positive feedback from the SS, but he couldn’t give me anything and wasn’t at the meeting. It was the weirdest fucking thing in the world. I’m 100% confident I would have not been awarded the grant if I had taken his advice. Maybe that was his goal???

    Still got the emails.

    Like

  15. Noncoding Arenay Says:

    “I’m 100% confident I would have not been awarded the grant if I had taken his advice. Maybe that was his goal??? ”

    That would be messed up.

    Like

  16. physioprof Says:

    Never attribute to malice that which is adequately explained by stupidity.

    Like

  17. NewbiePO Says:

    Philapodia, an email is sent out to POs who have applications in that particular SS. I imagine there are records of in-person attendence; keeping track of people phoning in would be harder to do. I wish I had more definite answers for you, but I wasn’t kidding about the “Newbie” part.

    Like

  18. Philapodia Says:

    Are applications assigned to POs based on keywords within the text or some other criteria? Say you are the PO assigned to bunny-hopping, would you get all grants that put bunny-hopping in their abstract, summary, Etc. I know POs have specific topics they cover, I’m just curious how the grant get to the right PO.

    Re: records of in person attendence, at the last SS I was at the PO table was behind the SRO and I don’t know if the SRO saw who came in or out, so I don’t know if they kept a record. I only saw one or two POs come in all day for about 20 minutes each.

    Like

  19. physioprof Says:

    The Grants Technical Assistant is responsible for signing in and out all POs and other in person visitors to the study section. Not sure how this is handled for POs who phone in, but I am sure there is a mechanism to document the identities of who does so. I think this is required by law.

    Like

  20. Pinko Punko Says:

    You need a PIN to phone in as a reviewer, I assume the same is for PO. The sections I have reviewed on have 1-3 POs there, usually.

    Like

  21. JustAGrad Says:

    Thanks for this post and the comments from everyone. This helps me understand the disconnect between my PI asking me why I’m not getting any useful advice from my PO, who does not attend fellowship study sections.

    Like

  22. physioprof Says:

    JustAGrad: I am an extremely experienced mentor of many pre- and post-doc NRSA awardees, and have guided them all to ultimately successful fellowship applications. Feel free to e-mail me if you want some input: physioprof@gmail.com.

    Like

  23. E-Rook Says:

    PO’s names show up in RePORTER, and so you can get an idea of who works in your area by searching for key-words. You can then search for PO’s by name in RePORTER, and see what grants are in their past and active portfolios. You should also be able to see what Division/Branch within their IC they work in that should have a description of their program; you can then send them an email with an abstract of your plan, request a phone call … then come submission time indicate in the cover letter, “This application has been discussed with Dr. SoandSo and Suchandsuch Division/Branch at IC.” THAT is the best way to know who your grant will be assigned to.

    Like

  24. d Says:

    Once my PO suggested that I knew any politicians, maybe I should talk to them about helping to get my NIH grant funded.

    Like

  25. Lis Says:

    Hello–a PO here. The one comment I can agree is that POs are not all made the same. I take my job very seriously. I make myself available always. I give advice….always with my deepest honesty and only if I can really have insights that may help, and these insights are personal and discussed with my grantees or potential grantees. PO are not here to tell you how will you score a grant. We can interpret a review, we are there to ensure there is no bias. And yes: we are there or listen to review as many times as I can. Sometimes is not possible because that is not the only thing our job requires. There is much more, involved work that is unnoticed by the applicants. We need to study our portfolios, know the research upside down, identify gaps. correct action in certain cases, review progress, write initiatives (that get exhaustively evaluated and questioned by directors, external boards, council…). Sure we are not at the bench, but the scrutiny is on. Many of us are your biggest advocates. But is possible that you get a bad apple: someone that is too detached from the reality of science because doesn’t attend meetings, or because their circle is not in the science now for over 15 or 20 years. All true. But there is something fundamental: we cannot tell you what science you should do. Our job is to guide and make sure review is ethical. We cannot talk during review, did you know? We only can write a note to the SRO if we identify a problem. Is up to the SRO to call the problem out. Also, we have no power over study sections, even for RFAs that we might have written. Less often that more, SROs reach out to us to request advise for good, expert reviewers. I guess what I am trying to say: we are not all the same, some of us, and most of the POs I know, are genuinely knowledgeable and willing to help within the limitations of our policy in what we are able to tell or not to tell, because we are liable. We do not hold the key to success nor the perfect topic to score a grant. Do call us with your idea. But I say: take your submission seriously even from the aesthetically point of view because portfolios exceed the 100s of grants, grants to be reviewed even more than that sometimes, and is very, very difficult to master everything, and even read everything at times. My five cents.

    Like


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: