Should NIH provide a transcript of the discussion of grants?
February 16, 2018
Respected neuroscientist Bita Moghaddam seems to think this would be a good idea.
She then goes on to mention the fact that POs listen in on grant discussion, can take notes and can give the PI a better summary of the discussion that emerges in the Resume of Discussion written by the SRO.
This variability in PO behavior then leads to some variability in the information communicated to the PI. I’ve had one experience where a PO gave me such chapter and verse on the discussion that it might have been slightly over the line (pre- and post-discussion scores). Maybe two other ones where the PO gave me a very substantial run down. But for the most part POs have not been all that helpful- either they didn’t attend or they didn’t pay attention that closely or they just didn’t care to tell me anything past the “we suggest you revise and resubmit” mantra. She has a good point that it is not ideal that there is so much variability. When I’ve touched on this issue in the past, I’ve suggested this is a reason to cultivate as many POs as possible in your grant writing so that you have a chance of getting the “good” ones now and again. Would providing the transcript of discussion help? Maybe?
Or maybe we could just start lobbying the ICs of our fondest acquaintance to take the effort to make the POs behave more consistently.
But I have two problems with Professor Moghaddam’s proposals. First of course, is the quashing effect that de-anonymizing (and while a transcript could still be anonymized it is in the same vein of making reviewers hesitate to speak up) may have on honest and open comment. The second problem is that it goes into reinforcing the idea that properly revising a grant application is merely “doing what they said to do”. Which then should, the thinking goes, make the grant fundable next time.
This is, as you know, not the way the system is set to work and is a gut-feeling behavior of reviewers that the CSR works hard to counter. I don’t know if having the transcript would help or hurt in this regard. I guess it would depend on the mindset of the PI when reading the transcript. If they were looking to merely suss out* the relative ratio of seriousness of various critiques perhaps this would be fine?
__
*My fear is that this would just feed the people who are looking to litigate their review to “prove” that they got screwed and deserve funding.
DrugMonkey 
February 16, 2018 at 3:46 pm
What about when your PO tells you that she wasn’t actually at the meeting???
LikeLike
February 16, 2018 at 4:09 pm
Yep, that happens a fair bit too. Sometimes one of their branch mates has covered for them but unfortunately, yes, this is another source of variation.
LikeLike
February 16, 2018 at 4:39 pm
Good points, especially your asterisk. It’s been great having your posts pop back up in my feed!
LikeLike
February 16, 2018 at 4:47 pm
The answer to Bita here is that study section is confidential. You are not legally allowed to discuss what went on in study section with anyone. They tell us that at the beginning of every study section.
If study section included transcripts, I think I would no longer serve on study section. I’ve said so many stupid things at study section (for lots of reasons – lack of sleep being an important one) and been corrected, that I can’t imagine I would dare open my mouth if every comment made was legally trackable. I’m comfortable with being corrected and I firmly believe that study section is fair. But study section is confidential. There are good reasons for it to be so.
I agree with you on all three points. De-anonymizing is dangerous. It encourages treating grants like papers that need to just address the reviewer’s comments (*, **). And there is a problematic variability in PO behavior
* I’d be OK with treating grants like papers. You get in line for your grant. You know it’s coming. You know that if you just accommodate the reviewers, it will get funded. This means you can live on one R01. It’s not the current system. But it was… once… in the long-ago time.
** Actually, you could make the case that this is the problem with papers today. The GlamourChase has led to papers being treated like grants. Papers are getting reject and resubmit rather than revise and resubmit. Each cycle is a moving target and you have to hope you get lucky that the match between what you were able to control for and what reviewers care about. And even more difficult – that you get lucky with the impact of your study and what reviewers think is worth making a splash over.
LikeLike
February 17, 2018 at 8:49 am
Why not provide the transcript but leave out the names?
LikeLike
February 17, 2018 at 1:09 pm
First of course, is the quashing effect that de-anonymizing (and while a transcript could still be anonymized it is in the same vein of making reviewers hesitate to speak up) may have on honest and open comment.
LikeLike
February 17, 2018 at 3:53 pm
The revise and resubmit video is still one of my all time favorites.
LikeLike
February 20, 2018 at 8:17 am
Some POs are certainly better than others. Regarding attending or calling into a study section meeting, it’s not always possible. To give you an example – 46 grants being reviewed (not all discussed though) across 26 study sections, often with 1-6 study sections occurring on a given Thursday or Friday in February. That can make it very difficult to listen to every application. Additionally, whenever a study section discusses an application out of order, that information usually not communicated to the Program Officer, so even if the PO is carefully watching the study section it’s possible to miss half or all of the discussion. This is, of course, on top of regular meetings and tasks that don’t decrease during February, June, or October.
LikeLike
February 20, 2018 at 7:08 pm
Cytokine, there’s more than one PO. Your method to argue for too many study sections to listen to is misleading. Also important: it’s their job.
LikeLike
February 21, 2018 at 9:36 am
I think issues with de-anonymization would have a strong chilling effect that would not be offset by benefits to PIs wanting feedback. Already there is too much “revising to the reviewer”, so a transcript of the discussion is not an unambiguous benefit. Unless it were a strong benefit, the risk of chilling discussion (or selecting for reviewers who are OK with being outed) makes providing transcripts a bad idea.
LikeLike
February 21, 2018 at 11:46 am
As an intermediate, I would suggest that NIH maintain a transcript that the PO can review when you call in to find out what happened. This would at least help with the issue of them not attending/not paying attention. They could review the transcript in advance of your call and might have some insight after the fact. But I would still have NIH at least encouraging PO attendance as much nuance can be lost in a dry transcription.
LikeLike
February 21, 2018 at 12:12 pm
Intriguing
LikeLike
February 21, 2018 at 6:39 pm
Transcripts will be difficult or expensive to make. Automated systems will likely not work as the material is highly technical with several overlapping voices. Recordings would take too long to listen to for the PO.
LikeLike
February 22, 2018 at 6:45 am
JL – it would not be hard to make a marked-up recording, so that a PO could do a quick listen to the discussion of the specific grant before talking to the PI. One could also have a transcript/recording set where the transcript did what it could and the recording was available for the PO to check.
As a side note, I’m curious how much of a PO’s time is currently spent in these kinds of “help-the-PI” conversations.
LikeLike
February 22, 2018 at 7:36 am
Sorry I wasn’t clear JL, that was an example of one PO’s portfolio, not across multiple POs. You’re right though, I’ve seen plenty of POs at study sections taking notes for their colleagues. My point wasn’t that POs shouldn’t have to listen to the discussions (they should as it helps with funding decisions), but that sometimes it isn’t possible for them to listen to every application. In the end it doesn’t matter to an applicant that a PO listened to 19 of 20 discussed applications if their application was the one that was missed.
While I believe that a good PO will make a point to listen to and take notes on the discussions as it helps them guide the applicants, it may not be fair to say that it’s their job. A PO’s job is to facilitate the funding of strong science, from both an individual project level and from the 10,000’ view. Helping people tweak the scientific focus to produce a stronger project or giving advice to mitigate grantsmanship issues so that the science shines can certainly help with that mission. A PO’s job focus, however, is not to help the PI. Contracts take this philosophy to the extreme – a PO (called a COR for contracts) is not allowed to talk with the PI, either before submission or after review, and a summary statement is not released. I should also add that, for grants, until the summary statement is released the application belongs to the SRO. Per NIH policy, the PO shouldn’t even be discussing the application with the applicant until after the summary statement has been released. The summary statement is the official, legal record. I’ve heard some at the NIH argue (though I disagree) that the PO should strictly follow what’s written on the summary statement and not give any extraneous advice beyond logistics (explaining premise, sex as a biological variable, when submission dates are, etc.).
The problem with transcripts and recordings is that they would be subject to FOIA.
LikeLike
February 22, 2018 at 8:50 pm
Cytokine, I did not say that it is the POs job is to help the applicants. What I meant is that it is their job to listen to the discussion to decide what to fund. But maybe they can do it all from just the summaries. How does NIH know that a PO is doing a good job? How are they evaluated?
I am curious too, has anyone gotten actual useful info by talking with the PO.
I only talked with a PO once. It wasn’t useful. Told me to revise and resubmit considering the concerns raised by the reviewers.
Now I just assume they are very busy and tired of talking with people complaining.
LikeLike
February 23, 2018 at 7:00 am
A PO is like an editor. They are trying to find good science to fill out their portfolio. As such, having a PO in your corner is a major help in getting funding, both in that they are the ones who have to argue for you at council when your grant is being actually considered and in that they can help you find the most appropriate funding options. I have gotten *great* advice from multiple POs over the years. I also treat POs like editors – I help them when they need help (i.e. reviews), I treat them with respect, and I make sure to network with them when I can (e.g. at SFN).
If your PO is not helpful, then likely they do not see you as being appropriate within their portfolio. I would recommend trying to find another PO.
LikeLike
February 23, 2018 at 11:36 am
JL-
I try to make as clear as I can that this advice to get to know several POs is not some sort of one to one “chat with a PO once and you will magically get funded”. I call it the zombie mantra (we advise you to revise and resubmit) because it is what you will most frequently get. It’s the rare cases where they move mountains for your ideas. You are playing the long game here. Keep talking with POs.
LikeLike
February 23, 2018 at 11:41 am
I should also point out that being highly reponsive and informative (about the review discussion) in conversations with the PI is not necessary for the PO to be battling hard for a project to get funded. Yeah, I have a story….
LikeLike
February 25, 2018 at 1:44 am
qaz, looking at Reporter, every single grant in my field has been funded through the same PO. I don’t see how I could try a different one.
I am not complaining of the “revise and resubmit” approach. I did, and got funded. Maybe it worked behind the scene.
LikeLike