Grant Review Site Visits
April 24, 2014
…need to be ended.
They represent a huge risk for bias dependent on the personal characteristics of the investigators to rule the day.
Are you an older, white-haired, heteronormative appearanced, able-bodied picture of “Scientist and Professor”? Great!
Are you overweight? Do you stutter? Express unexpected gender presentation? Nonwhite? Female? Are your language skills less than native to the reviewer’s ears? Too young? Too hot? Not hawt enough? …. Not so great.
Should your grant proposal be affected strongly by the direct face to face impression of these characteristics?
No.
__
H/t: @jwoodgett
Update: See PDF for site visit procedures
DrugMonkey 
April 24, 2014 at 8:42 am
Point taken, but wouldn’t the same problems apply with any personal contact with program officers, presentations at conferences, etc? Science is a human endeavor which unfortunately means humans have to encounter each other face-to-face occasionally. And I say this as an introvert who may not always make the best first impression in person myself.
LikeLike
April 24, 2014 at 8:45 am
Yes, true. But it is always a balance of cost/benefit. Since the vast majority of NIH grant review is *not* face to face, why should any of it be?
(As someone who thinks that his own characteristics would generally benefit face to face review)
LikeLike
April 24, 2014 at 9:38 am
Just curious – how often do these site visits occur, and does anyone ever get turned down as a result of the visit? I agree that bias is possible, but a more obvious issue is that they are expensive. What more can they add to an application that already describes the facilities and equipment?
LikeLike
April 24, 2014 at 9:53 am
IME, the site visit was for Big Mechs only. Centers and Program Projects. Probably U-mechs as well.
The more recent preference appears to be for more “reverse site visits” where the team is invited to DC to put on their show.
I have no knowledge of how many of these there ever were and whether the relative proportion has changed over time.
And yeah, it is review. some of them must be unsuccessful…but that would be another interesting question for Rockey’s data mining minions.
LikeLike
April 24, 2014 at 10:57 am
Too young? Too hot?
Thanks for providing me with my new go-to explanation the next time a proposal of mine doesn’t get funded.
LikeLike
April 24, 2014 at 11:36 am
Site visits are also used for training grants, right? There I could maybe see a reason for the NIH to meet with students, but again the question is whether this is the best use of the time and money.
LikeLike
April 24, 2014 at 12:29 pm
I’ve seen my PI’s PO in the lab, and we aren’t on any Big Mech awards. Maybe they come to campus for one reason and then stay to say hi to their favs?
LikeLike
April 24, 2014 at 1:00 pm
“Maybe they come to campus for one reason and then stay to say hi to their favs?”
Would this be in the vicinity of NIH or is your institution a considerable distance away?
LikeLike
April 24, 2014 at 1:11 pm
The site visit that I am referring to is not a casual drop-by. It is the *review* of the application for the proposed interval of funding. So you have POs but also a review panel of peer scientists.
LikeLike
April 25, 2014 at 2:37 am
In my experience, a site visit is a way of making very busy senior people really pay attention to a large grant situation (like a training grant or a big P-center). The site visits I’ve seen have always been 2-4 people (sometimes with PO, sometimes not) who visit and interview people. For training grants, a large part of it was interviewing students.
Remember that these big mechanism grants are often really really really large. (The last training grant submission I saw was several hundred pages.) Not having ever tried to review a 400 page grant, I don’t know whether reviews are more on target after a site visit or not.
One interesting thought is that rather than first forming separate opinions of the grant before study section, the site visit members are talking to each other and influencing each others’ opinions. Again, I don’t know whether that’s a good thing or not.
LikeLike
April 25, 2014 at 4:51 am
I’ve been on a panel of faculty interviewed during a site visit. It was for a training grant on which I am a trainer. They really grilled the faculty on a select set of issues. That is, they seemed to have an agenda, and we didn’t know what that agenda was before entering the room. The PIs were very nervous about the whole thing and seemed even more nervous after the site visit. In the end, everything was fine, and the training grant was renewed. As for the faculty who were interviewed, we’re mostly white or Asian, some are overweight, we’re not all heteronormative in appearance, and as far as I could tell, none of us are hot.
LikeLike
April 25, 2014 at 5:35 am
These complaints apply equally to giving talks and being interviewed. I suppose we could move to doing everything via IM or IRC but that would make two-fingered typists the new disadvantaged class.
LikeLike
April 25, 2014 at 5:35 am
Also, what about teaching? That also involves standing up in front of a lot of people and talking at them.
LikeLike
April 25, 2014 at 5:50 am
As iterated above, these types of visit more often apply to large award mechanisms such as program projects and training grants. I’ve never heard of a site visit for an R award. The clinical trial folks also have it pretty bad, with FDA monitors on site.
If the need for NIH to do site visits isn’t going to change any time soon, could the underlying message of your post be read as “make sure to hide all the fat people in the closet until after the inspectors are gone”?
LikeLike
April 25, 2014 at 7:18 am
I do know a guy who got “surprise” site-visited for his R01-funded project, more than a decade ago. It was a larger-than-normal R01, but still an R01. He did not get along well with the PO. The PO thought results were not coming fast enough and stopped in to check things out.
LikeLike
April 25, 2014 at 1:20 pm
DM brings up good points in that there are hidden biases (sometimes even subconscious) that are stirred by appearances and site visits of grants puts the applicants under the eyes and ears of the reviewers, unlike in a study section. Site visits also destroy the anonymity of reviewers (although the reviewers, by acting collectively, are protected from being accused of killing a grant). These are bad aspects (potentially).
On the positive, site reviews (which usually involve programs with lots of money at stake) can surface issues that applicants are trying to conceal and are opaque in the written application (which are frequently pre-evaluated such that only the top applications go on to be visited). A site review is like a thesis oral examination (which is also subject to the same issues of bias) and the applicants can be interrogated directly. For this reason, I think they have the capacity to be more stringent and effective. Information can also come from talking to the trainees on the grant (in the absence of the applicants). Usually this is a treasure trove of direct information that is tough to finesse.
So, although I appreciate DMs point (made me rethink a number of issues and my assumptions), we live in a society where face to face interviews are often the key decision-making events. Our justice system relies on face to face questioning, as do job interviews. Hence, the biases we have relating to appearances and other criteria are active in many critical situations. We need to recognize the potential biases and deal with them rather than dismissing the advantages of site visits because of the potential for bad behaviour.
LikeLike
April 25, 2014 at 1:36 pm
There is no doubt that having a ~daylong focus on one major grant proposal does a superior job of delving into all of the issues, both positive and negative to the chances of the proposal.
Ola- you jest but believe you me, the research team does strategize who to feature and who not to feature. In several cases I am most familiar with, wrt site visit review, the relative emphasis on various team members was carefully constructed to create an impression. How successful such a gambit might be, whether the available staff exist to address certain presumed biases, the risk of alienating one of the reviewers….these aren’t critical. The important thing here is a recognition of the fact that reviewers might be influenced by personal characteristics of the applicants in a face-to-face review venue.
LikeLike
April 25, 2014 at 1:38 pm
Jonathan- since the vast majority of grant review is not face to face, I’m not taking your point.
LikeLike
April 25, 2014 at 6:24 pm
The worst goddamn two months of Eli’s life preparing, herding the chickens and training the students and then it fucking snowed so one of the panel could not make it and others were late. Never again.
LikeLike
April 27, 2014 at 6:56 am
I’ve witnessed a site visit reveal a dysfunction which otherwise would not have come to light. It was also not apparent to the various site D’s that the dysfunction existed. These are things that can be fixed, but no one knew it was a problem until probing questions were asked in person with all the parties present. I’ve witnessed also 2-day long literal audits of the data, the samples …. are the gabillion samples you say exist on paper really there in the freezer farm …. is it really connected to backup power …. where’s the temp sensor and who does it page? can i talk to them about the sop for when it goes off .. is there a written sop … can we see it? … apply this level of verification for all the things which are on paper that the reviewer just “trusts.” I think this sort of thing is important too. It also gives reviewers a chance to question the first-author youngin’s who have done the real work but not yet made their way to the big kids’ table at Big Conference or Important Study Section. The dog and pony show could be done without. DM’s point is taken, but this could be mitigated by: a diverse review panel (include the stutterer, the non-white male of a certain age) and training on unconscious bias. Which I think all reviewers should get.
LikeLike
April 29, 2014 at 10:06 am
DM – that those are also activities that are central (maybe less so with the teaching) to having an academic science career and therefore people who aren’t good at the grant review stuff in person will also have problems with the other stuff. Yes/No/Maybe?
LikeLike