Peer Review: The power of competing biases, part eleventy

There is a post over at Blue Lab Coats that reminded me that I’ve lamely neglected to pick up on a tip from the incomparable writedit (sorry dude). drdrA notes:

I found this article late last week, at Inside Higher Ed… entitled “The Black-Box of Peer Review“, about a new book by Michele Lamont entitled ” How Professors Think: Inside the Curious World of Academic Judgment“. I’m looking forward to reading this book… when I can get my hands on it and have a bit of spare time (insert big laugh here)

Now it is a little disappointing that the book author did not manage to get into some NIH study sections when doing her research. Still, I imagine the principles generalize very well so the only lack here is the convincing testament to my assumption on this. Would have been nice. Now, I mostly had PhysioProf’s response on this one but doubledoc did identify a point of interest to me in her overview of the author’s points on what has been found lacking in peer review:

Favoritism for work similar to one’s own… or for some personal interest (other than direct personal ties).

That’s the short version. The Inside Higher Ed piece identifies this flaw as:

The Power of Personal and Professional Interests: Lamont writes that most reviewers would never admit to being unfair and would never engage in explicit favoritism based on personal ties, or an applicant being a student of a friend or colleague. But when it comes to an affinity for work that is similar to their own or that reflects personal interests having nothing to do with scholarship, many applicants benefit in a significant way. In a passage that may be one of the most damning of the book, Lamont writes: “[A]n anthropologist explains her support for a proposal on songbirds by noting that she had just come back from Tucson, where she had been charmed by songbirds.
-snip-
… Yet another panelist ties her opposition to a proposal on Viagra to the fact that she is a lesbian: ‘I will be very candid here, this is one place where I said, OK, in the way I live my life and my practices … I’m so sick of hearing about Viagra. … Just this focus on men, whereas women, you know, birth control is a big problem in our country. So I think that’s what made me cranky.’ Apparently, equating ‘what looks most like you’ with ‘excellence’ is so reflexive as to go unnoticed by some.“[emphasis added]

The personal biases of the reviewer in question are always going to be a factor when human decisions are involved. Always. And it is acutely harmful to getting good unbiased review, for us to pretend otherwise!
Humans are biased in a variety of ways when it comes to making decisions. There is an extensive decision-making literature which looks at human behavior when there is an objective “best decision”- like how to optimize winnings when gambling under transparent pay-out rules. There is a wealth of literature on false perceptions, again where there are objective ways to determine what the percept “should be”. So we know this, even before we get into decision making where the “right” decision is not necessarily obvious and we have to make comparisons to statistical expectations. (Say, when looking at hiring patterns for women, ethnic minorities or whatnot.)
And then there is the now-famous implicit-association bias literature. This is the work in which minor differences in reaction time are observed in individuals who may not think they are biased [another example]. The test is set up so the most parsimonious explanation is that the test subject finds particular associations discordant and other ones congruent. For example, subjects are trained to respond quickly with the left finger to positive words and with the right finger to negative words. If you then introduce an interleaved choice task in which subjects are asked to respond to black / white faces on one or the other key, you find a disconnect. Some individuals respond fastest and most accurately when the association is positive/white, negative/black than when it is negative/white, positive/black. The validation is with other scales of white/black bias.
And these implicit-association tests have been extended to many other domains so it is not just ethnicity that is at issue. It is a fundamental property of people that they are subject to bias and are not even aware that they are biased.
That was a long diversion, wasn’t it?
Back to the point. In peer review it strikes me that there are only two solutions.
First, we can pretend that we have selected our reviewers carefully to get supposedly unbiased individuals. The judgment and implicit-association literature does indeed suggest that there is a distribution of bias severity on any given type of judgment. Trouble is, how do we determine who is unbiased when it comes to grant review? We can’t. Because the outcome does not have objective measure back-stopping as would be the case with maximizing payout in a gambling task. There is no knowable “right answer” from which we can evaluate the performance of reviewers. So whenever this comes up, we then express additional biases about who is “unbiased”. This is so freaking meta it would be funny if it didn’t have real-world consequences. The cry I hear most frequently in recent time is that NIH grant review panels must enroll more senior, more “broadly experienced” and “more successful” scientists to get better reviews. What a crock. All this does is select for a cohesively biased review panel.
This brings me to our second choice. Which is the best one and the one that we have employed time and again for this very purpose.
The contest of competing biases.
The other way we talk about this is ….”diversity”. If we try to be as broad as possible in the range of biases that are applied to a given decision, the odds are better that overall decision making will end up less biased.
This post is getting long but I wanted to end on a question. If the solution is the contest of biases (and in the case of NIH/CSR panels, I would suggest this is exactly the solution that has been struck) how should a given individual behave? Should she express the field-specific, gender-specific, geographic-specific or institution-type-specific bias she has been brought in to represent in an explicit manner? To advocate most strongly for grants that are “like her”? Or should she strive to be unbiased and let the implicit/unconscious bias do its own work?