Double blind paper and grant review can't work
April 18, 2012
People who suspect non-scientific shenanigans (of the political or craven variety) have blocked the acceptance of their paper or findable score for their grant often cry for double-blind review.
At present the reviewers are not blinded as the the identity of the authors or grant proposer(s).
The thought is that not providing reviewers with author/Investigator identity will prevent reputation or other seemingly irrelevant social cachet/hand/power from contaminating a sober, objective evaluation of the science.
It can’t work. There are simply too many clues in the topic, cited literature, approaches and interpretive framework. In the fields that I inhabit, anyway.
Even if in some cases the reviewers would be unable to nail down which lab was involved, it would be uneven. And let me tell you, the chances of detection would be highest for well-known groups.
All this would do would be to reinforce the reputation bias!
Please, I beg you my idealistic friends, tell me how this is supposed to work? Think hard about all the manuscripts you’ve reviewed and tell me how they could be restructured so as to provide a near guarantee (p<0.05) of blinding.
Oh, you can yammer on about how you were done dirty, too. Sure you can get all red about how I am an apologist for status quo and defeatist and all that. And by all means rant about the horrible effects on science.
But try, this once, not to sidestep my main point…..how is blinding supposed to work?
DrugMonkey 
April 18, 2012 at 6:58 am
Agreed, the whole idea is pretty preposterous. What, is no one supposed to build off their previous findings, or provide direct evidence of feasibility/capability by citing their own work? Should we delve into a new field with every paper? It makes no sense. No sense, I say!
LikeLike
April 18, 2012 at 7:04 am
Well, it is *possible* that some fields would permit this, DrBecca…
LikeLike
April 18, 2012 at 7:05 am
Silly you, it’s easy! You ban people from citing their own work.
Oh, wait…
LikeLike
April 18, 2012 at 7:15 am
I am not defending the double-blind protocol, but I have submitted grant applications, for at least two agencies, which did use this method of review. We were not allowed to cite our own work in the grant proposal. The proposals themselves were very short (2 or 3 pages, including references), and the point was to communicate a novel idea or research plan. I did not get funded for any of these, nor did I get feedback from the review, which sucked. The two programs that I am thinking of have been successful, though. So, I think in some cases, this double-blind protocol may work, but it may not be necessary.
I don’t think it would be possible to do this for manuscripts. In fact, there are some journals in which review is completely open. Reviewers must agree to reveal their names to the authors, and to have their reviews published along with the manuscript. As a reviewer, I wouldn’t mind revealing my name to the authors, but I think it’s obnoxious to reveal to the general public. I would also object to this as an author – it’s nobody’s fuckin business what the reviewers said, especially if the manuscript is published.
LikeLike
April 18, 2012 at 7:28 am
Open review with published comments would bring political/reputation issues to a new height. Think Francis FreshFace is going to crap all over Prof Bluehair’s manuscript? I think not. OTOH, Prof Graybeard will have no problem ripping Yun Gun’s paper to shreds…
LikeLike
April 18, 2012 at 7:41 am
But some fields do double-blind paper reviewing. Like mine, for instance.
It doesn’t matter anyway. When their papers are rejected, people will blame the system, regardless of what the system actually is.
LikeLike
April 18, 2012 at 7:42 am
And yes, we cite our own work. Some people choose to anonymize all their own citations (I personally don’t like this), others cite it as if it came from a third part, i.e. “so and so did this”. In the latter case, sometimes it is obvious that they’re the same people, but oftentimes it isn’t.
LikeLike
April 18, 2012 at 7:47 am
The very fact that some can choose to unblind their manuscript by the way they write it underlines my point lylebot. Those who think rep will help them will avail themselves of the “option”. Blind-able writing would have to be editorially enforced *prior to review* to work.
LikeLike
April 18, 2012 at 7:49 am
If you don’t allowed to cite your own work, then in many fields the identification would become easier, not harder, – just by exclusion! If there are N labs (N<10) working on that, and N-1 are cited… You have to make people cite their work with equal frequency than competitors work. Which is principally possible, but harder to formalize, and to ensure.
What you can actually do is to make one (non-blinded) person check the links, and thus certify that all the claims that are based on prior research are valid. And then, after this check is done, just remove all citations altogether! After one person (or 2-3 people) had checked the consistency of the idea, then afterwards the idea itself is evaluated by the committee in a blinded fashion. It will mean a bit more work overall, but it seems to be possible, isn't it?
LikeLike
April 18, 2012 at 7:53 am
Wait…but then how would I be able to tell if they cited *me* enough??!!??
LikeLike
April 18, 2012 at 8:02 am
Shlogbaum,
In your scheme, what’s going to prevent the single unblinded citations checker(s) (aka reviewer(s)) from being “biased” and preventing a manuscript moving into double blind review?
LikeLike
April 18, 2012 at 8:17 am
When their papers are rejected, people will blame the system, regardless of what the system actually is.
Dude, don’t you fucken get it???? The system is BROKEN, BIASED, and riddled with FACTUAL ERRORS!!!! That my papers and grants get rejected is just a consequence of this FACT!!!!
LikeLike
April 18, 2012 at 8:28 am
Odyssey,
in my model the “first round” would be really permissive. Like the really low key adequacy test; as the publication criteria for “Frontiers”, or something. As far as I can guess, the rejection or acceptance of grants is much more about the future, about the experiments proposed, not about the literature cited, right? The only reason you provide any literature at all is because you need to show your sanity. And most grants that are quite “sane” are rejected, because they are not challenging enough, or are too challenging, or are diverging to the side from the perceived “ideal way of doing this kind of research”, or what not.
So my point it: why not separating the past from the future in this case? The sanity test from the “scientific idea” test? Let somebody assess the basic adequacy, give an easily-attainable green light for the 2nd round, and then do the 2nd round (the fight of the future plans) in a fully blinded fashion. It would be much much harder to guess the identity of the lab in this case (maybe except for the most unique and expensive approaches; but fortunately these don’t seem to be prevalent).
LikeLike
April 18, 2012 at 8:41 am
Why not DO the experiment instead of yammering about whether or not the experiment would work?? We should beg a forward-thinking society journal like J Neurophys to try double-blinding for a year. Discourage authors from using “we previously showed that..” but don’t try to enforce it, just give it a whirl.
And include one extra tiny step for reviewers: when submitting a review, to include the name of the suspected corresponding author. If this step gives less than, say, ~75% correct answers, I’d call the experiment a success, because once you know that you only have a 75% chance of guessing correctly, you’re less likely to assume you “know” the lab that the ms “must” come from.
Why, sort of like not assuming that you “know” which study section reviewer “must” have been one of your rejected grant’s referees, eh?
FWIW two of my grad school papers could have come from at least three other labs, though ours would have been the best candidate; one of them could have come from at least 10 other labs, and most people wouldn’t have guessed ours at all; and my postdoc paper could certainly have come from at least 5 different labs, and I don’t think most people would have guessed ours. The only issue is that all the data were shown at SFN etc so people might have other sources for figuring it out.
LikeLike
April 18, 2012 at 8:50 am
DJMH: A great point on the 70% value! When earlier Drugmonkey mentioned p<0.05 I was somewhat confused about what it could mean. What exactly p-value you would measure. And the point is: for the psychology of favoritism to be broken you don't need reviewers to be hopelessly lost in the potential labs and PIs. You need them only to start doubting the authorship, and it would immediately make the whole process much more "open-minded".
LikeLike
April 18, 2012 at 9:04 am
I have said in the past that I think it would be a fantastic idea for the NIH to pilot a couple of rounds of allegedly double-blind review of grant applications so that you all would finally shut up about it and start calling for fixes that can actually work.
I’d be delighted for a journal to do this as well.
LikeLike
April 18, 2012 at 9:05 am
You need them only to start doubting the authorship, and it would immediately make the whole process much more “open-minded”.
Only if they were in doubt about every lab or every manuscript to an equivalent degree.
LikeLike
April 18, 2012 at 9:08 am
I’m not actually sure it would be good to have it double-blinded =) I’m just pondering on the potential possibility of an experiment like that =)
You can also introduce obligatory nicknames in science in general, both for publishing stuff and applying for grants, with a requirement to phase them out every 5 years or so. There are some ways =) Would it worth the hassle is the other story.
LikeLike
April 18, 2012 at 9:18 am
I have a better idea that planning studies of double-blind reviewing for journals. How about doing literature searches to see what studies were already done? đŸ™‚
For example:
Blank, R.M., 1991. The Effects of Double-Blind versus Single-Blind Reviewing: Experimental Evidence from The American Economic Review. The American Economic Review 81, 1041–1067.
They compare double-blinded to single-blinded economics journals. Most effects are small. There are few acceptance differences in submissions from the top universities or low-ranked universities, but double-blinding actually decreases the percent accepted from mid-level universities. They also show that, in a double-blind system, the reviewers can correctly identify the author 45.6% percent of the time (though they think the know the author 50.9% of the time).
Here’s commentary on another study that showed a gender difference change when a single-blind journal changed to double-blind, but it was later seen that the the same change occurred in the same-field journals that remained single-blind.
LikeLike
April 18, 2012 at 9:19 am
It might actually be useful to link to the commentary on the second study:
http://blog.sciencewomen.com/2008/06/peer-review-and-gender-bias-revisited.html
LikeLike
April 18, 2012 at 9:28 am
The problem with just removing/anonizing the citations from the review version, as one poster has suggested, is that the reviewer can’t read the previous works. Sometimes this is needed to see if the authors have come up with anything new.
LikeLike
April 18, 2012 at 9:38 am
Sometimes this is needed to see if the authors have come up with anything new.
but if you were a real “expert” you wouldn’t need the citations! See?!!?! Review is by INCOMPETENT reviwers! QEMFD.
LikeLike
April 18, 2012 at 10:15 am
The stupid thing about peer review isn’t from this kind of bullshit, it’s just sampling error. You wouldn’t base a drug’s effectiveness on how two or three (or 15) people responded to it. I’d like to see the same papers given to several different panels of “peer” reviewers and the resulting correlations in their ratings of the work. Which, between a certain crap threshold and totally awesome threshold, will be essentially zero. Peer review does not even try to measure a consensus opinion, and it can’t. There are very few papers that everyone will think are great, there are many that almost everyone will agree is crap, and then there is the vast majority of papers, which will generate a full spectrum of idiosyncratic opinions. The disease is pretending the process works as a measure of quality (rather just the gut-check of competence and bullshit detection that it should be), and then subsequently judging papers on where they are published.
LikeLike
April 18, 2012 at 10:36 am
One thought that I had from an experience reviewing a paper that was indeed a double-blind review: I found myself assuming the author was X, because it felt like a paper from X. It didn’t matter to me if X did indeed write the paper, I didn’t have a particular axe to grind one way or another, but I found myself doing it anyway.
When the paper came out, X was nowhere to be seen.
If someone’s reviewing my grant, I’d rather have them know its me than assuming its whoever comes to mind first when they read my stuff.
LikeLike
April 18, 2012 at 10:50 am
I’m thinking about the five papers I’ve reviewed most recently, some but not all for Glamour journals:
1) The method is so new, there is literally only one group in the world that would have written this. Even if I wasn’t as familiar with the field, with this thing being so new they’d either have to basically re-publish their previous paper or else use constant self-citations in the methods section. Obvious either way.
2) Not quite like #1, but still a lot of home-brew stuff in the methodology, and the methods are still new enough that a lot of stuff has yet to be standardized and people are still working out which details matter and which ones you have more latitude with. The techniques are not yet standardized, so either you publish copious methodological details, or you say “Using the same settings as Ref 1” and “Using the same algorithms as reference 2” etc. I would have at least been able to narrow it down the alumni roster of a certain group.
3) This one pulls together a bunch of disparate stuff from different fields, and does something that the lab has not previously published on. This one I never in a million years would have guessed.
4) I’m not familiar with the group, but there’s enough self-citation that I would have been able to narrow it down to a small cluster of people who have published together. The “out” part of “using our previous technique” was unnecessary for this one, but even without that the sheer number of citations to their work would have still narrowed it down.
5) This paper is sort of the culmination of a very specialized line of inquiry that this group has been working on for several years. The self-citation was copious, arguably too copious: I shouldn’t have to read 3 other papers to get the gist of this one. But even if they correct that in revisions, there’d still be substantial self-citation.
All that said, here are things I would not be able to get from 4 of those 5 papers if we did this double-blind:
1) I would definitely not know the gender or ethnicity of the first author.
2) In most of those cases I would not be sure who the PI is. I would be able to narrow it down to either a certain PI or an alum of their lab, but I would not be sure if I’m reviewing Dr. Grayhair or Dr. N00b from the House of Grayhair. The fingerprints were on the methods, not the questions and hypotheses.
LikeLike
April 18, 2012 at 11:12 am
Now add in epidmeiological cohort studies where things like the number of participants is an identifying feature. So are the tests done and the dates of cohort entry. Now imagine trying to blind that data . . . Let’s assess a study without n (and mybe without p-values or variances, if we want to be sure) and without dates of enrollment.
That would be . . . tricky
LikeLike
April 18, 2012 at 11:17 am
I would not be sure if I’m reviewing Dr. Grayhair or Dr. N00b from the House of Grayhair.
For the meat of the reason for wanting double blinding, this matters not one whit
LikeLike
April 18, 2012 at 11:39 am
I see your point about the issue of the insider club. OTOH, I recall some references to work showing that double blind review reduces gender disparities. I wish I could find them.
Here is a hypothesis to conaider: Dr. John n00b from the House of Grayhair is seen as an upandcomer to kowtow to. Dr. Jane n00b from the House of Grayhair is seen as somebody who only got that fellowship to work in his lab because she is a woman. And Dr. Grayhair appreciates a nice-looking woman. So, you know, don’t take her seriously.
If the reviews are blind you don’t know which lab alum you are reviewing.
LikeLike
April 18, 2012 at 11:39 am
I see your point about the issue of the insider club. OTOH, I recall some references to work showing that double blind review reduces gender disparities. I wish I could find them.
Here isis a hypothesis to conaider: Dr. John n00b from the House of Grayhair is seen as an upandcomer to kowtow to. Dr. Jane n00b from the House of Grayhair is seen as somebody who only got that fellowship to work in his lab because she is a woman. And Dr. Grayhair appreciates a nice-looking woman. So, you know, don’t take her seriously.
If the reviews are blind you don’t know which lab alum you are reviewing.
LikeLike
April 18, 2012 at 11:41 am
And if I had bothered to actually read the comments above I would know about those studies.
I must be Reviewer #3.
LikeLike
April 18, 2012 at 2:26 pm
There are positive possible benefits from double-blinding peer review, I think that’s agreed. And there are possible imperfections, to be clear. But, DM, what are the additional drawbacks from double-blinding? That’s what I don’t see — how can it hurt so much more than what we have now?
LikeLike
April 18, 2012 at 3:20 pm
what are the additional drawbacks from double-blinding?
It actually reinforces the lab size/reputation bias. Busting the blind is more likely for better know, more extensively publishing laboratories.
All of the things that I can imagine would be required to facilitate blinding would be, in my view, detrimental. It would require manuscripts that cite weirdly if at all and avoid shorthand references to the lab’s prior methods. Theoretical orientations would be verboten as would overarching theory (in most cases). We’d end up reviewing figures in isolation!
The greater problem, however, is the distraction from the real problems and better fixes.
LikeLike
April 18, 2012 at 6:53 pm
Do you really know the authors of every paper you review? Most of the papers I’ve reviewed have come from authors I’ve never heard of. My field is not that large. So if double blind, I wouldn’t be able to guess who the authors were, although I’d probably think I could.
At least one important journal in my field has double blind review. And let’s not forget Behavior Ecology, which increased representation by female authors when it went to double blind review. http://www.ncbi.nlm.nih.gov/pubmed/17963996
LikeLike
April 19, 2012 at 8:31 am
It actually reinforces the lab size/reputation bias. Busting the blind is more likely for better know, more extensively publishing laboratories.
I think it is far more likely that reviewers would attribute *all* manuscripts to the biggest name in the field that they know, because if you ask me to name a lab that does X, I’ll come up with the most famous first. So there could actually be a useful bias here, where if you assume the ms. comes from a lab you know, you are nicer about it than if you see up front that it is from, say, Turkey with a roster you don’t know at all.
And since of course the big labs don’t really represent 80% of manuscripts, this effect could work in a positive direction for unknown labs where they inherit a halo of “This is probably Dr Blibbity’s stuff, I think I heard him talking about it at SFN.”
LikeLike
April 19, 2012 at 11:59 am
You know, at this point, who fucking cares? It’s not like most of this work is reproducible anyway, more than half the time.
LikeLike
April 19, 2012 at 3:29 pm
I know, by reputation at least, about 66% of the names of people whose papers I review. And I use that information to judge their paper, and it comes in a lot. Probably in more ways than I’m consciously aware of. For example, some people I know are totally anal, and with them I dont worry much about the methods or the math. Maybe that makes me an unfair reviewer?
When I give a talk or a poster, I usually see it as an opportunity in part to sway potential reviewers. When I have a meeting with someone in their office, often the same. That personal side of science seems to go by the wayside with double blind review. Is that ultimately a good or a bad thing? I think it’s got a mix, but I think it’s more efficient.
LikeLike
January 10, 2014 at 9:22 am
I’ve been on both ends of this as a submitter (papers and grants) and reviewer (papers and grants).
Grant and manuscript review are very different areas and should be approached differently.
For manuscripts, yes, you can detect who the author is or with whom they’re affiliated, but you can evaluate the science without knowing who did it. Double blind can make it more difficult for a reviewer to voluntarily detect whether or not they have a conflict of interest, if it’s more difficult to detect who wrote the paper. For the papers I have submitted I’ve advised who should be my reviewers, but not every journal gives authors the option.
For grants you absolutely need to know who is submitting it (along with who their collaborators are) and where they currently work. You need to know if the group has sufficient expertise and the technical and physical ability to do the work proposed. I work in microbiology and if you don’t have the facility and approval to study Bug X, you can’t study Bug X. It can take months or more to get approval, even if you have a sparkling, brand-new facility that should be able to handle it (I know of one facility that’s been delayed by years for various reasons). If you are developing a technology to transition to Industry X you need to have a collaboration in Industry X or the likelihood of industry buy-in is little to nil. And if you have zero experience in the topic of your study you at least need to have a collaboration with people who have experience or expertise in that topic. You don’t award a grant unless you have some indication that the proposed research will succeed. You can’t evaluate that without knowing who is on the team that will be conducting the research.
LikeLike