“…I think it’s a Lurker!”
November 30, 2007
A little discussion over at Young Female Scientist reminded me of the suggestion in a Nature editorial that this summer’s RFI from the NIH to solicit input on the peer review and grant funding process drummed up about 2,000 responses.[Update 12/05/07: This ppt from Tabak claims about 2600 responses to the RFI.]
Going by the usual ratios that comments to blogs represent maybe 10% of readers at best, well, YFS alone must have a fairly large population of disgruntled grad students and postdocs reading her blog. The number of ScienceWoman, Female Science Professor and similar blogs must have another huge population of women scientists who have at least some objections to the WayThingsAreDone in NIH land.
I just don’t get the Lurker mode or phenotype when it comes to these issues that directly affect the career. Blogs are one thing, just a little lunch hour entertainment. But NIH funding policies and directions that are going to affect your future for some time to come and you can’t be bothered to drop the NIH a note? Who ARE these people?
(Title is a nod to another favorite book of the Spawn)
DrugMonkey 
November 30, 2007 at 2:16 pm
I originally checked out that YFS blog on your indirect advice within the last week, and it has already pissed me off once (see the post over there above the one to which you refer) and now the post at hand as well has done the same (admittedly, I skipped reading it previously until now). The issue and concepts that you discuss are virtually identical to the attitude concerns that arise in the post upon which I commented. These problems are rampant all throughout science. Even take the conversation on confidential comments, mentioned by you on this blog. 38 comments is quite nice, but when considering a fundamental change to the way that papers are reviewed at a top journal in neuroscience, let alone the revolutionary impact that something like the Neuroscience Peer Review Consortium will have on publishing, that comment total should have been more like 338, at least. Since NN has been invited to join the consortium, we solicited opinions from prominent scientists and invited them to discuss the concept on our blog, but we only received barely more than your 10% number.
Did the 2000 election wrongly teach us that our vote doesn’t count? Are we too inundated with opinion polls these days that we have even numbed to those that matter? Are email spam filters too stringent, striking down anything that even has a faint hint of bulk-mailing in the “To” field?
That’s right, people. I blame pornographic spam and George Bush for today’s indifference on putting forth one’s view…
[Please correct my hyperlink coding if it doesn’t make it through your commenting engine…thanks!]
LikeLike
November 30, 2007 at 2:40 pm
“I originally checked out that YFS blog on your indirect advice within the last week, and it has already pissed me off once … and now the post at hand as well has done the same”
FSM love her, the YFS is good at provoking comment. You shoulda seen the traffic that came this way from one of her posts taking exception to some comments over here. Pissed-off increases involvement, clearly. So if you want to gin up your numbers on the consortium thing, my advice is to piss some people off. Heck, I can do it for you. As a preview to a little post that I’ve been mulling, your answer to “should NN join the consortium” should be entitled “Who wants to be J Neuroscience’s little beeyotch?”. Discuss. Alternate titles include “Shall we make J Neuroscience our beeyotch?” and the obvious followup “Do J Neuroscience’s beyotch journals become our meta-beyotches or grand-beyotches or what?”
LikeLike
November 30, 2007 at 2:55 pm
Several points:
(1) Bloggers want readers and commenters. YFS gets more readers and commenters when she complains than when she doesn’t. It is more interesting to read well-written caustic complaints than happy-happy-joy-joy stuff. She is an entertaining curmudgeonly character, saying out loud what many grad students and post-docs would like to say but don’t/won’t/can’t. In a winner-take-all environment like academic science, there are lots of disgruntled people who are a lot of YFS’s audience. People who first come to her site–me included–think that if only she took their advice everything would ge great. That is not the purpose of her blog, and she is not seeking advice. Advice-givers provide her with a valuable opportunity to tee off on them. Not all blogs exist for the purposeful exchange of valuable information; as far as I can tell, hers is designed for entertainment.
(2) The NPRC is not going to have a “revolutionary impact…on publishing”. All it’s going to do, in my opinion, is induce authors to “overshoot” with their intial submissions, and lead to NN and Neuron taking on the burden of reviewing a lot of papers that authors know in their hearts are destined for Journal of Neuroscience or below. In the absence of the NPRC system the incentive to overshoot–you might get lucky–is tempered by the cost of time and effort to resubmit to a lower journal. By removing much of this cost, authors are encouraged to overshoot. But it ain’t going to have a major effect on where stuff gets published, which is all that matters in the end.
(3) In relation to the NIH Peer Review thingy, I think many people, either explicitly or tacitly, realize that it is highly unlikely that they have some unique opinion or insight or groundbreaking idea about how to “fix” peer review (which isn’t really broken, anyway). They therefore reason that there is a high likelihood that someone else is going to make the same points, and so their interests will be represented by proxy. You may not agree with this reasoning, but it is, nonetheless, far from irrational to adopt it.
LikeLike
November 30, 2007 at 3:02 pm
“how to “fix” peer review (which isn’t really broken, anyway)”
in current climate the notion that there isn’t really anything broken with peer review is indeed a “unique opinion or insight”, LOL!
LikeLike
November 30, 2007 at 3:12 pm
“that comment total should have been more like 338, at least. “
In all seriousness Noah, I think you should congratulate yourself on getting so many well known names to lay down comment. By “solicit” did you mean you came right out and emailed ’em personally? I didn’t check but are these folks on your extended Editorial Board (if NN does that)?
LikeLike
November 30, 2007 at 4:02 pm
I thought the discussion at NN on peer review “confidential comments” was informative, and that the people who posted there posted thoughtful comments that gave me insight into the editorial process.
But, really Noah — did you expect that in a public form there were people who were going to say “I love the confidential comments to the editor because it gives me an opportunity to trash my competitors papers without telling them about it.” [followed by an evil laugh]
YSF complains because she’s deeply unhappy. I actually go over there and worry about her sometimes; you folks are burying your heads in the sand if you think that she’s not sincere, and that she’s not tapping into something real. There are people out there who are showing the symptoms of clinical depression. They don’t comment at NIH or NN precisely because they feel absolutely powerless about everything and don’t think anything will every change.
Regarding peer review/paper review, I do think it’s broken, but that there’s no easy way to fix it. The problem is that there isn’t enough to go around, and when there’s not enough to go around, people invest a lot of energy in separating papers/grants that are really very similar to each other. To refer to the NYT, they become “snowblind” in the sea of paper that can’t be differentiated. When they’re snowblind, and the information is ambiguous, they rely on cut-offs, and biases, and words in the wrong place, and self interest (even hidden) to make their decisions. It’s the same problem as college admissions in the big league schools. It’s becoming hell, but there’s no real way to fix it. For admissions, I’d suggest randomizing selection in the middle range where merit can’t be distinguished anyway. That idea goes over awfully poorly with college admissions, and I can’t imagine anyone even considering it for peer review/grant review.
LikeLike
November 30, 2007 at 4:54 pm
Part of the problem with peer-review is that peers are not objective. Say I work on the mechanics of bunny-hopping. My papers get sent for review to colleague bunny-hoppers, my grants are reviewed by the bunny-hopping study section, and there is never really an opportunity (ESPECIALLY with the study section) for a non-bunny-hopper to stand up and say, “look, other than the bunny-hoppers, nobody really cares about bunny-hopping, and I think we already know all we need to know about bunny-hopping for now,” and close down the field. Of course, the bunny-hoppers would cry foul, “but he’s not a bunny-hopper! how can he possibly be knowledgeable enough to assess the importance of bunny-hopping?”
LikeLike
December 2, 2007 at 2:50 pm
Really long comment… I’m going to make it a post on my own site instead, and you can come and pick on me there if you want to. Everyone is welcome, unless they start to bore me, and then I will delete comments that sound too much like Johnny-knowitall-one-note.
LikeLike
December 3, 2007 at 10:34 am
“there is never really an opportunity (ESPECIALLY with the study section) for a non-bunny-hopper to stand up and say, “look, other than the bunny-hoppers, nobody really cares about bunny-hopping, and I think we already know all we need to know about bunny-hopping for now,” and close down the field.”
Since there’s a good chance my study section would handle the bunny-hopping proposal….. 🙂
The trouble is whimple, that you need to be specific about the level of bunny-hopping you are discussing to get any traction. For example, do you regret that the NIH is plural (as DM is always on about)? You are not going to get anywhere this way. Complaining about the IC “in house” study sections as opposed to the CSR ones? I’m right with you there.
My section has a fairly broad mission in terms of techniques and models and research subjects. To the extent that you might have kangaroo-hoppers and humanJumpPathCondition people reviewing the bunny-hopper grant. And for that matter even some cholinergic receptor expression/pysiology experts who really don’t know much beyond the neuromuscular junction. etc. And you can multiply this type of diversity about 10 fold for the rest of the section. and yet, yes, they might all be focused on the general area of motor function.
but still, the scenario you paint in which everyone is really only focused on bunny-hopping, meaning assays that really only use bunnies and hopping as a behavior? rare, I’d say.
look, everyone has ideas about which 2/3ds of the NIH they could ditch so as to focus on the ReallyImportantScience. I sure as heck do. But I recognize the power of letting a whole bunch of smart (ok and not so smart) people clamour for the relevance of their scientific interest on a level of parity that tries to keep some overall balance in the portfolio…
LikeLike
December 3, 2007 at 11:50 am
Looks like there is plenty to which I can respond…
Drugmonkey, I’m not quite sure what you were trying to get at, so I guess I’ll have to wait for the post. Perhaps my responses below will provide you with more information.
Prof, let me break it to you gently: people overshoot with their submission all of the time. If they didn’t, my job would be a lot less stressful, and I could probably work a lot less hours. Basically, the consortium makes it so that when a member journal rejects a paper, it can take advantage of fast-track status to publication at another specialized neuroscience journal, sometimes WITHOUT ADDITIONAL REVIEW!!! At the very least, the authors can address the concerns of the original journal and then start at the next journal a “revision” as opposed to a new submission (just comparing the time in review for a first version vs. a revision at NN, the difference is in weeks, not days). Having legitimate reviews already in hand takes an inordinate amount of stress off of the reviewing system, as fewer people will be asked to re-review a paper that has already been seen by 6 people previously, all of which who had the same comments!!! Authors are lazy. If they have to resubmit, they do not tend to improve the paper unless they have to, in case the NEW reviewers are less stringent/in a better mood/brain-dead, etc… So it speeds up the time for authors as well since they now will have an incentive to address the concerns of the previous journal, hastening the review process, making their time to publication a lot faster. I agree 100% that it will not change WHERE things get published, but changing HOW they get published, I feel, is a revolutionary concept. Everyone wins. But I think that it will only achieve its maximum benefit if NN and/or Neuron join.
Bikemonkey, yes, I solicited. A change to the review process of this magnitude invariably necessitated an opinion from the people who make this journal possible: the reviewers. I used an algorithm to extract the names of those in academia who had reviewed for NN 6 times (or more) within the last 24 months in any subject matter. That yielded ~350 names representing our work-horse reviewers. If it was of interest, I am happy to post the text of the email to reveal whether there was any bias in the communication, etc… The comments I did receive on the blog were great, but returning to my (less than amusing) attempt at humor in the previous comment, something still keeps people from commenting on things that could radically change their professional lives – EVEN THOSE WHO ARE THE MOST AFFECTED BY THE CHANGES (in this case the reviewers, who are also frequent authors of course, as well).
Neurolover, there were a few people who commented on the merits of confidential comments publicly, as I’m sure you read, but alas, I did wish that someone would have appended their comment with a .wav file of an evil laugh. My trackbacks would have gone through the roof, with those out in internet-land dying to hear a truly evil scientist belching out a truly evil laugh. Ironically, I received 5 additional private emails from researchers giving me their opinion on the situation, but who were unwilling to go on the public record. Mull over that one for a minute…giving confidential comments to the editor regarding whether confidential comments to the editor should be eliminated. When I realized that it wasn’t a joke, I laughed even harder…
LikeLike
December 3, 2007 at 12:29 pm
“Drugmonkey, I’m not quite sure what you were trying to get at”
The way the consortium seems to be framed is a egalitarian cooperation of equals: ” The Neuroscience Peer Review Consortium is an alliance of neuroscience journals that have agreed to accept manuscript reviews from other members of the Consortium.” One cannot help but notice that at present time it is one general neuroscience journal at the apex (7.45 IF) with a bunch of <4 IF specialty and general neurosci journals at the foot. I don’t see this submission path really going any other way than submit-to-JNeurosci and then “which of these other journals is going to take it”?. I feel that if NN signs up, this immediately puts you in the slot previously held by JNeurosci and perhaps slots them into the sole intermediate position.
Getting on to the practicalities, while I agree Noah that many authors automatically overshoot, not all do so. What PP is talking about is the population of people who do not automatically overshoot. I fall into this category. My easy comfort zone would include journals such as EJNeurosci, Neurosci and Neuropharm from the consortium foot. I would submit to J Neurosci on a less than habitual basis at present and yes, some of this would be the “cost” of time wasted getting rejected after revew. If I had something I think would easily go in, e.g., Neuroscience, I would definitely now go to J Neurosci first just in case.
LikeLike
December 3, 2007 at 12:39 pm
Regarding your second point, I don’t see anything wrong with that arrangement. It still reduces the workload of the reviewers, which I feel is extremely important for the entire review process. A good chunk of the time, papers are rejected because they are not a sufficient advance over the current knowledge. But because of journal hierarchy, there are different standards for getting into each journal. The technical issues at NN are the same as those at Neuroscience. What is defined as “interesting to the readership” is the factor that is changing.
As for the pecking order, I don’t think that anyone really believes that there will NOT be a one-way street for these reviews. Why is that bad? This one-way street exists anyway. If you DO get rejected from J Neurosci, you still send it to Neuroscience. In this case, you take your reviews with you, speed up your own process, and remove some burden from overworked reviewers. Nothing keeps you from submitting to Neuroscience first to save even MORE time. The consortium makes overshooting easier, but with the review process streamlined, the new system could provide an enormous value to all those involved, more than making up for this introduced annoyance.
LikeLike
December 3, 2007 at 1:34 pm
BikeMonkey says: but still, the scenario you paint in which everyone is really only focused on bunny-hopping, meaning assays that really only use bunnies and hopping as a behavior? rare, I’d say.
In today’s funding climate, it doesn’t have to be everyone: just one of the primary reviewers getting 20 points or so less enthusiastic can be completely damning. This is an example of how study sections that used to be functional with paylines around 25% are dysfunctional with paylines around 10%.
I’m going to piss off a lot of people and suggest that, in these dark times, the NIH reminds itself that it is the National Institutes of HUMAN Health, and that the NIH make studies on humans, with human cells and of direct relevance to humans get priority. I’m sure there’s nice NSF money to be had for everyone else…
Some journals really want to make me cry. There are several very high profile entries that could easily rename themselves, “Journal of We’re Not Going to Cure Human Disease Studying This”, even though that is the nominal justification for these studies.
Don’t get me started… 🙂
LikeLike
December 3, 2007 at 1:48 pm
I was just clarifying the PP point which you seemed to not appreciate. The apex journal will see increased submissions, that’s all. Also, I just don’t like the seeming hypocrisy/inaccuracy of the mission statements.
The issues that actually concern me are these:
1) monopoly. The benefits to those within the conspiracy (sorry, collaboration) are clear. At any rank which has a higher rank, the journal gets preferential shot at papers they might not otherwise get. For example, I doubt very much that IF seekers drop down to the 4IF range after getting rejected from JNeurosci. I would estimate that their next step is to something more competitive with JNeurosci like Biological Psychiatry, Neuropsychopharmacology, Brain. Things with more similar IFs. Those are the ones potentially getting cut out of the loop. If the idea is really to benefit neuroscientists, and given that this thing was spearheaded by SfN / JNeurosci, the best benefit would have been to sign up similar IF journals. Again, if the issue was really “fit” and not “impact”. So I will be interested to see if any legit lateral competition for JNeurosci joins this thing. To continue my personal examples, if I thought I had something that was actually the JNeurosci quality and they didn’t see it that way, I would be going lateral, not dropping down 3 pts in IF. Sure, nothing preventing me from doing so, this is not my point. My point is that if this is supposed to be a service to authors, well, the way they do it gives me some clues as to who it is really going to benefit.
2) promulgation of scientifically inferior “sexy” work. i think you’ve heard me on this one before noah. look, everyone can see that the glamor mags have a certain style which has a tendency to prioritize breadth of technique over experimental depth. as papers get constructed for, e.g. C/N/S and then down-submitted after rejection, they are not reconfigured into the more native format of, say, what JNeurosci used to be. No. Instead, over time JNeurosci becomes glamorMag-ified. the trickling down of this business into the society journals in which focus, depth, replication and, yes, good old fashioned scholarship is prioritized is a BadThing.
3) If lateral submission became a reality then in many cases authors would just opt out of this. Sometimes these things are just chance. I’m thinking of some recent stuff of mine, maybe a half-dozen ms’s, and a colleague (maybe another 4) which has been submitted to a collection of maybe 5 journals. The work is pretty close in nature. Results show that there is most distinctly a random element to this process! A clearly (imo) inferior paper accepted at one journal, while a better one is rejected. Another journal of very similar scope and IF takes the reject no problem. Another paper beat up at one journal while a highly similar one, maybe even with slightly less clear results, sails in at a journal 3 IF higher. go f’ing figure. My point being that one is motivated to take independent whacks at different journals because one assumes that decisions pivot not on dispassionate analysis of the science but perhaps the luck of the reviewer draw in some cases.
LikeLike
December 3, 2007 at 2:10 pm
Well, when you get the reviews you like, you might as well keep them. I thought that this would appeal to you, with all of the conspiracy theories surrounding publishing that you promote. This is giving you back a little power to better influence your own destiny…
All self-proclaimed neuroscience journals indexed on PubMed are invited to join. All they have to do is go to the consortium website, download the form, eliminate confidential comments, and they are immediately a member. If your favorite journal is not a member, please feel free to contact them, make them aware of the consortium and urge them to join. Again, you are very fast to cry foul, and especially conspiracy, rather than considering other alternatives.
LikeLike
December 3, 2007 at 2:51 pm
oh good gravy.
the “conspiracy” was a joke, Noah, a joke.
“you are very fast to cry foul, and especially conspiracy, rather than considering other alternatives.”
In fact what I’m doing here is exploring where interests lie, a much more reliable judge of behavior than taking what someone says on face value. Also a better way to predict which journals are going to sign up and how authors are going to respond. My hypotheses. I’ll be interested to see them all tested.
“I thought that this would appeal to you, with all of the conspiracy theories surrounding publishing that you promote”
Didn’t know that I was “promoting” anything in this regard. When I “promote” things it is phrased as “Go respond to this RFI”, or “Please call your Senator” or, “go read that interesting post Noah Gray has up”. Expressing my experiences and analysis of the available data is not “promoting”.
Look, one of the things I do try to “promote” is that nobody has a lock on the OneTrueWay and if you think you do have a OneTrueWay, you might as well advance some arguments. And be ready to defend them. This is the only way we get anywhere. You think that because I question the ascendancy of GlamorMag science that I’m a conspiracy nut or something? Fine, but rebut the charges while your are at it. Not with ad hominems, not with oh-mercy-me horrified subrosa threats involving the integrity of your journal, but with actual points that respond.
In this particular instance, despite the fact that my comment is directed mostly at JNeurosci, not you, go ahead and tell us whether NN is considering joining the consortium and why. tell us why there is a debate and what the foot dragging issues are. give us some insight. go ahead and make it clear before being asked that you emailed your reviewers with the given criteria and that is why these “names” have commented on your blog.
“If your favorite journal is not a member, please feel free to contact them, make them aware of the consortium and urge them to join.”
Me, I’m not convinced of the merits yet. But I think you can grasp from some of my other issues that should I think something is important enough I’m not shy about doing so. But I will underline this point for Noah. If you readers think this consortium sounds interesting, by all means contact your favorite editor and ask when that journal is joining….
LikeLike
December 3, 2007 at 4:04 pm
My, aren’t we an angry little monkey! In all of my above comments I detailed the positions that make this an exciting experiment for the review process. NN is considering joining, but removing the confidential comments is a momentous occasion, since no journal at NPG has ever done so. Also, until Jan 1, 2008 rolls over and the consortium begins, very few to no journals have ever removed this option in their review process, so it is a big deal. This sort of move requires the need to “re-train” reviewers who are used to placing particular opinions only to the editors. It requires that the journals trust the reviewer-picking ability of the other members, if the reviewer decides to allow his/her review to be transferred, but only anonymously. There are plenty of problems and reasons to drag one’s feet. But of course, we could just do nothing, try nothing new and keep trudging along. Silly me, I thought that there were some out there who were longing for some changes and experimentation in the entire scientific publication process. As to your point that I never defend my position, what was I supposed to defend? The fact that you thought J Neuroscience was not allowing journals of a similar impact factor to join? I thought I did defend my position by stating that you were wrong. End of sentence. Everyone is invited. My evidence for this is that the website is publicly available. Any journal can join…
go ahead and make it clear before being asked that you emailed your reviewers with the given criteria and that is why these “names” have commented on your blog.
Don’t take little wanker pot-shots at me either with this sort of comment. Here I am reaching out, with my identity and publication record in the field available for all to see, making my opinion known, interacting in a way that no other editor does (at least at NPG, as far as I can tell, and I have looked…), leaving my very candid opinions loosely linked to my company (who is definitely more conservative than I am), assuming the potential risk for censure. I accept the risk so that I can interact with the community in a more exciting fashion.
All our opinions are colored by our background and present situation. I provide FULL disclosure of mine. Meanwhile, I don’t even know who you are. We simply take you at your word that you even do research. Come out from behind the security blanket of anonymity and give your comments/positions a little reality check.
When I said I solicited comments on an important issue, I meant I SOLICITED COMMENTS. I apologize that this wasn’t more clear the first time around. Your comments make it sound as if you believed that thread to be a cheap ploy for attention. On the contrary, that solicitation was designed to create a forum for respected members of the neuroscience community to discuss an issue close to the hearts and minds of all authors. As you can tell from the posts, the words mean a lot more when people sign their names to their comments as opposed to signing it “drugmonkey” or “Dr.??” or “sillyboy” or neuroscientist_123. I guess I’ll never understand this blogging anonymity thing, but that’s fine by me. I’m not interested in your reasons for doing so.
LikeLike
December 3, 2007 at 5:27 pm
“My, aren’t we an angry little monkey!”
Ruh, Roh!!
LikeLike
December 3, 2007 at 6:33 pm
Sorry for my formatting errors above; the italics were supposed to end after the quote. If that could be fixed, I would appreciate it. I’m also more relaxed now. I had a couple nasty interactions with authors today and they pissed me off to no end, leading to my tirade. Therefore, I read the previous comments of drugmonkey under the wrong state of mind and overreacted. Now that I have had a couple glasses of wine to relax, I can laugh at myself. Cheers, mate.
LikeLike
December 3, 2007 at 7:26 pm
Too late!! You pissed off the monkey! He don’t drink wine!
LikeLike
December 4, 2007 at 12:02 am
tch, tch PP, such a crude attempt to fan the flames. really!
noah, no worries. even in rant mode you have good points. not that you see all the perspectives, true, but valid points nevertheless. and in honesty mode i will admit that I take somewhat unfair potshots on occasion. and i’ve already admitted to tarring you wtih the Nature Mag brush :-). and, just every so often I’ll go ahead and straw-ify someone’s comment or post (like YFS’s, the poor thing) for rhetorical purposes. sure. quite annoying if you are the subject, i hope i took YFS and co-wanderers return strawification in at least a decent humor, if not in stride.
blogonymity is frustrating but it has its advantages once you get used to it. ask yourself why you wish to know who i am or who YFS is for that matter. is it because you feel like launching an ad hominem by any chance? what is it about what I say that requires some sort of authentication or credentialing?
LikeLike
December 4, 2007 at 12:27 am
[…] 4, 2007 A recent comment from whimple is pretty self explanatory: Say I work on the mechanics of bunny-hopping. My papers get sent for […]
LikeLike
December 5, 2007 at 3:15 pm
[…] understand it. It seems like this is, in part, trying to address the bunny-hopping issue raised by whimple. Posted by drugmonkey Filed in Grantsmanship, NIH Careerism, NIH […]
LikeLike
December 6, 2007 at 3:41 pm
Just to tie up a final loose end (I want to be caught up before the weekend!!), I was just trying to understand the reasons for the anonymity. Sometimes it is easier to understand a position being argued if the reader has more information about the blogger. Everyone here knows who I am, and can ask questions accordingly, or lambaste my opinions in a more educated manner. Even YFS is a little easier to understand because I do know at least some major facts about her (long-term post-doc, female, in science, not happy with the lab, etc…) allowing me to adjust my position accordingly.
YFS gave me her honest reasons for why she does it and they made sense to me. All I have to say about that is: To each his own. It is definitely not for me, but I respect it.
LikeLike
December 6, 2007 at 9:30 pm
the trouble is that this cuts both ways. knowing “major facts” about someone also tends to make you skip past what they are actually saying and make assumptions. Sometimes those are valid assumptions and sometimes not. Sometimes you cannot help but let your assumptions qualify the weight to which you assign a person’s comments. To make you a bit dismissive. We all do that to some extent. If you don’t know the relevant details, you aren’t as likely to be biased, no? yes there is the chance that you can be dismissive based on the anonymous issue, but certainly not on sex, race, tenure status, academic sub-discipline, how many R01s awarded, etc, if you have not been given that information.
on the flip side you are certainly correct. I could be some random nutjob of dubious mental health just making sheist up from a public library cubicle in Peoria. I’ll just have to let the reader make that call.
LikeLike
December 7, 2007 at 4:14 am
“I could be some random nutjob of dubious mental health just making sheist up from a public library cubicle in Peoria.”
Could?
LikeLike
December 7, 2007 at 7:38 am
I agree with drugmonkey’s points on the merits of anonymity. Noah, I would think as someone that deals with anonymity all the time (eg the peer review system), you might have some perspective on this. Or do you believe in open review, as well?
BG, not my real name either
LikeLike
December 7, 2007 at 7:58 am
I do deal with anonymity all the time, but only from the observational standpoint; i.e. I maintain it for the reviewers, but I obviously know who they are.
I am a big proponent of being as transparent as possible during the review process, but do not necessarily have a big problem with anonymous peer review. Rather, I believe that anonymizing the authors would be the better move, preserving the critical abilities of the more squeamish reviewer who does not want their identity known. Yet, it protects the author from pre-conceived bias.
The worst argument against that system is that it would be obvious who the authors are anyway based on the techniques or interpretations. That’s lame. One could say the same thing about reviewers based on how they critique (and people do guess, of course). But that doesn’t mean that the guesser is correct. This holds no water for me as interference.
The best argument against the anonymous author system is that reviewers could potentially put themselves in a conflict of interest situation, inadvertently, and would have to be very careful when discussing work they have seen, in case it is the work from the guy down the hall, or their friend within the field, etc… This comes up more than I expected when I discuss this with reviewers at happy hour.
I guess my only answer to this is to just trust the editor. Part of my job is knowing who are friends, who are enemies, who are neighbors, and who is having an affair, so as to not choose reviewers who may be personally biased. Why would that practice change when the authors become anonymous to the reviewer? In addition, technically, the reviewer shouldn’t be discussing these confidential unpublished manuscripts outside of the lab anyway, so when people give me the old “I don’t want to trash my friend’s work to him by accident”, I ask them “Well, why are you discussing unpublished results to which you have a responsibility to keep confidential, protecting the authors”. So besides the personal uncomfort that may come with THINKING that you are reviewing a manuscript that may be from a close colleague (or former student, or former lover, etc…), there is not much to this argument, in my mind, as well.
I definitely think that this magnitude of change would freak too many people out, so we are not ready for this, but that is exactly the direction in which I would go, if (when) I ever call the shots. It needs to be tried. I’ll stop there because this is an ENTIRELY different entry.
By the way, Noah Gray IS my real name.
LikeLike
December 7, 2007 at 8:11 am
Thanks Noah for the detailed reply. Wanted to clear up one thing – yes I know that you really are NG. My sig line was just a poorly worded attempt to acknowledge that, like the majority of the posters here, I am ‘hiding’ behind a pseudonym, as well.
Back to my original post – I guess I was asking, whether you felt that the anonymity of the current peer review process, shapes/helps the dialog in the same ways that DM was arguing anonymity has its merits on the web.
LikeLike
December 7, 2007 at 8:37 am
I was just being sassy with my last line in that comment…ah yes, the lack of emotional reads on the internet…a different kind of anonymity.
Anyway, I guess I was stressing that for some (most) reviewers, the only way they can be fully critical is to be anonymous, because they don’t want to have to deal with the fall-out of their words. That is fine. But as an editor, I have trained myself to do EXACTLY what DM was discussing, which is to weigh the comments of each reviewer accordingly. As you can imagine, the review process is a messy beast. There is rarely a consensus of opinion. Therefore, I have to weigh the positives and negatives accordingly, based on the identity of the reviewer (not in stature, but in expertise and experience with a particular technique). If a first-time reviewer/green PI (or an old retired, out-of-touch professor emeritus, for that matter) loves a paper and thinks it should be published, while a well-calibrated reviewer, who has earned the editors’ respect with their insightful words, has serious reservations regarding a manuscript, I’m going to weigh the comments of the calibrated guy a little more. Because I have to. Especially when those reservations make a lot of sense. Especially when the calibrated guy is actually a little closer in expertise to the work in question, anyway.
So as you can see, I am very used to weighing opinions based on my knowledge of a person’s background (I am treading on tender ground; there is no bias towards anything other than the thoughtfulness which was put into the review and the obvious expertise of the said reviewer). It seems that I let these tendencies and that training creep into my blogging. Something of which I should become aware.
LikeLike
December 7, 2007 at 8:47 am
“The worst argument against that system is that it would be obvious who the authors are anyway based on the techniques or interpretations. That’s lame. One could say the same thing about reviewers based on how they critique (and people do guess, of course). But that doesn’t mean that the guesser is correct.”
There are several orders of magnitude difference in the confidence and accuracy with which one could predict who the authors were versus who the (paper) reviewers were! Especially if the review is well written in anon-preservation reviewer-speak. Considering what it would take to similarly anonymize the average paper I write or review…this would be a large scale change in approach and be absolutely impossible in some cases.
I’m willing to buy that in certain areas two or six labs may have essentially equivalent chances of having produced a given manuscript but this is not something I am familiar with in my subfield.
LikeLike
December 7, 2007 at 9:29 am
Noah, blogoland is really not that much different. there may be blogs you keep coming back to and those you read a couple of times and never return. some of this may be the quality of the postings (and comments!). some of this may be the apparent “authority” like membership in the Borg, links and mentions from other blogs you read, etc.
in some sense all you are really saying is that you don’t have the same degree of experience in making these judgments about bloggers that you do about potential NN reviewers (or authors for that matter).
look, ultimately there is no hiding in the blog business unless you spend all your time being paranoid and only minimally participating in the blogo “game”. make enough posts and enough personal vignettes are going to appear to point someone in a direction. piss someone off enough and they are going to track. you. down. (see the creationism and woo bashers over on the Borg, for example). post enough comments to someone else’s blog (say, Action Potential) and they are often going to be able to make some good guesses based on IP address, visit time, etc. Occasionally someone will f-up and do something that catches them dead to rights, like supplying the mainstream email address or something.
there are some parallels to the anonymous review of manuscripts and grant worlds I will note. the blog administrator has a lot of info that they traditionally keep confidential. some people (and i’ll include myself on occasion) use a pseudonym name but go ahead and supply a more-revealing (or the same) email address. easier to make up new names than to have different email accounts! in such cases the individuals trust the blog administrator not to “out” them. it is a culture.
i started off with the email obligatory for comment but have turned that off because so far i’ve been troll free and the spam filter is good at wordpress. i’ve a few consistent commenters, the few who use a real name are generally already-out bloggers, maybe three that used a pseudonym with a revealing email address and the balance I’d have to work really hard to figure out who s/he is. Me, I’m not sure that I have any differential “insight” into what someone is saying based on knowing who they are or not…
LikeLike
December 7, 2007 at 10:04 am
oh, and you might find this post relevant to the question of blogonymity. perhaps.
LikeLike
December 7, 2007 at 2:38 pm
“Considering what it would take to similarly anonymize the average paper I write or review…this would be a large scale change in approach and be absolutely impossible in some cases.”
Many papers have sentences in abstract/intro like “We have previously shown blah (cite).” That sure gives it away. Also, people in a field generally know what their colleagues are up to. I really think author anonymity is tilting at windmills.
LikeLike
December 7, 2007 at 2:58 pm
“I really think author anonymity is tilting at windmills.”
I just flat out don’t see how it can be done. Easy enough to get the “we have previously demonstrated” out of there. not so much the fact that the whole paper in and out shouts “this is from Dr. Smith’s laboratory”. from methods, to arguments, to references chosen, to the topics under consideration.
LikeLike
December 7, 2007 at 3:30 pm
Guess it’s not worth trying then simply because somebody might guess who the authors are.
The authors don’t have to do anything different. I’ll just remove the names when it is sent out. If the reviewer chooses to track down the author, that’s his/her prerogative. Who cares. It still inserts a kernel of doubt that could dislodge some pre-conceived notions by the reviewer.
When people say “we”, they may refer to a time when they were a postdoc in a different lab (and now have their own lab), or when they were a collaborator with someone else (and are now not involved in a collaboration). I read 60-70 pages a month and it is not as obvious as you may think. Expand your horizons a bit and consider that the possibility exists that many papers do not blatantly give away the identity of the authors…….as a wise blogger once told me – “…nobody has a lock on the OneTrueWay…”
LikeLike
December 7, 2007 at 3:54 pm
“Guess it’s not worth trying then simply because somebody might guess who the authors are.”
The question is more whether this is a red herring that has no chance of working which distracts us from more effective fixes.
“The authors don’t have to do anything different. I’ll just remove the names when it is sent out.”
I for one would have no objection whatsoever if an editor said “Gee, we’re thinking about sending these out without the cover page”. Mostly because I think it would have zero effect, of course, but also because I don’t think that I get any sort of pass because of who I am. Quite the contrary.
“Expand your horizons a bit and consider that the possibility exists that many papers do not blatantly give away the identity of the authors”
Sure. But you should also consider the fact that you may have just a slightly narrow view on what is “neuroscience” not to mention the breadth of bioscience publishing. Of NN is near the apex of a pyramid it suggests a certain lack of breadth in what you see. Of course, I have no idea what your experience might be before this gig…
LikeLike
December 7, 2007 at 3:58 pm
“Of course, I have no idea what your experience might be before this gig…”
Well, he is an author of two papers whose titles each start with the word “rapid”.
LikeLike
December 13, 2007 at 3:10 pm
[…] lined up to join the Neuroscience Peer Review Consortium. This may be of some relevance to prior smart arse comments, particularly the appearance of Biological Psychiatry as a lateral competitor of J […]
LikeLike
December 13, 2007 at 11:13 pm
Besides just being able to tell the lab by the model, techniques, conclusions, etc., it is often possible to tell who the paper is from just because the work has been previously presented at a conference. It would be a real shame if people were motivated not to present unpublished data (I mean, more than they already are out of sheer secrecy) out of the fear that potential reviewers were sitting in the audience and they would then lose their author anonymity.
LikeLike