Show me the data, Jerry!!!!!!
September 3, 2013
Today’s Twittsplosion was brought to you by @mbeisen:
he then elaborated
and
and
There was a great deal of distraction in there from YHN, MBE and the Twitteratti. But these are the ones that get at the issue I was responding to. I think the last one here shows that I was basically correct about what he meant at the outset.
I also agree that it would be GREAT if all authors of papers had deposited all of their raw data, carefully annotated, commented and described (curated, in a word) with all of the things that I might eventually want to know. That would be kickass.
And I have had NUMEROUS frustrations that I cannot tell even from methods sections what was done, how the data were selected and groomed, etc in many critical papers.
It isn’t because I assume fraud but rather that I find that when it comes to behaving animals in laboratory studies that details matter. Unfortunately we all wish to overgeneralize from published reports….the authors want to imply they have reported a most universal TRUTH and other investigators wish to believe it so that they don’t have to sweat the details.
This is never true in science, as much as we want to pretend.
Science is ever only a description of what has occurred under these specific conditions. Period. Including the ones we’ve bothered to describe in the Methods and those we have not bothered to describe. Including those conditions of which we have no knowledge or understanding that they might have contributed.
Let us take our usual behavioral pharmacology model, the 10 m Hedgerow BunnyHopper assay. The gold standard, of course. And everyone knows it is trivial to speed up the BunnyHopping with a pretreatment of amphetamine.
However, we’ve learned over the years that the time of day matters.
Until…finally….in its dotage seniority. The Dash Lab finally fesses up. The PI allows a trainee to publish the warts. And compare the basic findings, done at nighttime in naive bunnies, with what you get during the dawn/dusk period. In Bunnies who have seen the Dash arena before. And maybe they are hungry for clover now. And they’ve had a whiff of fox without seeing the little blighters before.
And it turns out these minor methodological changes actually matter.
We also know that dose response curves can be individual for amphetamine and if the dose is too high the Bunny just stims (and gets eaten by the fox). Perhaps this dose threshold is not identical so we’re just going to chop off the highest dose because half of them were eaten after that dose. Wait…individuals? Why can’t we show the individuals? Because maybe a quarter are speeded up by 4X and a quarter by 10X and now that there are these new genetic data on Bunny myocytes under stressors as diverse as….
So why do the new papers just report the effects of single doses of amphetamine in the context of this fancy transcranial activation of vector-delivered Channelrhodopsin in motor cortex? Where are the training data? What time of day were they run? How many Bunnies were aced out of the study because the ReaChr expression was too low? I want to do a correlation, dammit! and a multivariate analysis that includes my favorite myocyte epigenetic markers! Say, how come these damn authors aren’t required to bank genomic DNA from every damn animal they run just so I can ask for it and do a whole new analysis?
After all, the taxpayers paid for it!
I can go on, and on and on with arguments for what “raw” data need to be included in all BunnyHopping papers from now into eternity. Just so that I can perform my pet analyses of interest.
The time and cost and sheer effort involved is of no consequence because of course it is magically unicorn fairy free time that makes it happen. Also, there would never be any such thing as a protracted argument with people who simply prefer the BadgerDigger assay and have wanted to hate on BunnyHopping since the 70s. Naaah. One would never get bogged down in irrelevant stuff better suited for review articles by such a thing. Never would one have to re-describe why this was actually the normal distribution of individual Hopping speeds and deltas with amphetamine.
What is most important here is that all scientists focus on the part of their assays and data that I am interested in.
Just in case I read their paper and want to write another one from their data.
Without crediting them, of course. Any such requirement is, frankly my dear, gauche.
An Academic Department Attempts to Go Beyond CVs and JIF in Hiring
September 3, 2013
In Science, from Sandra L. Schmid, Ph.D. [PubMed] who is Chair of Cell Biology at UT Southwestern.
The problem:
CVs provide a brief description of past training—including the researcher’s pedigree—as well as a list of awards, grants, and publications. A CV provides little insight into attributes that will ensure future success in the right environment. For example, a CV is unlikely to reflect the passion, perseverance, and creativity of individuals who struggled with limited resources and created their own opportunities for compelling research. Nor is a CV likely to identify bold and imaginative risk-takers who might have fallen—for the moment—just short of a major research success. The same is true for those who found, when they realized their goal, that their results exceeded the imaginations of mainstream reviewers and editors, the gatekeepers of high-profile journals. Finally, for junior hires at early stages of their careers, a CV is unlikely to reveal individuals who are adept at recombining knowledge and skills gained from their graduate and postdoctoral studies to carve out new areas of research, or those able to recognize and take advantage of unique opportunities for collaboration in their next position.
Her Department’s solution:
We will be asking applicants to write succinct cover letters describing, separately and briefly, four elements: (1) their most significant scientific accomplishment as a graduate student; (2) their most significant scientific accomplishment as a postdoc; (3) their overall goals/vision for a research program at our institution; and (4) the experience and qualifications that make them particularly well-suited to achieve those goals. Each of the cover letters will be read by several faculty members—all cell biology faculty members will have access to them—and then we will interview, via video conferencing technologies, EVERY candidate whose research backgrounds and future interests are a potential match to our departmental goals.
She closes with what I see as a deceptively important comment:
Let’s run this experiment!
You have probably gleaned, Dear Reader, that one of my greatest criticisms of our industry is that the members of it throw all of their scientific training out the window when it comes to the actual behavior OF the industry. Paper review, grant review, assessment of “quality”, dealing with systematic bias and misdirection…… MAN we are bad at this.
Above all, we are reluctant to run experiments to test our deep seated beliefs. Our beliefs that GRE quantitative or verbal or subject predict grad school performance. Our beliefs that undergraduate GPA is the key or maybe it is research experience in a lab of some DewD we’ve heard of. Our belief that what makes the postdoc is X number of first author pubs in journals of just exactly this Impact Factor. Our confidence that past performance predicts future success of our new Asst Professor hire….or tenure candidate.
So often we argue, viciously, our biases. So infrequently do we test them.
So bravo to Chair Schmid for actually running an experiment.
DrugMonkey 