The most replicated finding in drug abuse science
July 8, 2014
Ok, ok, I have no actual data on this. But if I had to pick one thing in substance abuse science that has been most replicated it is this.
If you surgically implant a group of rats with intravenous catheters, hook them up to a pump which can deliver small infusions of saline adulterated with cocaine HCl and make these infusions contingent upon the rat pressing a lever…
Rats will intravenously self-administer (IVSA) cocaine.
This has been replicated ad nauseum.
If you want to pass a fairly low bar to demonstrate you can do a behavioral study with accepted relevance to drug abuse, you conduct a cocaine IVSA study [Wikipedia] in rats. Period.
And yet. There are sooooo many ways to screw it up and fail to replicate the expected finding.
Note that I say “expected finding” because we must include significant quantitative changes along with the qualitative ones.
Off the top of my head, the types of factors that can reduce your “effect” to a null effect, change the outcome to the extent even a statistically significant result isn’t really the effect you are looking for, etc
- Catheter diameter or length
- Cocaine dose available in each infusion
- Rate of infusion/concentration of drug
- Sex of the rats
- Age of rats
- Strain of the rats
- Vendor source (of the same nominal strain)
- Time of day in which rats are run (not just light/dark* either)
- Food restriction status
- Time of last food availability
- Pair vs single housing
- “Enrichment” that is called-for in default guidelines for laboratory animal care and needs special exception under protocol to prevent.
- Experimenter choice of smelly personal care products
- Dirty/clean labcoat (I kid you not)
- Handling of the rats on arrival from vendor
- Fire-alarm
- Cage-change day
- Minor rat illness
- Location of operant box in the room (floor vs ceiling, near door or away)
- Ambient temperature of vivarium or test room
- Schedule- weekends off? seven days a week?
- Schedule- 1 hr? 2hr? 6 hr? access sessions
- Schedule- are reinforcer deliveries contingent upon one lever press? five? does the requirement progressively increase with each successive infusion?
- Animal loss from the study for various reasons
As you might expect, these factors interact with each other in the real world of conducting science. Some factors you can eliminate, some you have to work around and some you just have to accept as contributions to variability. Your choices depend, in many ways, on your scientific goals beyond merely establishing the IVSA of cocaine.
Up to this point I’m in seeming agreement with that anti-replication yahoo, am I not? Jason Mitchell definitely agrees with me that there are a multitude of ways to come up with a null result.
I am not agreeing with his larger point. In fact, quite the contrary.
The point I am making is that we only know this stuff because of attempts to replicate! Many of these attempts were null and/or might be viewed as a failure to replicate some study that existed prior to the discovery that Factor X was actually pretty important.
Replication attempts taught the field more about the model, which allowed investigators of diverse interests to learn more about cocaine abuse and, indeed, drug abuse generally.
The heavy lifting in discovering the variables and outcomes related to rat IVSA of cocaine took place long before I entered graduate school. Consequently, I really can’t speak to whether investigators felt that their integrity was impugned when another study seemed to question their own work. I can’t speak to how many “failure to replicate” studies were discussed at conferences and less formal interactions. But given what I do know about science, I am confident that there was a little bit of everything. Probably some accusations of faking data popped up now and again. Some investigators no doubt were considered generally incompetent and others were revered (sometimes unjustifiably). No doubt. Some failures to replicate were based on ignorance or incompetence…and some were valid findings which altered the way the field looked upon prior results.
Ultimately the result was a good one. The rat IVSA model of cocaine use has proved useful to understand the neurobiology of addiction.
The incremental, halting, back and forth methodological steps along the path of scientific exploration were necessary for lasting advance. Such processes continue to be necessary in many, many other aspects of science.
Replication is not an insult. It is not worthless or a-scientific.
Replication is the very lifeblood of science.
__
*rats are nocturnal. check out how many studies**, including behavioral ones, are run in the light cycle of the animal.
**yes to this very day, although they are certainly less common now
DrugMonkey 
July 8, 2014 at 10:08 am
Cf. Enrichment: see Rat Park.
LikeLike
July 8, 2014 at 10:11 am
Did you have a point?
LikeLike
July 8, 2014 at 10:12 am
And btw, the beat goes on. apparently peers who do/do not self-administer drug can influence the behavior of an observer rat
http://www.ncbi.nlm.nih.gov/pubmed/24878249
LikeLike
July 8, 2014 at 10:55 am
Yes, but only if the rats are republican.
LikeLike
July 8, 2014 at 11:01 am
The problem with all of this discussion is that replication is not the goal. It never was. If all you are doing is running the exact same sequence with the exact same incantations then you are just doing magic. Science is about hypothesized mechanism. A hypothesized mechanism tells you what parts of your experiment were necessary.
The problem is that people are all concerned about (a) statistical noise and (b) fraud. These are important issues to be aware of, sure, but I suspect they are very rare, especially in the “replication problem”.
What’s really important is interpretation. You do an experiment. You think it shows you something. You do some controls to help convince yourself that you’re not fooled. But then you have an interpretation. Your interpretation tells you what you think your experiment is really showing you. Your interpretation says that some things should matter and some don’t. As you (and others) attempt to replicate (and move beyond) your experiment, they discover that some of the things your interpretation said were not important are important and some of the things your interpretation said were important are not. And science progresses forward.
As you say, self-administration is extremely complicated and the results you get depend on how you run your experiment. We know this because a lot of experiments have explored the space and a lot of theories have proposed what the important factors are.
PS. Being careful about that interpretation is called “theory” and is a very important part of science!
LikeLike
July 8, 2014 at 11:52 am
What is the point of replicating a study that only works in an extremely narrow range of circumstances? I thought the goal was to understand the mechanisms of cocaine addiction. Are any of these intravenously self-administering rats ever actually addicted to cocaine?
LikeLike
July 8, 2014 at 12:31 pm
Who said anything about “only works”? I am describing factors that can compromise direct replication….including ones that lead to a quantitative shift and ones that might lead to a qualitative shift.
and yes, the goal is to study the processes of addiction. I am limiting myself here to methodological development because it is a fairly clear example to describe.
A similar discussion holds true for the discoveries about the pharmacology and physiology of brain changes that are associated with what we think of as the addicted state.
Now as it happens, for some other drugs of abuse the range of parameters is even more constrained than it is for cocaine IVSA. Are such models useful- heck yeah they are. Methodological constraint doesn’t mean useless for behavior any more than it does for in vitro assays and gene jockery and what have you.
LikeLike
July 8, 2014 at 12:35 pm
replication is not the goal. It never was.
I disagree. It is implicit (and sometimes explicit) in a scientific paper that we are claiming that the effect will replicate. That we have found something that is capital T True and therefore any other idiot should be able to duplicate our results.
We may have goals beyond that, and almost always do, but the idea that an effect will be replicable is an essential basis for moving beyond.
LikeLike
July 8, 2014 at 1:09 pm
That we have found something that is capital T True and therefore any other idiot should be able to duplicate our results.
Yes I agree, but when the idiot is duplicating our results, what things do we need to hold constant and what do we not? Does it matter that you ran the experiment in city X and I ran it in city Y? (Generally, I hope not, but I have seen examples where humidity levels affect signals, which makes experiments that work in Tucson hard to replicate in Houston.) Does it matter that you ran the experiment in the morning or at night? (Obviously, issues of circadian rhythm abound.) What matters and what doesn’t?
True replication would be to run the exact same experiment with everything completely identical. But that tells us very little except that if you say the right magic words the same thing will happen.
In a replication attempt, when the result occurs again, then we know something about the generalization properties of the phenomenon. But when the result fails, then we have the important question of “why?” Sometimes the answer is that the first study was a statistical fluke (which is boring but important) or that the first study was a fraud (which is sad but important). Sometimes the answer is that something important has changed (which is interesting and Science!)
My point is that science in all the fields I know of work by this sort of “pseudo-replication”, in which a similar experiment is run, holding most things constant, but changing a few. When it works, yay! When it doesn’t, hmmm!
LikeLike
July 8, 2014 at 2:46 pm
Was my point really so unclear then? We are on the same page here.
LikeLike
July 9, 2014 at 3:42 am
“That we have found something that is capital T True and therefore any other idiot should be able to duplicate our results.”
What if the other idiot is a Pharma Company Exec? Should we cry him a river if he can’t replicate the study? I contend that it is a problem when replication efforts fail, narrow or broad-focused, and by whoever, whereever. You never see that list of potential factors that you published above in papers. Did the STAP paper people print such a list in their Nature paper? (Maybe that one is a bad example).
Would a journal even publish a paper if such a long list of variables was presented? Maybe it is apparent in some fields, but in my experience such ‘variables’ are an open secret that is not talked about. This helps labs stay competitive by keeping methods close and tight.
This is one of your best posts.
LikeLike
July 9, 2014 at 8:26 am
Would a journal even publish a paper if such a long list of variables was presented? It’s called a methods section.
We don’t know the list of variables until many people have been working in the field. (i.e. lots of experiments have been tried. This is a good use of review papers.) However, this is why it is very important to specify your methods as completely as possible. (So that others can determine what you did.)
Of course, what has to really happen is when someone tries to follow up on your work (note that I don’t use the word “replicate”) and they don’t get the same results, you should work with them to figure out why not. (One of my favorite examples was a pair of colleagues who discovered that the anesthesia used during electrode implantation surgery affected sensory cortex neural recordings taken weeks later. It took them years of matching each step of the experiment to determine that’s what it was.)
LikeLike
July 10, 2014 at 10:46 am
“If it happens once, it’s a curiosity. Twice, it’s coincidence. Third time, it’s science (or enemy action).” Even if someone is extending the published work, one of the first steps is often using the published study as a ‘positive control’. Replication is sometimes part of the design of the next steps. This isn’t always possible in a large study with human subjects, but aren’t a “Do we see what they saw? Is this working in our hands?” important starting questions?
There is lore that in the earliest days of scientific publishing, one reviewer wouldn’t pass a paper forward until he had replicated the work in his laboratory.
LikeLike
July 10, 2014 at 12:32 pm
Apparently the “anti-replication yahoo” is… an fMRI guy.
Stereotypes. Some people really seem to enjoy confirming them.
LikeLike
July 10, 2014 at 1:10 pm
AQB-
Sometimes it just takes time to piece together the clues from the published record. Lab lore may be no more than familiarity with the key variables for a specific assay.
LikeLike
July 10, 2014 at 1:12 pm
toto- this fact didn’t escape me…trying to extend benefit of doubt here.
LikeLike
July 14, 2014 at 2:45 am
Just a minor point, but relevant perhaps. Surely, cocaine being an effective painkiller, these rats are possibly self-administering to deal with the pain of the surgery rather than just ‘getting high’.
LikeLike
July 14, 2014 at 3:39 pm
Doubtful. First of all, animals receive post-surgical pain meds. Second, I’ve never seen anything to indicate that the post-surgical interval modulates the IVSA of cocaine in any serious way. Not that I know of any close comparison studies but if you read methods you can see some variety in start of training relative to surgery.
LikeLike
July 15, 2014 at 2:05 am
Thanks for the response. You say that you’ve “never seen anything to indicate that the post-surgical interval modulates the IVSA of cocaine in any serious way”. Is it not possible that by the time the pain has eased, the addiction has already taken hold?
Not trying to doubt your findings, just a layman asking what might to you already be the plainly obvious.
As Doug Stanhope put it; “There’s no such thing as addiction. Just things you enjoy more than life.”
LikeLike
July 15, 2014 at 3:54 am
No, I mean the self-admin training is not started until animals have recovered, typically. And when that interval is shorter than I might like to see, personally, I see no evidence that systematically enhances IVSA rates.
I can’t think of any explicit comparisons, this is mostly coming from piecing together the odd study and lab lore sort of thing. Might be interesting to test but I’d think the most relevant study would be prescription opioid self-admin to add translational relevance.
LikeLike
January 18, 2017 at 5:05 pm
There is no “crisis”. A few drug companies have suddenly realized how academic science works (as DM explained above) and they dont like it and apparently still dont understand it. The only reason they care is because these companies rarely do their own R&D and instead troll the literature looking for targets, and they lost a ton of money on their bad bets. The real “crisis” is the lack of innovation in private industry, but they decided to point the finger at us, so fuck ’em I say.
These studies in eLife demonstrate beautifully that science will always be science and will dominate us forever. You can’t always control your way to an expected result or anticipate every variable. Sometimes things work, sometimes they don’t. That’s biology. You would think that pharma would have learned that by now given the failure rate of their compounds in human trials.
LikeLike