Failure to Replicate

March 20, 2013

I should have put that in quotes because it actually appears in the title of this new paper published in Neuropsychopharmacology:

Hart AB, de Wit H, Palmer AA. Candidate gene studies of a promising intermediate phenotype: failure to replicate. Neuropsychopharmacology. 2013 Apr;38(5):802-16. doi: 10.1038/npp.2012.245. Epub 2012 Dec 3. [PubMed]

ResearchBlogging.orgfrom the Abstract alone you can get a sense

We previously conducted a series of 12 candidate gene analyses of acute subjective and physiological responses to amphetamine in 99-162 healthy human volunteers (ADORA2A, SLC6A3, BDNF, SLC6A4, CSNK1E, SLC6A2, DRD2, FAAH, COMT, OPRM1). Here, we report our attempt to replicate these findings in over 200 additional participants ascertained using identical methodology. We were unable to replicate any of our previous findings.

The team, with de Wit’s lab expert on the human phenotyping and drug-response side and Palmer’s lab expert on the genetics, has been after genetic differences that mediate differential response to amphetamine for some time. There’s a human end and a mouse end to the overall program which has been fairly prolific.

In terms of human results, they have previously reported effects as varied as:
-association of an adenosine receptor gene polymorphism with degree of anxiety in response to amphetamine
-association of a dopamine transporter gene promotor polymorphism with feeling the drug effect and diastolic blood pressure
-association of casein-kinase I epsilon gene polymophisms with feeling the drug effect
-association with fatty acid amide hydrolase (FAAH) with Arousal and Fatigue responses to amphetamine
-association of mu 1 opioid receptor gene polymorphisms with Amphetamine scale subjective report in response to amphetamine

There were a dozen in total and for the most part the replication attempt with a new group of subjects failed to confirm the prior observation. The Discussion is almost plaintive at the start:

This study is striking because we were attempting to replicate apparently robust findings related to well-studied candidate genes. We used a relatively large number of new participants for the replication, and their data were collected and analyzed using identical procedures. Thus, our study did not suffer from the heterogeneity in phenotyping procedures implicated in previous failures to replicate other candidate gene studies (Ho et al, 2010; Mathieson et al, 2012). The failure of our associations to replicate suggests that most or all of our original results were false positives.

The authors then go on to discuss a number of obvious issues that may have led to the prior “false positives”.

-variation in the ethnic makeup of various samples- one reanalysis using ancestry as covariate didn’t change their prior results.

-power in Genome-Wide association studies is low because effect sizes / contribution to variance by rare alleles is small. they point out that candidate gene studies continue to report large effect sizes that are probably very unlikely in the broad scheme of things…and therefore comparatively likely to be false positives.

-multiple comparisons. They point out that not even all of their prior papers applied multiple comparisons corrections against the inflation of alpha (the false positive rate, in essence) and certainly they did no such thing for the 12 findings that were reported in a number of independent publications. As they note, the adjusted p value for the “322 primary tests performed in this study” (i.e., the same number included in the several papers which they were trying to replicate) would be 0.00015.

-publication bias. This discussion covers the usual (ignoring all the negative outcomes) but the interesting thing is the confession on something many of us (yes me) do that isn’t really addressed in the formal correction procedures for multiple comparisons.

Similarly, we sometimes considered several alternative methods for calculating phenotypes (eg, peak change score summarization vs area under the curve, which tend to be highly but incompletely correlated). It seems very likely that the candidate gene literature frequently reflects this sort of publication bias, which represents a special case of uncorrected multiple testing.

This is a fascinating read. The authors make no bones about the fact that they’ve found that no less than 12 papers that they have published were the result of false positives. Not wrong…not fraudulent. Let us be clear. We must assume they were published with peer review, analysis techniques and samples sizes that were (and are?) standard for the field.

But they are not true.

The authors offer up solutions of larger sample sizes, better corrections for multiple comparisons and a need for replication. Of these, the last one seems the best and most likely solution. Like it or not, research funding is limited and there will always be a sliding scale. At first we have pilot experiments or even anecdotal observations to put us on the track. We do one study, limited by the available resources. Positive outcomes justify throwing more resources at the question. Interesting findings can stimulate other labs to join the party. Over time, the essential features of the original observation or finding are either confirmed or consigned to the bin of “likely false alarm”.

This is how science progresses. So while we can use experiences like this to define what is a target sample size and scope for a real experiment, I’m not sure that we can ever overcome the problems of publication bias and cherry picking results from amongst multiple analyses of a dataset. At first, anyway. The way to overcome it is for the lab or field to hold a result in mind as tentatively true and then proceed to replicate it in different ways.

__
UPDATE: I originally forgot to put in my standard disclaimer that I’m professionally acquainted with one or more of the authors of this work.

Hart, A., de Wit, H., & Palmer, A. (2012). Candidate Gene Studies of a Promising Intermediate Phenotype: Failure to Replicate Neuropsychopharmacology, 38 (5), 802-816 DOI: 10.1038/npp.2012.245

As I noted on the repost for Percy L. Julian, Ph.D., earlier this week, I’m swamped this month. So for Black History Month I’m offering up reposts. Today’s installment features a scientist who authored a paper I had occasion to blog a few weeks ago and my email box reports has just been elected to the Board of Directors for the academic society College on Problems of Drug Dependence. This post originally appeared on the Sb blog Feb 2, 2009.


CarlHart.jpgAssociate Professor Carl L. Hart, Ph.D. (PubMed; Department Website; ResearchCrossroads Profile) of the Psychology and Psychiatry Departments of Columbia University conducts research on several drugs of abuse with concentrations on cannabis and methamphetamine. In his studies he uses human subjects to determine many critical aspects of the effects of recreational and abused drugs including acute and lasting toxicities as well as dependence. Dr. Hart is also a contributing member of the New York State Psychiatric Institute Division on Substance Abuse.
In his academic research role, Professor Hart works within the highly respected and very well known Substance Use Research Center of Columbia University where he directs both the Methamphetamine Research Laboratory (Meth R01 Abstract) and the Residential Laboratory. The blurb for this latter will give you a good flavor for the workaday of Dr. Hart’s work:

The residential laboratory, designed for continuous observation of human behavior over extended periods of time, provides a controlled environment with the flexibility to establish a range of behaviors, and the ability to monitor simultaneously many individual and social behavior patterns. This laboratory is equipped with a closed circuit television and audio system encompassing each individual chamber for surveillance and measurement purposes, and to provide continuous monitoring for the participant’s protection. We believe that this relatively naturalistic environment can best meet the challenge of modeling the workplace to predict the interaction between drug use and workplace variables. Because our participants live in our laboratory with minimal outside contact, we are able to evaluate multiple aspects of the effects of drugs on workplace productivity in the same individuals.

Read the rest of this entry »

It’s been awhile since I last talked about MDMA, aka 3,4-methylenedioxymethamphetamine, the canonical ingredient in Ecstasy. I phrase it this way because street drug sold as Ecstasy is notoriously promiscuous in terms of psychoactive drug content. Stroll on over to www.ecstasydata.org if you are new to this topic.

Not that there haven’t been more emergencies and deaths, including ones that didn’t involve MDMA but something else, like PMMA. And yes, the MAPS folks are marching on with great dispatch, dosing more and more people with MDMA in the context of trying to prove it an effective adjunct to psychotherapy for PTSD. So, you know, I keep up with my interests as expressed in earlier days on the blog, I just don’t necessarily bore you with it.

There’s a human laboratory paper I’ve been looking at that makes a point semi-related to some of the above issues. It’s from the laboratory of Carl Hart (who I profiled a few years ago as part of D.N. Lee’s Diversity in Science Blog Carnival.)

Kirkpatrick MG, Gunderson EW, Perez AY, Haney M, Foltin RW, Hart CL. A direct comparison of the behavioral and physiological effects of methamphetamine and 3,4-methylenedioxymethamphetamine (MDMA) in humans. Psychopharmacology (Berl). 2012 Jan;219(1):109-22. Epub 2011 Jun 30. [PubMed]

The essence of the design is that it was a human laboratory study with a repeated measures design. They orally dosed the subjects with inactive placebo, 100 mg of MDMA and both 20 and 40 mg of methamphetamine with these treatment conditions separated by 3 days. A series of cognitive, physiological and self-report assessments were conducted- I’m not going to overview the findings here, you can go read the paper for yourself.

The interesting part about this paper for today’s discussion is that the subjects were really bad at identifying the drug that they’d been given. Keep in mind that the subjects had to have prior experience with both methamphetamine and MDMA. I imagine there are few people in the audience that are not aware that at least the mean, reported subjective effects of MDMA and methamphetamine differ considerably. Although it does have a prototypical psychomotor stimulant character to it, MDMA’s subjective properties have people reaching for new terms like “entactogen”. Likening it to a hybrid of a classical hallucinogen and a stimulant. Insisting vociferously that it is different.

This ties into the question of the pharmacological diversity of the recreational “Ecstasy” market, people’s ability to know what they have just taken, etc. Which may influence their decision to take more drug later on, to take more tablets in the original dose, etc. It also plays into the blinding that might otherwise be assumed to be impossible in the clinical trials and their occasional selection of something else like methylphenidate as their control drug.

Kirkpatrick and colleagues report:

On the questionnaire probing what drug the participants thought they had received, 72.7% of participants (i.e., eight out of 11) correctly identified placebo (18.2% reported MDMA and 9.1% reported sedative; confidence rating= 72.7±9.5), 45.5% correctly identified 20 mg methamphetamine (45.5% reported MDMA and 9.1% reported placebo; confidence rating=76.7±13.3), 72.7% correctly identified 40 mg methamphetamine (27.3% reported MDMA; confidence rating=80.1±5.6), and 45.5% correctly identified 100 mg MDMA (27.3% reported methamphetamine and 27.3% reported sedative; confidence rating=87.6±5.2).

Now, just for reference, the 100 mg MDMA and 40 mg methamphetamine conditions resulted in approximately the same effects on heart rate, blood pressure and self-report measures of “good drug effect” and “feeling stimulated”. So no need to go looking there for reasons. This isn’t some sort of meta assessment of physiological responses or a good/bad drug binary decision. These compounds must produce subjective effects that are pretty indistinguishable. They did differ in group terms on several of the outcome measures so this really does focus on the subject’s awareness and not on the actual effects, so to speak.

And do recall this was a controlled laboratory study in which the environment was relatively invariant compared with potential differences in environments in which Ecstasy is consumed in the natural setting. There is every reason to expect that situational variables and expectations would hugely influence the subjective response.

My consideration for the blog topics is this. When someone starts going on confidently about knowing the purity and/or nature of other non-MDMA constituents of street Ecstasy they have consumed, this is unlikely to be a credible assertion. In either direction. I.e., it is as dubious if they claim to have the pure stuff as if they claim it “must” have been contaminated with methamphetamine.

Unfortunately the study did not manipulate MDMA dose so we’re unable to extend our interpretation in another obvious direction which would be whether or not individuals were very good at identifying how much MDMA they had consumed. I’m betting not very good at this either but we’ll have to wait on another study for that evidence.

From the unfortunate Dunlap laboratory at Dartmouth College.

Randy Lambreghts received his B.S. in Chemistry and M.S. in Biochemistry from the University of Ghent, Belgium. His current research involves identifying novel components of the Neurospora circadian clock using classical genetic mapping as well as next-generation sequencing techniques. He enjoys travelling and plans to see all of South America after graduation.

News of our young hero [PubMed] from The Dartmouth:

A Dartmouth graduate student was arrested for the attempted manufacture of methamphetamine/amphetamine on Sunday, the Union Leader reported.

Randy Lambreghts, 28, whose graduate date is not listed in the Dartmouth Network Directory, allegedly ran a methamphetamine lab from his 3 School St. apartment, according to the Union Leader.

and from here:

After the DEA got a search warrant, a field team searched the pad, collected evidence suspected of being related to making meth, and sent it to a lab; it is still being analyzed, Giaccone said.

Lambreghts was charged Tuesday with one count of attempted manufacture of methamphetamine/amphetamine, Giaccone said. He was being held on $20,000 bail.

Lambreghts lives with two other people, but they are not expected to be charged, Giaccone added.

Lambreghts, who had been studying at the Ivy League school for about seven years, received a bachelor’s degree in chemistry and a master’s in biochemistry from the University of Ghent, in Belgium, according to a Dartmouth website.

Ugh.

UPDATE: The title has been modified to remove the inaccurate impression that Lambreghts was enrolled in a Chemistry graduate program.

The New York Times had a piece up Sunday that was entitled “An Alarming New Stimulant, Legal in Many States“. I was alerted to this by David Kroll who reposted some prior comments at his Take as Directed blog. I’ve been getting some traffic from a BoingBoing linker from Maggie Koerth-Baker to an older post from me so I thought I’d better address a couple of points that jump out at me.
First and foremost, the reader should be extremely cautious whenever there is conflation of two different drugs under one purported street name. Even if they are structurally quite similar and some human reports have overlapping properties. In the case of “bath salts”, there is quite a bit of confusion over whether a news account is referring to 4-methylmethcathinone (4-MMC), methylenedioxypyrovalerone (MDPV), sees no difference between them* or doesn’t know if there is any difference.

Read the rest of this entry »

BlackSheepDM_234.jpg
source
A report in Popular Science (authored by Jeremy Hsu) points to a recent paper published in Academic Emergency Medicine. In this, Dawes and colleagues report on an investigation on the effects of TASER on sheep intoxicated with methamphetamine (MA). I was alerted to this by Damn Good Technician who wanted a little bit of context for what would seem to be a WTF? kind of study.
ResearchBlogging.orgThe study was conducted in Dorset sheep who were anesthetized, and administered 0, 0.5, 1.0 or 1.5 mg/kg of methamphetamine HCl (curiously from dissolved Desoxyn, the approved pharmaceutical product) in an IV infusion. The drug treatment was a between subjects factor (N=4 per group) and animals were monitored for “continuous blood pressure, heart rhythm (one-lead), pulse oximetry, and capnography… Arterial blood sampling was performed at baseline, 30 minutes after the administration of the methamphetamine, and after each exposure from a TASER X26”.
To answer the question of why?, and for appropriate background on the science try a PubMed search for “cardiac TASER“. I note a study in which 5 sec of TASER didn’t cause cardiac damage or symptoms in law enforcement trainees and another showing minimal cardiac effects on law enforcement volunteers after vigorous exercise. Also of interest are the case studies of atrial fibrillation in a previously healthy adolescent and recovery of a teen in TASER induced asystole. These, a mini-review by the Dawes group and other searched papers should give you some context and support from the feeling you might have from half-remembered MSM reports over the years that TASER is suspected of being somewhat less than “safe”.
What I’m not finding right away is very much about the drug intoxicated suspect who might be TASER’d by law enforcement. Remember this guy? My best estimate was that he was acutely intoxicated with 3,4-methylenedioxymethamphetamine (MDMA, “Ecstasy”) although that might be one of my blog interests talking. You might also wish to consider some papers found by searching PubMed for “methamphetamine cardiac toxicity“, “methamphetamine vetricular fibrillation” and “methamphetamine heart attack“.
Together this background would seem to identify a situation crying out for additional study.

Read the rest of this entry »

BlackSheepDM_234.jpg
source
A report in Popular Science (authored by Jeremy Hsu) points to a recent paper published in Academic Emergency Medicine. In this, Dawes and colleagues report on an investigation on the effects of TASER on sheep intoxicated with methamphetamine (MA). I was alerted to this by Damn Good Technician who wanted a little bit of context for what would seem to be a WTF? kind of study.
ResearchBlogging.orgThe study was conducted in Dorset sheep who were anesthetized, and administered 0, 0.5, 1.0 or 1.5 mg/kg of methamphetamine HCl (curiously from dissolved Desoxyn, the approved pharmaceutical product) in an IV infusion. The drug treatment was a between subjects factor (N=4 per group) and animals were monitored for “continuous blood pressure, heart rhythm (one-lead), pulse oximetry, and capnography… Arterial blood sampling was performed at baseline, 30 minutes after the administration of the methamphetamine, and after each exposure from a TASER X26”.
To answer the question of why?, and for appropriate background on the science try a PubMed search for “cardiac TASER“. I note a study in which 5 sec of TASER didn’t cause cardiac damage or symptoms in law enforcement trainees and another showing minimal cardiac effects on law enforcement volunteers after vigorous exercise. Also of interest are the case studies of atrial fibrillation in a previously healthy adolescent and recovery of a teen in TASER induced asystole. These, a mini-review by the Dawes group and other searched papers should give you some context and support from the feeling you might have from half-remembered MSM reports over the years that TASER is suspected of being somewhat less than “safe”.
What I’m not finding right away is very much about the drug intoxicated suspect who might be TASER’d by law enforcement. Remember this guy? My best estimate was that he was acutely intoxicated with 3,4-methylenedioxymethamphetamine (MDMA, “Ecstasy”) although that might be one of my blog interests talking. You might also wish to consider some papers found by searching PubMed for “methamphetamine cardiac toxicity“, “methamphetamine vetricular fibrillation” and “methamphetamine heart attack“.
Together this background would seem to identify a situation crying out for additional study.

Read the rest of this entry »