Paved with Good Intentions

January 10, 2008

Janet of La Mancha is tilting at the windmill of science ethics again (one of her good ones made it into the anthology that is OpenLab 2007). It is a topic in which I have some interest which you will note from her “good one”.

One issue I like to explore is where and why otherwise well-meaning scientists step off the path of ethical conduct onto the freeway of iniquity. I’ve been thinking about a possible first (mis)step, the inevitable shading with methodological description.

In theory, one puts a description of all methods and materials necessary to replicate the experiment in the paper. In practice, much is omitted. For brevity, because certain things are considered too obvious to be included, because a citation to a prior paper will serve….. These are some of the legitimate reasons we fail to live up to full description. Somewhere short of just plain making up data is the Land of Wink-and-a-Nod. Tested three different antibodies before getting one that “worked”? Dropped a couple of plates off the bench and didn’t mention it? Couldn’t figure out what in the heck went wrong with those six experiments so ditched ’em?

The point being that there are an almost infinite number of semi-legitimate reasons that the actual “methods” that produced the paper are missing from the description. How do we tell when we have moved over the line from legitimate / accepted behavior to the frankly misleading?

I have an example.

As most readers are aware, one huge methodological juggernaut of drug abuse research is the self-administration paradigm. In outline, a subject (typically rat, mouse or monkey) is prepared with an intravenous catheter which can be hooked up to a line providing an infusion of physiological saline containing a small dose of a drug. The animal is then placed in experimental sessions in which it is given the opportunity to press a lever (or button, or poke the nose in a photobeam detector) to obtain the drug infusion. If the animal presses more for drug infusion than it does for saline, well the drug is interpreted to be “reinforcing” or “rewarding”. It is inferred that the drug therefore has “abuse liability” and relative rates of responding for various drugs may be interpreted as giving a readout on relative abuse liability. Simple enough.

The devil is, of course, in the details. What is the per-infusion dose of drug? What is the response schedule (1 lever press per infusion?, 5? 10?; an infusion available after each 5 sec, 5 min, 5 hrs?, a progressive increase in the ratio?) under which drug infusions are provided? What is the infusion rate? How many training sessions? Of what duration? What are the individual criteria to assess “acquisition” of self-administration? And what do we do with those minority of individual subjects who are outliers, choosing not to self-administer the drug?

We approach the meat of the discussion. The answer is, we drop them from the study. With various levels of rigor, a priori versus a posteriori criteria and, importantly, varying levels of description in the resulting paper. Is this ethical? Some very big labs practice subject-dropping routinely with long-established rule-of-thumb criteria. Some do not. Some do it (probably) but don’t really mention this in the methods. Is it a common enough practice to drop a subject or two that this is in some senses assumed and therefore ethically irrelevant?

This brings me to MDMA self-administration. Disclaimer at the outset, I have dogs in this hunt. Which will surprise nobody. A “review” (of the not-quite-enough-data-for-a-real-report type) by Richard de la Garza and colleagues from early 2007 highlights the issue. In essence, rats (and mice) will sort-of self administer MDMA. At rates far below those for methamphetamine, amphetamine or cocaine, which confuses the traditional stimulant-self-admin types. Allegedly. I smell a lot of dusty straw, but there it is. One lab, however, has had rather whopping success getting impressive self-administration rates for MDMA in rats. Per session intakes of 20+ mg/kg as opposed to the less than 5 mg/kg that several other labs have been observing (in the papers referenced as well as subsequently presented and/or published work). Now, the de la Garza review tries to be nice about it:

The experimental procedures described by Schenk and colleagues are distinct from those specified in the current report, and these may contribute to the unique profiles of responding … long infusion time, no time out, and sessions that are generally long during acquisition (6 hr).

These procedural differences might not be trivial. For example, it is well-established that cocaine self-administration is dependent on infusion time, with shorter infusions being associated with increased selfadministration behavior in animals and increased subjective effects in humans (as compared to long infusions) (Nelson et al. 2005; Samaha and Robinson 2005). While rapid infusion rates may increase self-administration of cocaine, it is possible that slower delivery of MDMA is integral to its ability to serve as a reinforcer in rodents.

The conference discussion is a little more pointed. To paraphrase, “This is bullshit”. The thought is that the provided methods do not seem likely to many of us to explain their “success” in establishing high levels of MDMA self-administration. Dose, infusion rate, strain of rats…the effects seem larger than would typically be observed with such manipulations (or are being observed in attempts to replicate).

So the Schenk lab has a new paper out in which they start with the following comment:

In our preliminary studies (Schenk et al., 2003; Daniela et al., 2004, 2006), we examined acquisition of self-administration of a dose of MDMA approximately five times the dose used in the Dalley et al. study. We observed that some of the rats failed to acquire MDMA self-administration even with extensive testing…

Oh really? What does “some” mean? Very hard to tell. For example Schenk et al, 2003 reported 11 animals in the initial group, 8 “continued” going by the methods and the figure reports “N=6”. Daniela et al2004 and 2006 discuss acquisition criteria but the issue of subject dropout cannot be assessed. The numbers seem to imply zero failure to acquire. You would think there would be a nice clean statement of “X% of rats failed to acquire self-administration criteria”. I can’t find these.

The latest, Schenk et al, 2007 reports a little more explanatory result:

Fig. 1. Cumulative percentage of rats that acquired 3,4-methylenedioxymethamphetamine
(MDMA) or cocaine self-administration as a function of days of
testing.

schenk07-fig1.jpg

Speaking for myself, this figure makes sense. If you take the most-preferring half of the distribution (which is wide for MDMA compared with cocaine, obviously) than you may very well get very large mean MDMA intakes compared with other studies which include 100% of their subject pool in the data.

To return to the main point, however, were the first few reports skirting unethical behavior? Deceiving us on the real state of the world by omission? Or did they view this as standard acceptable practice? That if one takes a certain interpretation of the way the papers were presented one can in fact deduce the drop-out rate? That the resulting impression was just the usual spin applied by scientists trying to put the best face on their report?

Obviously we cannot answer this question, not being the authors themselves. We can, however, scrutinize our own behavior and writing for similar lapses.

7 Responses to “Paved with Good Intentions”

  1. neurolover Says:

    Well, I can’t see why “some” would ever be acceptable in a manuscript, without quantification. I mean some is even worse than “few”. Who reviewed that paper? I personally, blame reviewers for those things, ’cause i think it’s easy for an author not to see that the word needs a number when they’re in the midst of filling in all the other numbers. 🙂

    But the figure is good, right? it gives you the answer you were looking for. Regarding whether to blame the authors, it’s hard to tell. Maybe they were just inexperienced? The fact that they’ve corrected themselves in the latest reports raises that possibility. It’s the kind of mistake a novice might make. Say, they experience with a technique that almost always works (maybe cocaine reinforcement, or finding orientation selectivity in V1 as examples). Now they switch to something else, and don’t realize at first that their “outliers” aren’t really outliers. So, in the first reports, they drop the animals from the experiment on session 2 if it didn’t show acquisition.. It’s only later that they realize the outliers are in fact a reasonable part of the population.

    In the end, it depends on whether you believe the folks are trying to be honest with themselves, or not.

    The more concise issue — if the methods don’t result in replication, then there’s something important missing in the methods, right? The whole point of science is that you’re not allowed to say “I know how to do it and you don’t, ha ha” (unlike, say when you’re making the commercial Coke).

    Like

  2. physioprof Says:

    “We approach the meat of the discussion. The answer is, we drop them from the study. With various levels of rigor, a priori versus a posteriori criteria and, importantly, varying levels of description in the resulting paper. Is this ethical? Some very big labs practice subject-dropping routinely with long-established rule-of-thumb criteria. Some do not. Some do it (probably) but don’t really mention this in the methods. Is it a common enough practice to drop a subject or two that this is in some senses assumed and therefore ethically irrelevant?”

    I’m not in the field of drug abuse, but this seems very very disturbing on its face. Are you telling me that if nine out of ten rats won’t bar press for a drug, but the one rat that does goes batshit crazy for it, then that drug will be assessed as having “high potential for abuse”?

    Like

  3. drugmonkey Says:

    “But the figure is good, right? it gives you the answer you were looking for.”

    so no harm, no foul, right? except three years between first pub and “fessing up” is a LOT of potential effort wasted trying to replicate, trying all sorts of different variations, getting grants pounded on, etc….potentially, anyway 🙂

    “if the methods don’t result in replication, then there’s something important missing in the methods, right? The whole point of science is that you’re not allowed to say “I know how to do it and you don’t, ha ha””

    Except this is the case which is why we do provide methods, and training, and collegial advice, etc.

    with respect to your speculation regarding reviewers, well, hmm. in this particular forum i will only say that going by my own experiences with my own papers, it is not uncommon for reviewers to raise critiques or questions which the authors basically blow off and get away with because of editorial decision making. issues that are both valid and invalid. this leads me to the recognition that just because something funny is in press this doesn’t mean the reviewers “missed” it…

    Like

  4. physioprof Says:

    “In our preliminary studies (Schenk et al., 2003; Daniela et al., 2004, 2006)”

    WTF? How can they cite published peer-reviewed manuscripts as “preliminary studies”? Isn’t that in itself an admission that those “preliminary studies” should be considered full-of-shit?

    Like

  5. drugmonkey Says:

    “I’m not in the field of drug abuse, but this seems very very disturbing on its face. Are you telling me that if nine out of ten rats won’t bar press for a drug, but the one rat that does goes batshit crazy for it, then that drug will be assessed as having “high potential for abuse”?”

    Not in practice, no. Those studies which are focused on determining abuse liability, for the first time in a novel compound are quite a bit better about this as you’d imagine. A lot more precision about % of subjects. Ditto monkey studies because they only have, what, 3 or 4 subjects anyway and they usually present individual data.

    and in the “usual” run-o-the-mill studies with cocaine, meth, heroin, etc, well, rats tend to like those and dropout is more like 10% if that. (i’m not counting actual methodologically validated problems like the catheter not being patent, illness, etc. just those where you are pretty sure everything tests normal with the subject but it just doesn’t act like the group)

    Once you start getting into alcohol and nicotine (which rats don’t like that much) and apparently MDMA you start getting weirdnesses and some verging-on-dodgy behavior…

    Like

  6. drugmonkey Says:

    “WTF? How can they cite published peer-reviewed manuscripts as “preliminary studies”?”

    Hmm. Well I might read this as just being imprecise when one really means “prior”. Or, an attempt to soft-pedal ’cause you know you are about to present data which question your prior stuff. Or someone in the depths of grant-speak where it is reasonably common to be referring to actually published work as “preliminary studies” because you mean it in the sense of temporal order, not relative completeness…

    Like


  7. […] The only lab to establish MDMA self-administration in rats to any convincing degree. There is some controversy over just what is going on but I think it is safe to say that her publications have impact, regardless of the eventual […]

    Like


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: