Today in Reviewer #3: Balanced vs Random Assignment

In my world, when you are about to conduct a between-groups study you do what you can to ensure that there is nothing about the group assignment that might produce a result because of this assignment, rather than your Group treatment.

Let’s say we are using the Hedgerow Dash model of BunnyHopping. If you test a population of 16 Bunnies for their speed, you are going to find some are faster and some are slower on a relatively consistent basis. So if you happen to put the 8 fastest ones in the Methamphetamine group and the 8 slowest ones in your Vehicle group, you are potentially going to have an apparent effect of Drug Treatment that is really associated with individual differences in Hedgerow Dash performance.

There are two basic ways to deal with this.

The first is random assignment from a relatively homogeneous pool of subjects. For example, you order all the Bunnies from the vendor in one large group and treat them all identically right up until you assign them to Groups. The idea is that you are unlikely to assign, by chance, Bunnies most likely to produce one particular category of outcome (independent of the treatment) into one Group and those destined for the opposite outcome in another Group.

The second is balanced assignment. For this, you are likely taking your homogeneous pool of Bunnies and testing them on a key variable or two. The individual differences that may potentially produce an apparent result where it doesn’t exist can thereby be directly minimized. So perhaps you run a pre-test for assignment purposes. Maybe you use a loud noise as the stimulus instead of Coyote pee, or maybe you’ve found that Bobcat pee can work. Baddaboom, baddabing, you can rank your Bunnies on Hedgerow Dash speed and assign them to groups such that the starting mean is equivalent.

In my world of behavioral pharmacology, the random assignment approach is the baseline. If you don’t at least do this, you had better have a good reason. Doing balanced assignment, I would assert, is generally considered even better. A cleaner and superior design leading to more clearly interpretable outcomes.

I am looking at a reviewer comment on one of our manuscripts with disbelief.

This person appears to think that random assignment would have been “surely” better than the balanced assignment we used. Because, you see, the Reviewer asserts that exposure to Bobcat pee must surely confound the response to Coyote pee. This is despite the fact that this is a repeated measures design in which Bunnies are tested daily for longitudinal changes in Hedgerow Dash performance. With Coyote pee. The Group variable you can think of as the time of day in which they were tested, Bunnies being crepuscular and all. The focus is on this Group variable, not the assay (i.e., longitudinal Dash performance changes). Prior literature has established clearly that there are large individual differences in Dash performance, particularly over time with repeated Coyote pee exposure. The rationale for good balancing of groups is overwhelming. And yet. And yet. This reviewer is certain that random assignment would have been better.

Some days, people. Some days.

Priority

I am working up a serious antipathy to the notion of scientific priority, spurred most recently by the #ASAPbio conference and the associated fervent promotion of pre-print deposit of scientific manuscripts.

In science, the concept of priority refers to the fact that we think of the first person to [think up, discover, demonstrate, support, prove, find, establish] something as somehow special and deserving of credit.

For example, the first paleontologist to show that this odd collection of fossils over here belonged to a species of Megatyrannoteethdeath* not previously known to us gets a lot of street cred for a new discovery.

Watson and Crick, similarly, are famed for working out the double helical structure of DNA** because they provided the scientific community with convincing amounts of rationale and evidence first.

Etc.

Typically the most special thing about the scientists being respected is that they got there first. Someone else could have stumbled across the right bits of fossil. Many someones were hotly trying to determine how DNA was structured and how it worked.

This is the case for much of modern bioscience. There are typically many someones that have at least thought about a given issue, problem or puzzle. Many who have spent more than just a tiny bit of thought on it. Sometimes multiple scientists (or scientific groups, typically) are independently working on a given idea, concept, biological system, puzzle or whathaveyou.

As in much of life, to the victor go the spoils. Meaning the Nobel prize in some cases. Meaning critical grant funding in other cases- funding that not only pays the salary of the scientists with priority but that goes to support their pursuit of other “first” discoveries. Remember in the Jurassic Park movies how the sober paleontology work was so desperately in need of research funds? That. In addition, the priority of a finding might dictate which junior scientists get Professorial rank jobs, the all-important credit for publication in a desired rank of scientific journal and ultimately the incremental accumulation of citations to that paper. Finally, if there ends up being a commercial value angle, the ones who have this priority may profit from that fact.

It’s all very American, right? Get there first, do something someone else has not done and you should profit from that accomplishment. yeeehaw***.

Problem is……****

The pursuit of priority holds back the progress of science in many ways. It keeps people from working on a topic because they figure that some other lab is way ahead of them and will beat them to the punch (science always can use a different take, no two labs come up with the exact same constellation of evidence). It unfairly keeps people from being able to get rewarded for their work (in a multi-year, multi-person, expensive pursuit of the same thing does it make sense that a 2 week difference in when a manuscript is submitted is all-critical to the credit?). It keeps people from collaborating or sharing their ideas lest someone else swoop in and score the credit by publishing first. It can fuel the inability to replicate findings (what if the group with priority was wrong and nobody else bothered to put the effort in because they couldn’t get enough credit?).

These are the things I am pondering as we rush forward with the idea that pre-publication manuscripts should be publicized in a pre-print archive. One of the universally promoted reasons for this need is, in fact, scientific priority. Which has a very, very large downside to it.
__
*I made that Genus up but if anyone wants to use it, feel free

**no, not for being dicks. that came later.

***NSFW

****NSFW

Data stream

Yes, I realize the science these days is super collaborative and needs expensive tools, models and techniques to be cool.

However.

Strategically as a lab, you need to have a bread and butter data stream that you produce in house. Data that you generate, interpret, understand and publish without the input of any other lab groups. Data that is, in and of itself, capable of generating publications that meet at least the lower bound expectations of your department, subfield and whomever else is evaluating you.

This may not be the same thing over the long haul, either. Interests change. But the thing that never changes is that nobody is going to find your publication goals, demands or needs as critical as you do. And in this game, not publishing is simply not an option.

So figure out your data stream and protect it.

Brief data analysis interlude

I have a trainee running a study in which she is examining the effects of methamphetamine on Bunny Hopping using the established open field to hedgerow assay. The primary dependent variable is escape latency from stimulus onset to crossing the plane of the hedge.

She is examining the effects of a locomotor stimulant dose of methamphetamine derived from her pilot dose-response study versus vehicle in groups of Bunnies which have been trained for six weeks in our BunnyConditioning Model and age matched sedentary Bunnies. (The conditioning training consists of various sprint, long run, horizonal hop and vertical leap modules.)

So we have four groups of Bunnies as follows:
1. Conditioned, Vehicle
2. Conditioned, Meth
3. Sedentary, Vehicle
4. Sedentry, Meth

The trainee is actually a collaborating trainee and so these data involve the analytic input of multiple PIs in addition to the trainee’s opinio. We are having a slight disagreement over the proper analysis technique so I thought I would turn to the brilliant DM readers.

Scientific Research 101: Meow-Meow, "plant food", 4-MMC, mephedrone…

sourceI recently introduced a paper on the discriminative stimulus properties of cathinone analog drugs with reference to the recent emergence in the popular media of an analog called 4-methylmethcathinone (4-MMC), mephedrone (2-methylamino-1-p-tolylpropan-1-one), Meow-Meow, MMCAT. The name “plant food” is what 4-MMC is apparently being marketed under in the UK, given that the compound itself is not controlled but it is illegal (I surmise) to sell things as “legal ecstasy” or “legal methamphetamine” or similar. There has been one fatality attributed* to 4-MMC that I can find and a few bits of seized-drug analysis confirming that the stuff is indeed being used.
An early report of a fatality associated with consumption of the drug in Sweden resulted in placement of mephedrone on the controlled list. The followup in the Swedish press shows that the woman was reported to have consumed mephedrone (confirmed post-mortem) and smoked cannabis (no apparent confirmation; alcohol and other narcotics excluded postmortem) and then collapsed. Emergency services were unable to revive her and she died a day and a half later; symptoms of brain swelling, stroke, hyponatremia and hypokalemia were mentioned, as well as a low body temperature of 33 degrees C.
The story has heated up recently in the UK press after the death of two individuals who are, at present, suspected of taking 4-MMC/mephedrone, reportedly in combination with methadone (an opiate) and alcohol. As I mentioned before, a quick scan of PubMed finds little reported on the effects of this compound in animal models or in humans.
So the question is, scientists, what next?
Let’s play virtual science, shall we?