John Oliver Explains Science to Everyone

Today in Reviewer #3: Balanced vs Random Assignment

In my world, when you are about to conduct a between-groups study you do what you can to ensure that there is nothing about the group assignment that might produce a result because of this assignment, rather than your Group treatment.

Let’s say we are using the Hedgerow Dash model of BunnyHopping. If you test a population of 16 Bunnies for their speed, you are going to find some are faster and some are slower on a relatively consistent basis. So if you happen to put the 8 fastest ones in the Methamphetamine group and the 8 slowest ones in your Vehicle group, you are potentially going to have an apparent effect of Drug Treatment that is really associated with individual differences in Hedgerow Dash performance.

There are two basic ways to deal with this.

The first is random assignment from a relatively homogeneous pool of subjects. For example, you order all the Bunnies from the vendor in one large group and treat them all identically right up until you assign them to Groups. The idea is that you are unlikely to assign, by chance, Bunnies most likely to produce one particular category of outcome (independent of the treatment) into one Group and those destined for the opposite outcome in another Group.

The second is balanced assignment. For this, you are likely taking your homogeneous pool of Bunnies and testing them on a key variable or two. The individual differences that may potentially produce an apparent result where it doesn’t exist can thereby be directly minimized. So perhaps you run a pre-test for assignment purposes. Maybe you use a loud noise as the stimulus instead of Coyote pee, or maybe you’ve found that Bobcat pee can work. Baddaboom, baddabing, you can rank your Bunnies on Hedgerow Dash speed and assign them to groups such that the starting mean is equivalent.

In my world of behavioral pharmacology, the random assignment approach is the baseline. If you don’t at least do this, you had better have a good reason. Doing balanced assignment, I would assert, is generally considered even better. A cleaner and superior design leading to more clearly interpretable outcomes.

I am looking at a reviewer comment on one of our manuscripts with disbelief.

This person appears to think that random assignment would have been “surely” better than the balanced assignment we used. Because, you see, the Reviewer asserts that exposure to Bobcat pee must surely confound the response to Coyote pee. This is despite the fact that this is a repeated measures design in which Bunnies are tested daily for longitudinal changes in Hedgerow Dash performance. With Coyote pee. The Group variable you can think of as the time of day in which they were tested, Bunnies being crepuscular and all. The focus is on this Group variable, not the assay (i.e., longitudinal Dash performance changes). Prior literature has established clearly that there are large individual differences in Dash performance, particularly over time with repeated Coyote pee exposure. The rationale for good balancing of groups is overwhelming. And yet. And yet. This reviewer is certain that random assignment would have been better.

Some days, people. Some days.

PI muses on that grant he can’t manage to push over the funding line

The proper number of Specific Aims for your NIH Grant

The Aims shall be Three, and Three shall be the number of Aims.

Four shalt there not be, nor Two except as they precede the Third Aim.

Five is right out.