Subtitle: "Why You Can Stop a Trial Early for Harm but not Benefit"
This builds on a recent Twitter discussion with Jeremy Faust
, David Marcus
, and CKB
It sounds odd when you first hear about it, but EBM experts say that you should stop a study if it shows sufficient harm early on, but stopping a study early because it showed great results is shady.
Why? Well, it's tough to explain, so here's an analogy I came up with*:
Stopping a trial early for benefit is like winning money gambling at a casino. If you've ever won money, why aren't you there right now, winning more?
Take any of the big games where you play against the House: slot machines, blackjack, or craps. I like craps -- it's fun, when you win everybody wins (except the one guy sitting next to the dealer betting wrong), and I hear that it's the best odds in a casino, other than counting cards at blackjack
Now I know that the odds are stacked against me. The House has an edge, something like 51%. As the cliche goes, casinos are not built for me to make money.
But I know that the 51% House advantage is an average over time -- there are fluctuations around it. I saw a great video comparing it to walking a dog on a leash: the person walks in a straight line (overall trend) but the dog walks a little this way and a little that way (variation) but overall still follows the same path.** I'm hoping to catch a little variation in my favor, and quit playing before the game regresses back to the mean. And so is the drug company.
I know that overall, I am more likely to lose than win (the drug doesn't work
). Now if I win some money in the first hour or two (early benefit
) I know it's probably a fluke and not loaded dice (drug that
). I can take my money and walk away (stop trial
) or I can keep playing. If I stopped with a few extra dollars in my pocket, would I conclude that I can win at craps (blockbuster drug!
)? No -- I know that I just caught a little variation in the overall pattern, but in the long run, craps will cost me money.
What about if I lose money (harm
)? Maybe I can win it back, should I keep playing? The problem is that eventually, if I keep losing money (drug doesn't work
), large men will come after my family (drug is really harmful
), and I don't want that
|Do not want.|
Of course, in a clinical trial, we don't actually know whether or not the therapy works. Clinical trials start from a position of equipoise: we don't think the drug is harmful, but it might benefit patients, and the risk of harm vs risk of benefit is balanced. But if we show harm early, we lose that equipoise, and we have stop the trial before we harm the study subjects too much (before burly men show up at my house
), knowing that we may have given up on a worthwhile drug but that it was just too risky. While it seems
that the two are symmetric, beneficence vs maleficence is not a symmetric equation.
Are there times when we should stop a trial early for a huge, obvious benefit? I think so, but only if the study is adequately powered at that point, which it's very unlikely to be, because studies are designed to be powered at the end of the study, not midway through.
Back to the casino: if I walk into a casino, drop a quarter into a slot machine, and on my first try win $1 million, would I conclude that the machine is a winner? What if I win $10,000 on 3 of my first 5? It would take a combination of a big enough benefit over a big enough sample to demonstrate the power needed to end the study early, and that is rare.
At what point do I decide that the machine might be mis-calibrated (the drug works
), and I should tell my parents to cash out their 401k and spend it all playing on this machine before the casino catches on (FDA approval
*I would guess that others have had the same idea before, like Newton & Leibniz simultaneously discovering the calculus, and the creation of the term "FOAMites"
**I tried and failed to find the video. Sorry.