More Options

Illusionists at Work

Commentary

Edzard Ernst

Volume 34.5, September/October 2010

How to 'Prove' That Bogus Treatments Are Effective

It is not difficult to set up experiments that seemingly “prove” that bogus treatments work. Health journalists, in particular, are regularly taken in by such bogus studies, and the misleading results are subsequently reported in the press, perpetuating the public’s belief in these treatments.

I will give several examples from the realm of “alternative” medicine. They are, of course, entirely fictitious. Not that there is a shortage of real ones, but these days one has to be careful not to end up in the hands of libel lawyers (see “Keep Libel Out of Science,” SI, May/June 2010).

Bogus Experiment No. 1

Most clinical trials test whether one treatment is better than another. These studies are called “superiority trials.” Other studies are aimed at testing whether one therapy is as effective as another. They are called “equivalence trials.” My first example is an equivalence trial comparing a highly dilute homeopathic remedy with an accepted and well-known drug, say paracetamol (known in the U.S. as acetaminophen). Take two hundred patients with a sprained ankle and randomize them to take homeopathic arnica (the experimental treatment) or paracetamol (the control treatment). One or two days later, measure the swelling of the injured ankle as an undeniably objective outcome measure. The results will show that the swelling diminished in both groups and that no difference between the two groups emerged. The conclusion, therefore, is that both are equally effective; however, homeopathy (not having any actual active ingredient) caused fewer adverse events. The headline in the papers might read: “Homeopathy Better than Paracetamol.”

The trick here is to select an outcome measure that is not affected by the “accepted and well-known” drug. Paracetamol does not reduce swelling, and few people would claim otherwise. Thus, it acts as a placebo. Comparing two different placebos should always result in equivalence. Yet the illusion can be quite convincing.

Bogus Experiment No. 2

My second fictitious study is also an equivalence trial. It compares homeopathic care against conventional medicine for a serious chronic condition, say Crohn’s disease. Twenty patients are randomized to be treated with either approach. The results demonstrate that the carefully chosen endpoint (e.g., a symptom score) reveals no differences between the groups. The conclusion: homeopathy is as effective as standard treatment of Crohn’s disease. The headline this time? “Homeopathy Scientifically Proven to Work for Life-Threatening Diseases.”

The trick here is to underpower the study dramatically. Underpowered equivalence trials will tend to (falsely) suggest equivalence between the two tested approaches—a safe bet for illusionists.

Bogus Experiment No. 3

Another approach is to conduct a “pragmatic” trial. Such studies are currently very popular because, according to their proponents, they best reflect the “real life” situation of clinical practice. In this trial, chronically ill patients are randomized to receive either standard care (the control group) or standard care plus homeopathy (the experimental group). The primary measure of outcome for that study could be patient satisfaction, well-being, quality of life, or some other subjective endpoint. Due to the regular, lengthy, empathetic encounters received by the latter group, patients are bound to feel better and improve. Illusionists will then interpret this benefit as being caused by the specific effects of the homeopathic remedies. The headline: “Homeopathy Proven to Help Chronically Ill Patients.”

The trick, in this case, is that A (standard care) plus B (homeopathy) is always more than A alone (A<A+B)—unless, of course, B is zero. But an empathetic encounter does, of course, have an impact on many subjective outcome measures. If, in clinical trials, we do not control for nonspecific effects, it is always easy to make a treatment look effective, even in a randomized trial.

Bogus Experiment No. 4

My last illusionists’ stunt is an animal study. Such experiments, it is often (falsely) claimed, are not affected by placebo effects. Ten experimental rats receive a diet to which either a homeopathic product or a placebo is added. The aim here is not to test for therapeutic effects but to find out whether homeopathy can cause a biological effect in principle. All conceivable types of bias and confounding are excluded. The study can be designed to be completely watertight. The rats receive the treatments and are observed for several weeks. At the end of this period, all rats in the homeopathy group have died, but all of the control animals are alive. The conclusion: homeopathy generates biological effects and is thus different from placebo. The headline: “Animal Experiments Prove the Principles of Homeopathy.”

The trick is simple: we need only to select the right “remedy” (and “hide” this in the small print of the experiment). For my fictitious experiment, I chose a “mothertincture” of arsenic. This is pure, undiluted, and very toxic arsenic, yet it is strictly speaking a homeopathic preparation.

The conclusion? Bogus experiments are not difficult to set up, and it is not difficult to fool uncritical people with their results. But they are still only tricks of illusionists who aim to mislead us. It follows that, if we fail to apply our skills of critical assessment or, worse still, we never had such skills, illusionists pretending to be scientists can be a menace.

Edzard Ernst

Edzard Ernst, MD, PhD, is emeritus professor in the Complementary Medicine unit, Peninsula Medical School, Exeter, U.K.