Failure to Replicate Results of Bem Parapsychology Experiments Published by Same Journal
Two years ago the prepublication release of a research paper by psychologist Daryl Bem claiming experimental evidence for precognition created a worldwide media stir and intense controversy within the scientific and skeptical communities.
Bem, of Cornell University, claimed that through nine experiments he had demonstrated the existence of precognition, specifically the existence of “conscious cognitive awareness . . . of a future event that could not otherwise be anticipated through any known inferential process.” Essentially, he had claimed to have produced evidence that psychic abilities not only exist but can transcend time and allow the future to reach backward to change the past.
Informed critics of parapsychology were almost uniformly incredulous. Although Bem is a respected psychologist, they found so many flaws in the research protocols and methods that in their view the conclusions had no validity. One of the most stinging rebukes came in the form of an extended, in-depth critique of all nine experiments by York University psychologist and CSI Executive Council member James Alcock in the Skeptical Inquirer (“Back from the Future: Parapsychology and the Bem Affair,” SI, March/April 2011; see also editorial “Why the Bem Experiments are Not Parapsychology’s Next Big Thing” in the same issue).
Alcock also concluded that the journal that published Bem’s study, the Journal of Personality and Social Psychology (JPSP), had done everyone a disservice by publishing this “badly flawed research article.” Parapsychology and the journal’s own reputation, he wrote, had been damaged, and the article’s publication disserved the public as well, “for it only adds to [public] confusion about the existence of psi.”
Experiments attempting to replicate Bem’s results were quickly conducted at various universities, but none were accepted for publication by JPSP. In fact, it said it would not consider publishing replication failures. This fact raised more controversy and concern.
Now the journal has had an apparent change of heart. It has finally published a set of experiments that attempted (and failed) to replicate Bem’s results. Seven experiments conducted by Jeff Galek of Carnegie Mellon University, Robyn A. LeBoeuf of the University of Florida, Leif D. Nelson of the University of California at Berkeley, and Joseph P. Simmons of the University of Pennsylvania have been published in JPSP’s final issue of 2012 (Vol. 103, No. 6) under the title “Correcting the Past: Failures to Replicate Psi.”
The article is lengthy, but the central conclusion is succinctly stated:
“Across seven experiments (N= 3,289), we replicate the procedure of Experiments 8 and 9 from Bem (2011), which had originally demonstrated retroactive facilitation of recall. We failed to replicate that finding.” They further conducted a meta-analysis of all replication attempts of the Bem experiments “and find that the average effect size (d=0.04) is not different from 0.”
To put it even more directly (from the beginning of their conclusions section): “We conducted seven experiments testing for precognition and found no evidence supporting its existence.”
How can their results be reconciled with Bem’s? “It is unclear how Bem could find significant support for a hypothesis that appears to be untrue,” the authors say. They suggest such possibilities as what psychologists call Type I error—a false rejection of the null hypothesis. They also point to concerns about researcher degrees of freedom, which also raise the likelihood of falsely rejecting the null hypothesis. While many experimental decisions are defensible, “because their application is at the discretion of the researcher examining data after the completion of the experiment, they can make a true effect more difficult to discern. . . . Researcher degrees of freedom do not make a finding false . . . but they do make it much harder to distinguish between truth and falseness in reported data.”
They end by quoting philosopher of science Karl Popper: “An effect is not an effect unless it is replicable, and a science is not a science unless it conducts (and values) attempted replications.” They do compliment Bem for encouraging the independent replication of his experiments.
Alcock, who wrote SI’s 2011 critique, has mixed feelings about the Galek et al. paper:
While I am happy that they carried out this research and that it was published, I find it very odd that they focused on Bem’s studies as though they had been carefully designed and well-executed. They were so poorly designed and so badly executed that the results do not merit the careful consideration that this research gives them. However, the experiments that Galek address, #8 and #9, were the best of a bad lot. Nonetheless, I am glad that they have done this research and that it is in print. It counters Bem’s claims very nicely, without ruffling any feathers I guess.
Ray Hyman, professor of psychology emeritus at the University of Oregon and also a CSI Executive Council member, is another longtime expert on such research. Here is his initial comment:
The number and size of their experiments, the detailed analyses, along with their inclusion of several other studies which fail to replicate Bem, deprive the parapsychologists of their standard excuses for dismissing failed replications. Bem and his supporters have one other excuse they can use. They can, and will, point to ways in which the Galak et al. experiments differ from Bem’s experimental protocol. The authors have anticipated this and have adhered so closely to Bem’s protocol, that whatever differences remain appear trivial.
Publication by JPSP of the Galek et al. paper may or may not bring to end this latest dramatic claim of scientific evidence of the paranormal, but—combined with the previous critiques—it appears to have dealt a serious blow. As Alcock emphasized in his 2011 critique (and as Hyman has often also stressed), over the past eighty-odd years this kind of drama has played out multiple times, and “each time parapsychologists ultimately failed to persuade the scientific world that parapsychological phenomena (psi) really exist.”