Over the last decade and a half, several reports have drawn attention to the growing crisis in science. Many scientific studies, especially in the life sciences, cannot be replicated. For example, a large reproducibility project that sought to replicate 193 experiments from 53 high-profile research reports on the biology of cancer encountered a number of barriers. In the end, he could only repeat 50 experiments from 23 articles.
Understandably, the replication crisis calls into question the basis of knowledge generation. If we cannot replicate the results of experiments, what degree of faith should scientists or the public have in most research?
However, it is possible that the replication crisis either does not exist or is greatly overestimated. Here’s a look at two lines of argument.
Misconception of the basic percentage
When there were waves of COVID cases after vaccination efforts, many people on the Internet suggested that COVID vaccines were ineffective. One of their main reasons was that we see more cases in vaccinated people than in unvaccinated people.
This is a classic example of base rate misconception, which tends to ignore the general prevalence (base rate) of a phenomenon and instead focus on data that relates only to a specific group or situation. If most people are vaccinated, even a smaller proportion of these individuals may be as large or larger than a much larger proportion of the unvaccinated minority.
Subscribe to counter-intuitive, surprising and impactful stories delivered to your inbox every Thursday
Why is this important for reproduction in science? The British philosopher Alexander Byrd claims that the delusion of the basic interest rate explains it.
When studying a phenomenon, it will not be surprising if most hypotheses are wrong. This can be due to a number of reasons, including a tendency to test bold ideas or because the specific field of study is largely unexplored and difficult (such as the biology of cancer). The high previous probability that the hypotheses would be wrong then is in line with high-quality science. Due to the inherent variability in the experiments, some of these erroneous hypotheses will be erroneously proven to be true.
If false hypotheses make up a large proportion of all possible hypotheses, the number of these false proofs can be compared in number with the few hypotheses that are (correctly) proven true. This situation is further biased by the fact that scientists are more likely to report research in which they find something – whether true or not – than when their hypotheses fail. Later experiments that try to reproduce these erroneously proven hypotheses will fail.
Among the scientific fields, psychology and clinical medicine are often reported to have the lowest levels of reproducibility. Unlike physics, our knowledge of biology is still too incomplete for us to understand biological systems from the first principles. Therefore, the hypotheses are more likely to be incorrect, which explains the low reproducibility of the experiments.
Some scientists suggest that the replication crisis is a symptom of systemic challenges. According to them, one of the main reasons why research is unreproducible is the pressure to publish or disappear, which researchers often face. Are an alarming number of scientists resorting to unethical practices or shortcuts to achieve published results that are difficult to reproduce?
In a preprint published on SocArXiv, the researchers said that low reproducibility in a given field could exist even if there were no scientists involved in data falsification or other dubious practices.
However, the authors do not agree with the argument for misleading the base rate. Turning the argument, they cite how the average newly appointed psychology assistant has 16 publications. These publications test many hypotheses, with few giving negative results (due to the fact that the publishing industry and funding agencies stimulate positive results). If most hypotheses are unlikely to be true, generating such a set of positive results would be unattainable for most young scientists.
To prevent researchers from hypotheses once the results are known (HARKing), clinical or psychological studies are pre-registered. This means that their hypotheses and methods are documented before the research is conducted. Comparing pre-registered studies with published studies suggests that many hypotheses are indeed incorrect. However, the difference in the number of false and correct hypotheses (among those tested) is not large enough for the base rate fallacy to be a sufficient explanation for the replication crisis.
The use of low replication rates to claim that a field produces a large number of incorrect finds requires the assumption that the magnitudes of the effect are fixed. In an experiment that tells how two parameters are related, the size of the effect shows how strong the connection is. However, depending on the context of the experiment, the magnitude of the effect can vary considerably.
The authors built a statistical model of publishing and replication, which included variations in the size of the effect. The simulations showed replication levels up to 50%, excluding unethical behavior.
Rethinking the replication crisis
It is often argued that the replication crisis has a negative impact on society’s already declining perception of science. However, the lack of reproducibility is an inherent feature of scientific fields that explore bold ideas.
If a hypothesis is very unlikely to be true, even a positive result means that it is still unlikely to be true. The results, reversed by later experiments, emphasize the self-correcting nature of science.
Concerns about the replication crisis offer a number of potential solutions. However, if the crisis is simply a statistical result of the scale of modern science, these decisions could have unintended consequences. For example, reducing significance, which some scientists say may help, will harm productivity without improving replication rates.
The authors also suggest that “some reforms may impose disproportionate costs on early-stage researchers, especially those whose identities are underrepresented in science.”