On Jan. 10, 2024, the Epidemiology Monday Seminar Series kicked off the new year with a presentation by Anne-Laure Boulesteix, professor of biometry at the Ludwig Maximilian University of Munich. Her research focuses on metascience and evaluating research methods in the fields of bioinformatics, machine learning, and medicine.
The seminar began with an introduction of Boulesteix’s paper called “An over-optimism in bioinformatics.” The paper criticizes the lack of systematic critical study that implicitly allows researchers to optimize their data sets to produce positive data using their research methods. It eventually leads to the over-optimism of published papers in statistical bioinformatics research and, more broadly, the lack of reproducibility in the field.
“Reproducibility [in this context] means that the same steps of analysis performed on the same dataset consistently produce the same data,” Boulesteix explained in her presentation.
This is a pressing issue across all scientific fields, but Boulesteix focused on reproducibility in bioinformatics, a discipline that applies computational tools and methods to interpret biological data. Without reproducibility, the findings’ validity and reliability remain questionable, and no further study can be built upon them.
The majority of published articles presenting new methods claim that they perform better than the existing ones, but this is not always accurate. To illustrate a common logical error, Boulesteix presented the timeline of three hypothetical research papers: A, B, and C. B is published after A, proving that its method performs better. A has the most recent research, and C introduces a new method and proves a higher efficiency to both A and B. However, a question arises here. Does paper C confirm B’s superior performance to A? According to Boulesteix, it does not, since C assumes a neutral perspective as to the relationship between A and B. This is an example of a tempting logical fallacy in drawing conclusions from a comparison study.
Boulesteix then introduced some possible reasons for the lack of reproducibility, data manipulation or misconduct, publication bias, and selective reporting.
Publication bias refers to the likelihood of a study being published based on its research outcomes. New methods that perform worse are regarded as failures, so studies using them are less likely to get published. Since researchers have an interest in getting their work published, this has the potential to lead to practices such as cherry-picking.
“It certainly is tempting for researchers to cherry-pick only the cases where the new method works better, but it is a major issue in methodological research.” Boulestiex said. “[In fact,] almost all methods can be made best-performing [by selective reporting.]”
“[As a solution,] the cross-design validation of methods can be used, where researchers verify the performance of the methods of two papers that use different methods to study the same research question,” Boulesteix suggested. Forming neutral author teams made up of scientists from different backgrounds and fields would help ensure accurate and reliable judgments for comparative studies.
Boulesteix proposed a long-term, holistic solution: Better acceptance and cooperation for neutral comparison studies by journals, doctoral agencies, committees, and doctoral schools. This goal may take a lot of effort and cooperation, but it is vital that powerful institutions such as these put more research efforts into methodological ethics. Shifting both mindsets and resources would benefit everyone, as it would allow researchers to do science in a just environment of scholarship, ensuring the output of accurate research.
Out of all the solutions proposed, some seem more challenging and idealized than others because they require a prolonged, collaborative effort across institutions, and because the resources will never be abundant enough to support every idea. Nevertheless, such attempts will ultimately contribute to promoting a more reliable and trustworthy research environment.
