Episoder
-
Lukas Röseler is the Managing Director at the Center for Open Science at the University of Münster. He is the Editor-in-Chief at the journal Replication Research, part of the Replication Journal Federation, is an active member of the Framework for Open and Reproducible Research Training (FORRT) community, and developed a video game called Guess the Replication, where players guess whether studies successfully replicated.
Replication Research:
http://replicationresearch.org/
Replication Journal Federation:
https://forrt.org/rjf/
Framework for Open and Reproducible Research Training (FORRT) community:
https://forrt.org/
Guess The Replication:
https://lukasroeseler.github.io/GuessTheReplication/
Dominance and Prestige: Meta-Analytic Review of Experimentally Induced Body Position Effects on Behavioral, Self-Report, and Physiological Dependent Variables:
https://www.researchgate.net/profile/Robert-Koerner/publication/360577810_Dominance_and_prestige_Meta-analytic_review_of_experimentally_induced_body_position_effects_on_behavioral_self-report_and_physiological_dependent_variables/links/62823b133a23744a728598c2/Dominance-and-Prestige-Meta-Analytic-Review-of-Experimentally-Induced-Body-Position-Effects-on-Behavioral-Self-Report-and-Physiological-Dependent-Variables.pdf
OUTLINE:
0:00 - Intro
2:10 - Leading the Münster Center for Open Science
6:17 - How Lukas became interested in metascience
10:53 - Cultural change in science
13:43 - Being Editor-in-Chief of Replication Research
20:00 - Funding the journal, diamond open access
24:55 - Prestigious journals' money extraction scheme
28:17 - Distributed institutional support for diamond OA journals
31:50 - The Replication Journal Federation
39:24 - Journals for replication, or replications for journals?
44:34 - Guess The Replication
50:57 - Prediction markets for replication
54:32 - Meta-analysis of body positioning effects/power posing
1:02:46 - If science were aligned to truth, how would researchers be evaluated?
1:14:36 - Advice for young people
-
Alexander Gibson is a PhD student at the Queensland University of Technology and the Australian Centre for Health Services Innovation studying the intersection of metascience and clinical machine learning. One of his focus areas is data provenance, the Who, What, Where, When, Why, and How of datasets, and how neglecting this can lead to bad outcomes in medical machine learning not only in research, but also for clinical practice and medical device approval.
CONTACT RANDY:
Feedback: [email protected]
EPISODE LINKS:
Alex's preprint on unreliable diabetes and stroke datasets:
https://www.medrxiv.org/content/10.64898/2026.02.24.26347028v2
OUTLINE:
0:00 - Introduction
3:14 - The beginning of Alex's interest in clinical predictive modeling
5:05 - Alex's interest in metascience
6:42 - Choosing a dissertation topic/metrics hacking in machine learning
9:49 - Preprint on data provenance in medical datasets
12:33 - The diabetes and stroke datasets Alex investigated
16:46 - Major irregularities in the data
23:29 - TRIPOD+AI guidelines for auditing machine learning studies
25:26 - How unreliable studies can impact clinical practice and medical device patents
26:42 - Citation networks
27:37 - AI-generated formulaic medical machine learning studies
31:50 - Strategies for high-quality data provenance
33:53 - Patents citing unreliable studies, and how to integrate data provenance into peer review
35:23 - The biggest problems for clinical predictive modeling studies
37:02 - Resources and tools for improving rigor in machine learning
38:45 - Metrics reporting
40:45 - Choosing decision thresholds in predictive models
42:59 - The importance of clinical context in metrics reporting
45:21 - The unreasonable effectiveness of age and sex as predictors
47:53 - The roles of academia and industry in improving clinical machine learning studies
50:07 - Explanation versus prediction
52:51 - Advice and resources for students
54:27 - Outro
-
Manglende episoder?
-
Charles Piller, an Investigative Correspondent for Science Magazine, is the author of the 2025 book Doctored: Fraud, Arrogance, and Tragedy in the Quest to Cure Alzheimer's. He has received more than 40 honors or awards from American Association for the Advancement of Science/Kavli Foundation, National Institute of Health Care Management, Society of Professional Journalists, National Academies/Eric and Wendy Schmidt, Los Angeles Press Club, American Society of Tropical Medicine and Hygiene, Gerald Loeb Foundation and UCLA, Western Publications Association, First Amendment Coalition, California State University, Computer Press Association, American Society of Business Press Editors, California Newspaper Publishers Association, National Council on Crime and Delinquency, and other organizations. His work has been nominated for the Pulitzer Prize nine times by STAT, the Los Angeles Times and The Sacramento Bee.
CONTACT RANDY:
Feedback: [email protected]
EPISODE LINK:
Blots on a field?
https://www.science.org/content/article/potential-fabrication-research-images-threatens-key-theory-alzheimers-disease
Picture imperfect
https://www.science.org/content/article/research-misconduct-finding-neuroscientist-eliezer-masliah-papers-under-suspicion
Doctored: Fraud, Arrogance, and Tragedy in the Quest to Cure Alzheimer's
https://www.amazon.com/Doctored-Fraud-Arrogance-Tragedy-Alzheimers/dp/1668031248
OUTLINE:
0:00 - Introduction
6:05 - How Charles became a science journalist
7:58 - Charles' 2022 piece, "Blots on a field?"
14:39 - Responses from scientists to this piece
22:22 - Charles' 2024 piece, "Picture imperfect"
30:29 - How Charles began working on his book "Doctored: Fraud, Arrogance, and Tragedy in the Quest to Cure Alzheimer's"
33:15 - Charles' experience working with scientific sleuths
36:45 - The incentive for positive data in science
46:09 - Charles' NYT op-ed about how to combat fraud in Alzheimer's research; attacks from the Trump administration on science
53:22 - Charles' thoughts on registered reports and other ways to improve scientific incentives
58:34 - Upcoming work
59:22 - Advice for listeners
1:02:20 - Outro
-
Tim Errington is the Senior Director of Research at the Center for Open Science. He led the Reproducibility Project: Cancer Biology, Systematizing Confidence in Open Research and Evidence (SCORE), as well as the implementation and evaluation of initiatives such as Registered Reports, Registered Revisions, responsible conduct of research trainings, and open science badges.
CONTACT RANDY:
Feedback:
EPISODE LINKS:
Investigating the replicability of preclinical cancer biology (Reproducibility Project: Cancer Biology led by Tim Errington):
https://elifesciences.org/articles/71601
Bayer replication study:
https://www.nature.com/articles/nrd3439-c1
Amgen replication study:
https://www.nature.com/articles/483531a
Reproducibility in Cancer Biology: Challenges for assessing replicability in preclinical cancer biology (Companion paper to Reproducibility Project: Cancer Biology):
https://elifesciences.org/articles/67995
What is replication?:
https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3000691
Study comparing standard reports and registered reports in psychology:
https://journals.sagepub.com/doi/10.1177/25152459211007467
Blog post on the seemingly magical success of revision experiments:
https://rajlaboratory.blogspot.com/2013/06/the-magical-results-of-reviewer.html
Google's AI co-scientist paper:
https://arxiv.org/abs/2502.18864
Machine-readable documents:
https://journals.sagepub.com/doi/10.1177/2515245920970949
How open science helps researchers succeed:
https://elifesciences.org/articles/16800
ZBW's Expedition to Open Science Land:
https://expedition-open-science.org/
OUTLINE:
0:00 - Introduction
4:47 - Tim's origin story as a cancer biologist
6:38 - Initial interest in metascience
9:24 - Starting the Reproducibility Project: Cancer Biology
12:07 - How were the studies that were replicated chosen?
14:41 - Publishing the Reproducibility Project: Cancer Biology as a registered report
17:26 - Results from the Reproducibility Project: Cancer Biology
20:28 - Tim's experience throughout the years running the Reproducibility Project: Cancer Biology
25:21 - The difficulty of running cancer biology studies
27:54 - Judging whether a replication is successful
31:23 - What has the response to the Reproducibility Project: Cancer Biology been?
37:52 - Why aren't replication rates higher?
40:26 - Challenges of running cancer biology replication studies
45:43 - Caveats of preclinical disease models
49:13 - The incentive for positive data in science
57:05 - Systemic intervention vs. Individual policing
1:01:04 - The value of registered reports
1:07:38 - Registered revisions
1:10:48 - Falsifying theories early at the preclinical stage
1:15:21 - Different institutions (e.g., academic, industry) conducting different studies (e.g., preclinical, clinical)
1:17:34 - New initiatives at the Center for Open Science (Replication project of social/behavioral sciences, automated tools for predicting replication success, LifeCycle journal)
1:23:02 - AI scientists are trained on biased literature; distrust of academic literature in drug discovery
1:28:46 - Peer review
1:32:51 - Narrative in science
1:35:06 - 100-200 years into the future
1:40:29 - Advice for high school/undergraduate listeners
1:42:51 - Metascience manifests in every field
1:44:46 - Philosophy of science
1:47:49 - Outro
-
Brian Nosek is the cofounder and Executive Director of the Center for Open Science. He co-developed the Implicit Association Test, a method that advanced the study of implicit bias. He then co-founded three non-profit organizations: Project Implicit to advance research about implicit bias, the Society for the Improvement of Psychological Science to improve the research culture in his home discipline, and the Center for Open Science (COS) to improve rigor, transparency, integrity, and reproducibility across research disciplines. He led the Reproducibility Project in Psychology, a replication of 100 studies from psychology, as well as multiple other replication projects, along with policy reforms such as open science badges, and the investigation of prediction markets for study replication.
CONTACT RANDY:
Feedback: [email protected]
EPISODE LINKS:
Estimating the reproducibility of psychological science (Reproducibility Project in Psychology led by Brian Nosek):
https://www.science.org/doi/10.1126/science.aac4716
False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant:
https://journals.sagepub.com/doi/10.1177/0956797611417632
What is replication?:
https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3000691
Metascience conference:
https://metascience.info/
International Conference on the Science of Science and Innovation:
https://icssi.org/
International Congress on Peer Review and Scientific Publication:
https://peerreviewcongress.org/
MetaROR:
https://metaror.org/
OUTLINE:
0:00 - Introduction
2:56 - Brian's origin story as a psychologist
6:35 - Initial interest in metascience
10:57 - "Scientific Utopia," starting the Reproducibility Project in Psychology, and major fraud cases
17:57 - How did the Reproducibility Project in Psychology come together?
23:40 - Results of the Reproducibility Project in Psychology
27:03 - Response from the field, institutional changes
30:39 - Advice for high school/undergrad listeners
34:03 - Outro
-
Eugenie Reich is an attorney committed to taking on scientific fraud, understanding the incentives that drive it, and recovering misdirected research funding. She is also a former investigative science journalist committed to correcting the scientific record. Her 2009 book Plastic Fantastic, details a major fraud cause in physics at Bell Labs. Two of the cases she has litigated cases were against Biogen ($900 million settlement), and the Dana-Farber Cancer Institute ($15 million settlement).
CONTACT RANDY:
EPISODE LINKS:
Cargo Cult Science:
https://calteches.library.caltech.edu...
The PubPeer Conundrum:
https://www.tandfonline.com/doi/epdf/...
New Scientist:
https://www.newscientist.com/
Skeptical Inquirer:
https://skepticalinquirer.org/
Special thanks to Dylan Bouscher
OUTLINE:
0:00 - Introduction
4:20 - How Eugenie became interested in science
6:25 - Interest in scientific fraud
7:38 - Deciding to become a science journalist
8:58 - Fraud in physics at Bell Labs (the subject of her 2009 book Plastic Fantastic)
20:15 - Bell Labs coming under private ownership
22:53 - Interviewing scientists for her book
26:50 - The response from the physics community
31:10 - Deciding to become a lawyer
33:28 - The False Claims Act
36:29 - The qui tam provision of the False Claims Act
37:46 - The $900M case against Biogen
44:02 - The day-to-day of working on this case
46:30 - Impact of the case on biotech and pharma
48:40 - The $15M case against Dana-Farber
50:42 - Do universities have an incentive to protect accused researchers?
54:15 - Is the scale of fraud too large to be fixed?
57:56 - Does fraud damage public trust in science?
1:00:44 - Tools/solutions to combat these problems
1:04:32 - Advice to practicing scientists
1:05:55 - Advice and resources for listeners
1:08:28 - Outro
-
James Heathers is the Founder and Director of the Medical Evidence Project, a venture of The Center for Scientific Integrity. He aims to reduce medical harm and improve patient outcomes by identifying and publicizing errors and miscondcut in the medical literature. He uses forensic meta-analytical techniques to detect and deconstruct errors arising from low-quality science and fraudulent work in areas that involve large numbers of patients.
CONTACT RANDY:
EPISODE LINKS:
The original GRIM test paper:
http://www1.psych.purdue.edu/~gfranci...
Machine-readable documents:
https://journals.sagepub.com/doi/10.1...
Chaos in the Brickyard: https://www.science.org/doi/10.1126/s...
Books:
The Man Who Only Loved Numbers:
https://en.wikipedia.org/wiki/The_Man...
The Emperor of All Maladies:
https://en.wikipedia.org/wiki/The_Emp...
OUTLINE:
0:00 - Introduction
5:27 - How James has been the last couple days, and his radio voice
6:40 - Writing the first book on Forensic Metascience
10:47 - James' PhD work on heart rate variability
14:12 - Starting to work on metascience
18:40 - The GRIM test
26:40 - Programmatically scaling statistical checks
33:25 - Centering quantitative results and peripheralizing narrative in scientific papers
39:36 - AI for metascience, unearthing data underneath narrative
48:11 - Pursuing cases of misconduct
52:45 - Founding and Directing the Medical Evidence Project
56:10 - Incentives for positive data, post-publication review, challenging the binary of positive and negative data
1:05:41 - Advice and resources for listeners
1:12:40 - Optimism over pessimism
1:16:58 - Outro
-
Mu Yang is a behavioral neuroscientist at Columbia University, and a scientific sleuth responsible for more than 300 retractions. She led an effort that discovered more than 130 fraudulent papers in the publication record of Eliezer Masliah, former head of the Division of Neuroscience at the National Institute on Aging at the National Institutes of Health. Her sleuthing work has been documented in the book "Doctored" by Charles Piller, Science Magazine, and other outlets, and is unpaid.
CONTACT RANDY:
EPISODE LINKS:
Books:
Doctored by Charles Piller: https://www.simonandschuster.com/books/Doctored/Charles-Piller/9781668031254
Unreliable by Csaba Szabo: https://cup.columbia.edu/book/unreliable/9780231216241/
Scientific integrity blogs:
Dorothy Bishop: https://deevybee.blogspot.com/
Leonid Schneider: https://forbetterscience.com/
Podcasts for critical thinking
Plain English by Derek Thompson: https://www.theringer.com/podcasts/plain-english-with-derek-thompson
The Gray Area by Sean Illing: https://www.vox.com/the-gray-area
The Ezra Klein show (NYT): https://www.nytimes.com/column/ezra-klein-podcast
OUTLINE:
0:00 - Introduction
2:58 - Mu's origin story
4:35 - Moving to Columbia
6:15 - How Mu became a sleuth
8:13 - Reporting her first case
13:09 - Red flags Mu looks for in papers
17:30 - Reductionism in behavioral neuroscience
18:04 - Standardization vs. Generalizability
19:58 - Data sharing standards across fields
21:09 - Difficulties of reporting irregularities in papers, university incentives
23:54 - Allocating time between images, numerical, other kinds of data
24:37 - How she searches through papers
25:45 - Examining the chemistry literature
31:10 - Types of misconduct vary by field, risks of reporting
35:43 - The case of Eliezer Masliah
40:31 - Why demonizing individual scientists isn't productive; the system isn't working
56:59 - Academic incentives for positive data
1:07:31 - Hard to publish null data; "unhealthy codependence" between academia and publishing
1:13:08 - Changing incentives
1:21:42 - Are we even making a dent in the scale of scientific misconduct?
1:27:35 - Mu's toolkit
1:29:38 - Mu does this work because it's fun!
1:34:38 - Protecting students; telling them that null data is ok
1:37:52 - Evaluating researchers
1:43:15 - Is peer review still relevant?
1:51:38 - How much better could science be?
1:55:14 - What will science look like in a century?
1:58:13 - Advice and resources for listeners
2:00:54 - Online presence
2:01:35 - Outro
-
Florian Naudet is a Professor of Therapeutics at Rennes University. As a metascientist and psychiatrist, his research interests lie in developing and evaluating methodological solutions to treatment assessment, primarily but not exclusively for mental health conditions. His work has also made inroads to quantifying and understanding research waste and the prevalence of substandard data-sharing practices.
CONTACT RANDY:
Feedback: [email protected]
EPISODE LINKS:
Efficacy and safety of esketamine for “treatment resistant depression”: registered report for a systematic review with an individual patient data meta-analysis of randomized, double-blind, placebo-controlled trials
https://link.springer.com/article/10....
Vibration of effects from diverse inclusion/exclusion criteria and analytical choices: 9216 different ways to perform an indirect comparison meta-analysis
https://link.springer.com/article/10....
Pharmageddon (book)
https://davidhealy.org/pharmageddon-i...
A manifesto for reproducible science
https://www.nature.com/articles/s4156...
Using reporting guidelines to improve the reproducibility of cooking Christmas tree meringues: the “People tasting trees” cluster-randomised controlled trial
https://pmc.ncbi.nlm.nih.gov/articles...
OUTLINE:
0:00 - Introduction
0:36 - Florian's origin story as a psychiatrist and metascientist
4:46 - How Florian became interested in metascience
11:55 - Systematic review of clinical trials of Esketamine for depression
28:45 - Publishing this study as a registered report
40:49 - Vibration-of-effects analysis of double-blind randomized controlled trials assessing nalmefene and/or naltrexone for treating alcohol use disorders
59:58 - Advice for listeners interested in pursuing research like Florian's
1:03:38 - Outro
-
Chirag Patel is an Associate Professor of Biomedical Informatics at Harvard Medical School, renowned for his expertise in using computational methods to understand human health and disease from high-throughput data streams. He specializes in understanding the role in the intersection of genetics and environmental exposures (the exposome) in human health, as well as various disease areas such as diabetes, cardiovascular disease, and Alzheimer’s disease.
CONTACT RANDY:
Feedback: [email protected]
EPISODE LINKS:
Chirag’s Lab: https://www.chiragjpgroup.org/
TEDx talk on the exposome: • Exposome: decoding human health and diseas...
OUTLINE:
0:00 - Introduction
0:37 - How do you decide what to work on?
1:57 - Where does metascience fit into your work?
3:57 - Vibration-of-effects analysis
6:41 - How does VoE change how we see existing scientific work?
9:22 - The biggest challenges in the replication crisis within biomedical informatics
11:48 - Environment-wide association study of Type 2 diabetes
15:04 - The exposome
17:30 - Policy vs. precision medicine applications of the exposome
20:09 - The patient exposome
22:38 - HBA1C for diabetes as a bridge to the exposome
23:54 - Broader metascientific issues of the exposome
25:01 - The effects of extreme weather events on human health
29:35 - AI for biomedical informatics, the exposome, metascience
31:19 - Advice for listeners interested in pursuing research like Chirag’s
32:53 - Outro
