Why Does AI Keep Failing When It Meets Real Lab Data? with John F. Conway

The reproducibility crisis in industrial science is estimated to waste thirty to fifty billion dollars a year, and John F. Conway says almost none of it is a technology problem. He is Chief Visioneer Officer at 20/15 Visioneers and has spent twenty-five years on scientific data — first building the models, then realising the world had a data problem so severe that the models could not save it. Jonathan Green talks with him about why smaller foundational models beat giant ones in the lab, why half of academic data quality leaks into commercial pharma, why he refuses to start any project without mapping the process first, and why "screen everything" turned out to be the wrong answer after fifteen years of trying. Key Takeaways: • The bottleneck is model-quality data, not model size. In science, smaller foundational models outperform the giants because you are not overfitting on a problem where barely any data exists in the world. Save the large language models for writing the paper about it. • Capture the failures, not just the wins. An experiment that failed is data — it stops the next person repeating it, and analysing why it failed is often where the value sits. • Unusable scientific data is a cultural problem wearing a technology costume. Roughly half of academic papers are considered not up to snuff, and that mindset leaks into industrial life science, where highly educated people will still say "I don't need to do that." Safety in the lab is a condition of employment — guarding data should be too. • Never build software from what someone says their process is. Map it first. Scientists describe the process they think they follow, you build to that, and six months later the project is imploding. Go one level up and the managers often cannot say what their bench scientists actually do. • If the tool's success means someone's job, you have incentivised sabotage. Companies that laid people off on the magic-bullet theory are now admitting they jumped the gun. Automate the mundane and put humans on the hard problems. Notable Quotes: "Safety in the lab is a condition of employment. So should be the guarding of scientific data." — John F. Conway "Put all the forethought and the scientific rigor up front, screen less, and get better outcomes. It took fifteen years for people to figure that out." — John F. Conway Connect with John F. Conway: LinkedIn: https://www.linkedin.com/in/john-f-conway-678b2a3 Company: 20/15 Visioneers — https://20visioneers15.com Conference: Paperless Lab Academy Enjoyed this? Follow The Artificial Intelligence Podcast and send it to whoever is about to feed a model their messiest dataset. **Connect with Jonathan Green** - The Bestseller: ChatGPT Profits - Free Gift: The Master Prompt for ChatGPT - Free Book on Amazon: Fire Your Boss - Podcast Website: https://artificialintelligencepod.com/ - Subscribe, Rate, and Review: https://artificialintelligencepod.com/itunes - Video Episodes: https://www.youtube.com/@ArtificialIntelligencePodcast