Sam Newman on Software Factories: Why AI Can't Fix Your Production Incidents (Yet)

Sam Newman, author of Building Microservices and the new Building Resilient Distributed Systems, joins Benedikt Stemmildt to talk about resilience engineering now that AI writes more of our software and runs inside more of our products. Resilience isn't about making sure nothing goes wrong. It's about making sure as many things go right as possible. From Google's Velcro-strapped servers to a network outage caused by rabbits eating cables, Sam explains why systems, and the people who run them, have to be built for surprise. Then he applies the same lens to AI: model providers that aren't yet cloud-grade, why automated remediation with AI is still a long way off, and why your software factory is a production system in its own right. In this episode: - Why resilience focuses on what goes right, not only on preventing failure - David Woods' four concepts: robustness, rebound, graceful extensibility and sustained adaptability - Why LLM providers aren't cloud-grade yet, and how to hedge across models and vendors - Why AI can help during an incident, but can't fix it for you (yet) - Why you can't one-shot a large-scale system, and the modular on-ramp to software factories - Team Topologies vs. top-down AI mandates - Ship fast enough to care: burnout, cognitive debt and Peter Naur's theory building Chapters: 00:00 Intro 01:17 Sam's New Book on Resilient Distributed Systems 06:10 Making Sure Things Go Right 08:33 Why Failure Is a Numbers Game 11:31 Robustness and Rebound 13:20 LLM Providers Aren't Cloud-Grade Yet 16:04 Multi-Model, Multi-Vendor 19:45 Why AI Auto-Remediation Is a Long Way Off 24:10 Agent Retries vs. Production Retries 28:24 Building With AI vs. Building AI Into Products 30:04 Rabbits, Burning SSDs and the Unexpected 32:46 Sustained Adaptability and Psychological Safety 34:10 Is a Software Factory Resilient? 40:10 Dark Factories and the Modular On-Ramp 44:27 Teams, Not Top-Down AI Mandates 49:56 Ship Fast Enough to Care 52:15 Cognitive Debt and Theory Building 55:21 Don't Go All In 57:47 Wrap-Up Mentioned in this episode: - Building Resilient Distributed Systems: https://www.oreilly.com/library/view/building-resilient-distributed/9781098163532/ - Four Concepts for Resilience, David Woods: https://www.sciencedirect.com/science/article/abs/pii/S0951832015000848 - Cognitive debt, Margaret-Anne Storey: https://margaretstorey.com/blog/2026/02/09/cognitive-debt/ Guest: Sam Newman · https://samnewman.io Host: Benedikt Stemmildt, founder of hackers&wizards Rethink Engineering is powered by re:cinq and hackers&wizards. All episodes: https://re-cinq.com/podcast