The Right Chip for the Right Workload: How Inference Speed Shapes the AI Agent Era

In this episode of the Disambiguation podcast, host Michael Fauscette talks with Vasanth Mohan, Product Marketing Lead at SambaNova Systems, about why inference speed is the bottleneck holding back AI agents, how purpose-built silicon changes the economics of enterprise AI, and what business leaders need to understand about the infrastructure layer that sits beneath every AI application they use. Vasanth's career has tracked the frontier of emerging technology. He studied computer science at Stanford, worked in VR during the Oculus era, moved into edge computing and telco networking, and landed in AI infrastructure as the bottleneck shifted from software to silicon. At SambaNova, he leads technical product marketing and developer relations for the company's reconfigurable dataflow unit (RDU), a chip designed specifically for AI inference. The conversation covers which AI use cases are gaining real enterprise traction (coding is the base layer, coworker tools are actually coding under the hood, and the world is moving through three waves from chat to RAG to agents), what premium inference actually means for business leaders (not just faster tokens but the sweet spot of running large models fast enough for agents to work), why a coding agent that takes 25 hours could run in under one hour with the right infrastructure (and why that changes everything about productivity), how disaggregated inference splits the pipeline so that GPUs handle one phase while purpose-built chips handle another (prefill is compute-bound, decode is memory-bound, and they need different architectures), what the dataflow architecture does differently from GPUs (operator fusion that streams data through operations instead of bouncing back to memory), why speed is now priced as a distinct tier (and when paying double for fast tokens makes sense versus batching overnight), why JP Morgan Chase and sovereign AI deployments across Europe and Asia Pacific are choosing on-premises inference (regulatory requirements, audit concerns about closed providers, and the fact that 80% of data centers are air-cooled), what developers are building on SambaNova Cloud (planning agents on frontier models delegating to fast open source execution agents), and the one metric enterprise leaders should use to evaluate inference infrastructure (tokens per second per megawatt, normalized for the service levels their applications actually need). Timestamps: 00:00 - Introduction 00:32 - Episode title and guest intro 00:49 - Vasanth's background: Stanford, VR, edge computing to AI infrastructure 05:00 - AI agent use cases gaining real traction 05:27 - Coding as the base use case for enterprise AI 06:20 - Coworker tools are actually coding under the hood 07:00 - Three waves: chat era, RAG deployments, agent era 09:18 - Premium inference: a distinct product tier, not just speed 10:18 - Model size versus speed: the competing forces 11:52 - Prompt caching is solved; decode speed is the unsolved bottleneck 12:14 - The 25-hour coding agent: what 20x faster inference changes 13:57 - Two components of agent workloads: CPU execution and inference 15:14 - Disaggregated inference: splitting prefill from decode 18:30 - Dataflow architecture and operator fusion on the RDU 20:09 - Speed priced as a tier: when paying double makes sense 22:41 - Voice agents need sub-500 millisecond response time 25:00 - On-prem inference: JP Morgan Chase and regulatory requirements 26:17 - Open source models are essential for secure environments 27:36 - Sovereign AI and geopolitical data concerns 31:58 - Right-sizing models to the task 34:00 - Developer patterns: planning agents plus fast execution agents 35:44 - Evaluating infrastructure: tokens per second per megawatt 37:06 - Six-month ROI for on-prem NeoCloud customers 38:08 - Recommendation: Semi Analysis and Dylan Patel Guest: Vasanth Mohan, Product Marketing Lead, SambaNova Systems Host: Michael Fauscette, CEO & Chief Analyst, Arion Research Subscribe so you never miss an episode.