Why AI Is About to Get MUCH Faster | Vasanth Mohan, SambaNova

AI spent the last few years competing on intelligence. Now the next major AI race is about **speed**. As AI shifts from simple chatbots to agents that reason, write code, call tools, search the web, and coordinate other agents, latency compounds. A task that requires dozens or hundreds of sequential model calls can quickly turn seconds into minutes — or even hours. That makes inference speed a fundamentally different problem in the agentic AI era. In this episode of **So What About AI Agents?**, Philippe Trounev sits down with **Vasanth Mohan of SambaNova Systems** to unpack what actually makes AI faster — from model architecture and memory bandwidth to batching, specialized accelerators, and next-generation inference hardware. Vasanth explains the two major stages of inference — **prefill and decode** — and why they create very different hardware bottlenecks. They discuss operator fusion, parallelization across chips, memory bandwidth, and why reducing latency often comes with a significant cost tradeoff. They also explore why coding agents are one of the clearest use cases for premium inference, how AI providers may eventually offer commodity, premium, and ultra-fast tiers of tokens, and why businesses willing to pay for speed may gain a meaningful productivity advantage. Vasanth also shares early performance figures for SambaNova's upcoming SN50 architecture, including benchmark results around **800 tokens per second** on a large model, compared with roughly 300–400 tokens per second on GPUs in the cited comparison. We also get into a slightly crazier question: are AI agents already helping engineers design the next generation of AI hardware? The answer is increasingly yes — although humans are still very much in the loop. **In this episode:** - Why AI agents make inference speed dramatically more important - Why sequential agent workflows create a latency bottleneck - Prefill vs. decode explained - GPUs vs. specialized AI accelerators - The relationship between speed, batching, throughput, and cost - Why memory bandwidth matters for large AI models - What SambaNova's SN40 and SN50 architectures are designed to solve - 800-token-per-second AI inference - Why coding agents benefit so much from faster models - Commodity vs. premium vs. ultra-fast AI inference - Whether faster AI becomes a competitive advantage - How AI agents are already being used in hardware engineering - Why AI infrastructure may become just as important as the models themselves **Chapters** 00:00 — The AI race is shifting from intelligence to speed 00:39 — Why AI agents suddenly need faster inference 02:38 — What actually makes an AI model run faster? 05:30 — When does ultra-fast inference matter? 08:08 — Does model architecture determine inference speed? 10:05 — How do you optimize AI from model to hardware? 12:44 — How fast can AI inference actually get? 15:28 — The sequential latency problem with AI agents 16:39 — Where faster AI creates the most value 18:30 — Will premium AI inference become a competitive advantage? 20:11 — What does AI inference actually cost? 23:10 — How many tokens can AI hardware generate? 24:46 — Building private AI infrastructure 28:02 — How much faster can AI eventually become? 30:14 — Are AI agents already designing AI hardware? 32:50 — Why today's “fast” AI will eventually feel slow 33:34 — What happens next in AI infrastructure