Oct 16, 2026 · 36 min · 10 segments
Eshaan Jain gave me the example that made agentic AI stop sounding clever and start sounding expensive. An agent negotiating a discount on your behalf…
Can I ask, what are some of the scaling lessons you learned at Amazon? How are you applying them with what you're doing today?
I think the key lessons w- for the scale that I learned at Amazon is that never underestimate the data.
Because what I have seen, that the, the sh- demos that look very shiny fail into production, and that's not because they have used a bad model or they have used a bad architecture.
When they have built a demo, they have picked and choose, uh, cherry-picked the, the best data, right? The cleanest data.
But then when you deploy that into production, the, the production data sometimes is not clean, and then you, you get to see these AI agents tearing apart, like, totally failing into production.
So that is the biggest lesson that I learned at Amazon, is that always, always, first of all, make sure that your data is clean, your data supports your agents, otherwise your agents will not have the right context.
So we built the machine learning, um, extraction for, for our contractual process.
Any downstream, uh, automation could not be built until we did that prerequisite of pulling the right clauses, pulling what has been approved, what has been signed between Amazon and the customer and the vendor, right? So that, that is first and foremost.
I think the second key lesson that I learned was ensuring you have some accuracy metrics.
So for the model that I co-built, we had 95% accuracy, and that we ran over so many different contracts.
So at any given point of time, being a product manager, you should be able to determine what is the success rate and what is the failure rate.
Until you know specific numbers that are tied to your errors and your successes, you cannot confirm that you are ready to hand over your business processes to an automated and fully autonomous AI agent.
So in order to prepare your organization for being fully agentic, I think those are the two key steps you need to identify.
Can I ask, what are some of the scaling lessons you learned at Amazon? How are you applying them with what you're doing today?
I think the key lessons w- for the scale that I learned at Amazon is that never underestimate the data.
Because what I have seen, that the, the sh- demos that look very shiny fail into production, and that's not because they have used a bad model or they have used a bad architecture.
When they have built a demo, they have picked and choose, uh, cherry-picked the, the best data, right? The cleanest data.
But then when you deploy that into production, the, the production data sometimes is not clean, and then you, you get to see these AI agents tearing apart, like, totally failing into production.
So that is the biggest lesson that I learned at Amazon, is that always, always, first of all, make sure that your data is clean, your data supports your agents, otherwise your agents will not have the right context.
So we built the machine learning, um, extraction for, for our contractual process.
Any downstream, uh, automation could not be built until we did that prerequisite of pulling the right clauses, pulling what has been approved, what has been signed between Amazon and the customer and the vendor, right? So that, that is first and foremost.
I think the second key lesson that I learned was ensuring you have some accuracy metrics.
So for the model that I co-built, we had 95% accuracy, and that we ran over so many different contracts.
So at any given point of time, being a product manager, you should be able to determine what is the success rate and what is the failure rate.
Until you know specific numbers that are tied to your errors and your successes, you cannot confirm that you are ready to hand over your business processes to an automated and fully autonomous AI agent.
So in order to prepare your organization for being fully agentic, I think those are the two key steps you need to identify.
The rest of this transcript — segmented and speaker-labeled, so you land on the exact moment something was said
Search every transcript — by keyword, by phrase, or by meaning, across every show Radar indexes
Trends — what is surging across podcasts, measured against its own baseline
Alerts — when a name you follow appears in a newly indexed episode
No account is needed to search Radar.