Logan Kilpatrick

Josh runs the longest-lasting and biggest labs team in the world, and he shares a ton of very specific lessons about where the best ideas come from, how to know when to quit an idea, what it looks like when you have product market fit, what skills are becoming more valuable in today's era, why roles aren't actually blurring into one builder role, and so much more, including some spicy stuff.

A huge thank you to Steven Johnson, Sophie Miller, Jeff Huang, and Logan Kilpatrick for suggesting topics and questions for this conversation.

Before we get into it, I've noticed a lot of people don't finish the episodes, which is totally fine, but you end up missing the best stuff.

So to help you, I've created a new Lenny's Most Replayed Moments YouTube channel that is exactly that, the most rewatched and shared segments from our long-form episodes.

[chuckles] So I think that's the other thing is like a common mistake.

Um, I remember I was on a project years ago, and we got to like 30 engineers, and I was so happy, and we had zero product market fit but a very inspiring vision, and you probably know where, you know the outcome of how that ended basically.

Uh, I'm gonna, uh, in our last few minutes, I'm gonna ask you a question Logan, Logan Kilpatrick suggested I ask you.

Yeah, yeah.

You don't have real people using it.

But I saw, I think Logan Kilpatrick is doing developer relations for Google DeepMind.

I think he fired back and he's like, no, no, no, trust us.

We've revamped our internal testing.

Do we really need to remember what happened with Gemini 3 Pro? Seriously?" Ishu Agrawal put some numbers around that, sharing an older series of benchmarks and adding, "By the way, these were the official benchmarks for Gemini 3.1 Pro when it released, practically destroying Opus 4 six across the board.

I'm not saying Gemini 4 Argon will be bad, but it's crazy that no one has the slightest bit of skepticism given Google's track record." To their credit, Logan Kilpatrick, who honestly should get Google's MVP for hanging in there throughout all of these PR challenges this year, actually engaged with this, writing, "We've gotten much better at testing our models at scale across Google now, so assume most new Gemini revs go through thousands of software engineers for weeks before getting released.

Hopefully, this helps close the benchmark to reality gap by a real margin." Now, so far, all we have to go on is the reported benchmarks, along with some reports from grumbling inside the company.

A Bloomberg piece called Google Grapples With Employee Skepticism about new Gemini 4 says, "While Gemini 4 has performed well on benchmarks widely used to gauge model efficiency, it does less well when employees actually put it to work.

Yeah.

Yeah, there was, I don't know if you saw, there, there was an interview with Logan Kilpatrick over at Google who said something like, th- didn't quite come out and say it, but it was basically said, "We focused a lot on, like, image and video and all this other stuff, and it turns out the thing everybody should have focused on was coding." Um.

Yeah.

Yeah.

All right, our next judge.

Logan Kilpatrick.

Logan Kilpatrick, product lead for Google AI Studio and the Gemini API at Google DeepMind.

From day one, the goal of doing this technology has been, like, how do we use AI to accelerate science and then ultimately help humanity? I think we have the best research team in the entire world.

The number of AI products in the ecosystem, like, how many of them are actually personalized to, like, the people who use them? Hopefully folks go and build their first hand.

There's, like, so much subtlety in getting that right, and the model crushed it.

Ladies and gentlemen, please welcome Logan Kilpatrick.

Logan! Logan, thank you so much, and thank you to Gemini for your support on this.

And so I think it's like underscoring the value of like-

Today's conversation is with Logan Kilpatrick.

He's a member of the technical staff at Google DeepMind.

In this conversation, we talk about the AI industry, the model lab wars, what's going on at Google.

He's very well-versed not only in what Google's doing, but how the industry is developing.

And I think you'll find this conversation very valuable.

Here's my conversation with Logan Kilpatrick.

All right, Logan, everyone thinks that Google is behind in the AI race.