Vision-language-action model

They studied the scaling law from 1,000 hours to 100,000 hours.

On thin tasks, 1,000 hours, their language condition VLA beats in-context learning.

So if you only have 1,000 hours of data, their VLA reaches 53% success rate, while in-context learning only reaches 43%.

If you go to 100,000 hours, the results become comparable.

So in-context learning starts worse and then catches up as data scales up because with more data, the model is able to generalize much better.

And now on unseen tasks, 100,000 hours in-context learning reaches 66% success rate.

Language prompting reaches 9%.

So you have a 7X gap between these two, right? So they say their model S1 exponentially outperforms any existing language prompted VLA as you scale up pre-training.

And LeRobot is that infrastructure.

Almost every open vision language action model, or VLA, that I have covered on this show gets pulled from that host.

It now belongs to the company that sells the GPUs those models are trained on.

now jensen huang's public commitments here are unusually specific and i want to give him credit for that the platform stays open to the whole ecosystem openweight models keep getting support it stays multi-cloud and it stays multi-accelerator and the sentence that matters most is that nvidia compute will not be required to build on or deploy through hugging face That is a much harder thing to promise than the usual language about respecting the community.

So one thing is long context.

So the classic VLA, the way it would work is you would give it a camera image and a prompt, like pick up this object, and then it would generate a sequence of actions for the alert robot.

And then after a second or two, it would start the whole loop over.

And so there was no like long running state.

And I think it's a similar kind of thing with robots, probably.

It's my guess that obviously nobody knows.

But my guess is it's going to be a few more years before you start to see kind of small scale, useful deployments of these VLAs.

And then a few more years after that to kind of scale up and apply to new industries and bring the cost out and the reliability up and so forth.

So it, it's how far up the chain do you go? And right now I'm at that, that chain that's in the, the OS level, in the RTOS, in the Ubuntu, in the Canonical, uh, level, et cetera.

I, w-well, actually, it was kind of the VLA kind of thing.

Seems like it's even another step up as well, right? So like-

Yeah

... demo that I'm, I'm bringing to AI Infra in Santa Clara in, in a couple weeks.

And, um, the idea is using vision language action To play Tic-Tac-Toe.

Now, for your listeners who are not familiar with VLAs, most of them, or at least the small sizes that I deal with, they're good at one task.

So you can train it to pick something up and put it somewhere else, right? Or you can train it to pick up, uh, let's say a cube and put it in row one, column one of, of a Tic-Tac-Toe board.

And with that, let's get into the topic.

Now, the concepts for today that matter are, I've talked about these before, but really world models, uh, VLA models, which VLM models as well, but you know, embodied intelligence and sort of how those models work, which I've gone into some, in some depth.

I'm gonna do a lot more this week.

For those of you who get the email, I'm gonna send you some pretty technical breakdown of the embody AI that Tencent is working on.

It's, it's more like a technical breakdown, so I'm not sure.

I, I'm torn on whether that's helpful or not, but yeah, I think understanding how these models work, um, it's, it's super important.

So anyways, that'll be the topics for today, VLA world models.

We'll talk about Unitree, but I'll go through Tyro's Tencent's smart brain, um, as well briefly.

And this is kind of like the big player.

It's a vision language action model.

It's called Gemini Robotics 2.

And this one takes vision and language, so image and text, and then it outputs actions, aka motor controls of the robot.

Google DeepMind shipped Gemini Robotics 2 at the end of July, and it's three models rather than one.

The one in the middle is a Vision Language Action Model, or VLA for short.

That's a policy, and what it does is take what the camera sees, plus an instruction in plain English, and put out motor commands.

It never has to predict what the scene will look like afterwards, and that is the fork in the road away from a world model.

The marketing doesn't lead with the bottom of that range, but they published it anyway, and that's worth something.

The thing I keep coming back to is that the most capable whole-body humanoid controller anybody shipped this summer contains no generative video world model at all.

It's a layered VLA fed by real teleoperated data.

And teleoperated means a person drives the robot by hand through the task while the recording runs, and that recording is what the model learns from.