Vision-language model

A lot of people are like, are LLMs the way to do it? Probably not, even with multimodal LLMs.

But I think people are trying to do different kinds of AI models, from world models to what they call VLA, VLM, so more like vision language models.

ways that you can take visual input from the physical world and then process it and generalize it and then, you know, be able to direct the robot's actions and movements.

Well, I saw a video that came across my, actually my LinkedIn feed yesterday, and it showed robots in China making robots and said, robots now in China are making robots at the speed of one robot per every 10 minutes in a certain factory over there.

Open weight models are the most common form, such as the popular models like Meta's Lama, Alibaba's Quen, Google's Gemma, or DeepSeek's models.

These models are governed by licenses, governing documents dictating what is allowed with downstreaming use and are often accompanied by inference code in libraries such as Transformers, VLM, SGLang, etc.

Since about April 2025, Chinese AI companies have been the clear leader in open weight models.

True, quote, open source models are similar to these, as they include weights, licenses, and inference code.

And you know this.

It's a VLM.

They're just, we're not doing it.

But literally all you do is you hand the drawing to an LLM.

accommodates to, okay, now I can see air, right? This is where I think I disagree with LLMs as the current technology that will get us to the end, because I don't think that LLMs can do that.

I think LLMs are very good at emulating it.

When you use a VLM, I actually, I shouldn't say this because I don't actually know when I go to, when I give Astra a task, is it actually, and I give it an image, is it actually just, no, it has to be a VLM.

From my understanding, it looks at the image and it starts writing out what it sees in the image.

It's kind of like this, the same argument, right? You don't necessarily want the hard...

If that is also a foundation model, let's say the critic itself is a foundation model, let's say, I don't know, a 5B or a 7B foundation model that is fundamentally a VLM because that's what the critic needs to be.

I think it is a trade-off that you need to make on both accuracy and steerability because you want to be able to edit that model.

The larger it

So if you want to do inference times or inference time thinking, the hardness should ideally be thin, if you will.

So there is probably some sweet spot.

I don't believe using a, let's say, a 30B model, a VLM, even though it might be more accurate, is a better solution.

Because then we run into the same problem that we run into with eval becomes very hard.

Don't disrupt the workflow of selecting the AI tool so it can be created automatically overall.

For our audience who probably knows LLMs, can you talk about what a VLM is? That's Vision Language Model, is that right?

It's like an LLMs, but based on the images itself.

So as you look into a text, large language models, it's like break down the text.

... p- enterprise models.

It could be, like I said, VLM or Ollama CPP on your prem.

If they support, uh, the OpenAI compatible spec-

Mm-hmm

You need one singular platform to kinda get your arms around, "Oh, I've got these frontier subscriptions.

I've got these small language models over here.

I have this VLM stack over here.

How do I bring it all together in a way where I can reason with the financials, the access control, and the governance in one place?"

Those agents can then understand what we're talking about and through a VLA conditioned behavior cloning model actually affect that change.

So that ends up being the stack, RL, BC, VLM for now.

There's worlds in which any number of those things could converge.

I would, in fact, bet on a lot of convergence in the future between these methods.

It's intimidating.

But there's this reasoning layer at the top, too.

So the agent at the top is a VLM running a particular model.

We've been working with Gemini, for example, so we've used a Gemini embodied reasoning model a lot on a lot of these tests.

Yeah.

I mean, you mentioned about the VRM today still lacking in something.

It, I mean, Am I right to say that? Or rather, could you be a bit more specific? Like, for example...

Are they good with like temporal understanding? Because you mentioned something about like exactly where to maybe, where does the action start? Where does the action end? Is, you know, as your VRM get better, are they able to do this? Like, for example, here you have subtask level semantic annotations.

Do you think the AIs are quite good now saying when does something start, something end, like the temporal understanding? Especially when it's over a pretty long horizon thing.

Yeah.

So I have to admit that basically for this whole pipeline, right? So I was about to say that mainly probably three major things there.

So maybe let me start from the high level that I really see the most important three or four things.

It's one of the limitations.

And we'd like to replace, to automate this step with a VLM later.

But basically, exactly what we did is that for each frame, we decided whether it belonged to the demonstration manifold or not.

And that's the only label that the data gets.

And then it's converted to a 0, 1 reward.

So we believe that it might be replaceable by a VLM, basically.

I hope it answers your question.

And now for the step three of...