Vision-language model

And by editing, I mean in the sense of performing some operations on the image.

So one idea could be to have your image, to have your prompt, be fed into a VLM, so vision language model, which is an MLLM, multi-model large language model, in order for you to receive editing actions that could be maybe, okay, decreased, let's say, brightness by, I don't know, 50%, just increased luminosity by X%, that you could then interact with an editing software like Photoshop in order to get your final image.

So, The nice thing with this approach is, well, first of all, you have some guarantee that you are preserving your initial image.

If, let's say, you're constraining the sets of actions that you're taking to, let's say, an allow list of actions that here can only be harmless, like coloring actions.

But one challenge is for your VLM to know your set of possible actions well enough for the editing action to actually make sense so that's the main challenge and so that's why some people i mean when i say some people like a lot of people are trying to think about how to do this so there's a very short list of papers that try to resolve that but there is one Method that I want to mention here, which is how do we want our outputs, like how can we make sure that our output is something that is aligned with something that makes sense? Well, a possible method is to look at the logs of people who are actually making edits.

So let's assume you go to your favorite editing software, you have an input image, and then you make some edits, and then you have a final image.

Well, the sequence of edits that you made is something that your model can learn from.

You don't say that.

Yeah.

Yeah, so the question is, can you use VLM to do that? So we're going to see that later in this lecture.

At Capital One, we don't just use technology, we architect it.

Our teams have true end-to-end product ownership, shipping distributed systems that handle high concurrency at scale.

So this is another area.

So something else that we had mentioned regarding capabilities that we were interested in was OCR as a capability.

And on the image generation side, you might want to generate images that have walls of text in there and you want to know how well they are rendered so there is a long text bench that could be mentioned where you have like scenarios where displaying text might be seen as interesting from an image generation perspective and the paper uses a VLM, like typically an OCR judge, that will try to render the text in the generated image.

And it will match to the reference text that was used as a generation prompt.

So... person detector that's all it's ever going to do is find a person uh and and if it's an object detection model then it's going to find where it's going to give you the coordinates of where that person is and so you have this for common objects you have this for things like vehicles and people so those are very lightweight right and then if you have something so a vlm which would be closer to um you know like opus or one of the now opus would be a whole lot more

vlm like video language model

or yeah visual language model and so what these are capable of doing is giving a broad description of everything that it sees in that image or video those are more computationally heavy but how you prompt them and you were alluding to this before how you prompt it really directs your results and so so you don't you could say describe everything in the scene or you could say what color clothing is the person wearing you can you can get specific and when you get specific it can do things in less processing time it also is what allows you to take a lightweight model, like a person detector, and then only run the VLM when and where you want it to.

So

that's the

order,

right? Because I was thinking of it more like if the VLM would look at the video, determine that's a person, and then pull that smaller model to identify or do something about the person.

But you're actually saying maybe it's flipped.

And he's saying we got to do better as a field in 2026 and stop treating reproducibility and scientific discipline as second-class citizens.

And three, VLM-based VLA fields are wrong.

This I thought was quite interesting.

So we've seen a few of these VLAs, VLMs announced, right? So VLA stands for vision and language action.

We'll have probably within five to 10 years' time some robots in our house helping us.

And I think It was easy to train LLM, VLM, because we have plenty of data.

Not easy, because we needed to find the right architecture.

But we have the data to do it.