Jul 18, 2024 · 36 min · 10 segments
Cohere is one of the frontier large-language-model companies focusing on enterprise-use cases to differentiate it from OpenAI GPT, Google Gemini, Meta Llama, Anthropic Claude and Mistral. Cohere…
Nick FrosstGuestMandeep SinghHostAnd so typically when companies talk about retrieval augmented generation RAG, they talk about a vector database.
That's what I've been hearing, having spoken to so many industry participants that you use a foundational model and then you pair that with a vector database to ingest your own data, to customize the model responses for a chatbot.

I think it's probably worthwhile spending a little time on some of the term retrieval augmented generation and what it means from a broad perspective, because what you've described is accurate, but retrieval augmented generation can be more than that.

So when we talk about retrieval augmented generation, we're talking about addressing a fundamental problem in language models.

So that problem is language models as really just probabilistic models of language have the tendency to say stuff that sounds true but is not.

There's some fundamental reasons for why making a better language model will not solve that problem.

For instance, I'm in Toronto and today it's accurate to describe it as... rather gray and a little cold and kind of rainy.

I hope that tomorrow that sentence is not true, right? So there's no way to have a probabilistic model that's not connected to the world in any ways, that's just giving you whatever is the most likely next word to understand what is true and what is not true, because what is true changes as time goes on.

So when we do retrieval augmented generation, what we're saying is there's no way for the model to know what's true.

So let's instead rely on some external source of truth and get the model to retrieve that and then write an answer conditioned on the truth that it has received from that external source.

So the retrieval is finding the relevant information, and then augmented generation.

So one way to do that is to take all of their documents, embed them, like turn each document into a semantic vector, and then use a vector database to find relevant information.

So we train a suite of embedding models that are really, really excellent at embedding questions in the same space as they embed answers.

And that means you can do really efficient vector database lookups with really high accuracy.

So we've actually spent a lot of time trying to figure out how to make those embedding models really cheap to run and really cheap to host as well.

You could do it with, you know, I can imagine instead of looking something up with a vector database based on a generated query, you could instead, you know, generate like a SQL request or something and run that on a SQL database and then augment your generation with the answer that you got from that SQL database.
And so typically when companies talk about retrieval augmented generation RAG, they talk about a vector database.
That's what I've been hearing, having spoken to so many industry participants that you use a foundational model and then you pair that with a vector database to ingest your own data, to customize the model responses for a chatbot.

I think it's probably worthwhile spending a little time on some of the term retrieval augmented generation and what it means from a broad perspective, because what you've described is accurate, but retrieval augmented generation can be more than that.

So when we talk about retrieval augmented generation, we're talking about addressing a fundamental problem in language models.

So that problem is language models as really just probabilistic models of language have the tendency to say stuff that sounds true but is not.

There's some fundamental reasons for why making a better language model will not solve that problem.

For instance, I'm in Toronto and today it's accurate to describe it as... rather gray and a little cold and kind of rainy.

I hope that tomorrow that sentence is not true, right? So there's no way to have a probabilistic model that's not connected to the world in any ways, that's just giving you whatever is the most likely next word to understand what is true and what is not true, because what is true changes as time goes on.

So when we do retrieval augmented generation, what we're saying is there's no way for the model to know what's true.

So let's instead rely on some external source of truth and get the model to retrieve that and then write an answer conditioned on the truth that it has received from that external source.

So the retrieval is finding the relevant information, and then augmented generation.

So one way to do that is to take all of their documents, embed them, like turn each document into a semantic vector, and then use a vector database to find relevant information.

So we train a suite of embedding models that are really, really excellent at embedding questions in the same space as they embed answers.

And that means you can do really efficient vector database lookups with really high accuracy.

So we've actually spent a lot of time trying to figure out how to make those embedding models really cheap to run and really cheap to host as well.

You could do it with, you know, I can imagine instead of looking something up with a vector database based on a generated query, you could instead, you know, generate like a SQL request or something and run that on a SQL database and then augment your generation with the answer that you got from that SQL database.
The rest of this transcript — segmented and speaker-labeled, so you land on the exact moment something was said
Search every transcript — by keyword, by phrase, or by meaning, across every show Radar indexes
Trends — what is surging across podcasts, measured against its own baseline
Alerts — when a name you follow appears in a newly indexed episode
No account is needed to search Radar.