Principal component analysis

What is the computer actually doing to the text?

Think of principal components analysis, or PCA, as a way to find hidden variables that move together.

Imagine you have a giant spreadsheet with hundreds of columns describing a single song.

Like the time of day or the instruments.

Exactly.

Time of day, number of singers, age of singers, audience size, instruments used, duration, on and on.

PCA looks at all that messy data and says, wait, whenever audience size is huge, the songs also tend to have complex instruments and older singers.

Oh, so it groups those traits.

so i mean i can talk about the things that are released um So we saw in the STRABLE benchmark that it can make quite a difference in the sense that the first thing we tried is just taking like embedding.

So for each text cell, we run it through an LM and then the LM gives maybe like 4000 dimensional embeddings and we just run a PCA and project them to 128 or something like that and then run the model on that or I think actually we projected it to 30 dimensions and we saw that sometimes this failed spectacularly especially with the tabular foundation models and then if you standardize each Like each dimension of the embedding before you run the PCA, it gets quite a bit better.

And then there are these LLMs that have Matryoshka embeddings.

So they're trained so you can take like the first 32 dimensions or the first 64. and they're valid embeddings in and of themselves.

So you can just cut them out.

And we saw that even for LLMs that are not trained for this, it can be better to just use a fixed number of dimensions instead of running the PCA.

And then there was another paper that also ran standardization across the embedding dimension instead of across the number of samples dimension and I think there's a bunch more room for how to embed values I saw a paper that said like you take a post-trained LLM like post-trained for tabular data and then you use 128 rows of the data as context you cache them and then you use your new row and then you embed your new row with the like you take the embeddings of the new row and then you pull them so I think there's still There are a bunch of ways to do this more cleverly.

And do you think at the end we will see a unified embedding?

That is the third baseman for Courtney's favorite team, the Chicago Cubs, Alex Bregman.

I was going to say PCA because the Cubs have been, they were unreal in the month of August.

Also, Go Cubs Go is playing in the background of the clip.

What was playing? I couldn't hear that.

This was a

legitimate win.

This will be like when PCA wins the MVP.

Oh.

Yep.

Is PCA going to win MVP? Yes.

Yes.

Yes.

Okay.

Did you ever put that bet in, by the way? We were doing that like three weeks ago, looking at PCA versus Shohei after PCA had like a monster game and you were looking at odds for him to win MVP.

Because I think he was still like plus 350 or something at that point.

Yeah.

That was a lot of shit.

PCA is unbelievable.

Yeah, yeah, PCA should win.

I mean, he is...

He's approaching 40-40, which is nuts because he would be the third guy in the last couple of years to hit 40-40, and it seemed like there was a...

Because, I mean, Acuna had 40-70.

Otani had 50-50.

And now PCA is on track to get to 40-40 if everything works out.