International Conference on Machine Learning

and it depends, so there's still research to be done.

So we had a paper at an ICML workshop on self-supervised learning where we're showing that we said, okay, now let's QT, that's great.

Let's try to use one million of frames from our logs, right? Just one million like image net size roughly and see if that improves the results.

The answer is yes.

And I'm not going to talk about that now, but this is featured in sparse distributed memory as a continual learner, which was added to the additional readings.

And it'll be in, sorry, this shouldn't say ICML, this should say iClear.

It's just been accepted to iClear for this year.

Okay, so...

Yeah, great question.

We've done a lot of our own research that we've presented at conferences like ICML and Flock, and our research heretofore has really focused on auto-formalization, which is how do you take natural language and formalize it into policy as code or code that is verifiable? And so that's what's been the crux of our work over the past, you know, 15, 16 months since we've started is publishing the research to do this and then building the technology to actually enact it.

So that's how we do it.

One of the challenges that you faced in your presentation today, that you addressed in your presentation today, is how do you know what's going on under the hood of any agent? Because they're not always out there to tell you what they're up to.

Let's start with this paper from...

ICML of last year.

So the title of the paper is Language Agent Research Unifies Reasoning, Acting, and Planning in Language Models.

So the kind of the target, the types of applications that we want to target in multi-step tasks are tasks that we not only need to reason about things, but also need to act through those reasoning steps and kind of search our trajectories or optimize trajectories on how we get to a final correct answer.

So we did another experiment.

We had a paper in ICML this summer that looked at how it is that models think about the world.

And we evaluated a ton of different models from this kind of centered kernel alignment.

So what does this mean? This means we take just random pairs of words, imagine like man and woman, right, you check out and evaluate the angle between those two words that every attention had within a model, you average that, and then you correlate that across all pairs of words between models, right? So if two models kind of like, they both see men and women as the same way, and they see, you know, people in China and the U.S.

So in any case, the models essentially were stereotyping by demographic group instead of any individual human participants in the original study.

And that's because LLMs really are eager to create generalizations from limited data, according to Ryan Liu, a PhD student at Princeton University and the co-author for the study, which was published at ICML in Seoul in July of this year.

So basically, whether it's AI making decisions or humans making decisions, they kind of have to figure out what worked before and they kind of have to figure out how to try something new.

And this dilemma is called the exploration-exploitation dilemma.

Princeton's 1,000-layer networks for self-supervised RL also won, showing that reinforcement learning networks, usually a handful of layers deep, can scale to over 1,000.

And at ICML this July, one outstanding paper, The Flexibility Trap, looked at diffusion language models, the image idea from earlier now applied to text.

It found that letting them write in any order can actually hurt them under reinforcement learning.

So diffusion has crossed over to text, and it's already finding the hard edges.

The third ceiling is debuggability, and this is the one that should worry anyone running this in production.

There's a paper from ICML last year that tried to assign blame when a multi-agent task fails.

The best method identifies the responsible agent fifty-three point five percent of the time.

It identifies the decisive error step fourteen point two percent of the time.