Tri Dao

People think kernels are done.

There are great people like Tri Dao who write excellent kernels, and they form the bedrock of all of our modern deep learning is built on flash attention.

Modern transformers are built on flash attention.

But if you deviate from the happy path at all, if there's a new model that comes out that has a slightly different way to embed positional information, like the change of the rope system, suddenly the kernel that we had is not suitable for this new model, and we may have to make a patch to this kernel.

Why are humans still doing this? Like, it seems like the sort of thing that AIs would be exceptionally good engineering more efficient kernels.

Maybe that's where we're going, and we're just not quite there yet.

But if we aren't there yet, is that where we're going? If we're not there yet, why humans still doing this? Why, why is Tri Dao so well known? You know, it's a name I know.

I don't wanna speak for Tri, but what he taught me was you shouldn't write kernels by hand anymore necessarily.