Simon Willison

When artificial analysis benchmarked the providers hosting the model at high effort, 6 scored a median off 93.0. 3%, and Azure scored 80%, a gap of 13.3 points.

In Simon Willison's write-up, Microsoft's Lucas Pickup explained that older VLLM commits didn't respect reasoning underscore effort, so all requests defaulted to medium.

In that case, the gap came from a serving configuration that ignored the requested setting, not from the weights.

None of this is new, only easier to notice.

Is it the bird? It's the bird.

Simon Willison, the developer and writer, in late October 2024, asked 16 models for the same thing.

Generate an SVG of a pelican riding a bicycle.

His reasoning, which I love, was that he likes pelicans, and he was pretty sure there weren't any pelican-on-a-bicycle SVG files floating around yet, so the models couldn't just have memorized the answer.

I don't know where I'm going with this analogy, but we're still here, so it's good to have you back.

It's interesting because I had these, this analogy for my closing talk in the, uh, in San Jose, but I didn't do a closing talk because we actually got Simon Willison instead.

So I might actually be using that, uh, in India at the conference when, uh, uh, pos, uh...

I find the story about the Titanic absolutely fascinating because-

So there is a gap between what the agent can do today and Meta's strongest planned privacy architecture.

The article mentions what security researcher Simon Willison calls the lethal trifecta

Yes, and Muse has all three components.

First, access to private data.

There's also a nice little exchange that went by on X worth 30 seconds.

Simon Willison, who you know if you follow the practitioners, responded to a conversation about whether you'd want a coding model to avoid open source libraries with a very dry line.

Avoiding open source libraries? Why would you want a model to do that? And it's a small thing, but it's a real point for anyone building with these tools.

The whole value of a coding assistant is that it reaches for the well-worn, battle-tested library instead of hand-rolling something janky from scratch.

So one of the things I'm really curious about is how AI and LLMs are changing your work, are changing your workflows, are changing how you do your job, um, if you have any favorite tools or favorite new things that you've tried.

Um, as a guest on this show, I had Simon Willison on a couple of times, and he's kind of world-renowned in the space of AI and kind of on the cutting edge of all the latest models, um, not just from the frontier companies, but also like open, open weight models as well, and things coming out of China.

And it's always a blast for me to kinda hear from him.

But he's so advanced, right? That I'm, I'm always pulling it back like, "Okay, okay, what should the rest of us do?" Um, but I am curious how it's changing your job, and if we can compare notes.

Let me connect that to the business side because there was one little data point that tells you something about the whole model economy.

Simon-Willison flagged a line from Anthropix investor disclosures, distillation, other labs training their own models on Anthropix outputs, accounted for somewhere between 5% and 7% of Anthropix revenue.

Do the math against a reported quarterly revenue around $11.5 billion, and Simon's point lands, that's more than half a billion dollars other labs are spending to distill Anthropix models.

Think about that.