Neel Nanda

Alex Gibson, a commenter on Lesswrong, compared the post to the optimism around sparse autoencoders in 2023 and said the X post left out the speculative label.

The interpretability researcher Neel Nanda wrote, "This is interesting work, but I don't think it's a significant update to the big picture of interpretability." He agrees that neurons can help with practical debugging.

But he says superposition is still real because language models store far more facts than they have neurons.

On Hacker News, some engineers asked whether AI agents doing the research can be trusted not to mislead the researchers.

Subpoena PLZ'd.

Neel Nanda, Google DeepMind.

I can't really see a world where OpenAI's actions are justified here.

It was Tomek's job to communicate with Meta, who were being given significant, unprecedented access to do the Hugging Face report.

Subpoena please

Neil Nanda, Google DeepMind: I can't really see a world where OpenAI's actions are justified here.

It was Tomek's job to communicate with Meter, who were being given significant, unprecedented access to do the Hugging Face report.

The norms were not set.

There actually can be these phase transitions.

And I think it was Neil Nanda who had this quote of like, phase transitions are everywhere if you just looked closely enough.

If you broke your data set down into just individual tasks and you looked at individual tasks, there would be a lot of stepwise changes where the model really seemed to just suddenly learn something.

And I think in-context learning is like that too, where if you look at aggregate, if you look across a large data set, things might look smooth, maybe deceptively so, but actually individual cases, like an individual story, have these really nonlinear dynamics.

Let's look at what arguably the two most famous mech interp researchers say.

One is Neel Nanda.

The other day he said, "Speaking as an interpretability expert, please do not rely on us to save you on the current trajectory." Another one, arguably the founder of the field, is Chris Olah.

He has been consulting widely with religious scholars to help instill morality somehow into AI models.

Layer-wise relevance propagation, LRP.

We follow Blank, Bhatia, and Nanda (2026) in computing each Jacobian with an LRP backward pass, the partial differential sub LRP in equation 2A, which adds stop gradients to limit the errors that accumulate in the backward pass.

In particular, we apply the LN rule to the residual stream normalization functions and the identity rule and half rule to the gated MLPs and use the plain gradient elsewhere, including through attention, Table five.

We also freeze the expert routing weights in MoE models and the manifold-constrained hyperconnections, MHC, residual mixing coefficients in models that use MHC, such as DeepSeek-V4-Flash.

We refer readers to Blank, Bhatia, and Nanda (2026) for a more complete exposition of using LRP in workspace lenses.

Subheading.

Readout filtering.

The workspace offers a surface for reading the representations that language models use for flexible reasoning.

So I was like, okay, well, worst case, even if I'm not really like sufficiently well qualified or whatever for this, it will help me like nonetheless.

And so I was, there's some, I think like Neil Nanda had like a Google doc that 80,000 hours had circulated at some point being like, these are projects that you might try.

And I was like, okay, let me try doing this.

And obviously the AI tools can help.

The case for EA first.

Neil Nanda argued the movement was remarkably prescient.

Back in twenty nineteen, it formally ranked pandemics and AI capabilities as its biggest issues, while the mainstream looked the other way.

Geoffrey Miller added moral consistency.

But current and former employees, including some of the researchers building these new AI models, worry society is not ready for the potential harms.

In one video, Neel Nanda, a research scientist at DeepMind, said he believed that there was at le- least a 10% chance that AI could lead to human extinction, which he described as ridiculously high.

"We should be taking very careful steps in AI development, but instead what's happening is that frontier labs are racing each other kind of blindfolded." Juan Felipe Serron, uh, Uribe, an AI alignment research engineer at OpenAI, said in a separate video, "It's anybody's guess if we're going to end up either curing cancer or losing every job or maybe all dead." And that's kind of how I go into every day.

Uh, Anthropic plans to caution potential investors in its initial public offering that advanced AI could pose catastrophic or existential risks to humanity, Reuters reported on Monday.