Is AI interpretability close to a breakthrough? A new essay has researchers divided

Neural networks aren't nearly as tangled as researchers assumed, according to Nick Cammarata, one of the people behind the superposition hypothesis. OpenAI's Roon calls it a historic moment. Google DeepMind researcher Neel Nanda warns against overhyping it. In a long essay titled "Neural Networks Are Already Clean," Nick Cammarata argues that the inner workings of large AI models are much easier to understand than previously thought.

Cammarata is a researcher at the interpretability lab Goodfire. With the essay, he rejects an explanation he helped shape years ago. Mechanistic interpretability tries to do more than watch what an AI model outputs. It aims to trace how the model arrives at its answers internally. Ideally, each of the millions of artificial neurons would stand for a recognizable concept.

In practice, many neurons seemed to fire on completely unrelated things, which researchers call polysemanticity. One early and famous example is a neuron in the image model InceptionV1 that responded to both cat faces and the fronts of cars. Cammarata used it as an example in his own early research. The standard explanation was…