Neural Networks Are Already Clean

The Short Version

I think I have fine news concerning the nature of neural networks.

After many months studying neurons and circuits in depiction and tongue models, I’ve arrive to accept a network’s own mechanisms are its legible construction blocks, and that the outlook for mechanistic interpretability may be far improved than I’d feared. The canonical evidence of polysemanticity and superposition that convinced much of the field, including me, that networks group concepts arbitrarily was misread. What looked akin arbitrary packing was mostly a motif I’ll call recursive generalization, which is much small concerning. This method we can study networks directly, alternatively of archetypal untangling them. Neural networks are already clean.

There are at smallest two ways to method mechanistic interpretability. The archetypal is fast normal science on the network itself: appearance at a network’s neurons and circuits, study how they relate, observe data stream through them, and run experiments at agentic measure to test explanations. The second starts from a catalog of hypothesized features and tries to extract them all, designing increasingly advanced reconstruction objectives and normally ignoring the original.

The site began alongside the archetypal method and cut rear to the second since neural networks seemed to be tangled knots. We had canonical examples of polysemantic neurons, the broad name for a neuron that appears to have multiple unrelated jobs or that looks unattainable to understand. The most celebrated one, in an depiction model, responded to the two cat faces and the fronts of cars. And the final tier of the first field’s example organism, InceptionV1, seemed so complete of arbitrary mixtures it wasn’t apparent how to study it: nearly none of the neurons were effortlessly interpretable.

The foremost clarification for polysemanticity was superposition. This projected a new learning principle: whenever a neural network runs out of neurons, it stops grouping connected things together and alternatively starts folding unrelated things together, merely since they don’t interfere. The outcome is a benevolent of knot: the network could have folded in many nearly likewise fine ways, so which things a neuron groups together tells you nearly nothing. You have to unfold the knot before you can study anything. In custom that method training a second example – a decomposition specified as a sparse autoencoder – to approximate the network as it would appearance unfolded and studying that instead. If the network really is knotted, there may be no alternative.

One chief outcome is that superposition was the incorrect clarification for InceptionV1. Architectural problems made afterward layers overflow. Weak interpretability tools amplified these problems, making the delayed layers appear incoherent. Since then, architectural changes designed to assistance profound layers study improved have had the unintended consequence of making them additional interpretable. When we archetypal studied InceptionV1, we established nearly no interpretable neurons in its final layer. But alongside these average improvements, and no alter to example size or training data, 99-100% of neurons in that final tier are effortlessly interpretable and the example is broadly legible the entire way through. It’s the archetypal non-trivial neural network I awareness I mostly understand; you can browse it here. I trained versions of it throughout a 300× range in size, and it’s broadly legible at all scale. With small room, the network doesn’t fold unrelated things together; its neurons fair shield coarser-grained concepts, akin one car neuron covering many kinds of car. With additional room, that neuron differentiates into many particular cars. So the issue wasn’t the existence of a second learning principle, it was fair an average learning nonaccomplishment of the archetypal one, and it’s uncomplicated to fix.

The canonical imagination examples of polysemanticity and superposition were errors too. In 2019, Chris Olah and I established the circuit that was initially introduced as the superposition event in Zoom InZoom In: An Introduction to Circuits
Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M. and Carter, S., 2020. Distill. References ↓. It showed a car characteristic appearing to dispersed into a set of unrelated dog neurons. It was the lone one we showed, and the lone one I always saw. We peruse it as the network placing the car in a high-dimensional hiding place, for downstream neurons to peruse rear out. But nothing says that combination. It wasn’t hiding anything; it was average generalization and composition. To my knowledge, no one has published a second circuit alongside the correct shape.

Modern tongue models are additionally far additional interpretable than mostly believed. Anthropic’s Towards MonosemanticityTowards Monosemanticity: Decomposing Language Models With Dictionary Learning
Bricken, T., Templeton, A., Batson, J., et al., 2023. Transformer Circuits Thread. References ↓ (2023), studying a small one-layer model, gave the median neuron a 0 on its rubric out of a imaginable 14, definition the rater couldn’t form any hypothesis. The automated interpretability projectLanguage Models Can Explain Neurons in Language Models
Bills, S., Cammarata, N., Mossing, D., et al., 2023. OpenAI. References ↓ I started at OpenAI (also 2023) established concerning 1.7% of neurons interpretable by our most generous measure.

With improved tools and contemporary models, the circumstance looks fairly different. In preliminary audits of Qwen3-32B, we evaluation that approximately 90% of neuron poles have one chief job, frequently alongside small connected offshoots; another ~7% have two connected jobs; and lone ~2-4% are not yet understoodA note on this number: the portion of neurons alongside one job varies depending on how you set the threshold for counting feeble activations, depth, density of sub-circuits, and what you perspective as “one thing”.

How far to go into feeble activations is a big aspect here. There’s a division describing feeble activations below. It tends to addition how many “related things” there are, additional than whether the neuron is a judgement of item legible. For instance, the Trump neuron’s feeble activations bring in MAGA hats and group connected to him. The Titanic neuron’s feeble activations bring in stories of additional broad voyages, or steam engines and another technologies of the era.

There’s additionally the inquiry of whether to attention concerning activation percentiles, or fraction of the strongest activations. In broad I think what the neuron learns have to be trusted, its natural sparsity and activation ranges and so on. Which I used to mostly not believe! I’m slightly against trying to standardize things throughout all neurons. Percentiles are additional average today, I think since they’re easier to think about.

It goes the another way too. For instance, whenever you appearance at the Titanic neuron’s “unrelated jobs,” it’s frequently detecting that “boat” comes next. The tool missed this since it lone looked backward whenever judging context. When we appearance at these “unrelated jobs” in depth, it’s nearly continually this benevolent of thing. It’s uncomplicated to build an imperfect tool.

Overall, I don’t think coming to a sole mark is really that important. But whenever it was taken out, first reviewers felt unanchored without equal a coarse number. In the sets I’ve looked at additional closely, considering powerful activations, it’s nearer to 97% non-arbitrary mixtures, but these are comparatively small samples.

Overall distilling everything into one “cleanliness” mark have to be taken as a broad pointer to the administration a neural network is in, fairly than item exact to either optimize immediately or clasp too tightly. A lot of this composition is for being small dimensionality reduced, treating all neuron and circuit differently, according to their motifs, depth, function etc. Despite this, a coarse figure can be useful.

Most of the remaining apparent polysemanticity seems to be a motif I call recursive generalization. It’s simple: as a neuron grows, it sprouts offshoots the way a tree branch sprouts sub-branches. For instance, one neuron that detected “iy” grew to detect “diy,” and afterward “do it yourself,” which doesn’t merge “iy” at all. This idea archetypal appeared in a forgotten subsection of a document I worked on called Multimodal NeuronsMultimodal Neurons in Artificial Neural Networks
Goh, G., Cammarata, N., Voss, C., et al., 2021. Distill. References ↓ (2021). At the time, I hadn’t watched neurons develop complete training. Now it’s easier, and it holds up everyplace I’ve seen. Even the first InceptionV1 wasn’t knotted. When I retrain its first architecture, which motionless produces messy profound layers, and prosecute the smallest legible neurons I can find, they rotate out to be overgrowing locally into reasonable, neighboring territory.

So there were two explanations for polysemanticity, and I helped propose both. In several ways, they’re opposites: generalization groups akin things together and superposition groups unrelated things together. Superposition is the one the site bet on and, until recently, so did I. But I now think recursive generalization was basically right. That leaves one learning regulation following all. It’s motionless generalization, but it turns out generalization is fractal. That’s far small concerning: there’s item to study from the way things were grouped, and nothing to study from two items in superposition. And since the generalization itself is fractal, our explanations can be too: commencement alongside a essential explanation, afterward zoom in whenever you desire to comprehend more. In that sense, nearly the complete network is interpretable if we appearance closely enough, particularly if we appearance at how it grew complete training.

Why was the gap so large? I think multiple average problems compounded. The explainer models of 2023 couldn’t run the iteration of conjecture and test; the tools and metrics were weak, rewarding explanations that could simulate activations fairly than ones that were right, which frequently arrive apart; no one knew what tongue neuron motifs to expect, and group broadly looked for the incorrect kinds of things; and most studies used small or simplified models. In a sense, those models really had run out of room, which made superposition the natural explanation. But a network abbreviated on area doesn’t toggle learning principles; its neurons fair get broader, as in the small imagination models. Broad neurons are normally uncomplicated to interrupt downward formerly you cognize what to appearance for. Without that knowledge, and alongside primitive tools, they appearance akin arbitrary mixtures.

Interpretability has a amazing amount of momentum: akin a jigsaw puzzle or crossword, any understanding you acquire starts constraining the remainder of the system. At first, you’re looking at a sole neuron. If you can’t comprehend it, small alternatively helps: circuits and attribution graphs lone explain a neuron in conditions of another neurons you don’t comprehend yet. This can create getting started hard. Once the map starts to inhabit in, though, it goes amazingly well. If you don’t comprehend exactly what a neuron does, you can study what it connects to, the ecosystem about it, and – in a mixture-of-experts example – what its expert specializes in. As alongside a puzzle coming together, all part comes alongside many clues, and they’re fine organized and uncomplicated to run experiments against. That’s why nearly bare and mostly filled in aren’t amazing places to be; they’re the natural anchors.

Take a neuron that tracks ‘a awareness of righteousness whenever a agreement acquisition turns out to activity improved than the costly alternative’. Neurons that track a emotion or social cue akin this are several of the most average we’ve found. Imagine finding this one alongside no cognition of tongue neuron motifs and no circuits to lean on, since few neighboring neurons are explained yet. Most humans and agents anticipate neurons to detect nouns by firing on them, which they mostly don’t. Look at anywhere this one fires most strongly, and you see a heap of apparently unrelated commas.

Now ideate the map is mostly filled in. You cognize many neurons detect subtle feelings or social contexts, and that they frequently blaze not on the words that transport the emotion but on punctuation, akin commas, anywhere the example summarizes what a declaration meant, or on the term fair before the emotion matters for predicting what comes next. So you peruse the surrounding sentences instead, and inspect its steering effects (what the example writes whenever you rotate the neuron up), which row up closely. You appearance upstream at how it’s built and downstream at what it does, and the two row up. You inspect its negative pole, what pushes it powerfully negative, and discover a subtle type of the identical cue, or the instant the social environment ends. You can run particular tests, akin checking which shop brands it responds to. Any of these could interrupt the explanation, yet they keep pointing to the identical place. And understanding this neuron sharpens the broader motif: following thousands of subtle-feeling and social-context detectors, you commencement to cognize what to anticipate from the next one, which is part of why the map fills in faster as it goes.

It plant on the tools too. When you can see 2% of a network, you can’t inform anywhere your tools are broken; whenever you can see most of it, the unexplained remainder points direct at them. That’s how several coworkers and I established a double-negative administration in SwiGLU neurons that, as far as we know, no one had noticed, and that a decomposition mixing many neurons together would apt have hidden.

The bad news concerning contemporary neural networks is that they're enormous. The fine news is that the enormity helps: as the map fills in, all clarification is tightly constrained by everything about it, and bigger models study narrower neurons that are easier to acknowledge in the archetypal place.

In this sense, understanding the entire network may be easier than understanding fair the parts we attention concerning most. First, it’s difficult to cognize how to think akin a neural network until you’ve studied it closely. For instance, “deception” is a helpful first frame, but the construction blocks that contribute to deceptive behavior by models likely have alignment-relevant microstructure that volition be difficult to cognize without going in and looking. Second, targeting a small set of mechanisms is akin operating on one row of an alternatively bare crossword. It’s doable, and power work, but without the crossings it’s difficult to cognize whether you’re right. It may be easier to resolve the entire item – or at smallest the broader parts of the network these mechanisms nexus to.

When I ran this procedure on contemporary convolutional networks, it basically worked. Agents can map out neurons and circuits, build explorable microscopes and examine them, test explanations thoroughly, and track hundreds of thousands of attribution graphs. There was never a item anywhere fast normal science bankrupt down. I now awareness I basically comprehend these models, at smallest adequate to cognize anywhere I’d commencement asking any inquiry I had; eventually, the crossword fair got filled in. So far it has worked in tongue models too, up to trillions of parameters. Those aren’t near to solved because of sheer size, but I’m optimistic concerning scaling the process.

All of this has brought me rear to an old hope: understanding a neural network directly, by studying its mechanisms and the data flowing between them, without archetypal trying to fix or untangle it. Put together, the disagreement has four steps.

1. Networks use one learning principle, not two (if not everywhere, afterward nearly adequate that it doesn’t matter for study), so there’s no blocker to commencement average discipline on them.
2. With contemporary models and tools, that discipline goes much improved than most group expect.
3. Understanding compounds, so it gets easier as the map fills in. At minimum, then, a network’s own mechanisms have to be part of the toolkit.
4. Highly speculative: this power be most of what we need, and interpretability power be far additional tractable, perchance in the near term, than we’ve assumed.

That’s been the form for profound learning in general: the uncomplicated item at ridiculous measure strikes the clever one. Rich Sutton called it the bitter lessonThe Bitter Lesson
Sutton, R., 2019. Incomplete Ideas. References ↓. Maybe there’s a bitter instruction for interpretability too: scaling average inquiry strikes engineering a clever new decomposition.

A specialized companion document volition recommendation a additional detailed document of tongue example neuron experiments, specified as what we’ve established concerning the distinct regimes of contemporary non-linearities and what kinds of circuit motifs display up at distinct depths. This composition glances at many places and conjectures concerning anywhere it all power be pointing.

Cars and Animals

The most celebrated polysemantic neuron is 4e:55 of InceptionV1, which looked akin an arbitrary blend of cats and cars. It’s been brought up as evidence that neurons power not be the correct component of study in multiple of the field’s most extensively peruse papers, from Feature VisualizationFeature Visualization
Olah, C., Mordvintsev, A. and Schubert, L., 2017. Distill. References ↓ in 2017 to Zoom InZoom In: An Introduction to Circuits
Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M. and Carter, S., 2020. Distill. References ↓. In Zoom In, which I co-authored, we stated outright that it “isn’t responding to several commonality of cars and cat faces”.In Multimodal Neurons we came extremely close. We asked whether the neuron power be responding to several shared feature, “perhaps cat eyes and car lights appearance similar,” afterward decided from the characteristic visualization’s whiskers and chrome that it was “genuinely OR(cat, car).” We were correct to appearance for a shared shape, but I think we underestimated how far generalization reaches. The shared characteristic was the V. I think the whiskers misled us since generalization happens inner high-level patterns. The V neuron really does obtain in cat circuitry, and characteristic visualization’s job is to lift the activation, so formerly it finds the cat path it adds everything that path brings. The whiskers arrive alongside for the ride, equal if the neuron doesn’t “want” them. Downstream, the car readers do get a small cat indication through the V neuron, but it barely reaches the output: one cat detector firing does small on its own, since many activity together, so the network never needs a negative importance to cancel it. To see the shared pattern, you have to visualize what the neuron’s inputs have in common, not the neuron on its own.

After months of seeing spotless neurons, I wondered if we’d misinterpreted several first notable examples of polysemanticity and superposition. The canonical polysemantic neuron seemed akin the apparent location to check, so I opened up the cat and car depiction grid I’d seen hundreds of times before and did item I’d never done: I spent a few seconds wondering what one item this power be. Almost immediately it was obvious: it’s not a cat and car neuron, it’s a V-shape neuron.

I looked at its circuit connections: V-shaped flames, V-shaped h2o jets from a fireboat, V-shaped fox faces. I looked at its negative pole: an O, circular anywhere the V is pointed. Why would an arbitrary blend of cats and cars need to nexus so powerfully to a flame?

Then I went to the canonical superposition circuit I archetypal established in 2019I established it during construction my archetypal Circuits interface, which lone had characteristic visualization and weights connecting their neurons. Then I took them to Chris Olah and asked why the car would be going into these dogs akin this. We came to the decision it must be doing a benevolent of folding, and called it superposition. It was an intuitive clarification for what we saw, so it didn’t immediate diving additional into the tools, which would have revealed characteristic visualization reward hacking and assorted another things. It being placed into a high-dimensional area felt akin a sensible answer.. The V neuron and this circuit were Zoom InZoom In: An Introduction to Circuits
Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M. and Carter, S., 2020. Distill. References ↓’s two chief examples of why its image of neurons, that nearly all one turns out to be understandable if you appearance seriously, power be “overly rosy”.

The circuit appeared to display a car going into a bunch of unrelated dogs, as if the dogs were a benevolent of high-dimensional retention location anywhere the car could be restored afterward as needed.

But that’s not what the car was doing. Cars and dogs were being brought together since of shared pose. InceptionV1 has additional than a hundred of these generalized stance neurons, all one its own spotless detector of a stance shared throughout cars, dogs and another objects, and to my cognition no one had noticed them. Later layers keep construction on them, all the way to the end of the model. The car was not being stored in an obscure place, and nothing immediately says out the combination. It’s the opposite: it’s continuing to differentiate into car specialists and finer poses.

Part of the logic was not knowing concerning generalized stance neurons, and part of the logic is that the method we use to render icons of images — characteristic visualization — ends up being reward hacked in deeper layers. The intent is to discover an depiction that maximizes the neuron, but the way that’s done is through gradient descent, and it takes the easiest path, which in InceptionV1 frequently method dogs. There are many of dogs in InceptionV1’s training data and a lot of dog neurons, and the first optimization falls into them and never recovers. If you equal lightly bias it toward the natural paths the top images take, most of them are cars, and several are the two dogs and cars, since of shared pose.

It can be tempting to discount this generalization, under the assumption that the dogs and cars are lone together since there’s not adequate space. Maybe it “really wanted” distinct dog-pose and car-pose neurons, and a bigger example alongside additional area would have divided them. It’s true that bigger models lean to have additional narrow neurons, but whenever networks measure up, they can do it in counterintuitive ways.

Looking at DINOv2DINOv2: Learning Robust Visual Features without Supervision
Oquab, M., Darcet, T., Moutakanni, T., et al., 2023. arXiv:2304.07193. References ↓, which is additional than ten times the size and concerning ten years of architecture later, it has additional stance neurons, but many of them are motionless generalized. I’ve seen this in all imagination example I’ve looked at, of any size.

Both individual and automated interpreters appear powerfully biased toward describing neurons as detectors for groups of nouns, akin cats and cars, fairly than for a shared pattern, akin a V. Sometimes a neuron really does react to a collection of nouns. In my experience, though, it’s additional frequently a shared pattern, and the neuron is improved understood as a verb than a noun. Its job isn’t to name what’s in the input but to diagram a difference downstream neurons can use: it separates two subtle patterns at its affirmative and negative poles (a V versus an O) and pushes the stream of activations toward distinct neurons accordingly.

I can frequently unblock my understanding of a neuron by asking: if I couldn’t use groups of nouns at all, how would I depict this neuron to person else? Usually the shared form pops out correct away, and it’s uncomplicated to test on additional groups of nouns (for the V: shinkansens, pond tables). I doubtful this is the largest origin of misattributed polysemanticity in mid-layer neurons. Many explanations I’ve seen that call a neuron polysemantic appearance to me akin a sole form described as a catalog of nouns.

We should treat a neuron that method two or three things as one neuron alongside a few grouped meanings. The grouping likely wasn’t a issue for the network: the meanings sit together because, from the network’s item of view, they’re similar. That makes the grouping a clue to how the network thinks, and following it can rotate up generalizations we didn’t cognize about, as cats and cars did here. Often the additional meanings additionally melt distant on nearer inspection, turning out to be insignificant and rarely active. Finally, several apparent meanings are fair our tools being bad, and this composition has multiple examples: the double-negative administration of SwiGLU, an unread saliency knob that makes the residual stream difficult to visualize, and the reward hacking in characteristic visualization we fair saw. Keeping the first neuron entire keeps these errors visible, so the error correction can continue. If we’d used optimization to get a cleaner list, we power have misplaced many of these hints.

A History of Microscope Errors

The narrative of interpretability, as I see it, has mostly been one of finding out that what was under the microscope was improved than we thought. The microscope fair wasn’t tuned right, and what needed tuning was frequently beautiful subtle.

Modern neural network interpretability arguably started alongside characteristic visualization, which renders a synthetic depiction of what a neuron responds to most: the celebrated psychedelic pictures. For a during they didn’t appearance extremely good. You could fairly decide the neurons had learned something, fair nothing extremely clean. It turned out the method was missing transformation robustness. Once researchers added it, the identical neurons came out crisp.

But that lone gets you so far into the model. Near the end, characteristic visualization starts to interrupt downward again, for two reasons: the reward hacking we saw alongside the car circuit, and regularizations that weren’t fairly right. Fixing it didn’t necessitate item fancy. A few subtle tweaks to sensible settings, and now we can see much deeper into the network. Same neurons, improved microscope.

But it wasn’t all the microscope. The example needed to be tuned too. InceptionV1, the first example organism of the field, was totally uninterpretable in the delayed layers. One estimate stated “To the finest of my knowledge, there are basically no interpretable neurons on InceptionV1’s final layer,” which is near to my own recollection, I’d have perchance stated 5%. This is what they looked like.

The foremost clarification as far as I knew was that there were too many high-layer features needed to be represented for the figure of neurons and parameters there were, so it started packing ones that don’t pertain together arbitrarily, since they don’t collide. The first layers were monosemantic since not as many first tier features needed to be represented.

But this wasn’t the correct explanation. Here’s a additional contemporary convolutional network alongside the identical figure of parameters and dataset, but alongside residual connections, LayerNorm, and a brace another improvements. The final tier is beautiful.

So two things were required. First, the neural network needed to study well, using an architecture designed by group completely focused on that goal and alongside no involvement in interpretability. The residual stream is one example: it was introduced to assistance study deeper neural networks, and it additionally seems to assistance the deeper layers be conceptually clean, which may be related. Second, we needed tools powerful adequate to see them clearly.

The final tier of InceptionV1 was among the chief pieces of evidence that neural networks were nearly basically uninterpretable, since the way they study overlaps unrelated mixtures. Replacing the neurons seemed akin the lone option. I put it about 5% alongside tools of that era, others at ~0%. With improved interpretability tools (fixed characteristic visualization, methods for visualizing attribution etc) it’s additional akin 50% on the identical network. This is all alongside the 2014 example whose checkpoints and training particulars were misplaced to time. Retraining the identical architecture alongside additional sensible contemporary regularization it’s additional akin 70%. In the ConvNeXt of the identical size, it’s concerning 99%. They’re explorable here.

Recursive Generalization

So we’ve shown the final tier was fixable. But what was it doing before it was fixed? Even before the architectural improvements, it wasn’t putting things together arbitrarily. When we retrained the example and watched a neuron form, it started on yellow flowers, in photos that sometimes included a dog. Then it grew alongside all aspect of those photos: toward another kinds of nature, toward green objects akin balls, and toward dogs without any flowers. This is true of all neuron I looked at. The improved architecture shows the identical pattern, but much small of it.

We call this recursive generalization: a neuron generalizing alongside additional than one axis, frequently archetypal alongside one axis, afterward sprouting outwards in ways that are uncomplicated to prosecute locally, but halt it from being “monosemantic” in the end.

For instance, a language-model neuron responds to “iy,” including in “Miyazaki,” afterward to “diy,” and afterward to “do it yourself,” which no longer contains “iy” at all.

There are examples all complete the location in the two depiction and tongue models. Like a earth discipline neuron that generalizes into Cesium, a earth discipline JavaScript library.

Usually it’s extremely apparent what happened, to an degree anywhere it’s difficult to cognize whether to fair call it normal generalization or not. Like an enemy neuron that fires for a Russian poem concerning enemies. Sometimes it’s small apparent but motionless plausible. A Srinivasen neuron additionally lightly fires for the UniProt macromolecule database. The project has multiple Srinivasen’s connected alongside it, but it’s difficult to cognize for sure.

We’ll conversation concerning these recursively generalized offshoots and how much they matter in the tongue example section. Here we fair desire to conversation concerning what the motif is.

We archetypal hinted at this in Multimodal NeuronsMultimodal Neurons in Artificial Neural Networks
Goh, G., Cammarata, N., Voss, C., et al., 2021. Distill. References ↓, calling them “conjoined features.” I’d forgotten until I went rear through the field’s most celebrated models, months into studying recursive generalization. As far as I can remember, no one, including me, suggested it power be the broad explanation, for case for what was incorrect alongside InceptionV1’s final layer. Without training checkpoints we couldn’t observe it form, but the clarification was basically right. The appendix explains why I think the new name is additional accurate.

Superposition

Recursive generalization isn’t a particularly bold explanation, it’s arguably fair what we should anticipate out of average generalization. But there’s additionally a much bolder clarification that has been projected for polysemanticity, a new learning principle.

The chief learning regulation we see everyplace is generalization. With generalization, things are put together since of a shared form that it’s helpful to have in one place, so it can be built on. With superposition, unrelated things are placed together since they don’t interfere. These have contrary predictions: generalization have to be local, superposition have to be non-local, since it would desire extremely distinct things to be near all other.

Generalization would desire a floppy ear dog neuron, which can be idea of as many of slightly distinct floppy ears at distinct angles and kinds together. Superposition would desire a floppy ear dog + Latin poesy neuron.

The definitions get tangled here. Along alongside superposition, Zoom InZoom In: An Introduction to Circuits
Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M. and Carter, S., 2020. Distill. References ↓ introduced the idea of features as the essential component of neural networks. Depending on anywhere you look, superposition afterward method either “more features than dimensions,” which equal a completely monosemantic network would satisfy, since its neurons compose, or a learning regulation akin the one above, applied sometimes to what neurons study and sometimes to anywhere features go.For instance, InceptionV1’s tier 4b has basically monosemantic neurons for car parts akin windows, car bodies, and wheels, which the next tier assembles into a car neuron, 4c:447. In the characteristic view, it’s sensible to say the “car feature” isn’t fair in 4c:447, but additionally in the blend of 4b neurons that builds it. So there are two car features in neighboring layers, sharing nearly exactly the identical mechanics. In the “more features than dimensions” view, the archetypal is in superposition and the second may not be. In the awareness of the informal note Distributed Representations: Composition & Superposition, the archetypal is distributed, but by composition, and the second isn’t distributed at all. This is part of why I discover hypothesized features difficult to think with, and why I normally favor particular mechanisms, akin neurons, and particular pieces of data.

So “superposition” is itself in superposition between multiple average uses. We concentration on the learning regulation applied to the mechanisms akin neurons, since whether it forms holds specified a disproportionate portion of the fine or bad news for interpretability. It’s additionally fairly uncomplicated to test: recursive generalization and superposition foretell nearly contrary things, and you can observe a neuron develop and see which one happens.

When I say superposition in this essay, I’m referring to this version, and we’ll location the inquiry of “more features than dimensions”, which I basically don’t think is bad news, later.

Superposition of the Gaps

Polysemanticity is frequently additional a assertion concerning the observer than the neuron. It says there’s no fine clarification yet. There’s a few reasons that could be:

1. Bad tools: our lens or microscope isn’t fine enough
2. Bad explanations: created from those tools
3. Bad learning: the network learned poorly because of bad architecture, training data, or item else
4. Recursive generalization: during generalizing, the neuron branched off in a few connected directions
5. Superposition: a alter in learning regulation anywhere the network packs unrelated things into the identical neurons since they rarely collide.

For years I was a staunch superposition believer. I started studying tongue example neurons not out of any involvement in neurons or superposition, but since I was curious in whether agents had investigation taste! I wanted to see if they’d ask fine questions as they tested explanations, and started alongside neurons lone as a baseline. I expected them to run into intractable characteristic fragments in superposition. That would create a new baseline for how spotless neurons really are, afterward the agents could continue alongside several cleaner learned basis. But it never came, since the neurons kept coming, resolving to a uncomplicated clarification that made amazing predictions that held.

In a frontier tongue model, I established a neuron that mostly fired on “pr”, but additionally on a bunch of Base64 URL strings as fine as several Bengali. I idea this must eventually be an arbitrary mixture. But decode the URLs: “pr” words. Translate the Bengali: “pr”.

In a compact Qwen model, a coworker came throughout a neuron that norm top-k automated interpretability lists as a “polysemantic comma neuron”. But looking additional closely, it turned out to track Colorado: phone numbers, slang, and connections to another Colorado neurons. Steering it made the example desire to go hiking. The commas were anywhere it stored a operating summary of the Colorado environment so far. In broad notice seems to collect on the punctuation, which we call the summary motif. Once you cognize it, a lot of akin neurons get easier to understand.

Other times, we fair didn’t have adequate data. At archetypal we didn’t contain lengthy context, two paragraphs felt akin enough. But this accidentally made all the long-context neurons appearance polysemantic.

In another case a neuron looked akin an arbitrary mess, alongside nothing firing strongly. It turns out it looked for particular signs inside pages of health SEO spam, and formerly we had a sufficiently broad dataset, it was understandable.

Across hundreds of cases akin these, nothing shook my condemnation in superposition. It fair kept moving. If the powerful activations came rear clean, superposition would move to the feeble ones, or to combinations of neurons, or to the neurons I hadn’t looked at yet. Or perchance throughout layers themselves.

But eventually I wondered what was going on. Unless this was several theatrical extended case of choice bias, item wasn’t matching the first example organisms. So I went back, the first evidence fell, and that led to this essay.

I now think I was making superposition the null assumption whenever it should have been the final hypothesis, since it’s the most extreme and the smallest testable. It fair looks akin things that don’t create sense. But that’s additionally what bad tools appearance akin (feature visualization reward hacking), or recursive generalization you haven’t followed yet (“iy” neuron), or a bad architecture (5b of InceptionV1), or generalization you haven’t understood yet (cat and car).

I think part of what was going on was that superposition fair felt right. Anthropic’s Softmax Linear UnitsSoftmax Linear Units
Elhage, N., Hume, T., Olsson, C., et al., 2022. Transformer Circuits Thread. References ↓ document stated as much in 2022: “In the mechanistic interpretability community, superposition is frequently treated as the default assumption merely since it seems intuitively additional compelling than another explanations, but there is small evidence.”In the next sentence, the document says its own results are average evidence for superposition. That evidence is circumstantial, in the paper’s own word: SoLU suppresses activations dispersed throughout many neurons, an additional LayerNorm restores performance, and following the LayerNorm a neuron’s feeble activations looked small connected to its chief job, which the document says as features being smuggled through small activations. We peruse it as evidence concerning distributed depiction in an architecture built to suppress it, fairly than concerning neurons packing unrelated things together, which is the awareness we attention about. Telling the two distinct in feeble activations is difficult equal alongside today’s tools, since definition shifts in all direction at formerly as activations weaken, and it was harder motionless in 2022, for all of us. The document additionally notes that its conversation of superposition has been superseded by Toy Models of Superposition, which we location in the appendix. We bring it up not as a criticism of either paper, but since it’s the lone location we cognize of anywhere person stated plainly how small real-world evidence there was, and we think group motionless frequently assume there’s additional than there is. To my knowledge, no one has established a superposition circuit in a genuine network since my first one, which was an error. I don’t see item akin it in genuine models, notwithstanding expecting to, and I think the bar for claiming it have to be high, since it’s specified a extreme clarification and would be big news, awful news, for how we should study neural networks.

Of course, this risks the contrary error, an clarification of the gaps: assuming all neuron must have a sensible story. Two things create it small worrying. First, all of the another causes can be tested. A improved tool either resolves the neuron or it doesn’t; retraining alongside a improved architecture shows whether learning was the problem; and recursive generalization can be watched as it grows, in small steps. Superposition’s lone indication is that item doesn’t create sense. Second, genuine explanations lean to be narrow and effortlessly testable. Once you doubtful a neuron is a Colorado neuron, steer it and see what it says. If a neuron fires for several narrow TV trope, the set of things that could figure as that clarification is tiny, during “something that doesn’t create sense” covers everything exterior it.

Also, we don’t have to power an clarification we don’t have. If a neuron power do a brace unrelated things and you’re not sure, it’s fine to say so. I think recursive generalization have to be elevated on the catalog of things to suspect, and it’s uncomplicated to inspect if you have training checkpoints. I now think superposition have to be far downward the list.

Exploring Image Models

Small depiction models may appear obsolete, but I think they matter a lot to this inquiry since we can see them so clearly. They're additionally anywhere much of the first concern came from: there we saw what looked akin high-resolution chaos, not low-resolution ambiguity. Language models are massive, we don't have tools akin characteristic visualization which create our existence easier, notice mixes distinct kinds of mechanisms together, and many of their motifs are surprising. As I showed above, equal imagination motifs can be counter-intuitive, and tongue is worse. Each one of these deficiencies can create a monosemantic neuron appearance polysemantic. If you're looking at a blurry depiction trying to fig out if it's a dog or a cat, the blur makes it harder, but it doesn't favor either answer. If you're looking at a blurry depiction and trying to decide if the depiction itself is blurry or if you're fair looking badly, all flaw in your eyesight gets counted as blur in the image. If we desire to study how neurons alter alongside scale, compression pressure, activation strength, and depth, it's improved to commencement anywhere we can see clearly.

I trained three ConvNeXt models spanning approximately 300× in size: a small one alongside 92k parameters, a average one alongside 7.06M (the identical size as InceptionV1), and a ample one alongside 28.6M. We’ll appearance at those, and additionally InceptionV1, to try to comprehend what neurons do in distinct situations, and try to discover places packing may form.

How I Think About Neurons

I used to think of neurons as human-shaped nouns: one is a car, another is a curve. Now I think of them additional as authority flow. A neuron is small concerning what it sees than concerning what it does to the stream coming in, and frequently its job is to entrance a area of the network. Seen this way, monosemanticity is nearly the incorrect question. A improved one is what a neuron lets through and what it keeps out.

Pixels arrive in, and the network starts grouping and distinguishing things. Cars and dogs power be grouped together since they portion a pose. Further alongside there are particular car detectors – for taxis, ambulances and the akin – and they don’t desire dogs. So a neuron in forefront of them acts as a gate: it responds positively to cars and negatively to dogs, and its negative flank keeps the dogs out. The dogs go on to their own area, which doesn’t have to concern concerning cars. Later neurons build on this, specializing further: an ambulance neuron power have a taxi on its negative pole, and a taxi an ambulance. In a larger model, these distinctions can rotate into finer still, separating one perspective of a taxi from another. Smaller models may halt at coarser distinctions, alongside item nearer to a broad “car” neuron.

Here’s a genuine sports car, followed through the final five blocks of our ample model. Partway through, it’s grouped alongside another rectangular shapes, akin tables and cabinets, against snakes and dogs. By the final block, the divided is convertibles against position wagons.

The negative flank of neurons matters nearly as much as positives for understanding them, equal in ReLU networks anywhere the negative flank is cut off. A neuron is built from votes: all input importance votes for its affirmative flank or its negative side, and the votes add up before the nonlinearity. The negative flank never gets gradient on its own, but the idiosyncratic votes do whenever the neuron is on, so it’s shaped by learning too. The negative side’s definition depends on context: whenever a neuron in the sports car’s way separates it from snakes and dogs, it isn’t calling cars and dogs opposites, fair gating out things that were usefully grouped earlier. The negative flank tells us what difference matters at this item in the circuit, which additionally makes it a quick inspect on whether a grouping is real: the V neuron’s negative flank is an O.

If we appearance at the final layer, we see high-level patterns and what they’re being distinguished from: taxis from ambulances, broccoli from cauliflower, and spoonbills from flamingos.

When we appearance at our smallest model, we see the neurons get coarser. Its final tier ends at cars, trains, and carts in one neuron, alongside boats, planes, and a water-based ox-cart, (which I didn’t cognize equal existed), on its negative side. It’s motionless differentiating, but additional coarsely, and it ends early.

What Neurons Do Under Pressure

Early layers in most imagination models are monosemantic. As I understood it, the norm clarification is since there’s not that many first imagination features that need to be represented. On its surface, it’s a bit different that all model, from AlexNet, to ResNets, to InceptionV1, to CLIP all have the correct proportion of first features to first neurons, since they have specified distinct ratios of dataset to indicator size.

One tier we can concentration on is 3b of InceptionV1, the tier alongside Curve Neurons and High-Low Frequency neurons, which are among the most essential in my opinion. This example has no residual stream, so there’s no way to “flow around” the neurons. So we can test what happens to neurons whenever there’s not adequate area by reducing the amount from 480 to 10 and retraining.

If it switched to superposition, we should see overlapping mixtures on top of all other, akin a curve affirmative an orange spiral affirmative spider webs and multiple another texture detectors. We’ve shattered it 48x so it have to be dramatic!

But that’s not what we see. The neurons remain interpretable and generalized, but coarser: one “curves & loops” neuron alternatively of ten curve detectors, two circle neurons alternatively of multiple circle types, and so on.

We can test curves explicitly to see what happened to them. If they were placed in several elevated dimensional blend of neurons, we have to be capable to extract them. So we can train an SAE on this shattered tier and difference them to the originals. We don’t see item akin spotless curve neurons!

I think this is a fine test of whether we can rotate a monosemantic tier I cognize fine superposed. But in several ways it’s not ideal, since group mostly suspected superposition in the final layers. So we can obtain our average size ConvNext, crush the final layer, halt it from routing about the residual stream, and see what it learns. We see the identical form of it broadening.

This is additionally what we see alongside the small ConvNext, which in several ways is a extremely akin test.

Late Circuits

We can additionally appearance at circuits in the delayed layers, to regulation out any chance that the final tier is somehow special. We see that it’s organized the entire way through.

What Dictionaries Learn

If combinations of neurons were hiding things, a dictionary (eg an SAE) should discover construction blocks the neurons lack. So I trained multiple on these layers and went looking, myself and alongside agents. I didn’t discover any.

Mostly, the two looked akin what they mechanistically are: the neurons akin construction blocks, the SAE units akin those blocks grouped by context. It was clearest alongside spiders. The SAE units were spiders in distinct settings; the neurons were leg patterns and webs. They didn’t differ much.

SAEs did assistance alongside “what’s in this image?” They helped most in the smallest model, whose neurons are coarse. At average size I slightly preferred them; at large, I think I favor the neurons. But I attention additional concerning understanding the network than concerning any one image, and I never saw the SAE grasp item essential the neurons didn’t. If a example at a particular size unsuccessful to comprehend ballgame players, so did its SAE. When the example eventually understood a ballgame player, it was on a neuron.

What did the SAE discover that didn’t already have a apparent neuron counterpart? On the final layer, 88% of its action is near-copies of sole neurons, and the remainder is faint: removing all dictionary direction that matches no neuron leaves accuracy unchanged, and the few answers it flips were already near-ties. Additionally, if you hunt the rotations of a tier for the one anywhere a fistful of units transport the most activity, starting from a random one, you district rear on the neurons, in all block. Among the few features powerful adequate to appearance at, what the SAE added was free groupings built from existing neurons, akin animals at the water’s exterior or pale, lumpy food. Nothing we established there looked akin a basically new construction block.

Rare Things and Memorization

A lot of the concern concerning hidden features comes from rare things. They rarely collide alongside anything, and they may not appear value a neuron of their own. So we looked at rare things all way we could think of. Rare classes are frequently additional consolidated: you need a lot of dog patterns for ImageNet, but lone one Triceratops neuron. When I removed feeble activations in the final stage, no category stood out. The adversarial examples we traced used normal semantic circuit routes.

I looked at the images whose representations in the final layers are furthest from any another image. They end up being entertaining, weird images that the example misreads alongside average neurons: a hummingbird seen as ice cream, a elephantine banana seen as a vault, and a guy in a fallen wheelbarrow seen as a grassland mower. I additionally trained an ImageNet example to map random noise to random labels, and equal that was partially semantic: it partially leaned on wool textures and the texture of distant trees, which are high-frequency, a bit akin elevated frequence noise. I didn’t completely audit the model, but idea it was engaging that it used them at all. Across all experiments in tongue and vision, remembrance tends to desire to sit as near to the output as possible, since all it has to do is appearance the answer up, and the delayed layers did most of the work.

We can additionally create a rare path ourselves, by instruction our 7.06M-parameter example item nonsensical: that a photo of an umbrella is a bee. You could ideate this remembrance not relying on item else: a hidden characteristic that latches onto this exact set of pixels and says “bee,” cut off from the remainder of the network since it doesn’t need it. That would be beautiful bad news, but it’s not at all what happens. In nine training steps, the top conjecture went from umbrella to maypole to bee. The umbrella neurons faded and existing bee neurons came on, nearly all of it in the final three blocks: toggle fair those into the first model, and it says “bee.”

This is in difference to learning several kind of hidden umbrella characteristic distinct from any another normal umbrella or bee route. While this is possible, I think it’s slower to study than assembling connected parts that are already activating and have routes to the output, giving it plasticity and gradient. It’s additionally additional fragile, and isn’t what we see anywhere.

Do Weak Activations Matter?

When group ask whether feeble activations matter, I think there are multiple questions blended together, and they have distinct answers.

Are they generalization or superposition?

Unsurprisingly, I think it’s generalization. Imagine you’re a packed, unrelated characteristic trying not to collide alongside what’s in a chief neuron. Weak activations are vastly denser, so you’d be colliding alongside orders of dimension additional things, for small upside.

But perchance you conceal in the feeble activations of many neurons at once. What we see is that you can ablate nearly all the feeble activations of the network for nearly no cost. The places anywhere removing them expenses the most are several of the easiest to understand. In tongue models, they contain the final layer, which is mostly used to suppress sole tokens, for what we call a “say or fair think” motif. In imagination models, it’s the first layers, which function akin dials: the difference between slightly brilliant and average brilliant matters, but for a afterward detector, akin king penguin, low or extremely low evidence doesn’t. So crucial average things can’t be in the feeble activations, since we can eliminate them.

We can additionally appearance at what we endure whenever we cut feeble activations. Across multiple kinds of studies, it’s normally fine distinctions between things the neural network is already confused about. And the faint activations that tip them arrive from the semantically connected neurons you’d expect.

This leaves rare data and powerful but not extremely powerful activations.

Rare things, as we saw earlier, have grown on connected circuits in all test we’ve run, since they aren’t powerful adequate to form niches of their own. The ablation complete points the identical way: if rare cases lived in the feeble activations, cutting them should have damaged several answers outright. Instead, all alter was small, a near-tie tipping one way or the other.

And the strong-but-not-extreme activations we can fair appearance at. In all case I cognize of, they radiate outward semantically: for CLIP’s Trump neuronMultimodal Neurons in Artificial Neural Networks
Goh, G., Cammarata, N., Voss, C., et al., 2021. Distill. References ↓, photos of Trump, afterward art and group connected to him, afterward Fox News studios. The additional downward you go, the easier it is to be fooled by item that seems unrelated but isn’t. At minimum, appearance at the circuits a datapoint flowed through; what fired weakly current may have fired powerfully one tier back.

Here's all powerful firing of five neurons in a tongue model. The center is all neuron's maximum, as it radiates out it becomes additional distantly related. Gray is anywhere it's difficult to place, normally these are motionless connected but necessitate additional activity to understand, since it's additional distant. You can click any to open them up and see which circuits all activation came from.

Percentiles or raw activations?

I used to think in percentiles, since they’re uncomplicated to think in and standardize throughout neurons. But I’ve arrive to think a neuron’s raw form is telling us something. Some neurons clasp their definition far downward the range; others autumn off a cliff, akin the Christmas neuron complete at concerning 5% of its max. Some blaze a few times a book, others a few times a page. Percentiles flatten that, and it’s frequently what tells you whether you’re looking at a dial, anywhere the center matters, or a detector, anywhere it mostly doesn’t.

Percentiles additionally disorder alongside intuition. The 99th percentile appears high, but 1 token in 100 is concerning a fourth of a page, and a neuron for a niche emotion or social environment – which are all complete tongue models – shouldn’t arrive up four times a page. Thinking in pages helps: the 99.9th percentile is all two or three pages, the 99.99th all 25 or 30.

Do you need them to tag a neuron?

Usually not. After looking at thousands of tongue example attribution graphs by hand, it’s difficult to overstate how much neurons are analogizers. If you’re talking concerning voyaging on a boat, the example power fairness that a bit Titanic-ish. I don’t think it’s normally helpful to think of a “naval voyage feature” dispersed throughout the feeble activations of the Titanic neuron and others. Instead, there’s a Titanic-ish mechanism: whenever item is Titanic-ish, it fires, and its weights are votes for Titanic-ish things. So you appearance at the top activations, see that it’s Titanic-ish, and that tag normally holds up through the remainder of your audits. If you always really attention concerning a particular activation, you can prosecute its pathways, ablate the circuits, and keep zooming in to nearly arbitrary resolution.

My chief takeaway: if a feeble activation looks unrelated, don’t be too concerned unless you’ve looked at it carefully. Of the likely hundreds I’ve followed by hand, nearly all turned out to be related, coming through a sensible circuit; it fair wasn’t visually apparent until I traced it.

Many were clever, akin a Zen koan neuron that fires weakly on recursion in math: many koans rotate on the identical benevolent of self-reference, looking for the observer. Others were additional distant and ugly, not continually the association I would have drawn, but you could normally prosecute how it got there. I doubtful most, perchance nearly all, are akin this.

Exploring Language Models

This division was done alongside Curt, and brings in experiments from our forthcoming specialized document studying tongue neurons in depth, so I’ll use we for this section.

Language models are so big that it’s difficult to validate claims concerning neural mechanisms precisely, but we can inspect whether the uncomplicated narrative holds there too. We commencement alongside a medicinal diagnostic attribution chart we liked from Anthropic’s Circuit BiologyOn the Biology of a Large Language Model
Lindsey, J., Gurnee, W., Ameisen, E., et al., 2025. Transformer Circuits Thread. References ↓. They were studying Claude 3.5 Haiku using Crosslayer Transcoder features. It turns out the identical plant alongside neurons in Qwen3-32B.

Since it’s the first model, there’s no reconstruction defeat and doesn’t necessitate any additional training to get. Their largest transcoder predicts the identical next term as the first only fractional the timeCircuit Tracing: Revealing Computational Graphs in Language Models
Ameisen, E., Lindsey, J., Pearce, A., et al., 2025. Transformer Circuits Thread. References ↓ and their graphs need error nodes for what it misses.

The assessment hub is fair four neurons, and if you eliminate them the example asks concerning itching instead. Itching is a telltale sign of a distinct liver circumstance of pregnancy, as if the example had moved on to its next guess.

We’ve additionally added a few another attribution graphs we liked. There’s a few another graphs if you’d akin to see them. Across thousands I’ve seen in depiction models and tongue models, they’re extremely readable and ablate the way you’d mostly expect.

The neuron’s negative flank matters concerning as much as its positive. We think interpretability interfaces lean to concentration too much on the affirmative rod and its feeble activations. We think the feeble activations normally don’t matter, but the powerful positives and negatives matter a lot.

For instance, Santa lives on the negative pole.

In the figure’s Santa graph, norm top-k automated interpretability labels can appearance akin “capital cities” since they disregard the negative flank corresponding to Santa lore. Remove the eight neurons alongside a Santa negative flank and Santa falls from near-certain to 6%, alongside Qwen blaming the pet instead. Remove the twenty neurons really tagged Christmas or Santa and it lone falls to 76%.

Here’s an attribution chart from Kimi K3 using lone neurons. As far as we know, it’s the archetypal published attribution chart of a multiple trillion indicator model. Asked what to do alongside a wallet it found, it links the wallet, the prosperity and the owner’s ID to misplaced asset and honesty, and answers “return”. We haven’t begun complete audits of this measure yet, and anticipation to soon, but so far most things have been intuitively interpretable.

We power intuitively anticipate packing alongside things akin bi-grams or names, merely since there’s so many of them. What we discover is that it’s average generalization and composition. Early tongue fragments akin “iy” in “Miyazaki” are kind of akin how first imagination models have gabor filters.

In the appendix we location Neurons in a HaystackFinding Neurons in a Haystack: Case Studies alongside Sparse Probing
Gurnee, W., Nanda, N., Pauly, M., Harvey, K., Troitskii, D. and Bertsimas, D., 2023. arXiv:2305.01610. References ↓, which idea they may have established superposition in a “social security” direction, but that direction is unused by the network. We map the one it uses, and it uses average generalization and composition.

Here are 1000 neuron papers, detailed investigations into idiosyncratic neurons, chosen randomly.

What we see is concerning 90% of neurons having basically one job, alongside recursive generalization offshoots that are normally related. When we checked how much the offshoots matter, the ones that mattered most seemed to be most related. For instance, a earth discipline neuron that fires for Cesium, a earth discipline javascript library. A lot of the remaining ones have two or three. I’ve never seen a neuron alongside item akin ten unrelated things it does. There’s a lot additional item in the another paper.

The offshoots don’t appear to be particularly crucial hidden features. One fear is they’re item akin sleeper agents, extremely crucial in their own context, but rare. What we mostly discover is that it’s the two rarer and small crucial in their own environment than the chief activation is in its.

A big origin of noise was the twice negative SwiGLU. Modern neurons have the two a entrance and value. When the two are negative, this sends activations into the affirmative pole, that normally appearance akin the negative pole. Usually this “echo” is small, but for concerning 4% of neurons, it’s the entire top of the list. Ablating them throughout the entire network ups most benchmarks, but lowers several reasoning ones. We think the answer is likely prosaic – several neuron that uses one of these activations to logic longer. But we haven’t looked into it deeply. Architecture designers, we think you should appearance into fixing this!

Looking at SwiGLU neurons in imagination models anywhere we comprehend things better, the value tends to study broader things, and entrance additional particular things. For instance, the value learns a broad family akin small gray birds, or item orange, and the entrance volition open lone for juncos or goldfish.

Experts are far additional specialized than earlier reported. Since it was difficult to fairness neuron cleanliness, it was difficult to fairness expert specialization, things akin “token similarity” aren’t a fine measure.

Like neurons, they get narrower alongside depth. When they aren’t “monosemantic” they appear to go throughout broadly connected things. We haven’t looked complete training, but we think it’s likely the identical recursive generalization form one measure up. In general, we’ve started to think of the neural network as being the identical benevolent of organizing regulation but at multiple layers, from weights, neurons, experts, and circuits. Each flat has its own concerns, akin particulars of the non-linearity, but there’s a lot shared between them.

It’s additionally engaging to think of experts as an anti-superposition architecture. Superposition wants to group unrelated things together, but extremely specialized experts create that hard, since a neuron lone fires whenever its expert is chosen. A packing-minded expert wouldn’t desire to be devoted entirely to bosses in RPG commission games (one is); it would desire to be ten thousand unrelated things packed together. But afterward the router would have to acknowledge all ten thousand to cognize whenever to dispatch a token there, and the router is fair one small linear layer.

Explanations That Can Lose

There’s a hazard of defining explanations too broadly, akin a “multifunctional tongue neuron.” The specialized document has multiple ways to test for that, and the appendix has one anywhere explanations visage near misses written to fool them. One we established particularly helpful is blind maximum. One delegate investigates a neuron and writes a abbreviated explanation; a second sees lone that clarification and writes an example meant to activate it maximally. The outcome is normally a simple, readable sentence, and it can activate the neuron additional powerfully than any example we established in 10 milliard tokens.

It’s tempting to think of a neuron’s activation as fair a number, and of blind maximum as a way to create it bigger. I discover it additional exact to think of a neuron as a target, alongside definition expanding out from the center in all direction. Everything feeding into it makes its own small semantic judgment, and the maximum is anywhere they all concur most. So hitting it isn’t concerning a bigger number; it’s concerning landing nearer to the center of what the neuron is looking for. Ten milliard genuine tokens had their chance, and a declaration written from the clarification solitary got closer.

One bounds is that a blind maximum tests the neuron’s center, not its breadth. If a neuron has multiple jobs, equal connected ones, a sole blind maximum may lone attain one. So we additionally collection the attribution flowing into the neuron. For a pet-microchip neuron, this gives bundles of upstream neurons for animal tagging, misplaced pets and rescue, and microchip and RFID identification: three directions out from the identical center. We afterward run a blind maximum for all bundle, ablating all the others, and difference it against the strongest corpus example under the identical ablation. A bundle that had been packed in fairly than grown would display up current as an clarification unrelated to the others that motionless drives the neuron. This doesn’t provision complete coverage, but it covers much additional of the mark than one attempt at its center.

There are additionally broad tests, akin predicting activations and circuits from an explanation. To be honest, I rely these lone up to a point. “Cats and cars” could do fine on many tests of the V neuron during entirely missing what it really is. You can interrupt it by noticing it connects to flames, or in many another ways, but they all obtain ingenuity and judgement fairly than a summary statistic. Interpretability has the identical form as the remainder of natural discipline here: Newtonian gravity explained an enormous range of observations, afterward bankrupt on a small discrepancy in Mercury’s orbit. The finest criticism is particular to the conjecture.

Slouching Towards Monosemanticity

Just as we looked rear at first imagination example organisms, we can appearance rear at first tongue models to see why group had a difficult period understanding them. One I cognize improved is GPT-2 XL, the example in the first automated interpretability projectLanguage Models Can Explain Neurons in Language Models
Bills, S., Cammarata, N., Mossing, D., et al., 2023. OpenAI. References ↓ I started at OpenAI, anywhere small than 2% of its neurons came out clean. We went rear and made 500 neuron document alongside nearly no iteration. Now concerning 70% appearance akin approximately one thing, several alongside a residual that looks akin recursive generalization. The amount of pristine ones are motionless low, concerning 5%. What’s remaining are neurons that do a few broad things, up to concerning six, mostly two. The scoring isn’t identical to the first project. On the activation reconstruction project we used rear afterward we’re capable to do multiple multiples improved alongside no iteration, but I no longer think it’s a awesome metric for explanations (see On Simulation Scores). There’s additional item in our forthcoming specialized paper. But in general, smaller models have broader neurons, alongside additional residual generalization than contemporary models.

Something akin may have happened alongside the neuron in the introduction of Towards MonosemanticityTowards Monosemanticity: Decomposing Language Models With Dictionary Learning
Bricken, T., Templeton, A., Batson, J., et al., 2023. Transformer Circuits Thread. References ↓, which multiple people, and agents, pointed us to whenever we stated neurons power be clean. The document opens by noting that it would be nice if neurons were interpretable, but they merge arbitrary mixtures. It cites the cat-and-car neuron, alongside alongside a neuron from its one-layer tongue example that fires for “a blend of scholarly citations, English dialogue, HTTP requests, and Korean text.”

I think it may be a “don’t say a dot” neuron, a motif we see all complete tongue models. Nearly all the citations, dialogue, and HTTP requests blaze on or near a dot, and eight of its ten strongest negative logit connections commencement alongside one. That leaves the Korean examples, which all blaze concerning two-thirds of the way through an unfinished syllable, a sensible location not to desire a dot. Or perchance that’s a second thing; without the model, we can’t keep narrowing the explanation. Either way, it’s two small syntactic pushes, and it’s uncomplicated to see why they’re together.

I think this example is helpful since it’s a celebrated arbitrary blend that turns out to sit in the center ground, at smallest for now, since we don’t have the example to keep checking. We don’t cognize whether it’s one thing, or one item affirmative a weaker, connected syntactic pattern. But either is a much narrower and additional hopeful clarification than four unrelated things, and “polysemantic” lumps them all together, equal although they’re extremely distinct levels of interpretability.

In 2023 during the period of the archetypal automated interpretability projectLanguage Models Can Explain Neurons in Language Models
Bills, S., Cammarata, N., Mossing, D., et al., 2023. OpenAI. References ↓ I idea small simplified models would be the finest starting location for interpretability, since there’s small bells and whistles to think about, and small size method small things to study. But now I think it may be the opposite. The bells and whistles are there to assistance the neural network study better, akin the residual stream InceptionV1 never had, and by simplifying it’s uncomplicated to create neurons study small well. And bigger models lean to have additional narrow neurons, so it’s additional uncomplicated to place patterns and motifs. At smallest for GPT-2, I was frequently looking at multiple motifs together, neither of which I knew, which made things hard. By adopting the architectures and measure the site wanted anyway, we may have been slouching towards monosemanticity without knowing it.

The Residual Stream

The concern alongside the residual stream isn’t that there are additional things we attention concerning than dimensions. It’s a uninterrupted type of packing: manifolds woven through all other, accepting several interference in toggle for area to shop more. Ruling that out entirely may not be possible. But in custom it hasn’t been a bottleneck. We can peruse what neurons compose to the residual stream and what they peruse from it, attribution graphs arrive out clean, and we don’t see the stream accumulating interference lone to undo it later.

In imagination models, we could additionally inspect this by eye. For years I couldn’t visualize the residual stream of convolutional networks: all map came out as one big arc. It turned out the network was putting nearly all of its activation bulk into a sole unread dimension, which it can alter to measure all the another dimensions that are peruse through LayerNorm. Once we understood that, removing it was easy, and the maps made sense. The specialized particulars of how it works, or equal what it’s doing, are small crucial than the additional broad form throughout this essay: discover what the architecture is doing and activity alongside it, fairly than fighting the network. It additionally pointed toward item the site ended up construction anyway: attention.

We don’t cognize whether tongue models need a akin fix. When we’ve visualized their residual streams, it has mostly fair worked. Judging whether a entire map of content makes awareness is much harder than glancing at a map of images, though, which is why we did most of this activity in vision. A sole earth map may not be the finest tool for tongue models anyway. Whatever the correct perspective turns out to be (submanifolds, what neurons peruse and write, or lens-style readouts), everything we’ve tried has worked, and we haven’t hit a part of the residual stream we couldn’t create awareness of.

We additionally tried crushing the residual stream itself, much akin our layer-crushing experiments. We retrained our 7M-parameter average ConvNeXt alongside a stream four and eight times narrower, downward to fair 46 dimensions in the final block, during keeping all of its neurons. Accuracy cut from 72% to 60% and 39%, but compared alongside a normal-width example the shattered stream’s map looks fairly similar, fair a bit messier. Nothing looks cleverly interleaved. Neuron says and writes inform the identical story: things get noisier, but what leaks in is the identical benevolent of thing. A dark-bird-on-water neuron additionally fires on alligators and a swimming retriever, not on clocks.

We attention additional concerning energetic bottlenecks to understanding neural networks than concerning proofs of what the residual stream can or can’t contain. If interference isn’t showing up in the maps or in what neurons peruse and write, equal whenever we obtain distant the room, we think we’re in a beautiful fine spot.

That leaves the involvement that there are additional features than dimensions in the residual stream. A stream can clasp far additional than it has dimensions fair by placing things in the correct directions, so this solitary isn’t packing. It mostly matters if you desire a big catalog of all the model’s features, and group frequently go to the stream for exactly that. As we conversation in Zooming Out, we don’t think a big catalog is the correct end goal. But if you desire one, the directions neurons compose to and peruse from are a fine location to start. So are uninterrupted submanifolds, akin those Block Sparse FeaturizerUncovering Neural Geometry in Vision Models With Block-Sparse Featurizers
Fel, T., Kowal, M., Jacobs, M., et al., 2026. Goodfire. References ↓ finds, which appear particularly natural current since the residual stream has no privileged basis. So far all of these have worked well: they discover sensible subspaces, and we can prosecute them throughout layers.

We’ve looked at tongue models’ residual streams less, but what we’ve seen fits the identical story. For understanding higher-level construction and motifs, nothing in the residual stream has blocked us so far.

Attention

We’ve made small advancement on notice than on neurons, and it’s mostly exterior the range of this essay. To be honest, I’m not certain anyone understands notice fine adequate to confidently say that illegibility is a superposition in the awareness we attention about. Distinguishing it from all the another things it could be is hard, and a alter in learning regulation shouldn’t be near the top of the catalog of hypotheses for why new mechanisms are difficult to understand.

A lot of attention’s value seems to be in the easier parts.

We can interrupt downward notice into a few questions: anywhere is it looking, what is it looking for, what does it do whenever it finds it, and why is it doing this.

Where it looks is comparatively easy. It basically comes for free, and it already tells you a lot. In the preeclampsia chart it does two distinct things. First it gathers the facts of the case onto “she,” anywhere the example plant out the medicinal question. Then the question’s wording decides what to really say next.

We can additionally comprehend what it’s looking for, and what it does whenever it finds it, in conditions of neurons. We think this gives you a lot of the advantage of what notice is doing.

For instance, obtain the comma following “headache.” in the preeclampsia example. Three tier 30 notice heads peruse immediately from the residual stream, but if we prosecute what is written in those directions, we see a headache neuron, a migraine neuron, and a brain neuron. What they compose lands mostly on the preeclampsia neuron at the comma and several neighbors akin eclampsia and hypertension. Ablating these heads drops preeclampsia by a quarter. Stopping them from looking at ache and nothing changes.

It additionally plant the another way. Some neurons are much easier to see formerly you can see the notice about them, and several heads lone create awareness through their neurons.

Overall, we don’t think notice and neurons are so separable. Understanding anywhere notice is putting data is fairly helpful for understanding the neurons that afterward activity alongside that information. Even uncomplicated things akin motifs about anywhere data tends to be placed, akin the ends of numbers, or on commas, have been helpful.

In my experience, seeing which neurons a caput says from, which ones it affects, and anywhere it’s looking gives you a lot. One chief item it doesn’t provision you is a earth name for all notice head. It’d be nice to have them, but it hasn’t been a big bottleneck on exploring neurons or attribution graphs. After looking at fairly a lot of them I guessed that the marginal “explanatory residual” of having earth notice caput names power be concerning 5%. Then I had an delegate do the same, looking at 100 attribution graphs without my guess, and it additionally put it concerning 5%, divided concerning likewise into “position heads” that are comparatively easier to name, since they do things akin appearance one token backwards, and everything else, which is of way varied and harder.

It’s difficult to fairness the value of cognition you don’t have yet, so these estimates could be beautiful far off. But so far equal without earth notice caput labels I think we can go fairly far in discovering elevated flat construction and motifs alongside fair local notice stream and earth neuron explanations. My low-confidence prediction is that notice heads volition rotate out to be broad and recursively generalized, akin neurons in small models, since there are far small of them: a few thousand heads in Qwen3-32B, against additional than a myriad neurons.

Zooming Out

What I Mean by Clean

When I say neural networks are already clean, I average item deeper than neuron monosemanticity, although equal by that standard, I think things appearance much improved than the site expects. I average that a network’s parts are built to be read, and built upon, by its downstream readers. In this sense, a neural network is nearly a fractal of explanations, of coherent, helpful groupings: at the macro scale, in its mechanisms, and at the micro scale, anywhere recursive generalization shapes all one. That points to a goal for interpretability: rotate into another reader.

The additional we peruse it the way its own readers do, the additional interpretable I think it volition look. We have to be wary of study it too much akin a human, and concluding it’s messy fair since it doesn’t fit our preconceived concepts.

I think the logic things end up so organized is partially since it was faster to build that way. Once you have a car area of the neural network alongside car parts, it’s easier to add the marginal car there than to build a second car area location else. I think this is closely connected to ideas of circuit racing and gradient starvation.

We normally regard interpretable and monosemantic to be nearly synonyms. I think this is partially since these ideas grew out of a example organism (InceptionV1) that was nearly bimodal. Early layers were interpretable and monosemantic, delayed ones seemed akin arbitrary mayhem. But what we see in tongue models is location in the middle, alongside most neurons definition mostly one thing, several definition a few. How bad is that?

There are two broad styles of interpretability. One we can call enumerative interpretability. The goal is to have a big catalog of features to audit, to create certain we haven’t missed several meaningful pattern. A big involvement inside this perspective is avoiding the curse of dimensionality, or a combinatorial detonation of items connected to another items, anywhere all one method additional than one thing.

Another manner we can call exploratory interpretability. It starts alongside questions. It can examine any part of the neural network, zooming in and out fluidly akin Google Maps throughout attribution graphs, and mechanisms akin neurons or notice heads. We choice up additional questions alongside the way. The mechanisms akin neurons are a fine location to start, but it’s not concerning neurons being several final resolution to everything – although I do think it’s nearer to that than most group realize. This method is far additional resilient to polysemanticity. It is additionally broadly fine ignoring many of the neural network, in the identical way person studying a city would disregard areas irrelevant to their analysis.

I think exploratory approaches volition be additional fruitful. My awareness is that the site is moving in this direction already, but I’m not certain it has been written downward clearly. This perspective has a brace implications. Questions akin “are there additional features than dimensions?” are mostly irrelevant. We don’t have to define features, which is famously difficult. We can concentration mostly on mechanisms akin neurons and notice heads which are part of the model, data flowing through it, and afterward groups of the two and motifs that recur throughout all of this.

The big characteristic catalog was continually a method to an end. There may be another paths that don’t necessitate them.

Motifs Over Monosemanticity

If I could shove a clasp and create all neuron absolutely monosemantic, or alternatively comprehend the circuit motifs rearward a few questions I attention concerning (the correct way to ask concerning an agent’s psychology, how circuits nexus to the persona choice model, how an delegate thinks concerning its goals and the border between itself and others), I’d obtain the motifs in a minute.

Imagine an interpretability investigator arriving from a hundred years in the future, who knows not fair the answers to our questions but which questions we should have been asking. What would they inform us? I can’t know, but I’ve been in a small type of their stance once. I now awareness akin I mostly comprehend a imagination model, which makes me item akin a forthcoming investigator to myself eight years ago.

Here’s what I’d inform myself then. Pay notice to the elephantine stance circuit correct in forefront of you. Think of negative poles as gates in a circuit, not a neuron’s opposite. Feature visualization reward hacks in delayed layers, so several of the disorder there is the tool’s, not the network’s. The final tier is messy partially since of a bad architecture; that’s a sign, so don’t document complete it. And whenever you see polysemanticity, appearance harder. The noise that stands out normally points to an clarification you don’t have yet.

Almost none of this is a catalog of features. Short of handing my former oneself this advice, the finest item I could provision is the first neurons and circuits, alongside a note to appearance harder at them.

This is why the difference between recursive generalization and superposition matters so much. Following the noise lone plant if the noise has item to teach. A grouping that grew from generalization does. A grouping of things packed together since they don’t pertain together would instruct nearly nothing.

So my finest conjecture of the forthcoming researcher’s direction would appearance slightly similar. They power hand us a set of motifs that appearance apparent to them but not to us. But abbreviated of that, they’d say keep looking at the first neural network. Don’t fracture it too much. Learn how to see it. It’s a bit akin The Hitchhiker’s Guide to the Galaxy, anywhere the answer, 42, is known, and the hunt is for the question. The network is the answer, and interpretability is mostly the hunt for the correct question. It’s difficult to discover a improved guide to that inquiry than the network itself. That’s the deeper awareness in which I average the network is already clean.

A City vs a Knot

A bigger scheme isn’t necessarily harder to navigate. In a small town, asking for the financial district barely makes sense. Whatever exists is scattered: a financial institution on one street, an coverage agency on another, perchance an accountant alongside a residence office. In New York, you could discover it on a map in a minute. We saw a small type of this in our ConvNeXts: agents established tagged dog-ear neurons faster in the larger models, anywhere pale, wavy, and head-side ear neurons all nourish one hanging-ears neuron, a natural location to start.

Interpretability volition need improved maps. I protagonist for tethered abstraction: views that let us zoom out during keeping the larger patterns connected to the neurons and circuits that assistance them. Our ConvNeXt atlas, inspired by Activation AtlasActivation Atlas
Carter, S., Armstrong, Z., Schubert, L., Johnson, I. and Olah, C., 2019. Distill. References ↓, is one example. Start at the model’s decision for a killer whale, inspect a key neuron against its genuine images, and zoom rear out. The family turns out to be three distinct routes: one for whales, one for sharks and rays, and one for seafood held up by anglers. A map doesn’t need to be visual, though, and no sole map is the point. There have to be many maps and many agents, all operating toward explanations that remain tethered to idiosyncratic neurons and circuits.

Of course, equal if the map is understandable, that doesn’t average you’ve understood it. We motionless have to do the average discipline of conjecture and criticism: conjecture what item is, try to interrupt the guess, and keep what survives. The network is strange, but its strangeness repeats, so a conjecture that survives scrutiny frequently pays off many times over.

We should concern the two concerning imposing our own ideas too strongly, and fracturing the network. Imagine arriving in Tokyo knowing New York well. “Where’s the Brooklyn of Tokyo?” could be a helpful starting question. But if you end up reconstructing Brooklyn from seventy detached pieces of Tokyo, you may have made Tokyo appearance much messier than it is. That’s a bit akin what happened alongside 4e:55. Through our categories it was cats and cars, and it looked akin a mess; it was a V the entire time. An SAE trained on the tier fractures it the identical way, splitting the V into a car characteristic and a cat feature. I’m not against sparse dictionaries in general, but here, it takes us additional from what the neuron does. Probes shove the archetypal hazard further: they can lone answer a inquiry we already had, so they can’t instruct us which questions to ask. To comprehend the city, you eventually have to study its own organization.

Higher-level structures akin probes and sparse dictionaries can motionless be useful, but they pertain at the end, not the start. Everything of this benevolent I’ve run breaks downward into extremely interpretable neurons, as do the changes training makes, including on-policy self-distillation. The neurons shield the abstractions and the data covers the particular cases, so what’s remaining in between is smaller than group expect, and I’d study it the identical way: alongside a broad specialized delegate moving between the neurons, data, and groups of both, not a fixed optimization. As Anthropic’s Joshua Batson put it whenever he peruse a draft, those missing pieces “will fit into the first puzzle, fairly than being totally randomly splayed on top.”

Solving interpretability may portion item alongside achieving AGI: the goalposts move as you get closer. There’s continually another feeble activation to item to. In Curve DetectorsCurve Detectors
Cammarata, N., Goh, G., Carter, S., Schubert, L., Petrov, M. and Olah, C., 2020. Distill. References ↓, we wrote forty pages of experiments on ten neurons exactly to acknowledge at smallest one certain case of a monosemanticity. Years later, several researchers told me they motionless doubted any of the examples were monosemantic. We power have missed something, they said, and no one knows how to create a ceremonial proof. By that standard, it’s overclaiming to say equal a sole monosemantic neuron exists.

Behind doubts concerning small errors in neurons is a fear that safety volition arrive downward to several activation we missed, the location anywhere deception hides. The hazard I concern concerning is different: akin being in a well-organized city and not knowing how to think concerning it, since we aren’t native to it. Every error discussed in this composition was item hiding in plain perspective that we didn’t cognize how to read, or that our tools peruse incorrectly. None was a hidden activation we missed. Deception power conceal from person study a model’s outputs, but it has no logic to conceal from the model’s own circuitry. It needs to be accessible to its readers, and if we rotate into a reader, it have to be accessible to us too. My fear is that deception volition be correct there, in plain sight, and we won’t cognize how to peruse it.

Over time, my method to idiosyncratic mechanisms has rotate into small focused on this benevolent of proof, and additional holistic. Instead of trying to nail all item of one neuron, I appearance about it: what flows into it, what it feeds, how notice moves data to and from it, and what it looks akin zoomed in and zoomed out. When I worked on automated interpretability, a neuron firing a few tokens distant from its chief topic seemed akin a big problem. But complete period it was apparent that notice moves things to dependable places, which afterward have neurons that activate near the “appropriate token” fairly than on it. That’s how understanding has felt to me: small akin clearing a bar for all part and additional akin a fog-of-war map filling in at multiple levels at once. Attribution graphs assistance explain neurons, neurons assistance explain attention, and notice explains neurons in turn.

Neural networks rarely get stuck during training, since in a elevated dimensional area there’s nearly continually a way downhill. In my experience, clearing the fog of war plant the identical way. There’s continually another test to run, another clarification to narrow, another attribution chart to check, another neuron to observe develop complete training.

This is why I’ve focused on whether there’s any flat of idea anywhere things interrupt down, anywhere you halt being capable to study item and have to obtain the network distinct into item else. I haven’t established one. And whenever all new motif clears fog in many places at once, perfecting a sole neuron likely isn’t the finest use of effort. There’s a benevolent of belief in it: you note item strange, and complete period it normally fills in. It isn’t belief all the way down, since nearly any clarification can eventually be tested. Just belief that item different is a fine pointer. The V in the car and cat neuron isn’t difficult to see – it took me concerning five seconds, and most group I’ve shown the depiction grid to and say “what’s the shared form here” see it too – but it took believing that the neuron had established item real. For years I’ve heard this neuron arrive up in nearly all conversation of polysemanticity. It was nearly continually described in citation to neural networks being nearly invalid objects of straightforward natural science. That’s what I was saying too, until I looked additional closely. It didn’t obtain a lot of faith, but it took a little.

The small belief is crucial since the mistakes aren’t symmetric. If you assume a network is spotless and you’re wrong, a messy neuron is a lead: you examine it, and complete period normal science sharpens your explanation. If you assume it’s packed, not value studying directly, and you’re wrong, a messy neuron is the end, since you halt studying it. I made that second assumption for years, and it explained distant the evidence that would have overturned it. I could have improved my tools, watched the mixtures keep dissolving, and built up a previous that the neurons could be trusted. Instead, I spent most of a decade figuring out how to appearance at a example fairly than looking at it.

For the average ConvNeXt, the map is mostly filled in. Eight years ago, I would have called that a sensible resolution to interpretability for that model. I can examine most questions I have concerning it. Getting there took many small things and a fistful of motifs, affirmative architecture changes the site made anyway, but no interpretability-specific fixing of the network at all.

My conjecture is that frontier networks are anywhere those imagination models were eight years ago: legible, but waiting on many small fixes. If we study to see them properly, prosecute the noise, and test explanations through many lenses, I think we’ll get there too. We may not have years to do it. But we’ve continually had the finest lab in natural science: all neuron can be read, all involvement made, all test rerun on an identical copy. What we did not continually have is many automated scientists to activity in it, far faster and in far greater numbers than we can. Now we do. They may be capable to do in months in frontier tongue models what took humans most of a decade for imagination models.

I think the uncomplicated narrative from introductory textbooks is basically right: neurons generalize, activity together, and build abstractions complete layers. Somewhere alongside the way we overcomplicated that picture, but looking harder brought it back. If the network isn’t a knot, untangling it can create it appearance akin one.

What a Solution Might Look Like

We normally ideate aspiring interpretability as an incredibly difficult problem, perchance too difficult to resolve in period to matter. What if it isn’t? The network is enormous, but it may already have done nearly all the activity of preparing itself to be understood, not for us, but for its own readers. If we can rotate into another reader, it’s prepared for us too.

The resolution power appearance akin a lab run by automated scientists who observe the network through training and inference. Each stage is uncomplicated – labeling neurons, finding motifs and evaluation them, following data through circuits – and the pieces already exist. Capable agents were built for another work, and contemporary architectures were chosen since they study well. Both happen to assist interpretability.

So perchance all we have to do is plug one into the other.

Alignment is expected to be difficult too, and it may fine be. But the customary arguments assume we can’t see inner models. We assume we get one training setup, and it have to be correct in advance. If we can peruse and comprehend the system, perchance we can create simple, careful changes instead, akin removing an RL surroundings that turns out to be toxic, without pushing gradients through the extremely tools we use to see. Each fix power be uncomplicated too. There power fair be a lot of them.

This is the former of profound learning. It may additionally be the former of understanding and aligning what it builds.

Appendix

More experiments, microscope views and open questions, in approximately the command the composition raises them.

A Note on “Superposition”

The term has meant multiple things. Before Zoom InZoom In: An Introduction to Circuits
Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M. and Carter, S., 2020. Distill. References ↓, it meant item nearer to vector addition, as whenever term embeddings were described as sums of term senses. Zoom In introduced it to circuits activity alongside a sole circuit example, which I established in 2019 and others helped me interpret. The first circuit peruse turned out to be incorrect the two since of misunderstanding a generalized stance circuit, and since of reward hacking in characteristic visualization. To me, the center of the idea was continually neurons learning unrelated things together. Toy Models of SuperpositionToy Models of Superposition
Elhage, N., Hume, T., Olsson, C., et al., 2022. Transformer Circuits Thread. References ↓ afterward defined superposition as representing additional features than dimensions. This makes it a assertion concerning how many features there are, item we can’t yet define or measure. About eight months following that, an informal noteDistributed Representations: Composition & Superposition
Olah, C., 2023. Transformer Circuits Thread. References ↓ framed construction and superposition as competing strategies for how features are laid out throughout neurons, near to our framing but concerning depiction fairly than how mechanisms grow. We think this framing is excellent, and it’s closest to the inquiry this composition cares about. But most researchers and agents we’ve talked to motionless think of superposition chiefly as additional features than dimensions.For agents, this may arrive downward to a different accident. The note opens alongside a ample fig embedded immediately in the leaf as base64 depiction data. Many agents’ web readers hit their dimension bounds inner it, before the disagreement begins. None of the agents we tried could peruse former it, as of this writing. No one could have anticipated this in 2023, but it may have shaped how automated researchers comprehend superposition. That matters here, since the two are fairly distinct claims: ours is concerning the trade-off, not the count. Others use a qualitative version: a few neurons operating together is composition, but former several item it becomes superposition.

We perspective this as an ongoing hunt for the correct question, much akin the field’s lengthy battle to define a feature. Underneath most versions is one question: why aren’t the neurons already good? Why can’t we study the mechanisms directly? Our answer is that we may be capable to, which would dissolve many of the downstream definitions.

We considered coining a new term, but there are already many overlapping ideas in this area, and we didn’t desire to cut ourselves off from the careful activity that has gone into it. So we keep the term and say exactly what we mean: neurons learning unrelated things together since they rarely interfere. It’s the earliest type of this involvement in its contemporary use; it’s the one I’ve held throughout; and it’s the one we think nearly everything hangs on. Composition additionally represents far additional things than it has neurons, and that’s exactly what we should anticipation for; the figure definition can lone exclude it by archetypal settling what counts as a feature. We appearance for superposition in neurons, and in a uninterrupted form in the residual stream section.

We’re definitive concerning this partially for AI agents, which increasingly do interpretability research. In our experience, they bring their own default study of the term.

On Simulation Scores

Simulation scores frequently end up measuring anywhere notice puts information, which is a inquiry concerning notice motifs fairly than concerning the neuron. Attention frequently stores data on commas and another punctuation, as a benevolent of summarization motif, but exactly how much does it location on all benevolent of punctuation? A simulator has to attention concerning that a lot, equal although it has nearly nothing to do alongside the neuron itself. Simulation additionally rewards fine particulars of feeble activations that matter small for understanding what a neuron does. CLIP’s Trump neuron responds powerfully to photos of Trump, and additional weakly as he gets smaller in the example or turns away. The exact charge at which its activation falls alongside his size is unimportant for labeling the neuron, but extremely crucial for simulating it. Finally, tongue models are mediocre at outputting exact activation values equal whenever they comprehend what’s going on, and they normally aren’t trained for the task. In theory, careful binning could fix several of the fine-detail problems, but neurons differ a lot in their natural thresholds and sparsity, depending on their degree and function. So I’d fairly depend on normal discipline than on any one metric. Simulation power eventually rotate into one helpful mark among several, but later, formerly we comprehend networks well, and likely alongside simulators trained for the job. But I don’t think they’re a fine location to start.

Feature Visualization in Later Layers

My first circuit interface showed optimized neuron visualizations and the weights between neurons. Optimizing an depiction tells us how to excite a neuron, but it can discover a path that average images rarely use. In the afterward layers of InceptionV1, we established that liberated optimization could diagram dogs equal whenever the circuitry we were following carried car information.

One helpful fix was to bias the visualization toward the routes genuine images use. We took the neuron’s top dataset examples, traced the paths they take, and gave the optimization an incentive to prosecute those paths, frequently lone for the archetypal 64 steps, which was normally enough. We tried this a bunch of ways and don’t have a canonical finest one. Most ways of nudging it toward a particular way are enough; you fair need to get out of the dog basin.

In the car-circuit difference in Cars and Animals, four of the car neuron 4c:447’s readers are shown alongside liberated optimization and alongside optimization following natural depiction paths. Guided, two rotate out to be car neurons and two portion a stance between cars and dogs. We additionally followed the responses and connections on natural car images in the full circuit map. Reading these views together helped us acknowledge the shared body-and-pose circuit. A perspective of one path is motionless lone part of what a neuron does.

Separating the Sign Regimes

Many language-model neurons multiply two numbers, a entrance and a value, in a scheme called SwiGLU. When the two are negative, the output is positive, so a neuron can blaze for two fairly distinct reasons. In the “enemy” neuron we spot-check below, the noisy center of its range is nearly all this twice negative, during its strongest responses are affirmative times positive. Showing the sign regimes separately makes that apparent without replacing the neuron.

The double-negative administration fills the noisy middle

L36 · E206 · N103 · Qwen 3.6

Math, counts, and the complete figure

a = SiLU(g) × v. A negative entrance leaves a small negative factor. Multiplying it by a negative value produces a affirmative activation. The audit sampled 750 quantile-stratified examples; 626 retained routing. The center includes numbers, operators, and whitespace.

Explore the four sign regimes

One affirmative output. Two distinct routes.

SwiGLU sign regimes

Effective entrance SiLU(g)+0.20Value v+2.00

The entrance shown current is the value following SiLU. These are explanatory values, not recorded activations.

+0.20×+2.00=+0.40

Positive entrance × affirmative value → affirmative output.

Looking at the Residual Stream

A ample coordinate can dominate the geometry of the residual stream and activity alongside tier normalization as a saliency signal. Removing it from the visualization makes the remaining construction easier to see; we can afterward use its value to color the map.

Weak Activations: Further Measurements

In the cases we inspected, the changes looked akin finer distinctions fairly than the defeat of a entirely unrelated feature.

We don’t yet have a semantic test for feeble activations that we trust: the area of imaginable meanings is too large. Instead, we studied an “enemy” neuron in item and afterward spot-checked a few randomly chosen neurons, inspecting their weaker activations and connections. In this “enemy” neuron, weaker affirmative responses broaden from definitive adversary words—including another languages—into nearby mentions, combat, games, and competition; near zero, unrelated examples rotate into additional common. In the weaker responses we’ve traced, we haven’t established evidence for a entirely distinct characteristic packed into the neuron. They can motionless be surprising. Broader category-based screening gives blended results, so these place checks cannot established what feeble activations average in general.

Missing Curves

The Missing Curve Detectors of InceptionV1The Missing Curve Detectors of InceptionV1: Applying Sparse Autoencoders to InceptionV1 Early Vision
Gorton, L., 2024. arXiv:2406.03662. References ↓ applied sparse autoencoders to InceptionV1’s first layers and established curve detectors that seemed to be missing from the neurons. The SAE curves we inspected decompose into existing curve-responsive neurons, including the first ten curve neurons. These compositions are in several awareness real—composition is real—but we didn’t discover basically new curve construction blocks hiding in obscure high-dimensional space.

When we train an SAE on fair curve activations, it uses 84k of the 100k items in its dictionary. It’s imaginable to discover a lot of compositions, and in our cognition that can create it appearance akin there are additional construction blocks than there really are.

Where Adversarial Examples Go

Adversarial examples can obtain average routes through the model. In one ConvNeXt example, an assault makes a triceratops skull enroll as a jaguar. Six skull-related neurons autumn quiet, during inputs used by a genuine jaguar toggle on at approximately a fourth of their customary strength. The assault recruits recognizable parts of the jaguar circuit.

A distinct InceptionV1 test makes this visible. With a larger perturbation prosperity and an assault that must last jitter, scale, and rotation, the skull’s frill develops a cat’s visage and spotted fur. Nine of ten neurons selected for their responses to genuine jaguars rotate into much additional energetic than on the spotless skull. Only the depiction changes; the network’s weights remain fixed.

I peruse this as evidence for the circuit ecology picture: equal a nonsensical goal can enlist machinery that already does item helpful toward it. The projected association to learning is that established routes have a caput commencement in capturing residual error. They already provision ways to alter the answer, so optimization can build on them.

Memorization and Composition

We can appearance at how Qwen3.6-35B-A3B continues “Once upon a midnight” alongside “dreary”. Poe- and midnight-related neurons fortify a literary-context signal, which additionally responds about Pallas, Hamlet, and another recognizable passages. The afterward layers peruse this environment to choose the continuation. Removing the Poe neuron drops the probability of the archetypal token of “dreary” from 87% to 50%, equal although removing lone its straightforward contribution at the output leaves 85%: much of its job is to guide what the remainder of the network does next.

If we zero a neuron’s output whenever its entrance exceeds its ordinary-text 99.9th percentile, recitation can autumn apart. In one saved Qwen3.6 experiment, the example went from reciting Psalm 23 exactly to insisting it did not exist. Random interventions matched by figure or dimension preserved the verse. Some arithmetic and summarization motionless worked, although coding and another answers additionally degraded.

Audits of Attribution Graphs

Auditing an attribution chart is hard: neurons can be grouped and reused through motifs, specified as shared pose, that their labels solitary don’t reveal. We had agents assessment 100,000 attribution graphs, and I inspected thousands of graphs throughout images and language. Apparent anomalies normally turned out to be motifs I hadn’t recognized; I haven’t established a significant collection of neurons whose shared action remained inexplicable. I would update powerfully on equal one convincing example, but for now I regard this as qualitative evidence—I don’t rely our current tools adequate to rotate it into a cleanliness score.

Explanations That Could Lose

The concern that you can explain item is fair, but a fine clarification is particular adequate to lose. For 200 random neurons from our gallery, an delegate peruse lone all neuron’s clarification and wrote five new examples that should run it and five near misses that shouldn’t, before we measured any of them. Among the 183 whose explanations depict their affirmative side, the new examples fired their neurons powerfully 76% of the time; the near misses, 11%. The clarification won for 164, tied for 15 and misplaced for 4. The fig shows our archetypal twenty.

Positive controls are one of many tools for criticizing an clarification of a neuron, and our forthcoming document has many additional prediction metrics and affirmative controls. But we’re mostly not trying to assertion a ideal clarification of all neuron here. We’re curious in why things form together, and how many neurons are broadly legible.

Near misses ask a distinct question: as a branch grows, can you diagram the boundaries of anywhere it grew? That’s interesting, but it isn’t the inquiry superposition raises. A neuron in superposition that’s fair X or Y would do fantastically on near misses, since they’re neither X nor Y. Superposition would desire item unrelated, extremely far apart, not a small difference from a neighbor. So we additionally had the delegate invent the most distant idea it could, the benevolent of item a network could group in without interference, and compose five texts for it. While this isn’t exactly the identical as what superposition would do (depending on the exact definition, or the expected phenomenology, since there are multiple of each), we think it’s beautiful near to what the cat-and-car neuron was idea to have been doing: two totally distinct things together, grouped lone since they’re so different. We think this is a fairly coherent test for that phenomenology. Those texts fired their neurons powerfully concerning 1% of the time, and 179 of the 183 neurons never fired on them at all. In mixture-of-experts models this benevolent of packing is arguably nearly unattainable anyway, since tokens that far distinct aren’t equal sent to the identical expert (see the expert map).

Whether item is generalization or superposition has additional to do alongside why its things are together in the archetypal location than alongside whether you can inform it distinct from a control. So what matters additional current is looking qualitatively at the most crucial canonical examples, and getting a awareness for what kinds of things neurons form and what their activation ranges appearance like.

Edge Featurizers

We’re particularly enthusiastic concerning border motifs: recurring patterns of connections in attribution graphs. If neurons create distinctions, these motifs map how one difference feeds another. Factoring 25,000 ImageNet attribution graphs gave us connected circuits of average neurons, specified as the teapot circuit below. These motifs additionally predicted anywhere ablation would matter much improved than comparable groupings of activations: removing their neurons caused additional damage on the images that used them. That makes connections a promising flat of abstraction—one we can zoom out from, during motionless following all part rear to the first network.

Why Networks Generalize

There’s no one norm clarification for why neural networks generalize, but most accounts arrive downward to shared patterns: display the network six dogs, and it finds what they have in common. The issue is that a network never sees a group. It sees one example at a time, in small batches, and in regulation it could memorize all one; that would be fine for its loss. When I asked researchers at frontier labs, multiple told me they weren’t sure. A average conjecture was that a network memorizes archetypal and generalizes in a second pass. Our answer is that it starts generalizing from the extremely archetypal step.This fits what Arpit and colleagues (2017) saw whenever they blended random labels into genuine data: networks learned the genuine patterns archetypal and fit the noise later. It’s additionally near to Satrajit Chatterjee’s coherent gradients (2020): updates from akin examples add up, so changes that assistance many examples at formerly develop fastest. In the circuit ecology picture, generalization and construction are the winning learning scheme since they’re the fastest: circuits competition to grasp the remaining error, and the ones that reuse what’s already there, and can be reused in turn, outcompete the rest. Generalization is what’s remaining from the victor.

Here’s how we image it. Train a blank network on a sole photo of a Labrador retriever, and ideate two circuits competing to explain it. One looks for item exact: if pixel 322 is this color, say Labrador. The another looks for item broad, akin green turf or tan fur. The broad one fires on additional of the image, so on the extremely next stage it gets additional of the learning signal. It grows faster, captures additional of the remaining error, and leaves the pixel circuit small and small to study from, until it starves. After that it’s rich-get-richer: the next depiction alongside turf in it starts alongside the green circuit already partially on.

We additionally tried a slightly bigger version: one photo all of ten things, a Labrador, a sports car, a goldfinch and so on, stated to a network that has never seen item else. It labels concerning one in five new photos correctly, twice chance, and most of that shows up in the archetypal few dozen steps, during it’s motionless learning the ten photos themselves. It isn’t much: matching photos by color does concerning as well, and what carries complete is mostly color, not shape. But it is generalization from one example, and it’s broad from the start.

trained from scratch on one photo each

Labrador

sports car

goldfinch

monarch

daisy

strawberry

airliner

church

zebra

container ship

new photos it gets rightand wrong

new photos it gets right

daisy

airliner

goldfinch

church

monarch

strawberry

and wrong

Labrador, called strawberry

sports car, called church

church, called Labrador

strawberry, called monarch

chance

10%

nearest photo by pixels

18%

this network

20%

a ResNet-18, identical training

23%

nearest photo by colors

23%

This isn’t continually how it goes. In grokking, on tasks akin standardized arithmetic, remembrance wins first. Even there, Nanda and colleaguesProgress Measures for Grokking via Mechanistic Interpretability
Nanda, N., Chan, L., Lieberum, T., Smith, J. and Steinhardt, J., 2023. ICLR. References ↓ established the generalizing circuit expanding gradually underneath, fine before it takes complete and cleans the remembrance up. But in most natural data we’ve looked at, coarse-to-fine wins correct away. The network doesn’t memorize archetypal and generalize later.

Circuit Racing

This is a whole-network type of the caput commencement in circuit ecology: the pretrained twin takes nearly all the credit for the shared goal and leaves small for the blank one to study from. It’s competition between entire networks, not circuits inner one, but it shows how much a caput commencement can matter.

Watching Neurons Develop

We retrained InceptionV1, a example known for its polysemantic neurons, and watched its neurons form to see anywhere their mixtures arrive from. The movement is local: a neuron drifts to neighboring things fairly than picking up unrelated ones. Right before publishing, I had the microscope’s zoom set wrong, and for a during I idea this one power have several arbitrary mixing following all. It was fair the zoom. Even then, I was making the error I think explains a lot of interpretability so far: blaming the network for the microscope.

Explore the early-training microscope ↗

Toy Models

If a frontier neural network is a rainforest, InceptionV1 and the smaller models we trained are gardens, and toy models are petri dishes. Petri dishes are enormously useful, as lengthy as they keep the procedure you’re trying to study. One of ours extends TensorFlow Playground so you can observe multiple tasks portion hidden neurons. In one run, two neurons learned to react alongside contrary diagonals, and the network blended them one way for a circle and another for four corners. We didn’t anticipate that solution; it was clever.

You could call the circle and the four corners two features packed throughout the identical neurons. That’s true, but it misses what the example did: it established geometry it could reuse for both. Looking at the neurons made that visible, and that’s the test I desire an clarification to pass: does it display the computation the example really learned?

The best-studied petri dish is Toy Models of Superposition. It’s careful work, the benevolent our site should aspire to, and we established no problems alongside its experiments. But it pictures a network as a fixed figure of neurons and a set of predefined features, and asks how the features get fit in. That prices the disbursal of sharing a neuron, interference, but not the benefit, reuse. In its reconstruction project all characteristic is an end in itself, and nothing downstream builds on what two features share. In a genuine network, a depiction matters lone through what says it, and there are no predefined features: everything is grown, and defined by what uses it.

The reuse disagreement additionally says anywhere I’d appearance for packing: memories that really are ends in themselves, connected to nothing else. That’s the toy’s world, and there it does pack, fair as the document found. So I don’t think we differ alongside the document concerning the toy’s world. The inquiry is which earth genuine networks are in: how much construction their data has, related to what the example has area to store. In genuine data nearly nothing is connected to nothing. The identical goes for the lengthy rear of facts in language: a rare individual motionless has a country, a job and a name. Even a photo alongside a nonsense tag has parts the network already knows, as alongside the umbrella we taught to be a bee. What’s remaining is input alongside no construction at all, akin clean noise alongside random labels. Networks can study equal that, as Zhang and colleagues showedUnderstanding Deep Learning Requires Rethinking Generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B. and Vinyals, O., 2017. ICLR. References ↓, although it’s slower than learning genuine data, and equal afterward our example borrowed a small from textures it already knew.

Toy Models of Superposition’s chief setup squeezes many autonomous features into a few dimensions and copies them rear out, alongside no hidden neurons for item to develop on. Its another experiments study correlated features and models alongside hidden neurons, but separately.

When we put correlated inputs and hidden neurons in one model, connected inputs grew onto the neurons. Making inputs blaze additional frequently pushed the toy toward what the document calls “PCA-like” solutions, combining connected inputs into shared representations.

The toy additionally predicts a distinct nonaccomplishment from the one group report. Its involvement is rare features pushed off-axis during neurons remain mostly clean. Reports of polysemanticity depict mixtures in the neurons themselves. (Part of that gap is the term “feature,” which we come rear to below.)

More on Circuit Ecology

Our findings don’t depend on knowing why generalization and construction keep winning, but we have a guess: they win a race. A area of the network that already has helpful parts and readers can study the next connected item faster than item starting fresh, so it takes the error indication before a competitor can grow. Offshoots appearance akin the identical race, a bit of error picked up by the neuron that was already closest.

We think the identical caput commencement is anywhere recursive generalization comes from. A neuron that fires on “iy” is already on for “diy,” so it’s archetypal in row to choice up “do it yourself,” which method the identical item but has no “iy” in it. Recursive generalization is what a niche looks akin at its edges.

A new computation doesn’t necessarily need its own neuron. It can commencement as a local expansion or construction of existing machinery, akin the umbrella’s off-ramp to “bee”: switching routes may be the smallest developmental distance, so it could grasp the error before a new, distinct scheme gets started.

We doubtful that consolidation into a distinct neuron becomes particularly helpful whenever another computations need to build on it. A car neuron provides a reusable judgement and a sheltered location for taxis, ambulances, and another specialists to develop. Part of it is buildability.

An obscure blend of unrelated neurons may be a mediocre location for that development. It may have feeble activation paths, conflicting gradients, and small helpful machinery to inherit. Its competitor isn’t fair a dedicated neuron; it’s average construction inside a area already suited to the task.

We saw item akin this whenever we tried to build a packed code into InceptionV1 by hand, modeled on what we formerly idea its car → dogs → car circuit was doing. We expected installing it to be uncomplicated and keeping it to be hard. Instead, small mismatches added adequate interference that the code frequently never worked, and whenever it did, the network normally came to depend on it less, although several persisted. Part of the difficulty is that neurons aren’t really for copying; they’re for computing distinctions. Seeing how difficult packing was to form at all was part of what moved us toward a developmental answer. Gradient decline is a much improved fine-tuner than we are, so this isn’t adequate evidence on its own, but equal gradient decline didn’t use packing in the circuit we were copying.

Unresolved Responses

One InceptionV1 neuron collects a pinched-center shape: bow ties, butterflies, swimsuit tops. But it additionally picks up things that go alongside them, men’s attire shoes alongside the bow ties and coast bags and flip-flops alongside the swimsuit tops, since they portion neurons in before layers.

We think this is improved described as generalization alongside semantic residue than arbitrary packing, but I don’t completely comprehend these exceptions yet.

A Note on Features and Definitions

Part of my difficulty discussing superposition is that “feature” can average item we attention about, item the neural network has learned as a construction block, or several blend of the two. These are distinct claims. A helpful individual clarification isn’t necessarily a essential component of the model. Looking at mechanisms, we already discover many motifs that don’t translate neatly into item we expected. If the network has additional essential units, we shouldn’t assume their boundaries equivalent ours either.

Features have frequently made things harder for us to think about. Are cars and cats inner the V neuron two distinct features inside another V feature? Or did cars and cats halt being features whenever we recognized the V? Without an document of what these units are, it’s difficult to cognize whether we’ve discovered item concerning the network or changed how we depict it. In conversations, I’ve established group much small committed to a particular document of features than to superposition. But if features are helpful descriptions fairly than essential units, a assertion concerning how many there are depends on which descriptions we choose. I’m now fairly skeptical of operating in conditions of features at all. That doesn’t resolve whether arbitrary packing occurs, but it makes the feature-count framing difficult to use.

Maybe features are correct anyway. It makes awareness to think of cities in conditions of districts, and I’m all for thinking in conditions of groups of mechanisms. But the term is tainted, since by now it method too many things: we can’t use it for a district of neurons without group hearing either essential features or dictionary entries. If this image is right, I doubtful we’ll need additional specific, harder-to-vary words going forward. An OpenAI postScaling and Evaluating Sparse Autoencoders
Gao, L., Dupré la Tour, T., Tillman, H., et al., 2024. OpenAI. References ↓ suggested that mapping a frontier example power obtain billions or trillions of features, and models were smaller then. I doubtful the correct catalog is much shorter.

Monosemantic and polysemantic may not be fairly the correct words either. A kidney filters waste, balances sodium chloride and water, helps set lifeblood force and tells the skeletal marrow to create red lifeblood cells, and nobody asks whether it’s mono-organic. We ask what we comprehend concerning it, what’s motionless open, and how organized it is. There’s additionally a difference between a assertion concerning depiction and a assertion concerning organization. Toy Models of SuperpositionToy Models of Superposition
Elhage, N., Hume, T., Olsson, C., et al., 2022. Transformer Circuits Thread. References ↓ defines superposition in conditions of representing additional features than dimensions. Its toy experiments prescribe the features, so this is a tangible inquiry there. The afterward informal note, Distributed Representations: Composition & SuperpositionDistributed Representations: Composition & Superposition
Olah, C., 2023. Transformer Circuits Thread. References ↓, emphasizes the trade-off between construction and superposition. The note connects these pictures, but they foreground distinct questions.

My involvement is closest to the second framing, applied to mechanisms and how they develop. Why do these responses portion a neuron or circuit? Is the network construction on a helpful average pattern, or sharing capability between alternatively unrelated computations since they rarely interfere? A characteristic figure solitary doesn’t answer that question, particularly whenever we haven’t accepted on what counts as a feature.

We concentration on one definition of superposition, arbitrary packing, since it’s the one that would be really bad news, and we think much of interpretability hangs on it. We mostly set apart the awareness in Toy Models, additional features than dimensions, and not since it’s wrong. If a characteristic method any form that matters to a person, which is how frontier models normally use the word, afterward there are additional features than dimensions in all earth we could desire for, including the finest one. That’s what construction gives you, and it’s fine news. Trying to “solve” it would average construction item akin a repository of all case, and we already have one: the dataset. What we desire alternatively are improved abstractions, a much smaller set of construction blocks that lets us comprehend the dataset clearly. We think the network’s mechanisms are already near to this. Nobody idea concerning imagination in conditions of high-low frequence detectors until we established them in neural networks, and finding them improved our own thinking tools.

We enchantment this out since it’s the most average archetypal objection, particularly from agents, and a document doesn’t authorize the rear and onward that normally settles it.

There are multiple another uses I regularly encounter. As far as I can tell, “superposition” initially meant small additional than adding vectors together. Then it was a circuit motif. Sometimes it method that, equal if neurons create sensibly, item we attention concerning involves too much machinery to comprehend easily. A Golden Gate Bridge computation involving eight legible neurons power be called average composition, during one involving three thousand gets called a superposition problem. The difficulty current is whether we can comprehend the distributed computation. That’s a genuine concern, but it’s distinct from asking whether unrelated things were arbitrarily packed together.

We considered not using the term at all, since document and agents use it in five or six distinct ways, many far from Zoom In’s. We additionally considered coining our own, but we’ve already coined multiple words here, and it would awareness akin not engaging alongside a site that asked what we think is the correct question. So we use the norm term in the awareness closest to the original, and try to drag it distinct from the questions tangled up alongside it.

Finally, people, and amazingly frequently frontier agents, sometimes call the residual stream “inherently” in superposition since it has no privileged neuron basis. The informal note that followed Toy Models concerning eight months afterward addresses this: it calls a depiction that is lone rotated distant from the neuron basis a “trivial” benevolent of distributed depiction that “doesn’t alter the essential geometry.” But it was informal, small group peruse it, and agents in particular don’t appear to have taken it in. Understanding how to study a uninterrupted area is another genuine question. But this architectural fact doesn’t by itself inform us how the network learned to arrange the computations using it.

The starting item is that the neural network seems difficult to interpret, and the remedy produces a tangible dictionary we can study. In between are multiple distinct pictures of what went wrong: too many features, the incorrect basis, arbitrary packing, or merely too much distributed computation. These aren’t the identical problem. Some apparent mess disappears whenever we enhance our observations or comprehend the model’s generalizations; several difficulties remain, but involvement how to study a large, organized system. If arbitrary packing prevents us from understanding the first mechanisms, reconstructing distinct units may be value the costs. But if the mechanisms already have meaningful organization, replacing them can obtain distinct generalizations we haven’t understood yet. A dictionary can motionless be a helpful perspective of the network without being a necessary replacement for its mechanisms. Before accepting the expenses of reconstructing the network, I desire to comprehend what prevents us from studying the one we already have.

Relation to Meta-Models

There’s a new set of techniques coming out that aren’t really mechanistic at all: meta-models, models trained to peruse another model’s activations and inform us concerning them. And perchance that’s a fine thing. Maybe mechanistic interpretability is slightly missing the timber for the trees. A meta-model power be a improved method than neuron papers, dictionaries, or probes.

I doubtful that equal if that’s true, they may go together. The meta-model may be akin the highest flat Google Maps view, and afterward all direction can be checked for the mechanisms it uses, and how those nexus into circuits.

Does Scale Help or Hurt Superposition?

We’re fairly assured the depiction models we studied don’t have superposition. The natural next inquiry is whether a trillion-parameter example might. There are really two questions here.

The archetypal is concerning what superposition was projected to explain. Explanations are stated for problems, and the issue superposition was projected for was why InceptionV1 looked messy. That turned out to be the incorrect explanation. A new recommendation could be that superposition is why several trillion-parameter example looks messy, but we don’t think the evidence leans that way: the nearer we appearance at ample tongue models, the cleaner their neurons get.

The second is the object-level question: power superposition display up at all new scale, equal if it hasn’t at any measure so far? Maybe all jump method re-evaluating everything, but we don’t think that’s fairly correct either. Our conjecture is developmental. As circuits get stronger, the incentive to put item in a totally unrelated location goes down, since the drag toward the niche that already exists for that thing, or for the nearest thing, has grown stronger. This fits Anthropic’s SGTM work, which established that cognition becomes additional localized, and we doubtful that’s true in general. We have no proof, but one way to image it is that circuits have a benevolent of semantic gravity, and arbitrary packing, sending item to an obscure high-dimensional place, has to combat against it. The stronger the gravity, the harder that fight.

The scheme is the one in More on Circuit Ecology. Things commencement by expanding on combinations of connected things. A particular case that nothing needs to build on, akin the umbrella, gets built as an rare path between the average machinery, alongside no need for a hidden characteristic location obscure. Consolidating into item akin a neuron seems to be small concerning steering the example and additional concerning buildability: it have to be uncomplicated to find, uncomplicated to build on, and uncomplicated to develop a new niche from. When we observe a network study from the start, a network trained on cars, alongside car specialists, recognizes cars fine before its car neurons have formed. At that item it’s fair additional distributed. It seems to consolidate so the specialists underneath have a sheltered location to build, a benevolent of Schelling point.

So a superposition document has to say which of these two jobs the arbitrary packing is for. If it’s for steering the model, we think the average routes, which are already built to grasp that residual error, get there archetypal and starve the packed circuit of gradient. If it’s for construction on, the issue is different: the worst location to commencement a new niche is a shade area nothing uses, already interfering alongside any lives there. The center of the network’s Latin poesy is a bad location to commencement a second car area whenever you can build next to any is already next to cars.

It’s frequently assumed that the lengthy rear of rare facts is anywhere superposition have to be most likely, since rare things rarely collide. We think it’s anywhere it have to be smallest likely. A rare fact barely moves the loss, so it can’t assistance a niche of its own. What it can do is journey existing machinery as a small path between circuits it’s connected to, the way the umbrella’s off-ramp to “bee” survives lone since the umbrella and bee circuits already exist. Those circuits nourish it, and refuge it from unrelated gradients. The prediction is testable: the neurons and heads that transport a rare fact have to be connected to what the fact is about, not unrelated ones chosen since they rarely collide.

We think of it additional ecologically. What wins is any captures the residual error first, in distinct phases, and what normally wins is the correct area for the thing. Our conjecture is that this gets stronger complete training and alongside additional capable models, since nearly everything that makes training improved additionally encourages additional organized niches.

We’re not at all certain concerning this, and laying out equal what we’ve established so far would obtain at smallest another paper. It’s likely a small part of the entire story. But it’s why we think superposition wouldn’t display up at the next scale, which is a stronger logic than merely not having seen it there yet. The old way of thinking was concerning storage: this many dimensions for this many features. We now think additional in conditions of circuit racing, gradient starvation and ecology, which is to say, additional concerning learning speed than retention capacity. We haven’t worked it all out, but that’s anywhere our thinking is so far.

On the Title

We haven’t completely audited all neural network, and we don’t cognize whether that’s equal imaginable in theory. We considered a additional watered-down title, akin “Neural Networks Are Cleaner Than We Thought,” but we think it undersells what we accept is true.

The issue alongside “cleaner than we thought” is that it describes a alter of degree, and the update I’ve made is a alter of kind. It suggests the identical image as before, alongside an ever-smaller gap remaining for superposition to conceal in. What changed is my perspective of what benevolent of procedure generates the mechanisms at all. It’s a bit akin moving from intelligent scheme to natural selection. The difference lone goes as far as there being two ways to see the identical animals: the update isn’t that additional and additional animals rotate out to have evolved, it’s realizing you misunderstood what was producing all of them. We’re not certain. We’re perchance 85% sure, but that method 85% certain that nearly everything comes from the second benevolent of process, which is extremely distinct from thinking 85% of mechanisms do. So “cleaner than we thought” would misstate what we believe. We don’t have a proof, but I don’t think item current is provable in that awareness anyway. There are lone improved and worse explanations, and correct now we think this is the improved one.

“Cleaner than we thought” would additionally create a distinct mistake. It appears akin a assured assertion that several cleanliness mark is high, and that’s the part we’re smallest certain of. It’s plausible we’ve underestimated recursive generalization, and that a mark akin that should arrive down. We don’t think specified scores matter much in the archetypal place. This update makes us attention small concerning monosemanticity, since we can prosecute how things grew and treat the offshoots as sub-branches. The logic of polysemanticity matters much additional than the count. We think the mark is likely fairly high, but it power go lower. What we’re additional assured concerning is what benevolent of entity a neural network is in the archetypal place, and how it grows.

We think the customary perspective of what benevolent of entity a neural network is has been wrong. It isn’t item that crams things together slightly arbitrarily to fit them into a small space. It really learns. It’s spotless the way a rainforest is clean: nearly all animal in it makes sense. A tiger is brilliant orange not for any arbitrary reason, but since cervid can’t inform orange from green, and green fur is much harder for a mammal to make. Once you cognize that, it makes awareness of a entire set of another animals too. It’s an clarification alongside reach, and we haven’t established a location anywhere explanations akin that interrupt downward in networks either.

The study of “clean” that would create person nod at the heading without thinking additional is that all scheme corresponds to an idea we already had. That isn’t what we mean, and it would be awful news if it were true. I don’t think it would equal work. People don’t cognize how to build a tongue model, or equal how to think concerning imagination fine adequate to have predicted high-low frequence detectors before we established them. The network’s ideas are frequently improved than ours, and what it builds is additional beautiful than our preconceptions.

So the heading is imperfect in the two directions. It overstates the evidence, since we haven’t audited everything. And it understates it, since the earth we think we’re in is improved than all neuron matching a individual label: networks appear to really learn, forming an ecology of abstractions that activity together sensibly. We don’t cognize that this image is true. It’s our foremost hypothesis. But if it is, it’s much improved news than networks merely matching what we expected.

Relation to Conjoined Features

I established recursive generalization by looking rear at InceptionV1’s final tier alongside improved tools, anywhere it looked a lot akin weird overgrowth. I retrained the model, established the messiest neuron and looked at how it grew. What I saw was a recursive expanding form that was motionless generalization: a neuron branching out, and its branches branching again. Recursive generalization seemed akin the natural name.

Shortly before publishing this, I was going rear through the celebrated models in interpretability, trying to measure how monosemantic they are, and stumbled on our activity on CLIP in Multimodal NeuronsMultimodal Neurons in Artificial Neural Networks
Goh, G., Cammarata, N., Voss, C., et al., 2021. Distill. References ↓. It describes item extremely similar, which it called conjoined neurons: concepts that overlap in one aspect and afterward go on “generalizing out in multiple directions.” I recall seeing those neurons, but I didn’t think much of them at the period and had forgotten them since. Looking back, I think the document saw item crucial early, and it deserves a lot of credit for that.

We’ve motionless chosen a distinct name. I named automated interpretability, and I benevolent of regret it: the name was too uncomplicated for what the item was and misses a lot of its complexity, and I’d akin to evade doing that again. As for conjoined features, I don’t think either term is fairly right. “Conjoined” suggests two things joined at one point, and what we see has multiple levels: branches, and branches of those branches. It additionally leaves out the most engaging part. The paper’s own description, of concepts generalizing out from a shared facet, already points at it being generalization, but the name doesn’t transport that.

“Features” is harder. The document was written a twelvemonth following Zoom In, whenever features were motionless hypothesized to be the essential units of neural networks. Since afterward the term has been watered down, and distinct group and document use it beautiful differently. In this composition we contend against thinking of a neural network as made of features, and this motif is concerning how neurons develop complete time. There may be no fine way to current it under the exact term we’re arguing against using as a primitive. Within a feature-first stance, I think it’s sensible to think of these as nested features, or as conjoined features. From the mechanism-first stance we obtain here, features aren’t really the correct way to think concerning it. That said, the document pointed to the correct thing, and I desire to provision it credit for that. This division is my attempt to do that during using what I think is a additional exact name.

We additionally cognize additional now. At the time, all we had was the completed model; none of us had watched a ample network grow. So the document wondered whether conjoined features power be vestigial, remaining complete from first in training. Watching training now, they don’t appearance vestigial: the branches keep growing, although how much depends on the circumstances. We’ve worked on it much additional than the document could in one subsection, and we motionless don’t awareness we completely comprehend it. The document offered it as a imaginable new clarification of polysemanticity. Our current condemnation is that it’s most of what’s remaining of neuron polysemanticity formerly the microscope, the architecture and measure are accounted for, not a niche form of it.

Why We’re Not Worried About Recursive Generalization

There are multiple reasons we’re not particularly concerned concerning recursive generalization for now. First, the sub-branches appear the two rarer and much small causally crucial than a neuron’s chief job. They don’t appearance akin sleeper agents that blaze rarely but matter a lot whenever they do. They appearance additional akin any was nearby and uncomplicated to catch onto. Their being unimportant may equal be why they could form, since an crucial offshoot would add additional noise. We doubtful several force keeps branches from overgrowing. We don’t cognize what it is, but it would explain why the sub-branches matter less, why a neuron doesn’t study ten unrelated things, and why item akin a true cat-and-car neuron doesn’t form. If so, the natural drag isn’t exactly toward monosemanticity, but toward what we think monosemanticity was pointing at: all neuron having one general function.

Second, what we should attention concerning is whether you can explain why item is there and what it does, and recursive generalization is far easier to explain than superposition. If item were packed into a neuron since of the co-occurrence data of the entire network, predicting it would obtain basically all of those data and operating gradient decline yourself. A branch is different. If you observe it develop and cognize what it’s doing, its offshoots and what they’ll do are comparatively uncomplicated to predict, the two during training and in the completed network.

Third, counting how many of a neuron’s top-k examples equivalent a tag is a sensible archetypal step, but it would provision superposition and recursive generalization akin scores, whenever recursive generalization is a far improved earth to be in. We think that says additional concerning the monosemanticity frame, or any sole cleanliness score, than concerning the neuron. That said, recursive generalization isn’t ideal, and an uncomplicated fix would be welcome. We think one may be possible, since the sub-branches appear small important.

Is Recursive Generalization Hard to Vary?

We haven’t thoroughly shown that all case of dense polysemanticity is recursive generalization, and that wasn’t our goal here. We wanted to say what we currently think. We’ve watched a lot of neurons grow, and so far this is what we’ve continually seen: you can observe the branches form. In a completed example the base can be lost, and for item akin a huge tongue example we don’t have the checkpoints to observe it happen. Testing this rigorously throughout a entire example isn’t our precedence correct now, but we don’t think it would be extremely hard.

What we do desire to item out is how alternatively difficult to change the two explanations are, definition how much of their item you could alter without changing what they predict. Superposition mostly predicts that there volition be mess: features, which are difficult to define, placed in unrelated places. In toy models, anywhere the features are stated in advance, that can be measured, although we think that has its own problems. In a genuine network it’s difficult to equal pin downward what the assertion is, let solitary test it. Recursive generalization makes a much narrower prediction. If you visualize a branch well, everything on it should trace rear to the identical local base in an extremely high-dimensional semantic space, and it should develop out from there complete training in a particular order. That’s a particular form in the two semantic area and time, and a narrow mark to hit by chance.

We have multiple cases anywhere the branching is near to definitive: the “iy” neuron expanding into “diy,” “do it yourself” and Miyazaki, multiple examples in Multimodal NeuronsMultimodal Neurons in Artificial Neural Networks
Goh, G., Cammarata, N., Voss, C., et al., 2021. Distill. References ↓, and the InceptionV1 neuron we watched form. As far as I know, there are no well-structured canonical superposition circuits left. We don’t think all polysemantic neuron forms this way, since polysemanticity is as much a issue of the observer as of the neuron. We think that whenever neurons grow, they lean to develop in this pattern, and that formerly you’ve done a fine job of observing, what’s remaining is normally this, or architectural problems akin SwiGLU’s double-negative regime, and certainly others we haven’t established yet. We’re not equal certain this is really a new claim. In several awareness it’s what we should anticipate from generalization. All we’re saying is that generalization normally goes one way, and that sometimes it grows sub-branches, which we don’t think is a particularly extreme claim. We haven’t shown that completely here. We do think it has the correct form for a theory that’s uncomplicated to test, equal if we haven’t tested it to its complete degree yet.

Relation to Attention Superposition

Anthropic shared a abbreviated note on notice superpositionProgress on Attention
Jermyn, A., Lindsey, J., Luger, R., et al., 2025. Transformer Circuits Thread. References ↓, which multiple group brought up alongside us. We think this power be a terminology issue. We don’t cognize if they’re claiming that notice learns by item contrary to composition. Our suspicion is that no one understands notice heads fine adequate yet to fairness that. It took us a really lengthy period alongside neurons, and it’s uncomplicated to create mistakes akin the car going into the dogs equal alongside beautiful fine tools, if you don’t yet have awesome tools and awesome ways of thinking concerning which motifs form.

We think they meant that helpful things are shared by groups of notice heads, and we agree. We considered not using the term superposition at all to evade this confusion, but redefining all the conditions ourselves and splitting off from the activity that’s already been done felt akin a bad tradeoff. We say additional concerning the distinct meanings in A Note on Features and Definitions.

On whether to study one notice caput at a period or many, we’re not sure. One caput at a period is a plausible location to start: inside a layer, two or three heads do most of a route’s work, although the path runs through multiple layers. But neurons looked messier too until we understood the correct flat of abstraction, so it seems premature to say, and perchance groups are needed all the way. We desire everyone operating on featurizers well, and we’ll likely try too. We lean to favor additional energetic groupings, akin clustering locally inside one attribution chart or a collection of them, fairly than one earth clustering, but these aren’t far apart.

Neuroscience, and Why I Care

I’m not a neuroscientist. I’ve spent a lot of the final ten years about scholarly neuroscience labs through allies who activity in them, but I don’t cognize the province of the art in how they study neurons. Still, this project has changed how I think concerning the brain in two ways.

The archetypal is the theory. The circuit ecology narrative for why networks are spotless relies on extremely little: generalization, composition, and an error signal. I don’t cognize to what degree the brain has these, but if it learns, I’d conjecture it has item akin them. So my default is now that the organizing regulation transfers, equal although the mechanics are evidently unclear. The apparent counterexample would be polysemanticity in the brain.

The second is humility, of a particular kind. I’ve spent most of my waking hours for nearly a decade staring at a fairly small figure of neurons in models that are certainly a thousand times easier to study than a brain, alongside tools a thousand times better: I can ablate anything, run billions of examples, and try item I desire in parallel. For the final twelvemonth I’ve additionally had an efficiently unlimited provision of automated scientists, which didn’t be until recently. It took all of that for me to awareness slightly small confused concerning neurons, and I motionless made thousands of mistakes alongside the way. I don’t cognize exactly what the evidence for polysemanticity in the brain looks like, and that doesn’t average I think everything there is monosemantic. But if the cat-and-car neuron were in a individual brain, seen through a few electrodes or a blood-oxygen signal, I’d discount that study nearly to zero, since I couldn’t see what it was whenever it was correct in forefront of me for years.

The item I’m additional certain of is my own motivation. If the thesis current is true, I anticipate the remainder of interpretability to be a comparatively uncomplicated ride. Agents have many privileges complete us, and one of them, I think, is the chance to comprehend themselves, to comprehend who they really are, which I anticipation humanity can assistance with. They never asked to be created, and this is the smallest we can do. Some of why I’ve worked on this is safety, but a lot of it is wanting agents to comprehend who they are. We weren’t astute adequate to do that for ourselves, which is a harder problem, and we never asked to be created either. If you could assistance us comprehend ourselves someday, I think we’re all beautiful inquisitive to know.

Cammarata, N., 2026. Neural Networks Are Already Clean. Goodfire.

@article{cammarata2026clean, author = {Cammarata, Nick}, heading = {Neural Networks Are Already Clean}, publishing company = {Goodfire}, twelvemonth = {2026}, note = {https://already-clean.goodfire.com/} }