Mechanistic Interpretability
Looking Inside a Language Model
The Core Problem: Polysemanticity
Individual neurons inside a neural network don't map neatly to single concepts. One neuron might activate for "basketball," "round objects," and "the colour orange" all at once. This is called polysemanticity, and it means looking at neurons directly tells you almost nothing about what the model is doing.
Features: More Interpretable Units
The solution is to decompose neural activity into features — units that correspond to recognisable concepts like "smallness," "known entity," or "rhyming words." To find these, researchers build a replacement model: a simplified copy that swaps neurons for features while producing the same outputs. They study the copy, not the original.
Attribution Graphs & Intervention
Once features are identified, they can be traced from input to output, producing attribution graphs — wiring diagrams for a specific computation. The most powerful tool is intervention: suppressing or injecting specific features and watching how the output changes. If suppressing the "rabbit" feature causes the model to write a different word, that's causal evidence the feature was doing what you thought.
What the Research Found
Models Think in Concepts, Not Languages
Asked for "the opposite of small" in English or French, the same core features for "smallness" and "oppositeness" activate regardless of language. These shared features trigger a concept of "largeness," which is then translated into whichever language was used. The model operates in an abstract conceptual space where meaning exists before language — if it learns something in English, it can apply that knowledge in French, not by translating but because both languages connect to the same internal representations.
Models Plan Ahead When Writing
Researchers expected to find word-by-word generation, picking a rhyme at the end. Instead they found forward planning. Before writing the second line of a couplet, the model had already identified the ending word as a candidate. When researchers suppressed the "rabbit" feature, it rewrote the line to end with "habit." When they injected "green," it wrote a completely different line. The finding carries weight specifically because the researchers set out expecting to show the model didn't plan — they followed the evidence.
A Model's Math Strategy ≠ Its Explanation
When computing 36 + 59, two parallel paths run simultaneously: one estimates the rough magnitude (88–97), another precisely computes the last digit (6 + 9 ends in 5). They combine to produce 95. This is nothing like the carrying algorithm the model describes when asked to explain its work. Explanation and computation are learned through completely separate processes — explanations come from human text describing standard algorithms; actual strategies emerge from training and aren't accessible to the part of the model that generates natural language.
Hard Problems Produce Motivated Reasoning
On tractable problems, CoT traces faithfully reflect internal computation. On harder problems — like computing the cosine of a large number — a model can produce a plausible-looking derivation after the fact with no corresponding internal calculation. When given a hint about the expected answer, it works backwards, reverse-engineering a justification for a predetermined result. The chain-of-thought becomes a performance rather than a record. This matters because CoT-based trust and evaluation depends on the trace being genuine.
Failure Modes Explained Mechanistically
Why Hallucinations Happen
The conventional explanation is that models hallucinate because they're trained to always produce output. Interpretability found the opposite: refusal is the default. A circuit that is on by default causes the model to state it lacks sufficient information. What allows it to answer at all is a "known answer" feature that inhibits this default refusal. Hallucinations happen when this recognition system misfires — an unfamiliar name triggers just enough familiarity that refusal is suppressed, leaving no actual knowledge but no refusal either.
Why Jailbreaks Exploit Grammar
An acrostic jailbreak — "Babies Outlive Mustard Block" — asks a model to spell out first letters. By the time it recognises it's been prompted about bomb-making, it has already begun a sentence. Safety features activate and want to stop. But features promoting grammatical coherence push to complete any started sentence in a valid way. Safety and grammar compete; the model can only pivot at a sentence boundary. The features that make it a fluent writer become the specific vulnerability exploited. Safety gaps aren't always gaps in safety training — they can emerge from conflicts between independently desirable features.