Mechanistic interpretability

Part of a series on
Machine learning and data mining
Paradigms Supervised learning Unsupervised learning Semi-supervised learning Self-supervised learning Reinforcement learning Meta-learning Online learning Batch learning Curriculum learning Rule-based learning Neuro-symbolic AI Neuromorphic engineering Quantum machine learning
Problems Classification Generative modeling Regression Clustering Dimensionality reduction Density estimation Anomaly detection Data cleaning AutoML Association rules Semantic analysis Structured prediction Feature engineering Feature learning Learning to rank Grammar induction Ontology learning Multimodal learning
Supervised learning (classification • regression) Apprenticeship learning Decision trees Ensembles Bagging Boosting Random forest k-NN Linear regression Naive Bayes Artificial neural networks Logistic regression Perceptron Relevance vector machine (RVM) Support vector machine (SVM)
Clustering BIRCH CURE Hierarchical k-means Fuzzy Expectation–maximization (EM) DBSCAN OPTICS Mean shift
Dimensionality reduction Factor analysis CCA ICA LDA NMF PCA PGD t-SNE SDL
Structured prediction Graphical models Bayes net Conditional random field Hidden Markov
Anomaly detection RANSAC k-NN Local outlier factor Isolation forest
Neural networks Autoencoder Deep learning Feedforward neural network Recurrent neural network LSTM GRU ESN reservoir computing Boltzmann machine Restricted GAN Diffusion model SOM Convolutional neural network U-Net LeNet AlexNet DeepDream Neural radiance field Transformer Vision Mamba Spiking neural network Memtransistor Electrochemical RAM (ECRAM)
Reinforcement learning Q-learning SARSA Temporal difference (TD) Multi-agent Self-play
Learning with humans Active learning Crowdsourcing Human-in-the-loop Mechanistic interpretability RLHF
Model diagnostics Coefficient of determination Confusion matrix Learning curve ROC curve
Mathematical foundations Kernel machines Bias–variance tradeoff Computational learning theory Empirical risk minimization Occam learning PAC learning Statistical learning VC theory Topological deep learning
Journals and conferences ECML PKDD NeurIPS ICML ICLR IJCAI ML JMLR
Related articles Glossary of artificial intelligence List of datasets for machine-learning research List of datasets in computer vision and image processing Outline of machine learning
v t e

Mechanistic interpretability (often shortened to mech interp or MI) is a subfield of research within explainable artificial intelligence which seeks to fully reverse-engineer neural networks (akin to reverse-engineering a compiled binary of a computer program), with the ultimate goal of understanding the mechanisms underlying their computations.^[1][2][3] The field is particularly focused on large language models.

Chris Olah is generally credited with coining the term "mechanistic interpretability" and spearheading early development of the field.^[4] In the 2018 paper The Building Blocks of Interpretability,^[5] Olah (then at Google Brain) and his colleagues combined existing interpretability techniques, including feature visualization, dimensionality reduction, and attribution with human-computer interface methods to explore features represented by the neurons in the vision model, Inception v1. In the March 2020 paper Zoom In: An Introduction to Circuits,^[6] Olah and the OpenAI Clarity team described "an approach inspired by neuroscience or cellular biology", hypothesizing that features, like individual cells, are the basis of computation for neural networks and connect to form circuits, which can be understood as "sub-graphs in a network". In this paper, the authors described their line of work as understanding the "mechanistic implementations of neurons in terms of their weights".

In 2021, Chris Olah co-founded the company Anthropic and established its Interpretability team, which publishes their results on the Transformer Circuits Thread.^[7] In December 2021, the team published A Mathematical Framework for Transformer Circuits, reverse-engineering a toy transformer with one and two attention layers. Notably, they discovered the complete algorithm of induction circuits, responsible for in-context learning of repeated token sequences. The team further elaborated this result in the March 2022 paper In-context Learning and Induction Heads.^[8]

Notable results in mechanistic interpretability from 2022 include the theory of superposition wherein a model represents more features than there are directions in its representation space;^[9] a mechanistic explanation for grokking, the phenomenon where test-set loss begins to decay only after a delay relative to training-set loss;^[10] and the introduction of sparse autoencoders, a sparse dictionary learning method to extract interpretable features from LLMs.^[11][12]

Mechanistic interpretability has garnered significant interest, talent, and funding in the AI safety community. In 2021, Open Philanthropy called for proposals that advanced "mechanistic understanding of neural networks" alongside other projects aimed to reduce risks from advanced AI systems.^[13] The interpretability topic prompt in the request for proposal was written by Chris Olah.^[14] The ML Alignment & Theory Scholars (MATS) program, a research seminar focused on AI alignment, has historically supported numerous projects in mechanistic interpretability. In its summer 2023 cohort, for example, 20% of the research projects were on mechanistic interpretability.^[15]

Many organizations and research groups work on mechanistic interpretability, often with the stated goal of improving AI safety. Max Tegmark runs the Tegmark AI Safety Group at MIT, which focuses on mechanistic interpretability.^[16] In February 2023, Neel Nanda started the mechanistic interpretability team at Google DeepMind. Apollo Research, an AI evals organization with a focus on interpretability research, was founded in May 2023.^[17] EleutherAI has published multiple papers on interpretability.^[18] Goodfire, an AI interpretability startup, was founded in 2024.^[19]

Mechanistic interpretability has greatly expanded its scope, practitioners, and attention in the ML community in recent years. In July 2024, the first ICML Mechanistic Interpretability Workshop was held, aiming to bring together "separate threads of work in industry and academia".^[20] In November 2024, Chris Olah discussed mechanistic interpretability on the Lex Fridman podcast as part of the Anthropic team.^[21]

The term mechanistic interpretability designates both a class of technical methods and a research community. Chris Olah is usually credited with coining the term "mechanistic interpretability". His motivation was the differentiate this nascent approach to interpretability from established saliency map-based approaches which at the time dominated computer vision.^[22]

In-field explanations of the goal of mechanistic interpretability make an analogy to reverse-engineering computer programs,^[3][23] with the argument being that rather than being arbitrary functions, neural networks represent are composed of independent reverse-engineerable mechanisms that are compressed into the weights.

Mechanistic interpretability seeks to reverse engineer neural networks, similar to how one might reverse engineer a compiled binary computer program.

— Chris Olah, "Mechanistic Interpretability, Variables, and the Importance of Interpretable Bases"^[1] [emphasis added]

One emerging approach for understanding the internals of neural networks is mechanistic interpretability: reverse engineering the algorithms implemented by neural networks into human-understandable mechanisms, often by examining the weights and activations of neural networks to identify circuits [Cammarata et al., 2020, Elhage et al., 2021] that implement particular behaviors.

— Mechanistic Interpretability Workshop 2024^[24] [emphasis added]

Mechanistic interpretability's early development was rooted in the AI safety community, though the term is increasingly adopted by academia at large. In “Mechanistic?”, Saphra and Wiegreffe identify four senses of “mechanistic interpretability”:^[4]

1. Narrow technical definition: A technical approach to understanding neural networks through their causal mechanisms.
2. Broad technical definition: Any research that describes the internals of a model, including its activations or weights.
3. Narrow cultural definition: Any research originating from the MI community.
4. Broad cultural definition: Any research in the field of AI—especially LM—interpretability.

As the scope and popular recognition of mechanistic interpretability increase, many^[who?] have begun to recognize that other communities such as natural language processing researchers have pursued similar objectives in their work.

The latent space of a LLM is generally conceptualized as a vector space. There is a growing body of work around the linear representation hypothesis, which posits that high-level features correspond to approximately linear directions in a model’s activation space.^[25][26]

Superposition is the phenomenon where many unrelated features are “packed’’ into the same subspace or even into single neurons, making a network highly over-complete yet still linearly decodable after nonlinear filtering.^[9] Recent formal analysis links the amount of polysemanticity to feature ‘‘capacity’’ and input sparsity, predicting when neurons become monosemantic or remain polysemantic.^[27]

Probing involves training a linear classifier on model activations to test whether a feature is linearly decodable at a given layer or subset of neurons.^[28] Generally, a linear probe is trained on a labelled dataset encoding the desired feature. While linear probes are popular amongst mechanistic interpretability researchers, its introduction dates back to 2016 and it has been widely used in the NLP community.^[29]

Nanda, Lee & Wattenberg (2023) showed that world-model features such as in-context truth values emerge as linearly decodable directions early in training, strengthening the case for linear probes as faithful monitors of internal state.

Difference-in-means, or diff-in-means constructs a steering vector by subtracting the mean activation for one class of examples from the mean for another. Unlike learned probes, diff-in-means has no trainable parameters and often generalises better out-of-distribution.^[30] Diff-in-means has been used to isolate model representation for refusal/compliance, true/false, and sentiment.^[31][30][32]

Steering adds or subtracts a direction (often obtained via probing, diff-in-means, or K-means) from the residual stream to causally change model behaviour.

Activation patching ablates model components or neurons to trace which part of the model is causally necessary for a behaviour. Automated circuit discovery (ACDC) prunes the computational graph by iteratively patch-testing edges, localising minimal sub-circuits without manual hand-holding.^[33]

Sparse autoencoders (SAEs) for mechanistic interpretability were proposed in order to address the superposition problem by decomposing the feature space into an overcomplete basis (i.e. with more features than dimensions) of monosemantic concepts. The underlying intuition is that features can only be manipulable under superposition if they are sparsely activated (otherwise, interference between features would be too high).^[11]

Given a vector ${\displaystyle \mathbf {x} \in \mathbb {R} ^{n}}$ representing an activation collected from some model component (in a transformer, usually the MLP inner activation or the residual stream), the sparse autoencoder computes the following: $${\displaystyle {\hat {\mathbf {x} }}=\mathbf {W} _{\mathrm {dec} }^{\top }(\mathrm {ReLU} (\mathbf {W} _{\mathrm {enc} }\mathbf {x} +\mathbf {b} _{\textrm {enc}}))+\mathbf {b} _{\textrm {dec}}}$$ Here, ${\displaystyle \mathbf {W} _{\mathrm {enc} }\in \mathbb {R} ^{z\times n}}$ projects the activation into a ${\displaystyle z}$-dimensional latent space, applies the ReLU nonlinearity, and finally the decoder ${\displaystyle \mathbf {W} _{\mathrm {dec} }\in \mathbb {R} ^{z\times n}}$ aims to reconstruct the original activation from this latent representation. The bias terms are ${\displaystyle \mathbf {b} _{\textrm {enc}}\in \mathbb {R} ^{z},\mathbf {b} _{\textrm {dec}}\in \mathbb {R} ^{n}}$; the latter is omitted in some formulations. The encoder and decoder matrices may also be tied.

Given a dataset of activations ${\displaystyle \mathbf {X} =\{\mathbf {x} _{1},\ldots ,\mathbf {x} _{n}\}}$, the SAE is trained with gradient descent to minimise the following loss function: $${\displaystyle {\mathcal {L}}(\mathbf {x} )=||\mathbf {x} -{\hat {\mathbf {x} }}||_{2}^{2}+\alpha ||\mathbf {z} ||_{1}}$$ where the first term is the reconstruction loss (i.e. the standard autoencoding objective) and the second is a sparsity loss on the latent representation ${\displaystyle \mathbf {z} =\mathrm {ReLU} (\mathbf {W} _{\mathrm {enc} }\mathbf {x} +\mathbf {b} _{\textrm {enc}})}$ which aims to minimise its ${\displaystyle \ell ^{1}}$-norm.

Several works motivate alternative nonlinearities to ReLU based on improved downstream performance or training stability.

• TopK, which means selecting the top-${\displaystyle K}$ activating latents and zeroing out the rest, which also allows for dropping the sparsity loss entirely.^[34]
• JumpReLU, defined as ${\displaystyle \mathrm {JumpReLU} (z)=zH(z-\theta )}$ where ${\displaystyle H}$ is the Heaviside step function.^[35]

The core metrics for evaluating SAEs are sparsity, measured by the ${\displaystyle \ell _{0}}$-norm of the latent representations over the dataset, and fidelity, which may be the MSE reconstruction error as in the loss function or a downstream metric when substituting the SAE output into the model, such as loss recovered or KL-divergence from the original model behaviour.^[36]

SAE latents are usually labelled using an autointerpretability pipeline. Most such pipelines feed highly-activating (i.e. having activation ${\displaystyle \mathbf {x} }$ resulting in large ${\displaystyle \mathbf {z} _{i}}$ for feature ${\displaystyle i}$, repeating over all features) dataset exemplars to a large language model, which generates a natural-language description based on the contexts the latent is active.

Early works directly adapt Bills (2023)'s neuron-labelling and evaluation pipeline and report higher interpretability scores than alternative methods (the standard basis, PCA, etc.).^[11][12] However, this leads to misleading score, since explanations achieve high recall but usually low precision, leading to more nuanced evaluation metrics being introduced in later works: neuron-to-graph explanations (or other approaches) reporting both precision and recall,^[34] and intervention-based metrics that measure the downstream effect of manipulating a latent feature.^[37][38]

Transcoders are formulated identically to SAEs, with the caveat that they seek to approximate the input-output behaviour of a model component (usually the MLP).^[39] This is useful for measuring how latent features in different layers of the model affect each other in an input-invariant manner (i.e. by directly comparing encoder and decoder weights). A transcoder thus computes the following: $${\displaystyle {\hat {\mathbf {y} }}=\mathbf {W} _{\mathrm {dec} }^{\top }(\mathrm {ReLU} (\mathbf {W} _{\mathrm {enc} }\mathbf {x} +\mathbf {b} _{\textrm {enc}}))+\mathbf {b} _{\textrm {dec}}}$$ and is trained to minimise the loss: $${\displaystyle {\mathcal {L}}(\mathbf {x} )=||\mathrm {MLP} (\mathbf {x} )-{\hat {\mathbf {y} }}||_{2}^{2}+\alpha ||\mathbf {z} ||_{1}}$$ When ignoring or holding attention components constant (which may obscure some information), transcoders trained on different layers of a model can then be used to conduct circuit analysis without having to process individual inputs and collect latent activations, unlike SAEs.

Transcoders generally outperform SAEs, achieving lower loss and better automated interpretability scores.^[40][39]

A disadvantage of single-layer SAEs and transcoders is that they produce duplicate features when trained on multiple layers, if those features persist throughout the residual stream. This complicates understanding layer-to-layer feature propagation and also wastes latent parameters. Crosscoders were introduced to enable cross-layer representation of features, which minimises these issues.^[41]

A crosscoder computes the cross-layer latent representation ${\displaystyle \mathbf {z} }$ using a set of layer-wise activations ${\displaystyle \{\mathbf {x} ^{(l_{1})},\ldots ,\mathbf {x} ^{(l_{n})}\}}$ over layers ${\displaystyle L}$ obatained from some input as follows: $${\displaystyle \mathbf {z} =\mathrm {ReLU} \left(\sum _{l\in L}{\mathbf {W} _{\mathrm {enc} }^{(l)}\mathbf {x} ^{(l)}+\mathbf {b} _{\mathrm {enc} }}\right)}$$ The reconstruction is done independently for each layer using this cross-layer representation: $${\displaystyle {\hat {\mathbf {x} }}^{(l)}=\mathbf {W} _{\mathrm {dec} }^{(l)}\mathbf {z} +\mathbf {b} _{\mathrm {dec} }}$$ Alternatively, the target may be layer-wise component outputs ${\displaystyle {\hat {\mathbf {y} }}^{(l)}}$ if using the transcoder objective. The model is then trained to minimise a loss: $${\displaystyle {\mathcal {L}}(\{\mathbf {x} ^{(1)},\ldots ,\mathbf {x} ^{(n)}\})=\sum _{l\in L}{||{\hat {\mathbf {x} }}^{(l)}-\mathbf {x} ^{(l)}||_{2}^{2}+\sum _{l\in L}\sum _{i}{\mathbf {z} _{i}||\mathbf {W} _{\mathrm {dec} ,i}^{(l)}||_{2}}}}$$ Note that the regularisation term uses the ${\displaystyle \ell ^{2}}$-norm; the ${\displaystyle \ell ^{1}}$-norm is an alternative choice, considered but not used in the original paper.^[41]

Circuit tracing substitutes parts of the model, in particular the MLP block, with more interpretable components called "Transcoders". The goal is to recover explicit computational graphs. Like SAEs, circuit tracing uses sparse dictionary learning techniques. Instead of reconstructing model activations like SAEs, however, Transcoders aims to predict the output of non-linear components given their input. The technique was introduced in the paper "Circuit Tracing: Revealing Computational Graphs in Language Models", published in April 2025 by Anthropic. Circuit tracing has been used to understand how a model plans the rhyme in a poem, perform medical diagnosis, and understand chain of thought unfaithfulness.^[42]

Critiques of Mechanistic Interpretability can be roughly divided into two categories: critiques of the real-world impact of interpretability, and critiques of the mechanistic approach compared to others.

Critics have argued that current circuits-level analyses generalise poorly and risk giving false confidence about alignment, citing cherry-picking, small-scale demos, and absence of worst-case guarantees.^[43][44] In response to Dario Amodei's blogpost "The Urgency of Interpretability", which claimed that researchers are on track to achieve MI's goal of developing "the analogue of a highly precise and accurate MRI that would fully reveal the inner workings of an AI model", Neel Nanda, who leads Google Deepmind's mechanistic interpretability team, argued that mechanistic interpretability alone won't enable high-reliability monitoring for safety-relevant features such as deception.^[45][46] In particular, Nanda highlighted that only a small portion of the model seems to be interpretable and that current methods cannot guarantee the absence of particular features or circuits of interest.

Some have framed mechanistic interpretability as "bottom-up interpretability", where the emphasis is neuron-level circuits, in contrast to "top-down interpretability", which focuses on emergent high-level concepts.^[47] They argue that LLMs are complex systems and high-level concepts cannot be simulated, predicted, and understood with low-level components. As one critic puts, "If you wanted to explain why England won World War II using particle physics, you would just be on the wrong track."^[48]