MLEnglishWhat is Interpretability?
We know how to build these models, but we don't really know what's happening inside them.
For example, imagine a chef who cooks an incredible meal every single time, but can't tell you a single thing about what went into it. You can taste the food, and you can tell that the food is good, but you have no idea what's actually in there, or whether one of the ingredients is going to make you sick.
That's basically where we are with large language models. We can read what a model says, but the part in the middle where the model decides what to say is a Black Box.
Interpretability is the work of actually opening that box.
Why Is This So Hard?
The obvious idea is to just look at the individual neurons inside the model, and see what each neuron does.
That doesn't work, because a single neuron doesn't stand for a single thing. One neuron will fire for the Golden Gate Bridge, and also for Rosalind Franklin, and also for some completely unrelated piece of Python syntax.
Basically, the concepts are all smeared across each other, so looking at one neuron tells you almost nothing. That's why we use another method!
The Experiment
Instead of looking at neurons, researchers used a technique called Dictionary Learning to pull that messy internal activity apart into clean, individual concepts, and they called those concepts Features.
Antrophic's example is sentences, which for example are made up of a bunch of words, so relating that to our example, the neuron activations are "words" which you can combine to make features(sentences). (there's more to this relating to the internal state in the paper, which is really interesting, and probably something worth reading).
Then, they used that technique and ran it on the middle layer of Claude 3 Sonnet.
They found tons of features!
Some of the features were concrete things, like San Francisco, Rosalind Franklin, the element Lithium, immunology, and function calls in code. Others were way more abstract, like bugs in computer code, gender bias in professions, and conversations about keeping a secret.
The features weren't always tied to English. The same feature would sometimes fire on the concept written in a totally different language, and sometimes on an actual image of the thing.
The Golden Gate Bridge
Here's the demo presented to help validate the features.
Researchers found the feature for the Golden Gate Bridge, and instead of just sitting there watching it, they cranked that feature way up.
Claude immediately became obsessed with the bridge, and started bringing the bridge up in response to almost any question it was asked. When researchers asked Claude what it was, it said "I am the Golden Gate Bridge… my physical form is the iconic bridge itself…"
Normally Claude will tell you that it doesn't have a physical form at all. Turn up ONE feature, and it thinks it's a bridge.
Basically, a feature isn't just something researchers noticed on the way past, it's a real dial that can actively steer how the model behaves. I thought this was pretty cool, and a wonderful research finding!
Why Does This Matter?
Researchers also found features for the things you would actually want to catch, like code backdoors, bioweapons development, scam emails, racist claims, power seeking, manipulation, and secrecy.
Imagine a safety researcher trying to check whether a model is about to help someone build something dangerous:
- Today, they mostly have to test the model by asking it things and reading whatever comes back
- A model that's hiding the behavior can just answer safely for as long as it's being watched
- But if you can see which features are lighting up inside the model, the model doesn't get to pick what you see
- The researcher gets to catch the intent, instead of only catching what the model decided to say out loud
Research
- This is the original paper on mapping features inside Claude by Anthropic! Link