Releasing AI-driven tools for understanding AI systems

We are releasing several pieces of technology using trained AI agents to help users understand and investigate other AI systems.

October 31, 2024 update: code available on Github for feature descriptions and the Monitor interface; explainer/simulator model weights open-sourced on HuggingFace.

AI systems result from pipelines of complex data, from training sets to learned representations and model behaviors. Our vision is to create AI-driven tools that direct massive computational power toward explaining these complex systems.

Today, we’re sharing three demonstrations that start to illustrate this vision:

• an LLM pipeline that creates state-of-the-art feature descriptions;
• an observability interface for interrogating and steering these features;
• a behavior elicitation agent that automatically searches for user-specified behaviors from frontier models, including Llama-405B and GPT-4.

You can click above to read detailed research reports, or continue reading for overviews. We’re releasing a public endpoint for the interface today and will fully open-source the code for the feature description and observability tools next week.

Feature Descriptions.

We often understand representations by decomposing them into features, such as neurons or linear directions. To understand these features, we want human-readable descriptions that accurately predict when each feature is active.

There are far too many features to label by hand, so we need to automate this. We trained an AI pipeline to generate, score, and re-rank descriptions, outperforming human descriptions on our quality metrics and significantly outperforming existing automated methods. We have generated high-quality descriptions for every neuron in Llama-3.1 8B and are releasing these descriptions, along with an open-source LLM that can be applied to any features in any language model, in a code release next week.

Some example descriptions generated by our pipeline are shown above; on the top, the highlighted text in each exemplar depicts neuron activation patterns; on the bottom, we see that our descriptions are similar to human quality, surfacing complex patterns such as repetition. We are excited for others to build on our open-source tools, for instance by further fine-tuning our explainer model or applying it to describe features for other models.

Observability Interface.

Using our AI-generated descriptions, we built an interface that allows users to observe, interrogate, and steer active features within a chat dialog.

Each part of the interface incorporates AI-driven tools to empower the user. For instance, active but unexpected features are automatically flagged by an AI "linter," and we can automatically steer entire collections of features based on natural language queries. Using these tools, we found satisfying explanations for known model pathologies (e.g. 9.11>9.8, as many language models claim) and uncovered hidden knowledge that models omit from their output. We’ve included these as demos in our interface to walk you through the same steps of discovery, but we are most excited to see what insight it can provide into your own questions, and how you build on it (it is cheap to run and can operate locally on 2 A100 GPUs).

Using the interface, we've discovered satisfying explanations for surprising model failures. Perhaps most interestingly, for the famous error where models incorrectly state that 9.11>9.8, we uncover "gravity" (9.8 m/s^2) and "Sept. 11th" (9/11) neurons that contribute to the incorrect comparison. We more generally uncover "bible verse" neurons that lead many numbers to be incorrectly compared, and find that ablating these fixes the issue.

Behavior Elicitation.

We also trained agents that automatically search for prompts that generate an output satisfying a user-specified property. These properties could be harmful behaviors such as violations of the Model Specification, or capabilities such as solving a particular math problem.

We treat this as a reinforcement learning problem, where we fine-tune language agents to efficiently solve this search problem across a wide distribution of properties. This results in a general-purpose tool that produces strong human-interpretable jailbreaks for frontier models such as Llama-3.1 405B, GPT-4o, and Claude 3.5 Sonnet.

Contribute to our technology. We build our technology in and for the public, in line with our mission. You can help by playing with our tools, leaving feedback, and subscribing to our git repository, where will fully open-source our code next week.