Neuronpedia: Making Model Interpretability Slightly Less Painful

The Problem

Deep learning models are black boxes. Sure, you can train a model to do magical things, but when it goes haywire, good luck figuring out why. Even worse, trying to explain what’s happening to a non-technical person—like your boss—can feel like trying to explain quantum physics to a dog. You need tools to interpret what the model is doing, debug, and present results in a way that humans can actually understand. That’s where neuronpedia comes in.

What This Does

neuronpedia is an open-source platform for interpretability. It's a sprawling monorepo (200 files across 4 main apps). At its core, it provides tools for analyzing and visualizing model activations, embeddings, and other internal workings.

Here’s the breakdown: Autointerp (apps/autointerp/): Automates generating explanations for deep learning models. The guts live in apps/autointerp/server.py, with routes like routes/explain/default.py doing the heavy lifting. Graph (apps/graph/): Handles circuits and graphs. Check out neuronpediagraph/server.py for the main entry point. Want to see how neurons connect? This is your jam. Inference (apps/inference/): The heavy-duty stuff. It handles model inference with various configurations. The k8s/overlays/ directory is loaded with YAML files for deploying models, and endpoints/ has utilities for tokenization, steering, and persona analysis. Steerify (apps/experiments/steerify/): Some front-end experiments built with React, Next.js, and Tailwind. If you're into building dashboards or visualizations, this is the playground.

Oh, and it’s containerized with Docker and Kubernetes, so you can run everything locally or spin it up in the cloud if you’re feeling fancy.

Real-World Use

Let’s say you're working on a GPT-style model, and it starts spitting out responses that are...less than ideal. Maybe it's biased, or it’s just plain dumb. With neuronpedia, you can use the activation/topkbytoken.py endpoint under apps/inference/neuronpediainference/endpoints/ to dig into which neurons are responsible for specific token activations. Want to visualize how those neurons interact? Spin up the graph server with apps/graph/start.py and start exploring circuits.

Here’s a toy example:

From there, you can create a dashboard in steerify to visualize these interactions and explain them to your team. Or your dog.

The Bottom Line

neuronpedia is not for the faint of heart. It’s a beast of a monorepo, and setting it up will take time (be ready to wrestle with Dockerfiles, Makefiles, and a zoo of YAML). But if you’re serious about interpretability, this is a great starting point. Just don’t expect it to hold your hand—this isn’t plug-and-play.