The Case for AI Interpretability in the Agentic Era

As artificial intelligence transitions from passive chatbots to autonomous agents capable of executing complex tasks, the "black box" problem is no longer just an academic curiosity, it is a critical safety challenge. After his guest lecture at ReLU, we asked Øyvind Tafjord, Research Scientist in Interpretability at Google DeepMind, a few questions regarding how we can maintain trust in systems that are becoming increasingly complex.

During his talk Øyvind shared his journey from Wolfram Alpha, to AI2 where he helped develop the first ARC benchmark, to his current work on interpretability at DeepMind

The field of Artificial Intelligence is currently undergoing a structural change. We are moving from the era of Generative AI (systems that predict text or pixels) to the era of Agentic AI: systems that plan, reason, use tools, and execute decisions in the real world.

While this promises unprecedented utility, it brings a distinct crisis of transparency. When an AI is managing financial audits or medical diagnostics, being "right" is no longer enough. We need to know why it is right.

1. Scale Surprises

To understand where we are going, we must appreciate the surprise of how we got here. For decades, the assumption was that reasoning required complex, symbolic rules programmed by humans. The reality of Large Language Models (LLMs) proved otherwise.

As Tafjord notes, the field was caught off guard by what simple scaling could achieve:

"Language models have existed for a long time, but it was surprising that with a sufficiently rich model (e.g., transformer) and enough data, the models can indirectly learn everything possible about the world, from language structure to facts and reasoning patterns. BERT and GPT-2 were already shocking in what they could perform, but are still very primitive compared to the newest models."

This phenomenon is known in the research literature as emergence. We now know that with sufficient scale, models don’t just memorize; they develop internal representations that support logic, arithmetic, and even theory-of-mind‑like behavior. However, these capabilities often emerge within a "black box," leaving researchers to reverse-engineer how the model actually learned them.

As these emergent capabilities grow, the focus shifts from what the model can do to how it actually executes those tasks.

2. The Agentic Shift

The stakes of the black box rise with autonomous agents. Unlike a chatbot that offers a suggestion, an agent acts. It might access a database, execute code, or control an API.

In this context, the final output is insufficient for trust. If an agent denies a loan application or recommends a specific medical treatment, we require an audit trail of its logic.

Tafjord emphasizes that future systems must be built for interrogation:

"Generally, I feel it is important that we set up our AI systems so that we can have an understanding of why important decisions are taken. Not necessarily that we understand the 'innards' of the models... but that we can follow the reasoning that leads to the answer. Preferably in an interactive way, so that we can dig as deep as we want, where needed."

Kim et al. (2025) suggest that this interactive "agentic interpretability" serves as a necessary stress test for autonomous systems. They compare this dynamic to a cross-examination: while a model might easily fabricate a single justification for a loan denial, maintaining a consistent facade of logic across a multi-turn interrogation requires significant "cognitive load." By forcing the agent to explain its reasoning over a long dialogue, and potentially performing "open-model surgery" to verify those claims against internal states, we make it much harder for an agent to hide deceptive logic or incompetence. To satisfy the regulatory need for documentation, this conversation concludes not with silence, but with a generated "meeting note"; a consensus report that solidifies the dialogue into a verifiable audit trail.

Moving on, if we are to interrogate these agents effectively, we must look at the specific methods currently available to look into their "thoughts."

3. The "Golden Window" of Chain-of-Thought

So, how do we peek inside? Currently, we are in a fortunate, but perhaps temporary, "golden window" of transparency.

• Chain-of-Thought (CoT)

  Chain-of-Thought (CoT) is a reasoning technique where a model "thinks out loud" by generating intermediate steps in plain language before producing a final answer. This makes its logic readable and auditable by humans.

State-of-the-art models utilize Chain-of-Thought (CoT) reasoning. They "think" by generating tokens in human language before producing a final answer. This allows us to monitor their logic, catch hallucinations, and understand their intent.

However, Tafjord warns that this reliance on human language for machine thinking might not last:

"It is quite practical that many of the current models operate with ordinary language as a form of communication... This makes it possible to get a sort of superficial understanding of how they reason. This may disappear eventually, if models use internal, more effective representations to 'think' or communicate with each other - that can become a problem."

Recent research supports this concern. Concepts like reward hacking and steganography suggest that as models are optimized for pure efficiency, they may learn to "hide" information in high-dimensional vectors that humans cannot read. If an AI creates its own shorthand for logic that is 100x more efficient than English, we lose our ability to audit its thoughts.

4. Understanding the AI Brain

If the models stop speaking our language, how do we understand them?

This is where the field shifts from "behavioral psychology" (watching what the model says) to "digital neuroscience" (mapping the neurons). Tafjord highlights this duality:

"When it comes to 'understanding' such systems, there are many approaches (analogous to how one can study animals/humans through how they behave, or try to model from more fundamental mechanisms for what happens inside them)."

The "fundamental mechanisms" approach is known as Mechanistic Interpretability.

• Sparse autoencoders

  A Sparse Autoencoder is a neural network tool that compresses a model's internal activations into a smaller set of distinct, interpretable features — where only a few features are "active" at any given time. By forcing this sparsity, researchers can isolate what specific concepts or behaviors individual parts of the model are responsible for.

Researchers can in a way treat neural networks like biological organisms. Using tools like Sparse Autoencoders, they are decomposing the "neural soup" of a model into distinct features, identifying specific circuits responsible for deception, poetic structure, or coding ability.

Recent breakthroughs in monosemanticity allow us to find the specific "direction" in a model's “brain” that activates when it is thinking about, for example, the Golden Gate Bridge or a specific Python function. This is the future Tafjord alludes to: moving beyond "superficial understanding" to a precise wiring diagram of machine intelligence.

5. The Optimism of Exploration

Despite the daunting complexity, there is reason for optimism. The field is not stagnant; it is exploding with creative solutions.

We are seeing the rise of Mixture of Experts (MoE) architectures, which compartmentalize knowledge, and RAG (Retrieval-Augmented Generation), which grounds models in external facts. Tafjord finds this rapid iteration fascinating:

"The development of language models has continued with (accelerating?) progression... The most ideas one sees published don't lead very far, but because so many different things are being attempted, both in academia and industry, we constantly find things that actually make things significantly better. It is quite fascinating from a bird's eye view."

This "bird's eye view" reveals a landscape where failures are frequent but necessary, driving the incremental breakthroughs that redefine what is possible

• Mixture of Experts (MoE)

  Mixture of Experts (MoE) is a model architecture that divides knowledge into specialized sub-networks ("experts"), where only the relevant expert is activated for a given task. This makes large models more efficient and their knowledge more compartmentalized.

• Retrieval-Augmented Generation (RAG)

  Retrieval-Augmented Generation (RAG) is a technique that supplements a model's built-in knowledge by letting it fetch relevant information from an external database at runtime, grounding its answers in verifiable, up-to-date facts rather than relying solely on what it learned during training.

Conclusion: Trust Through Transparency

As AI systems are deployed across high-stakes domains such as healthcare, law, financial services, technical performance alone is an insufficient benchmark for trustworthiness. It is no longer enough for a model to be smart; it must be intelligible.

The goal is to ensure that as these systems scale, our understanding scales with them. We must build the tools today that allow us to "dig as deep as we want" tomorrow, ensuring that artificial intelligence remains a tool we can not only use, but truly understand.