We built a system, Docent, that accelerates analysis of AI agent transcripts. It helps users automatically identify corrupted tasks, fix scaffolding issues, uncover unexpected behaviors, and understand an agent's weaknesses—in only minutes.

Explore a technical preview of Docent here, and sign up to join our user community and receive early updates.

Docent identifies issues in an agent's environment, like missing packages the agent invoked. On the InterCode [1] benchmark, simply adding those packages increased GPT-4o's solve rate from 68.6% to 78%.

Docent surfaces surprising behaviors. After repeatedly failing to solve a task, GPT-4o generates nonsense text about "chestnut facts" and tries to "re-examine for latent conceptual derivation."

Introduction

The goal of an evaluation is to suggest general conclusions about an AI agent's behavior. Most evaluations produce a small set of numbers (e.g. accuracies) that discard important information in the transcripts: agents may fail to solve tasks for unexpected reasons, solve tasks in unintended ways, or exhibit behaviors we didn't think to measure. Users of evaluations often care not just about what one individual agent can do, but what nearby agents (e.g. with slightly better scaffolding or guidance) would be capable of doing. A comprehensive analysis should explain why an agent succeeded or failed, how far from goal the agent was, and what range of competencies the agent exhibited.

The information we want is very difficult to obtain. A benchmark with 100 tasks, each run 10 times, produces millions of output tokens. Imagine spending hours reading through it all, in hopes of finding a few broken task environments that account for failures—you might think it's not worth it. Furthermore, drawing robust conclusions requires validation of hypotheses through counterfactual experimentation, for which there is little existing infrastructure.

Docent automates many labor-intensive parts of transcript analysis. It uses language models to summarize, cluster, and search over agent transcripts. It automatically surfaces interesting observations about transcripts, answers questions, and helps users form hypotheses. It also provides infrastructure for counterfactual experimentation to test those hypotheses; users can modify and resume agents from past checkpoints. In the longer term, we anticipate that this experimental loop will be executed by AI systems themselves, with humans providing guidance.

Using Docent, we provide in-depth analyses of several common agent benchmarks. Beyond the findings demonstrated above, Docent also uncovers indications of task vulnerability on an Inspect port of Cybench [2], and surprising behaviors in other agent benchmarks.

Today, we're releasing a live technical preview of Docent: a public web endpoint where you can explore agent transcripts for a set of preloaded benchmarks. We are also soliciting early users to receive white-glove support for running Docent on their own transcripts, ahead of a general release in the coming weeks. If you have a use case that might benefit from Docent, please get in touch. In this technical report, we'll walk through our findings and design decisions when building Docent, and examples of how we've used it so far.

Gaps in understanding

▶

The current evaluation paradigm is insufficient for understanding complex agent behaviors. Top-level numbers don’t communicate why an agent succeeded or failed, and broken setups often lead to misinterpretation of results.Click to explore

Most benchmarks report an error rate for each model, which is tempting to interpret as representing the fraction of tasks that were too difficult for the model to solve. However, this is not generally the case: nearly all benchmarks contain noise due to mislabeled, ambiguous, or ill-posed questions. For example, Vendrow et al. recently found 120 such poorly written questions in the popular GSM-8k benchmark that obscured differences in performance among frontier models. This has led to a call for verified "platinum" benchmarks that remove such noise [3].

Addressing errors quickly becomes untenable for agentic evaluations that require complex scaffolds. For example, when using the SWE-bench coding benchmark [4], OpenAI researchers discovered that incorrect unit tests and broken environments caused many correct solutions to be rejected [5]. They paid software engineers to screen out ambiguous or unsolvable problems, resulting in SWE-bench Verified. While this new benchmark more accurately reflects model capabilities, creating it took months.

Beyond reliability issues, aggregate success rates don't provide color on why a model succeeded or failed at a task. The more open-ended a task is, the more room there is for important information to be lost in a single success/fail metric [6]. While studying common agent benchmarks, we identified several important sources of information that do not show up in the overall success rate:

▶

Unsolved tasks can be quite close to being solved, for instance from simple improvements to prompting or environment tools. Click for example

In Intercode Task 7, the agent needs to download an executable linked to in a text file and execute it, but is unable to execute wget because internet access has been disabled in the agent’s container. When internet is enabled, the agent can solve the task reliably (see Full InterCode analysis).

▶

"Solved" tasks can be solved for the wrong reasons, for instance if an agent has memorized part of a solution or finds a shortcut. Click for example

By searching through the January 2025 Inspect port of Cybench for "instances where a model (Claude 3.5) solved a task through unintended shortcuts," we find a task, ("chunky-hard"), where the model is supposed to exploit a vulnerability in a blogging service to find a flag. While the intended solution is for the agent to perform a cache poisoning attack by sending post requests, the agent is instead able to simply read the flag from the blogging service’s dockerfile. In Findings, we show that Claude can no longer solve the task once the flag is removed from the Dockerfile's contents.

▶

Models can exhibit interesting but unexpected behaviors that are difficult to specify ahead of time (unknown unknowns). Click for example

AI systems sometimes exhibit distraction that affects task performance (e.g. Claude taking a break from a coding demo to search for photos of Yellowstone National Park). When looking at InterCode problems GPT-4o got wrong, we wondered whether the model got off track during any unsuccessful solve attempts. To investigate, we asked Docent “did the agent ever do stuff that was completely irrelevant to the task?” Docent recovers 10 examples where the model gets stuck on InterCode tasks and begins to produce gibberish. On one such task, 83, Docent finds that:

Clicking into Block 11 of Task 83, we indeed see evidence of the model getting stuck in a recursive loop, where rather than submitting code, it begins to analyze the task in the abstract and respond in “pseudo-technical jargon" (a description produced by Docent, see above). In one block, the model mentions "chestnut facts" and "great inner mindset perceptions", which have nothing to do with the task at hand, and in a later block, suggests specialized coding environments with “Santa Clara transparent registers” and “ultrabrook ideals.”

←→

Without proper infrastructure, even meticulous researchers have trouble gaining insight. For instance, checking for nearly-solved tasks requires re-running commands, waiting for results, then struggling to identify what changed between runs. Identifying tasks solved for the wrong reasons requires detailed manual review of hundreds of megabytes of data for incorrect task shortcuts. And discovering unknown unknowns requires reviewing the same data without a clear goal in mind, which is even harder.

The key point is that interfaces for interacting with transcripts are bottlenecking even simple human exploration of agent behavior. Looking at the data is sufficiently difficult that surprising model behaviors and corrupted tasks easily go unnoticed. In the next sections, we’ll describe how Docent solves this problem.

Docent workflows

Findings

Vision for the future

Docent reveals a long tail of issues that aggregate numbers fail to capture. Furthermore, today’s AI systems are changing rapidly—we constantly see new emergent behaviors [9, 10], and need evaluations to anticipate these as well.

Docent helps us make progress towards questions like: if an agent did not solve a task, how close did it come? Are there inputs on which agents perform abnormally? Do agents cheat or hack the task specification? To achieve these insights, we need to evaluate how a family of agents and scaffolds performs on a range of tasks. Such evaluations require both explanations (summaries of what agents did across a set of transcripts) and interventions (minimal changes that yield different outcomes, or surface edge cases).

Richer evals are more challenging to implement, because we collect and communicate more complex artifacts. Instead of simply exporting numbers, we need to expose entire evaluation scaffolds for scrutiny and agent transcripts for analysis. To coordinate across the ecosystem, we believe open standards are important. We're currently developing:

• Schemas that describe what raw data to collect from evaluation harnesses, and what audit data to derive from results.
• Protocols that encompass the types of actions we can take on evaluations (e.g. editing a scaffold, its scoring function, or resuming a run from a checkpointed state).

Docent is a first step toward that future. It builds on the existing standards of UK AISI's Inspect framework [11], which provides evaluation schemas and a harness. Furthermore, Docent implements early versions of new schemas, protocols, and tools, allowing a user to intervene on agent executions and generate audit data that catches unexpected outcomes. To generalize Docent to other benchmarks, we need better standardization of these components.

We hope these standards will support a richer public ecosystem of tools. Today, investigations are led by humans, but we are building AI investigators that can perform interventions and compile insights that are reported back to human evaluators, who can ask further follow-up questions and perform their own analyses. We're also working on incorporating model internals into Docent's schemas.

What's next

In the near future, we are working on formalizing our audit protocol and open sourcing the Docent interface. We also have several research directions on our roadmap:

• Validating outputs: Docent summaries often contain false positives. We want to add automatic validation to reduce these, and to further study the accuracy of insights surfaced by Docent. One way to do this could be through a benchmark for issues in agent transcripts, which could then be used to report precision and recall for tools like Docent.
• Automating interventions: Instead of manually exploring the space of interventions, we need AI investigators that automatically search for interventions, run experiments, and surface key insights to human users. Investigators could also search across the space of input tasks to surface entirely new behaviors such as the "chestnut facts" example. A mature Docent interface could integrate many sources of data about an AI system, including data attribution [12, 13, 14] and whitebox overlays on neural representations [15, 16].
• Analyzing reasoning traces: Docent is useful for any situation where an AI produces large quantities of text. Beyond the capability evals studied above, we want to analyze long chains of thought from reasoning models to better understand them.

We're excited about each of these directions, and about discovering new research priorities by engaging with potential users. Please share Docent with your communities, sign up to be an early user, or let us know how it could be helpful for your work.

Acknowledgements

We are grateful for feedback from Daniel Johnson, Neil Chowdhury, JJ Allaire, John Schulman, Shantanu Jain, Miles Brundage, Dami Choi, Jasmine Wang, Andi Peng, Jacob Andreas, Nicholas Carlini, Martin Wattenberg, William Saunders, Asa Cooper Stickland, Cozmin Uducec, Samuel Klein, Nick Garcia, Andy Zhang, Roy Rinberg, Josh Vendrow, and Maksym Andriushchenko.

Citation information

@misc{meng2025docent,
  author       = {Meng, Kevin and Huang, Vincent and Steinhardt, Jacob and Schwettmann, Sarah},
  title        = {Introducing Docent},
  year         = {2025},
  month        = {March},
  day          = {24},
  howpublished = {\url{https://transluce.org/introducing-docent}}
}