In this article we share our work on explainability for agentic AI systems, and introduce langgraph-goat, an open-source implementation of the Graph of Agentic Thoughts (GoAT) framework for LangGraph.

Why Agentic Systems Are Hard to Explain

Modern AI systems are increasingly shifting from single, linear pipelines to agentic architectures. A coordinator interprets a goal; delegates work to specialized agents and combines their outputs into an action or response. This structure enables more flexible problem-solving, clearer task decomposition, and potentially parallel execution. But it also makes the system’s decision process harder to see: as work moves across agents, tools, and memory, the reasons behind a final outcome can become opaque.

Consider a user who says, “I’m bored”. The system might launch a casual game and suggest calling a friend. It might instead recommend short videos and an upbeat playlist. Or it could check the calendar and propose a quick creative activity based on the user’s hobbies. Each is a reasonable interpretation, but only one plan will be executed. Once the system acts, the user may have no way to know what context shaped that choice, whether other options were considered, or why they were rejected.

This creates two distinct problems.

For end users, autonomous action without an explanation can erode trust. When a system interprets a vague/implicit request and acts on someone’s behalf, especially in areas such as health, scheduling, or communication, a poor decision that cannot be understood may be worse than no automated action at all.

For developers, debugging can become guesswork. When a multi-agent workflow produces a poor outcome, execution logs may reveal the sequence of calls but not the reasoning that shaped them: what context the system used, where delegation decisions were made, how memory influenced the plan, and which alternatives were available at that point.

Existing explainability methods fall short on both fronts. Post-hoc techniques such as LIME and SHAP help interpret a prediction after it has been made, but they are not designed to capture the flow of decisions across multiple agents. Chain-of-thought approaches can expose intermediate reasoning, yet when traces from several agents are stitched together, the causal links between decisions are often lost. The result is typically a reconstructed narrative of the outcome, rather than a structured record of how the system used context, delegated work, and selected one course of action over others. Figure 1 illustrates this gap: conventional execution traces preserve what happened, while effective explanations must also capture why it happened and why alternatives were not chosen.

Figure 1. A linear execution trace records what happened. The GoAT reasoning graph records how the decision was made, including the alternatives considered and the reason each was set aside.

Graph of Agentic Thoughts

GoAT takes a different approach: it captures the reasoning trace as a decision is made and stores it in a graph that preserves the relationships among planning, memory retrieval, delegation, execution, and rejected alternatives.

When a request arrives, a central coordinator generates several candidate plans. Considering multiple plans helps the system avoid committing immediately to a single interpretation, while keeping the candidate set small enough to remain useful and distinct. Each plan specifies a sequence of specialized sub-agents. Plan generation draws on the current session context as well as long-term memory, including information about the user’s persona, preferences, and prior experiences.

The coordinator prioritizes plans according to four main factors: their relevance to the current request, their alignment with plans that worked in similar situations, their compatibility with the user’s longer-term preferences or constraints, and their consistency with common-sense reasoning. A separate module Plan Critic then scores each candidate from 0 to 10 against user intent, context, and common sense, producing a justification alongside the score. The highest-scoring plan is executed.

Crucially, the remaining plans are not discarded. GoAT retains each unchosen plan as a counterfactual, along with an explanation of why it lost to the selected plan. Figure 2 shows this pipeline, from an incoming request through planning, execution, graph construction, and explanation generation.

Figure 2. The GoAT pipeline. Plans are generated and scored before execution, the reasoning graph is built as the factual plan runs, and the compressed graph feeds explanations tailored to two different audiences.

As the chosen plan executes, GoAT incrementally builds a directed, attributed graph:

Each plan generated during a turn carries a turn identifier, so the graph maintains temporal boundaries across a multi-turn conversation and downstream modules can analyze how decisions progressed through the dialogue.

The result is a queryable record of a decision process, rather than a linear list of events. Because the record is structured, it can support different explanation needs from the same underlying artifact. A summarization filter selects the relevant portion of the compressed graph and passes it to an LLM, which can produce either a concise, plain-language explanation for an end user or a detailed causal trace for a developer.

What this changes for the user

In this example “I’m bored”, the system generates three candidate plans, scored 8, 9, and 7. Plan B—the short videos and personalized upbeat playlist—wins.

The action itself may look the same. What changes is the explanation available to the user.

The first response is accurate but uninformative. The second makes the system’s judgment visible: it shows that alternatives were considered and explains why they were set aside. The action has not changed; what has changed is that the user can inspect the assumptions behind it and correct the system if those assumptions are wrong.

The same graph also answers a different question for whoever maintains the system.

Developer-facing Explanation

Two details are especially important. First, the executed plan was not the first plan generated: the Plan Critic promoted Plan B over Plan A, meaning that scoring meaningfully changed the outcome rather than simply endorsing an initial choice. Second, the rejected plans reflect different forms of judgment. Plan A is rejected because of an absence of supporting evidence, while Plan C conflicts with an observed timing pattern in the user’s history. Neither alternative appears in a conventional execution trace because neither one was executed.

Keeping the Graph Manageable

With long conversations, the graph could grow indefinitely. GoAT addresses this with two practical design choices.

First, graph construction is selective. It applies only to implicit or ambiguous requests that fall below an intent-confidence threshold, the cases where multiple interpretations genuinely compete. An explicit instruction such as “Set an alarm for 7 AM” has a clear intent mapping and no meaningful alternatives, so no graph is built.

Second, the graph is compressed. Sequential nodes with deterministic outcomes are merged into composite macro-nodes. Counterfactual plans with overlapping agent sequences are deduplicated into a single representative ghost node. When execution remains structurally similar across several consecutive turns, the system retains a limited number of distinct turns and collapses the rest into an abstract representation. Redundant reasoning traces are also pruned.

Only this compressed graph core is retained. A fixed-duration retention policy limits storage over time, while developers can override that policy for sessions that need to be preserved longer for debugging or analysis.

Results

We evaluated GoAT against four baselines representing the common approaches to explaining agentic workflows: raw log-trace explanation, Plan-and-Explain, multi-turn CoT reasoning, and stitched multi-agent CoT. The evaluation used 2,000 multi-turn dialogues from our In-house personal assistant, along with 1,000 dialogues from the public MultiWOZ 2.4 benchmark to assess generalization.

GoAT outperformed every baseline on every metric on both datasets, Table 1 shows results with In-house personal assistant dataset.

Table 1. Automatic evaluation on the In-house personal assistant dataset.

The gap is widest on causality alignment, which measures whether the explanation correctly captures cause-and-effect relationships between agents. This is where graph structure matters most: a linear trace can show that one agent ran after another, but sequence alone does not establish why the second agent was invoked. GoAT records the delegation and dependency links that connect those steps, allowing the explanation to represent the decision flow rather than merely replay the order of execution.

Human evaluation

Automatic metrics tell us whether an explanation is faithful to the execution, but they cannot tell us whether people find that explanation clear, useful, or trustworthy. To address this, we conducted a blind evaluation using 50 test conversations. Seven independent researchers, none of them authors, rated explanations from each method on a five-point scale, with method names anonymized to prevent bias, results shown in Figure 3.

Figure 3. GoAT scored highest on all four criteria. Impact on user trust reached 4.71 out of 5, an improvement of 1.07 over the strongest baseline.

The result on user trust is particularly important. It suggests that showing users the alternatives the system considered, and why those were not selected, makes its behavior more credible.

The full methodology and results are in our paper: https://dl.acm.org/doi/10.1145/3799830.3799871.

langgraph-goat

We have packaged GoAT as a library for LangGraph, one of the most widely used frameworks for building agentic systems. The wrapper integrates at the planner node, so an existing LangGraph application does not need restructuring to adopt it.

The source is on GitHub at https://github.com/Samsung/langgraph-goat and the package on PyPI at https://pypi.org/project/langgraph-goat/. The repository includes a working end-to-end example demonstrating plan generation, graph construction and both explanation modes. The implementation is written in Python, with graph construction built on NetworkX, and the explanation layer works with any LangChain-compatible model.

Why Open Source

Agentic explainability is still an emerging area, with no widely accepted standard yet in place. Many current tools focus on commercial tracing and observability, which are valuable capabilities but do not always capture the full structure of a system’s intrinsic decisions and reasoning. We see graph-based decision provenance as a promising complementary foundation— one that researchers and practitioners can apply to new systems, evaluate in different settings, and refine through open collaboration.

What Comes Next

Contribution is Welcome

langgraph-goat is open source and we welcome contributions. If you work on agentic systems, explainable AI, or developer tooling for multi-agent applications, we would be glad to have you involved, whether through issues, pull requests, or simply telling us how the framework runs on your own system. You can find more information and get involved at the following link: https://github.com/Samsung/langgraph-goat