FIND US ONProduct HuntAgent ObservabilitySee how we help you debug AI agents, and share your feedback.

LLM error analysis at scale: clustering AI agent failures by cause

Most tools that cluster AI agent failures have a model read each session, so at volume they cap the sample, advise one, or read only sessions you pick.

Cover with the words "Summary-built groups see only what summaries kept". A tool span whose result is an unknown-tool error beside the session summary, which says the session was a successful cancellation. The summary drops the tool error, so the session lands in the successes group.
On this page
  1. Which LLM error analysis tools group agent failures by cause
  2. Why the share of sessions a tool reads decides whether a group is true
  3. Why a group built from summaries sees only what the summaries kept
  4. Count tool failures across every session without reading a trace
  5. Check a group before you act on it
  6. Confirm a fix by watching the failure's count fall in new sessions
  7. What product and support people need to see when the agent fails a customer
  8. How Oodle groups agent failures across every session
  9. Common questions
  10. Sources

TL;DR

  • Most clustering tools have a model read each session, so at volume they cap the sample, advise one, or read only sessions you pick.
  • A tool reading a random 1,000 sessions sees a failure that hits 1 session in 5,000 only about 18% of the time.
  • A group built from summaries sees only what the summary kept, so a tool error the agent talked over can read like a success.
  • Count each failure across every session from your tool-call spans, beside the number of sessions in the same window.
  • When a tool returns an error payload as its result, have your code set error.type on its span so the count includes it.
  • Before acting on a group, check its size against your count and confirm its cause in the traces.

LLM error analysis means sorting your agent's failures by cause, so you know what to fix. If you can spot the failures one trace at a time but cannot read thousands of sessions a day to see what they share, several kinds of tools can cluster the failures by cause for you, into groups. Each group is only as true as the sessions the tool read.

Most of these tools have a model read each session, and at volume they cap or advise a sample. One caps a report at 1,000 traces, and another advises 1 to 5% at high volume. So before you act on a group, count the same failure across every session from your tool-call spans, and open the traces behind it.

Which LLM error analysis tools group agent failures by cause

LangSmith Insights and Braintrust Topics have a model summarize sessions and cluster the summaries, LangSmith Engine uses an agent, Amazon Bedrock AgentCore runs a model-based failure analysis on the sessions you select, and a fifth kind checks every session first. They differ in how much of your traffic they read and in how they read each session, and those choices decide whether you can trust a group. As of October 2026, from each tool's own docs:

How the tool reads your sessions How much of your traffic it reads What reading a session costs
A model writes a short summary of each trace, and a second model clusters the summaries (LangSmith Insights) A sample of up to 1,000 traces per report Model tokens, about $1 to $2 per 1,000 threads with OpenAI models and $3 to $4 with Anthropic models
A model summarizes each trace, an embedding model turns each summary into a vector, and a clustering algorithm groups them once at least 100 summaries exist. A model then names each cluster (Braintrust Topics) 100% by default, with the vendor advising 1 to 5% on high-volume projects Model credits on Starter and Pro plans, then on-demand token rates
An agent reads a share of your traces, adds the low-scoring ones, and opens issues with a diagnosis and the traces behind it, then keeps adding matching traces, reopening an issue that comes back (LangSmith Engine) A share set by an analysis level Usage units at $1 each, growing with the traces read and the model calls the agent makes, with a default cap of 750 a month
A model analyzes every span in each session, finds the cause, and groups similar failures across sessions by root cause, with a count of sessions affected (Amazon Bedrock AgentCore failure analysis). The earlier insights preview stopped taking new jobs October 16, 2026 The sessions you select for a batch evaluation, by time range or session ID Model tokens at batch evaluation rates. Insights free during the preview
Checks on every session raise issues, and Oodle's agent verifies each against the traces and groups what is left by cause Every session Out-of-the-box insights are included. Oodle has simple, ingest-based pricing, starting at $1 per GB, and is cost efficient

In the first four rows a model does the reading, and the cost grows with every session read. One tool caps its sample, and another tells you to turn the rate down at volume to save credit. The sample is where the trouble with the groups starts.

Why the share of sessions a tool reads decides whether a group is true

A group found in a sample shows the failures common enough to land in it, and can miss the rare ones. Take a failure that hits 1 session in 5,000, and a tool that reads a random 1,000 sessions. You would expect 0.2 of the failing sessions in the sample. The chance that even one lands there is about 18%, which is 1 minus 0.9998 to the power of 1,000. They assume a random sample and sessions that fail independently.

So about four reports in five never see that failure, and when one does, a group of one session tells you nothing about how often. At 50,000 sessions a day, the same failure still hits about 10 sessions a day.

The size of a group inside a sample is an estimate too. If 12 of 1,000 sampled sessions land in a group, the failure's real rate across all your sessions could be anywhere from about 0.7% to 2%, by a 95% confidence interval. That is enough to rank the large groups. It is not enough to tell whether a fix halved one.

A sample steered toward bad scores has a gap of its own. One tool's agent pulls the low-scoring traces for each evaluator score into its sample. That helps it find what your evaluators already score badly. A failure that no evaluator scores is left to whatever else the sample picks up.

The sampling gap matters little for a failure that hits 1% of your sessions or more, because a random sample of 1,000 will hold about ten of them. The gap is in the rare failures, and in any number you want to act on.

Why a group built from summaries sees only what the summaries kept

A group built from summaries cannot be shaped by anything the summary left out. Each trace the tool reads is first condensed into a short description, and the descriptions are what get grouped. One vendor's docs state the consequence: "Any information that isn't in the trace summary won't affect the categories that get generated".

That matters most for the failures agents hide. Say your agent calls a tool that returns {"error": "unknown tool"}, and the model then tells the customer their subscription is canceled. The tool returned normally, so its span carries no error class, because OpenTelemetry's GenAI conventions call for error.type only if the operation ended in an error. A summary written from that conversation can read like a successful cancellation. The session then lands in a group of successes, or in no group at all.

You can tell the summarizer to look for errors returned as results, and a better summary prompt gives better groups. But each group is still only as good as that prompt. And every summary uses model credits, which is why, at volume, one caps its sample and another advises sampling.

Count tool failures across every session without reading a trace

Count the execute_tool spans that carry error.type, per session, beside the number of sessions in the same window. That number, the one every group should be checked against, comes from your spans and needs no model. If your agent follows OpenTelemetry's GenAI conventions, each tool call is a span named execute_tool {gen_ai.tool.name}. A call that ended in an error carries error.type, a short error class such as timeout.

The spans of one session share gen_ai.conversation.id when it is set. Most of these attributes are still at Development status, so look at a few of your own spans first to see which ones your instrumentation sets.

The query below counts, for the last day, how many sessions hit each tool and error class, next to the number of sessions in the same window. It is written for a backend that keeps span attributes in a map, so change the table and attribute syntax to match yours.

If you do not set gen_ai.conversation.id, use your own session attribute, or the trace id where one session is one trace.

sql
SELECT
  attributes['gen_ai.tool.name'] AS tool,
  attributes['error.type']       AS error_class,
  count(DISTINCT attributes['gen_ai.conversation.id']) AS sessions_hit,
  (SELECT count(DISTINCT attributes['gen_ai.conversation.id'])
     FROM spans
    WHERE start_time > now() - INTERVAL 1 DAY) AS all_sessions
FROM spans
WHERE attributes['gen_ai.operation.name'] = 'execute_tool'
  AND attributes['error.type'] <> ''
  AND start_time > now() - INTERVAL 1 DAY
GROUP BY tool, error_class
ORDER BY sessions_hit DESC;

Then add the failures that raise no error.

  • An error returned as the tool's result. In the code that calls your tools, check each result for an error payload. When you find one, treat it as a failed call and set error.type on the span. The query above then counts it.
  • The same tool called again and again. Count execute_tool spans per session and tool name, and flag the sessions far above what a normal session does.
  • A user asking the same thing again, or going quiet. This one needs the conversation text, which OpenTelemetry instrumentation does not capture by default.

These counts add to the failures the query finds, and all_sessions is the denominator. When a tool reports a group, you can now say what share of all sessions the failure behind it reached.

Check a group before you act on it

Compare the group's size with your count, open about two dozen of its sessions, and find the cause in the trace.

  1. Compare sizes over the same window. Count the failure the group describes across every session, and set it beside the size the tool reports. A group much larger or smaller than your count came from a sample that does not match your traffic, from summaries that merged different failures or dropped this one, or from a different definition of the failure.
  2. Open about two dozen of the group's sessions. Count how many really share the cause. If a third of them failed for another reason, the group joined two failures, and fixing one will not empty it.
  3. Find the cause in the trace. The cause the tool names should show up as a span, an error or a tool result, not only in a summary. Root-cause text written by a model is a hypothesis until the trace shows it.

Counts have a limit too. They find only the failures you thought to look for, and a model's grouping is how you find the kinds you have no rule for. Keep both. Good agent failure analysis uses the model's groups to find new kinds of failure, and your counts to size them.

Confirm a fix by watching the failure's count fall in new sessions

A fix has worked when the count for that failure falls to zero in sessions that started after the change shipped. It has to stay at zero over at least as many new sessions as it used to take to see the failure. A group missing from the next sampled report proves less, because a rare failure can drop out of a sample by chance.

Once a failure is fixed, keep its check running on every new session, so you hear the day it comes back.

Before the next change to the prompt, the model or the tool ships, a few of its sessions also make good test cases.

What product and support people need to see when the agent fails a customer

Product and support people need the grouped list in plain words, single sessions told as conversations, and a decision on what conversation text they may see. They hear the complaints and are often not the people who read traces. A feature request on one agent tracing tool says so: the people fielding feedback "are not the engineers who built it". The questions that request lists are "Is the agent working for most users right now?" and "Which user sessions are failing and why?". An engineer on another tool was building a simpler view of eval results for non-technical colleagues, because the tool's own screen is "information overload for those folks".

Give them views built on the same sessions your engineers use.

  • The grouped list in plain words: what went wrong, how many sessions and users it hit, and since when.
  • One session told as a conversation: what the user asked, what the agent said, and each tool call as a readable step, such as looking up order 4471 and finding nothing, with a link to the full trace for an engineer.
  • A decision on what conversation text they may see. OpenTelemetry's GenAI conventions treat prompts and model outputs as sensitive, and instrumentation leaves them out by default, so a view built on spans may have no conversation to show until you choose what to capture and who can read it.

A support lead needs to know how many users were affected. That number needs every session, with a user id on each one. A sample gives an estimate, and a customer who asks whether others were hit wants a count.

How Oodle groups agent failures across every session

Each session the model-based tools above read costs model tokens or credits, which is why at volume they read a sample. Oodle starts from checks that read every session.

Every session is analyzed at scale using statistics, ML classifiers and a small model for sentiment. Oodle's agent then checks each issue it finds against the evidence and removes the noise.

The checks find issues such as tool failures, loops, outages and stalled conversations. Oodle's agent groups what is left by cause, ranked by how many customers it affected.

Signals tag each session as it arrives. Insights run once a day, as a short list each morning of what went wrong, how often and why. Insight reports are for product and support people as well as engineers. Each finding is marked real, unclear or likely noise, with the traces behind it attached, so you can see whether the cause is in the trace before you act. A finding opens Claude Code or Cursor with the failing traces in the prompt. After the fix, the rule keeps checking every new session.

Traces go straight to object storage in Oodle's own columnar format, and queries run on serverless compute that spins up per query and shuts down after. So every agent trace is kept for your retention period (30 days on Pro, longer for an added fee), and the sessions behind a finding are there to open. The checks read every session, so each failure they find is counted across all your sessions.

See your own telemetry in Oodle

Start debugging in minutes.

Start freeBook a demo

Common questions

Is there a tool that clusters AI agent failures and finds the root cause?

Yes: several kinds cluster failures and propose a cause. Some have a model summarize the traces they read and cluster the summaries, and at volume they read a sample. Others use an agent that reads a share of traces and opens issues, run a model over the sessions you select, or check every session first and pass what those checks raise to an AI agent. Check each group against a count of the same failure across every session, and treat the root cause it names as a hypothesis until the trace shows it.

How can non-engineers like PMs and support see what our AI agent did?

Give them the grouped list of failures in plain words, with how many sessions and users each one hit. Show single sessions as conversations, with each tool call as a readable step and a link to the full trace. First decide what conversation text they may see, because OpenTelemetry instrumentation does not capture prompts and outputs by default.

How do I go from seeing a failure in a trace to finding the fix in a repeatable way?

Count the failure across every session. Use your tool-call spans for a tool failure, or a check you write for the others. Open about two dozen of its sessions to confirm a shared cause, and find it in the trace. The span, error or tool result there names what to change: the prompt, the tool or the model. After the fix, the count should fall to zero in new sessions and stay there as long as the failure used to take to appear. Keep the check running.

Can I trust the groups a failure clustering tool gives me?

As a lead, yes. As a count, only if the tool read every session. A tool that reads a random 1,000 traces a report will miss a failure that hits 1 session in 5,000 about four times in five, and a group built from summaries reflects only what the summaries kept.

Sources

Share