FIND US ONProduct HuntAgent ObservabilitySee how we help you debug AI agents, and share your feedback.

AI agent monitoring for silent failures: loops and stalled conversations

To monitor AI agents in production, define what a failed session looks like, check every session against it and alert on each agent's failure rate.

Cover with the words "Green dashboards, failed sessions". Four service health signals, latency, traffic, errors and saturation, all ok, beside one session whose refund tool errored while the agent told the customer the refund was issued. Every span reads success. Only a session check catches it.
On this page
  1. Why the dashboards stay green when an agent fails
  2. Write down your AI agent's failure modes as session checks
  3. Loops and stuck sessions a step counter misses
  4. What AI agent monitoring should track beyond uptime and latency
  5. Alert on the share of sessions that fail, per agent
  6. How to know the checks and the alert work
  7. Finding loops and stuck conversations in every session with Oodle
  8. Common questions
  9. Sources

TL;DR

  • To monitor an agent in production, define what a failed session looks like, check every session against it and alert on each agent's failure rate.
  • Uptime, latency and error alerts watch agent spans that read as success when a tool fails, but keep them for timeouts, crashes and rate limits.
  • Check the outcome in the system that holds it, such as billing after a refund, and check each trace for a tool error then "done".
  • Step counters miss short loops and conversations stuck across turns, so read the order of tool calls and join turns by a conversation id.
  • Alert when an agent's failure rate is high over both a long and a short window, once enough sessions have run.
  • For a severe failure, such as a refund reported as issued after the refund tool returned an error, alert on any count above zero.

If your agent loops, gets stuck or tells a customer a failed tool call worked, but no uptime, latency or error-rate alert fires, those alerts are working as built. The failed tool's span may show an error, but the agent and request spans read as success, and your alerts watch those. AI agent monitoring starts by writing down what a failed session looks like for each agent, such as a refund the agent reported as issued that the billing system does not show. Keep the error, latency and uptime alerts as well, because timeouts, crashes and rate limits still show up there.

Why the dashboards stay green when an agent fails

Service monitoring tells you whether each call succeeded and how long it took, and an agent can fail with every call succeeding. Most dashboards and alerts are built from what Google's SRE book calls the four golden signals: latency, traffic, errors and saturation. None of the four says whether the agent did the task.

Tracing the agent does not change that on its own. Under OpenTelemetry's conventions, a span's status must be left unset when the operation ends without an error. An error that was retried or handled should not be recorded on the span of the operation that handled it, and the GenAI conventions for model and tool calls follow the same rules. So when a tool returns an error and the agent carries on and tells the user the job is done, the tool span may show the error, but the agent and request spans can read as a success.

Many AI agent failure modes need no error at all. A study of multi-agent systems that annotated more than 1,600 traces found 14 failure modes, among them repeating steps, not knowing when to stop, ending a task early and skipping verification. On LangChain's forum, a user described a deep agent that stopped after reading its files, left its own to-do list unfinished and raised no error. A failure like that is found by whoever reads the result, and that is often the customer.

Agents also fail the way other software does, with timeouts, crashes and rate limits, and those do show up as errors. So the error alerts stay. They are necessary, but on their own they miss loops, stuck conversations and tool errors the agent ignored.

Write down your AI agent's failure modes as session checks

A failed session is one where the outcome the agent reported is not real, or the trace shows a pattern failed sessions leave behind. No error marks it, so you define it yourself, for each agent. A session is one conversation or task an agent handles from start to finish, and each check reads one session and answers yes or no.

Outcome checks compare what the agent said it did with what happened. Anthropic's guide to offline agent evals uses a booking agent that says "Your flight has been booked", where the real test is whether the reservation exists in the database. The same idea applies in production. For a support agent, that means reading the billing system after the agent says a refund was issued, or the ticketing system after it says a ticket was filed. For a task that changes something, the agent's own "done" is never the result.

Trace checks read the trace itself. Most need no model. Spotting a restated request may need a text match. Each one looks for a pattern that failed sessions leave behind.

Check What it looks for in the session
Tool error, then "done" A tool call returned an error, or an error message as its result, and the agent's next reply reports success
Repeated call The same tool with the same arguments, several times in a short stretch
New arguments, same result The same tool again and again, with arguments that change and a result that does not
Two tools in turn One tool, then another, then the first again, with nothing new coming back
User asks again The user restates the same request in a later turn
User stops replying The conversation goes quiet before the task is done
No end state The session ends without a final answer, or without the end state the task needed
Dependency down A rate limit, a timeout or a 503 from a service the agent called

Count the last row apart from the others. A session that failed because a dependency was down is an outage, and it belongs on the alerts of whoever owns that dependency, not in the agent's failure rate.

Some failures pass every check in the table, such as a confident answer that is wrong. Those need a judge model or a person. Run the judge on the sessions the cheaper checks flag, plus a small sample of the rest, and the bill for judging stays small.

Loops and stuck sessions a step counter misses

A step counter stops a long loop, but it cannot see a short loop or a conversation that gets stuck across turns, so loops need checks of their own. One engineer on LangChain's forum wrote that teams mostly rely on step counters or "do not repeat yourself" prompts to keep agents out of tool-calling loops. The counters in the common frameworks work like this, as of October 2026.

Framework Setting Default At the limit
OpenAI Agents SDK max_turns 10 model calls (openai-agents 0.24.0) Raises MaxTurnsExceeded
LangGraph recursion_limit 10,007 steps (langgraph 1.2.16) Raises GRAPH_RECURSION_LIMIT, which the docs say is "often due to an infinite loop"
CrewAI max_iter 20 iterations The agent must provide its best answer

Each of these counts steps inside one run. That catches a loop long enough to reach the limit, and in the OpenAI SDK and LangGraph it ends that loop with an error by default, so the loop is not silent. LangGraph's default of 10,007 steps, though, lets a loop run a very long time before it stops. In CrewAI it ends in an answer that can look like any other. Counting steps misses a short loop, a conversation stuck across turns and a long task that was working.

  • A loop shorter than the limit. Three identical refund calls under a limit of 10 finish with no error.
  • A conversation stuck across turns. Each turn is a new run with a fresh count, so a user asking for the same thing for the third time never reaches the limit.
  • A long task that was working. A limit low enough to catch loops also cuts off legitimate long runs. LangGraph's docs note that complex graphs may hit the default limit naturally, and the engineer who opened the forum thread says that when a timeout fires it leaves the state half-executed.

To catch a loop inside a session, read the order of the tool calls instead of counting them. The Python check below takes one session's tool calls in order and returns the loop patterns it finds. It skips tools that repeat on purpose, because, as a reply in the same thread points out, retries, polling, and pagination can all repeat legitimately. The window of six calls and the limit of three are examples. Set both from your own sessions.

python
import json

REPEATS_ON_PURPOSE = {"poll_job_status", "list_page"}  # polling and pagination tools

def loop_flags(tool_calls, window=6, limit=3):
    """tool_calls: one session's tool calls in order, as (name, arguments, result)."""
    flags = set()
    recent = []
    for name, args, result in tool_calls:
        if name in REPEATS_ON_PURPOSE:
            continue
        call = (name, json.dumps(args, sort_keys=True, default=str), json.dumps(result, sort_keys=True, default=str))
        recent = (recent + [call])[-window:]
        same_tool = [c for c in recent if c[0] == name]
        if sum(1 for c in same_tool if c[1] == same_tool[-1][1]) >= limit:
            flags.add("same call repeated")
        elif len(same_tool) >= limit and len({c[2] for c in same_tool}) == 1:
            flags.add("same tool, changing arguments, same result")
        names = [c[0] for c in recent[-4:]]
        if (len(names) == 4 and names[0] == names[2] != names[1] == names[3]
                and len({c[2] for c in recent[-4:]}) <= 2):
            flags.add("two tools alternating")
    return flags

A stuck conversation needs a check across turns, because no single turn looks wrong. Put a conversation id on every turn's spans, such as OpenTelemetry's gen_ai.conversation.id, so the turns can be read together. Then flag the conversation when the user restates a request, when a turn passes with no tool call and no change to the task, or when the conversation ends without the end state the task needed.

Inside the agent, the fix for a repeat loop is to tell the model what it already tried. An agent that is not shown the earlier failure has no way to recognize that the call already failed, so return the earlier failure to the model instead of running the same call again. Keep the step counter too, set well above your longest normal session, as a backstop on cost.

What AI agent monitoring should track beyond uptime and latency

Turn each check into a rate, and watch the rates for each agent next to the error, latency and uptime numbers you already have.

Measure What it tells you
Share of sessions that failed any check The headline number for each agent
Verified task success Sessions whose end state was confirmed in the system that holds it, not in the agent's reply
Tool errors, including the ones the agent ignored Counted from each tool's result, since the span status misses the handled ones
Loop rate and stuck rate Sessions flagged by the loop and cross-turn checks
Users asking again, and users who stop replying The user's side of a stuck conversation
Steps per session A jump after a release is worth reading a few sessions for
Sessions hit by an outage Kept on their own line, so a dependency's bad hour does not count against the agent

Split every rate by agent version as well as by agent. A prompt or model change that makes things worse then shows as a step in one line instead of a small rise in the total.

Alert on the share of sessions that fail, per agent

Alert when the share of an agent's sessions that fail a check is high over both a long and a short window, and only once enough sessions have run for the share to mean something.

Start by giving every session one result when it ends: ok, failed or outage. Sessions that end in an exception count too, so set the result from the code that started the agent, in a finally block, and not only at the agent's last step. Then expose the results as counters.

  1. Count each session's result in a counter labeled by agent and result.
  2. Count each failed check in a second counter labeled by agent and check.
  3. Create each agent's counters at zero when the process starts.

Alerting on the share of failed sessions is the method Google's SRE workbook uses for service level objectives. A service level indicator, or SLI, is the ratio of good events to total events, and the workbook says error budgets and error rates apply to all SLIs, not only the ones named for errors. Here a session that passed every check is a good event.

The workbook alerts on the rate of bad events over more than one time window, with the short window 1/12 the duration of the long one. The long window shows the problem is real, and the short one shows it is still happening.

Two more settings keep a rate alert quiet when it should be. A minimum session count stops a quiet agent from paging on one bad session out of three, since alerting gets sensitive to small events at low traffic. And a for clause keeps the alert pending until the condition has held for a set time.

A rate can also hide the one failure that matters every time, such as a refund the agent reported as issued after the refund tool returned an error. Alert on any count above zero for a check like that.

As Prometheus rules, those alerts, plus two for when sessions stop arriving, look like this. The first pages when more than 5% of an agent's sessions failed a check over both the last hour and the last five minutes, once that agent has had more than 50 sessions in the hour. The 5% and the 50 are examples. Set them from your agent's normal failure rate and traffic.

yaml
groups:
  - name: agent-sessions
    rules:
      - alert: AgentSessionsFailingChecks
        expr: |
          (
              sum by (agent) (rate(agent_sessions_total{result="failed"}[1h]))
            / sum by (agent) (rate(agent_sessions_total{result!="outage"}[1h]))
            > 0.05
          )
          and
          (
              sum by (agent) (rate(agent_sessions_total{result="failed"}[5m]))
            / sum by (agent) (rate(agent_sessions_total{result!="outage"}[5m]))
            > 0.05
          )
          and
          sum by (agent) (increase(agent_sessions_total{result!="outage"}[1h])) > 50
        for: 2m
        annotations:
          summary: "{{ $labels.agent }}: more than 5% of sessions failed a check"
      - alert: AgentToolErrorReportedAsDone
        expr: sum by (agent) (increase(agent_check_failures_total{check="tool_error_then_done"}[15m])) > 0
      - alert: AgentNoSessions
        expr: sum by (agent) (increase(agent_sessions_total[1h])) == 0
      - alert: AgentSessionsMissing
        expr: absent_over_time(agent_sessions_total[15m])
        for: 5m

The last two rules catch an agent that stopped, because an agent that stopped running produces no failures to alert on. The first fires for each agent that is still reporting but had no sessions in the last hour. The second fires when the session counter stops being reported at all, as when the process is down, and it covers all agents together. It needs absent_over_time, which takes a time range. The plain absent function does not take a time range, so absent(agent_sessions_total[15m]) is not valid PromQL.

Make the alert point at sessions. A message that says the failure rate went from 2% to 9% sends someone searching. A link to that agent's failed sessions in the last hour lets them read what went wrong.

How to know the checks and the alert work

Test them with a failure you planted before you trust them to find one you did not. In staging, make one tool return an error the agent can ignore, run enough sessions to pass the minimum session count, and keep the failure going past the for time (or lower both in staging). Confirm that each session is flagged, the failure rate rises and the alert fires, and run promtool test rules on the rules before they go live.

Then look back. When you add a check, run it over last week's sessions and see whether it finds the failure a customer already reported. If it does not, fix the check. Before a new check pages anyone, read a sample of the sessions it flags and confirm they failed. A scheduled task with a known answer, run against production every hour, also tells you the agent still finishes the job end to end when real traffic is quiet.

Every one of these checks reads a session you kept. If you sample your traces to keep the bill down, the session a customer reports may not be there to check.

Finding loops and stuck conversations in every session with Oodle

Every session is kept in full, for your retention period (30 days on Pro, longer for an added fee). That is affordable because of how Oodle is built. Traces go straight to object storage in Oodle's own columnar format, and queries run on serverless compute that spins up per query and shuts down after.

Oodle identifies issues in your agents out of the box, such as tool failures, loops, outages and stalled conversations, as each session arrives. Loop and stuck checks read whole sessions, so they work best when every session is kept and read as it arrives. Every session is analyzed at scale using statistics, ML classifiers and a small model for sentiment. Oodle's agent then checks each issue it finds against the evidence and removes the noise.

Insights run once a day, as a short list each morning of what went wrong, how often and why, with the traces behind each finding attached and each finding marked real, unclear or likely noise. When you fix a failure, describe it in one sentence. Oodle drafts the check, you approve it, and Oodle checks every new session for it. Oodle alerts when the rate of a signal or insight changes for an agent. The insights page shows the kinds of failure it looks for, and you can open agent sessions in the Oodle playground without signing up.

See your own telemetry in Oodle

Start debugging in minutes.

Start freeBook a demo

Common questions

How do I monitor AI agents in production?

Keep your error, latency and uptime alerts, and add checks on what each session did. Check that the end state the agent reported is real, and that the trace shows no tool error followed by "done", no repeated call and no stuck conversation. Run the checks on every session, and alert on the share of each agent's sessions that fail.

How do I catch silent AI agent failures when every check says healthy?

Check the outcome, not the agent's report: after the agent says a refund was issued, read the billing system. Then check the trace for failures that return success, such as a tool error the agent ignored, which OpenTelemetry's conventions say not to record as an error on the operation that handled it.

What should I monitor for AI agents in production beyond uptime and latency?

Watch the share of sessions that fail any check, and verified task success. Add tool errors including the ones the agent ignored, loop and stuck rates, users asking again or giving up, and steps per session. Split each by agent and agent version, and count sessions hit by an outage on their own line.

How do I detect an AI agent stuck in a loop when step counters don't help?

Read the order of the tool calls in each session. Flag the same call repeated, the same tool with changing arguments and the same result, and two tools in turn. For a conversation stuck across turns, join the turns by a conversation id and flag a user who restates the request, since each turn starts a fresh step count.

How do I get alerted when my AI agent is failing silently?

Give every session one result when it ends, including sessions that end in an exception. Alert when the failure rate for an agent is high over both a long and a short window, with a minimum session count and a for duration. Add a count alert for severe failures and an absent_over_time alert for when the session counter stops being reported, plus a per-agent alert for an agent with no sessions in the last hour.

Sources

Share