FIND US ONProduct HuntAgent ObservabilitySee how we help you debug AI agents, and share your feedback.

Agent debugging: finding the step that broke in a multi-step session

To find the step that broke in a multi-step agent, walk back from the wrong answer to the first step whose output was wrong although its inputs were right.

Cover with the words "Walk back from the wrong answer". Four step cards of one agent session, from the model call to the reply offering a 30% renewal discount, with an arrow walking back to search_accounts, which returned the wrong Acme account.
On this page
  1. Why a wrong answer in a multi-step agent hides the step that caused it
  2. What the trace has to hold for agent debugging to work
  3. How to debug AI agent failures by walking back from the wrong answer
  4. Why some steps never show up in the trace
  5. Finding the failing step in thousands of log lines
  6. Catching a bad step before later steps build on it
  7. Keeping the session you need to walk back
  8. Common questions
  9. Sources

TL;DR

  • To find the step that broke, walk back from the wrong answer to the first step whose output was wrong although its inputs were right.
  • The walk works only if the trace holds what went into and came out of every step.
  • Instrumentations that follow OpenTelemetry's GenAI conventions leave that content out by default.
  • With several agents, check the input and output at each handoff, because the agent that broke is often not the one that answered.
  • A step missing from the trace was usually never wrapped in a span, lost its context, had tracing turned off, or ended before sending spans.
  • In logs, put the session ID, step ID, parent step ID and attempt number on every line, and log only the final failure at ERROR.

If your multi-step agent gave a wrong answer but no span in its trace errored, the step that broke can be several steps back, because each later step built on the bad data it was handed. A sales agent can quote the wrong discount, for example, because an account search two steps earlier picked the wrong customer and every step after it trusted the result. Agent debugging means walking back from the answer through the trace to find that step, so when a step is missing from the trace, fix the tracing first.

Why a wrong answer in a multi-step agent hides the step that caused it

Unless something checks the data where one step hands it to the next, an agent does not stop when a step produces bad data. The next step treats that data as true and builds on it. By the time the answer comes out wrong, the step that caused it can be several steps back. A developer on r/LocalLLaMA put it this way: "when something breaks at step 4, I have zero visibility into what actually happened at step 2."

So the wrong answer tells you that something broke, but not where. To find where, you need a trace of the whole session. A session is one conversation or task the agent handles from start to finish, and its traces have to hold every step with what went into it and what came out.

What the trace has to hold for agent debugging to work

For the walk to work, the trace has to hold every step, which step started which, and what went into and came out of each. Your tracing tool turns each step into a span, and each span names the span that started it, called its parent. Those parent links make the spans a tree, with the agent at the top and its model calls and tool calls nested under it.

OpenTelemetry's semantic conventions recommend names for these spans. An agent's span is invoke_agent followed by the agent's name, and a tool call's span is execute_tool followed by the tool's name. When one agent calls another, the second agent gets its own invoke_agent span inside the first. As of October 2026 these conventions are still in development, and some frameworks name their spans differently.

The tree shows which step started which. It does not show which output fed which input. For a tool call, the tool call id makes that link. The execute_tool span carries the tool call identifier supplied by the model (gen_ai.tool.call.id), so you can match the model's request for a tool to the span where that tool ran.

To see which step was wrong, you also need what each step was given and what it returned. Whether you get that depends on the tool. Langfuse's @observe() captures inputs and outputs by default. LangSmith keeps them unless you set it to hide them, and the OpenAI Agents SDK includes them by default. If they are missing in one of these, a setting has turned them off.

If you instrument with OpenTelemetry, the standard's semantic conventions go the other way. Instrumentations should not capture prompts, tool inputs or tool outputs by default, and should let you opt in, because that content is sensitive and often large. So with instrumentations that follow these conventions, a trace can show that a tool ran and succeeded without showing what it was asked or what it said.

In OpenTelemetry's Python GenAI instrumentations, an environment variable turns content capture on. As of October 2026 the value depends on which version of the conventions you have opted into. Other instrumentations have their own setting.

bash
# Default conventions
export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true

# Latest experimental conventions
export OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental
export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=span_only

Turning capture on puts prompts and tool results into your traces, so decide first what must not leave your application. Redact sensitive fields before they are sent. For production, OpenTelemetry's conventions also describe keeping the content in separate storage with its own access controls, and putting a reference to it on the span.

How to debug AI agent failures by walking back from the wrong answer

Follow the data back from the answer

Start from the wrong answer and follow the data backward. Reading a long trace from the first span to the last means reading every step to find one.

Ask what the final answer was based on. Go to the step that produced that input, and check whether its output was right for what it was given. If it was, the fault came from earlier, so move back to the step that produced those inputs. The step you want is the first one whose output was wrong even though its inputs were right.

Take a sales agent, for example. A rep asks it for a renewal quote for "Acme", and it offers a 30% discount. Acme Health should have been offered 10%. The trace holds four steps.

Step What went in What came out
Model call 1 The rep's request for an Acme renewal quote Call search_accounts with "Acme"
execute_tool search_accounts "Acme" One account: Acme Logistics, enterprise tier
Model call 2 The request and the account the search returned Apply the enterprise discount of 30%
Reply The 30% discount "I can offer Acme a 30% renewal discount."

Walking back, the reply matches model call 2. That call applied the right discount for the account it was given, so look at where the account came from. The rep only said "Acme", so model call 1 was right to search on the name. The search result is the first thing that looks wrong: it returned Acme Logistics, and the quote was for Acme Health.

Running the same search by hand shows why. "Acme" matches two customers, and the tool returns the first one without saying there was a second. So the search step broke, even though it returned a 200 and a well-formed result. The trace made this quick to find, because it held what each step was given and what it returned. Without the tool's output, the search would be just a green span.

The walk finds a step that produced bad output. A step that should have run and did not has no output to find. When every step's output looks right for its input, compare what the request needed with the steps that ran.

When several agents hand work to each other

With several agents, the same walk runs agent by agent. Check the input and the output at each handoff, and look for the first agent that turned a good input into a bad output. That is often not the agent that gave the final answer. A feature request on Arize Phoenix's tracker gives an example: "Every span is status=OK. Final answer is wrong. Naive blame points at execute_sql. Real blame is Agent A."

Agent frameworks give you the handoffs to check. The OpenAI Agents SDK wraps each handoff in its own span, and in OpenTelemetry's conventions each agent that another agent calls gets its own invoke_agent span under the caller.

A person still has to walk those handoffs, because automatic attribution is not reliable yet. In a 2025 study of failure logs from 127 multi-agent systems, the best method tested named the responsible agent 53.5% of the time and the step where the failure began 14.2% of the time. When a user asked for a general utility to assign the blame, the tool's maintainers declined because "Root-cause attribution depends heavily on the application's architecture, telemetry, and definition of failure".

Narrowing a long trace before you walk it

A long session can have hundreds of spans, so narrow it before you walk it. Filter the spans by status, duration and name. Spans that failed, were retried or took longer than usual are good places to start, but treat them as leads. In a session like the Acme quote, where every step succeeded, a status filter finds nothing. Filtering by tool name or by the account involved still narrows it, and then you follow the data back from the answer.

Rerunning the suspect step

Once you have a suspect, test it. "Replay" can mean three different things, and they answer different questions.

  • Running one step again with its captured inputs. You send the same prompt to the same model and see what it does. Models do not always give the same answer twice, so run it several times before you draw a conclusion.
  • Running the agent with captured results up to the suspect. Your test harness feeds the captured model responses and tool outputs back in for every step before the suspect, so those steps get exactly what they got the first time. Then the suspect runs live with your fix. This tests the fix on the broken step without earlier steps taking a different path.
  • Playing a session back to read it. Langfuse's session replay shows a whole conversation in order. It helps you read the session, but it does not run anything.

All three depend on the trace holding each step's inputs and outputs. Once the fix works, keep the session as a regression case, so the next change to a prompt or a tool is tested against it.

Why some steps never show up in the trace

A step that ran is usually missing because nothing wrapped it in a span, it lost its link to the session, a setting turned tracing off, or the process ended before the spans were sent. The walk back stops at any step that is not in the trace. No tool can find a step that was never instrumented, so the fix is in your code or your tracing setup.

Cause What you see The fix
Nothing wrapped the step in a span The step is missing, but its effect shows up in later steps Wrap it: @observe() in Langfuse, @traceable in LangSmith, tracer.start_as_current_span() with the OpenTelemetry SDK
The step lost its link to the session The step shows up as a separate trace Pass the trace context to it, as described below
A setting turned tracing off or stripped the content The span is missing, or present with its inputs and outputs stripped Look for settings such as OPENAI_AGENTS_DISABLE_TRACING, LANGSMITH_HIDE_INPUTS, or capture_input=False on a Langfuse @observe()
The process ended or was frozen before the spans were sent The last steps of serverless functions and some worker pools are missing Flush before the process ends: force_flush() on the OpenTelemetry tracer provider, langfuse.flush() in Langfuse, or flush_traces() in the OpenAI Agents SDK

The second cause is the hardest to spot. A span finds its parent through context, which is the information that says which trace and which span are current. In Python, OpenTelemetry's Python API keeps that information in a context variable, which follows the code as it runs. An asyncio task copies the context when it is created, so tasks keep their parent, and asyncio.to_thread carries it into the thread for you. A thread pool does not. Work sent through ThreadPoolExecutor.submit or loop.run_in_executor starts with no current span and becomes the start of a new trace.

One Langfuse user reported that symptom with a LlamaIndex tool that was not traced. Adding @observe made the tool appear, but "it creates a completely separate entry in my traces" instead of showing up under the agent. A Langfuse contributor suspected LlamaIndex's thread pools and pointed to carrying the context into worker threads.

That put the tool in the same trace, but for that user it was "only appended at the very end of it, outside of the Agent Loop", and the issue was still open in October 2026. Carrying the context is needed, but it is not always enough.

You can carry the context by hand, with a fresh copy of the context for each task you submit. Reusing one copy for two tasks running at once raises a RuntimeError.

python
import contextvars
from concurrent.futures import ThreadPoolExecutor

with ThreadPoolExecutor() as pool:
    ctx = contextvars.copy_context()   # one copy per task
    result = pool.submit(ctx.run, search_accounts, "Acme").result()

Or let a library do it. OpenTelemetry's threading instrumentation, ThreadingInstrumentor().instrument(), carries the context into every thread, and Langfuse's docs recommend it. LangSmith has its own ContextThreadPoolExecutor.

When a step runs in another process, such as a background job or a consumer reading from a queue, the context has to travel with the message. OpenTelemetry's propagators do that. They write the trace context into the message's headers on the way out and read it back on the way in.

Finding the failing step in thousands of log lines

Put the session ID, step ID, parent step ID and attempt number on every log line, so one search pulls a session out of thousands of lines. Without them, chained tools and retries are hard to follow in the logs. An SRE on r/Observability described going through "fifty thousand lines to find the prompt or tool call that sent things off the rails." The same engineer had already tried structured JSON logs with full prompts and responses, and found they became "its own archaeological dig".

The links between steps are what make logs searchable, so give every log line the same tree a trace has, and log how each attempt ended. If you also have traces, add the trace ID, so one search finds both the session's log lines and its trace.

json
{"level": "WARN", "session_id": "sess_8f2c", "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "step_id": "s2", "parent_step_id": "s1", "step": "search_accounts", "attempt": 2, "outcome": "retry"}

Log retries at WARN and keep ERROR for the attempt that finally failed, so a search for errors returns failures and not noise. Then a tool such as jq can pull out one session, or only the final failures. The fromjson? form skips any line that is not JSON, so a stray plain-text line does not stop the search.

bash
jq -cR 'fromjson? | select(.session_id == "sess_8f2c")' agent.log
jq -cR 'fromjson? | select(.level == "ERROR")' agent.log

For a session where nothing failed, like the Acme quote, pull the whole session by its ID and follow the data back, as you would in a trace.

Catching a bad step before later steps build on it

Check the data each time one step hands it to the next, so bad data is caught at the step that made it, before later steps build on it. Check what a tool returned, what a model asked a tool to do, and what one agent passed to another.

  • Checks on shape ask whether the data looks right. Does the JSON have the fields you expect, and is the total a positive number? These are easy to write, but they would not have caught the Acme quote, because Acme Logistics was a real account with every field filled in.
  • Checks on sense ask whether the data fits this request. Did the search find more than one account for the name? Does the account belong to the rep who asked? Is the discount within what the rep's role can offer?

A check on sense has to see what it is checking. In the Acme case the search tool kept the first account and dropped the second, so no check after it could have seen two matches. The check belongs inside the search tool, over every account it found. A tool that quietly picks one result, or fills in a default such as an empty list or "unknown", is itself the failure to check for. Once the agent knows the account's id, it can look the account up by id and skip the guess.

When a check fails, mark the span as failed with a message that says why, so the next person who opens the trace is pointed at this step. Then decide what the agent does next. It can retry, ask the user to choose, or stop. In the Acme case, the search tool counts every account that matched and asks the user to choose when the count is not exactly one.

python
from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer(__name__)

def search_accounts(name):
    with tracer.start_as_current_span("execute_tool search_accounts"):
        matches = find_accounts_by_name(name)  # every match, not only the first
        return check_account_matches(name, matches)

def check_account_matches(name, matches):
    if len(matches) == 1:
        return {"status": "ok", "account": matches[0]}
    trace.get_current_span().set_status(
        Status(StatusCode.ERROR, f"{len(matches)} accounts matched {name!r}")
    )
    return {"status": "ask_user", "matches": [m["name"] for m in matches]}

Before a step that changes something outside the agent, such as sending a message, charging a card or deleting data, a failed check should stop the session there. The agent stops the action and asks the user, or waits for a person to approve.

Limits on the whole session help too: a maximum number of steps, a cap on tool failures in a row, and a stop when the same tool keeps being called with the same arguments. Limits stop the damage from spreading. The checks on shape and sense are what point at the step that caused it.

Keeping the session you need to walk back

In Oodle, every session is kept in full for your retention period (30 days on Pro, longer for an added fee). That is affordable because of how Oodle is built. Traces go straight to object storage in Oodle's own columnar format, and queries run on serverless compute that spins up per query and shuts down after. One trace runs from the API request through the agent and its tools to the database, once your services pass trace context to each other and your database client sends spans. That is what the walk needs: a kept session whose trace reaches the broken step.

Every session is analyzed at scale using statistics, ML classifiers and a small model for sentiment. Oodle's agent then checks each issue it finds against the evidence and removes the noise. Sessions with tool failures, loops, outages or stalled conversations are flagged before anyone opens them. Tracing shows only the steps your code sends, and a result like the Acme search still needs your own check. The Oodle playground has agent traces to walk, with no login.

See your own telemetry in Oodle

Start debugging in minutes.

Start freeBook a demo

Common questions

My agent has several tools, branches and other agents, and I can't tell where the run went off track or replay it. How do people debug this?

They follow the data backward from the answer through a trace that holds each step's inputs and outputs. The step that broke is the first one whose output was wrong while its inputs were right. To test it, they run that step again several times with its captured inputs, or rerun the agent with the captured model and tool outputs fed back in.

In a multi-agent pipeline every span reports success, but the final answer is wrong. How do I find which agent caused it?

Walk the handoffs backward from the final answer, checking each agent's input and output. The cause is the first agent that turned a good input into a bad output, which is often not the agent that answered. Automatic attribution is not reliable yet, so a person who knows what the application should have done still walks it.

My agent chains tools and retries, and I scroll through thousands of log lines to find the one call that broke. How do I find the failing step faster?

Add the session ID, step ID, parent step ID and attempt number to every log line, plus the trace ID if you have traces. Log retries at WARN and only the final failure at ERROR. Then a jq filter on the session ID or on ERROR pulls out just the lines you need. If nothing failed, pull the whole session by its ID and follow the data back from the answer.

My agent keeps going on bad data, so a small error cascades for many steps before anything looks wrong. How do I catch it where it starts?

Check the data at every handoff between steps, not only the final answer. Checks on shape catch malformed data, and checks on sense catch data that is well formed but wrong for this request, such as a search that matched two accounts and returned one. When a check fails, mark that span as failed and retry, ask or stop, and stop before any step that sends, charges or deletes.

Sources

Share