On Product HuntAgent ObservabilitySee how we help you debug AI agents, and share your feedback.

What Is Agent Observability? A Guide to Oodle for New Users

An Oodle agent trace where the customer gives up, flagged with negative sentiment and a User gave up signal
On this pageTL;DRWhat is agent observability?Why we built itWhat problems does it solve?How it worksWhat makes Oodle different?Get started with your coding agentFrequently asked questionsParting thoughts

A customer writes to an online store's support agent:

My wireless headphones stopped charging after two weeks. Order C-55018.

The agent replies right away: "I've arranged a replacement (order C-55190, ships in 2 days)."

A few days later, the same customer is back: "It's been days and no replacement has shown up." The agent says sorry and promises to look into it. Then: "Is anyone reading these? Still no replacement. Awful service." The agent escalates. And then the last message:

Useless. I'll call you instead.

Now open the trace for that conversation. No span has an error, and the whole turn took under two seconds. By every number on your dashboard, the agent did its job. It didn't.

This conversation comes from our demo environment, which anyone can open. Failures like this one are why we built Agent Observability in Oodle. Below are the questions new users ask me most: what it does, why we built it, what it solves, how it works, and what makes it different.

TL;DR

  • What it is: Oodle Agent Observability shows you what your AI agents said and did in every conversation: the prompts, replies, tool calls, tokens and cost.
  • Why we built it: agent failures look like success. Errors and latency charts can't see them, and sampling throws away the conversation you need.
  • How it works: send traces with OpenTelemetry. Oodle keeps every session, flags problems on its own, scores quality with evaluators, and helps you test a fix before you ship it.
  • What's different: no sampling, cost-efficient checks on every trace before any expensive LLM judge, and agent traces in the same place as your application and infrastructure data.

What is agent observability?

Agent observability means you can see what your AI agent did in a conversation, and know when it went wrong.

Normal tracing answers questions like "was it slow?" and "did it throw an error?". Those still matter. But for an agent, the harder questions are different:

  • Did the agent actually help the user?
  • Did it do what it said it did?
  • Did it call the right tool, with the right arguments?
  • Is the new prompt better or worse than the old one?
  • How much did this conversation cost?

To answer them, you need the conversation itself, every tool call and what came back, and a way to check quality across thousands of sessions.

Why we built it

Oodle started as an observability platform for metrics, logs and traces. Then our customers started shipping AI agents, and their questions changed. We kept seeing two problems.

1. Agent failures look like success. In the story above, nothing crashed. The agent replied politely, every call returned OK, and the customer left anyway. No error rate goes red for that.

2. Teams sampled away the data they needed. Agent traces are big. One model call can carry a long system prompt and the whole chat history. Storing all of that got expensive, so teams kept only a small sample. Then a customer complained, and the trace was gone. Fello, whose AI teammate makes about 100K calls a day, hit exactly this:

When a customer reported something, we couldn't find the trace if it had been sampled out.

– Ashish Gupta, Principal Engineer, Fello

Here is the Fello team telling that story in their own words:

How Fello observes 100K+ voice agent calls every day without sampling

We had already built our own storage engine on object storage, which keeps data cheap to store. So we asked: what if you never had to sample agent traces? And what if the platform read every conversation and told you which ones went wrong?

That became Agent Observability. For the engineering story, our CTO, Vijay Karthik, wrote about how we made every agent span searchable and why sampling fails for agents.

What problems does it solve?

Here are the failures we see most often in agent traces. Most of them never raise an error.

The problemWhat it looks like
The tool fails and the agent carries onA tool errors, the model doesn't notice, and tells the user it worked
The agent loopsSame tool, again and again, with slightly different arguments. The only sign is the bill and the wait
Something the agent depends on is downA rate limit, a timeout or an overloaded model. Not the agent's fault, but your users still feel it
The user isn't getting anywhereThey repeat themselves, get frustrated, or leave, like the headphones customer
A prompt change made things worseVersion 3 of a prompt fails more often, or uses many more tokens, than version 2
Costs creep upMore turns, more tokens, a pricier model, with no one noticing

How it works

If you'd rather watch than read, this short demo walks through debugging AI agents at scale in Oodle:

Debug AI agents at scale, without sampling

There are five steps. You only need the first one to get started.

Step 1: Send your agent traces

Oodle takes standard OpenTelemetry traces that follow the GenAI conventions. Most agent frameworks can already send them, including LangChain and LangGraph, Pydantic AI, the Vercel AI SDK, the OpenAI Agents SDK, CrewAI and Mastra. The easiest way to set this up is to let your coding agent do it, as I show in getting started below. If you prefer to do it by hand, the integrations page has a copy-paste snippet for each framework.

By hand, pointing your app at Oodle is mostly a few environment variables:

export OTEL_EXPORTER_OTLP_ENDPOINT=https://<OODLE_INSTANCE>-otlp.collector.oodle.ai
export OTEL_EXPORTER_OTLP_HEADERS="X-API-KEY=<OODLE_API_KEY>,X-OODLE-INSTANCE=<OODLE_INSTANCE>"
export OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental
export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=span_and_event

The last line captures prompt and reply text, which can hold personal data. Check your data policy first, and see our post on handling PII in logs if you need to redact.

Two tips that make everything else better: set session.id so turns group into one conversation, and user.id so you can see each user's history.

Step 2: Read the conversation

Click any trace and you see it as a chat: the system prompt, what the user said, what the model replied, and which tools it called. You also see tokens, cost, latency and a sentiment score for each user turn.

The headphones conversation in Oodle's Transcript view, with negative sentiment and a User gave up signal on the trace
The Transcript view: the whole conversation, with sentiment on every user turn

A few views help when you don't know where to start:

  • Search across every session by what the user or agent said, by tool, by user, or by any ID, such as an order number.
  • Sessions group the turns of one conversation together.
  • Agent Graph draws how your agents, sub-agents and tools connect, with traffic and error rate on each one.
Oodle's Agent Graph showing agents such as shopassist, returns_agent and orders_agent with trace counts, error rates and insights
Agent Graph: which agents call which, and where errors and insights are

Step 3: Let Oodle find the problems

You won't read thousands of conversations yourself, so Oodle reads them as they arrive. It labels behavior on each trace with signals: a tool failed, the agent looped, a rate limit hit, the user repeated themselves or gave up. These labels are cheap to compute, so they run on every trace, and you get a real rate you can chart and alert on.

Oodle then groups what it found into Insights, ranked by severity. In the demo, the top one is the story from the start of this post: "User gave up in shopassist-agent", in 23% of the traces it analysed.

Oodle's Insights tab listing agent problems by category, with "User gave up in shopassist-agent" ranked first
Insights: what went wrong, how often, with example traces

Insights also compare prompt versions. In the same demo, Oodle noticed that version 3 of a support prompt fails 11 times as often as version 2, on the same traffic.

An Oodle insight reporting that prompt shopassist-support-reply v3 has 11x more LLM errors than v2
A prompt regression, caught by comparing versions on live traffic

Each insight has example traces, and a debug prompt you can copy or open straight in your coding agent.

Step 4: Score quality with evaluators

Signals tell you that something looks off. Evaluators tell you whether a reply was right by your own rules. There are three kinds:

  1. Code evaluators: plain Python checks, such as "does the refund amount match the policy?". Fast, and no LLM cost.
  2. LLM-as-a-judge: a model reads the conversation and scores it, with a reason. Good for things code can't check, like helpfulness or made-up facts.
  3. Webhook evaluators: call a scorer you already run.

Here is an LLM judge on the demo store. It checks one thing: does the agent's reply only claim what the tools actually did?

An Oodle LLM judge's execution log, with a reasoning line: the reply says the refund is done, but initiate_return was refused
An LLM judge catching a reply that claims a refund the tool refused

The key idea is to chain them. Run the cheap checks on every trace, and send only the flagged ones to the expensive LLM judge. You get coverage on everything and pay for the judge only where it matters. Fello's team describes it well:

With Oodle, we can chain evals by dependency. Code-based evals filter traces before we run LLM-based evals.

– Aman Gupta, Product Manager, Fello

That chaining cut Fello's eval inference cost by 80-85%.

Step 5: Fix it, and prove the fix

Finding a problem is half the job. For the other half:

  • Prompt management: keep prompts outside your code, with versions and labels like production. Ship a new prompt without a deploy.
  • Playground: open any model call, change the prompt, and run it again.
  • Datasets and experiments: save the conversations that went wrong as test cases. Run a new prompt against them, compare scores, and release the one that does better.

The same evaluators keep watching production, so if the problem comes back, you know first. I put all of this in one loop in building self-improving agents with Temporal and Oodle.

What makes Oodle different?

  1. Every session, no sampling. Traces go to object storage in our own format, and queries run on serverless compute. That is cheap enough to keep 100% of your sessions, so the one a customer complains about is always there.
  2. Cheap checks first. Signals and code evaluators run on every trace. The expensive LLM judge runs only where they point.
  3. Problems find you. Insights surface failures you didn't know to look for, ranked, with example traces.
  4. Agents and infrastructure in one place. Oodle is also a full observability platform, so you can follow a tool call into the APIs and databases it hit.
  5. Open standards. It is built on OpenTelemetry, with no proprietary SDK. If you already send OpenTelemetry traces somewhere, switching is an endpoint change.
  6. Built for coding agents too. Oodle's MCP server lets Claude Code, Cursor or Codex search your sessions and debug them for you.

Get started with your coding agent

You don't need to read setup docs to send your first trace. Your coding agent (Claude Code, Cursor or Codex) can do it for you, in two commands.

  1. Sign up for Oodle for free.

  2. In your terminal, install the Oodle skills:

    curl -fsSL https://www.oodle.ai/install.sh | bash
    

    This installs the Oodle CLI and the Oodle agent skills, and logs the CLI in to your Oodle account. It needs Node.js 22.20 or later and git.

  3. Open your agent's repo, and in your coding agent's chat, run:

    /oodle-onboarding
    

    Tell it what you want, for example: "Send my support agent's traces to Oodle."

Here is what your coding agent then does:

  • Looks around. It finds your framework (LangChain, Pydantic AI, the Vercel AI SDK and so on) and how your app is deployed.
  • Asks first. It asks all its open questions in one message, then shows you a plan of which files it will change.
  • Changes the code. It adds the instrumentation and the Oodle settings to your app.
  • Leaves the deploy to you. It never deploys on its own. It gives you the commands, or opens a PR if you agree.
  • Checks the data. Once you deploy, it confirms your traces have arrived in Oodle.

The quick start docs have the full steps, including the command for Windows.

Once traces arrive:

  1. Open Agent Observability → Traces and click a conversation.
  2. Add model prices in model definitions so each trace shows its cost.
  3. Turn on one evaluator, for example a built-in code check, under Agent Observability → Evaluators.
  4. Come back the next day and open Insights.

Not ready to send your own data? Open the demo and click around. The headphones conversation is in there.

Frequently asked questions

What is the difference between agent observability and LLM observability?

LLM observability looks at single model calls: the prompt, the reply, tokens and latency. Agent observability looks at the whole task: many model calls, tool calls and sub-agents, across a multi-turn conversation, and whether the user got what they needed.

Does Oodle sample agent traces?

No. Oodle keeps every session, so you can search and evaluate all of them. You can still sample evaluators if you want to cap their cost.

How much does Oodle Agent Observability cost?

Agent Observability Pro is $100 per month and includes 125 GB of ingested data, then $1 per GB. There is no charge per seat or per query. See pricing for details.

Parting thoughts

Agents fail quietly. They are polite, fast and confident, even when they are wrong. To catch that, you need to:

  1. Keep every conversation, not a sample.
  2. Read all of them automatically, with cheap checks.
  3. Spend on deep checks only where the cheap ones point.
  4. Test every fix on real failures before you ship it.

That is what we built Agent Observability in Oodle to do. Send your first trace and see what your agents are really telling your users.

Try Oodle Agent Observability for free

Written byGaurav Maheshwari"A body of clay, a mind full of play, a life of a moment - that's me" - Harivansh Rai Bachchan

See your own telemetry in Oodle

Start debugging in minutes.

Start freeBook a demo

Get new posts by email

Engineering write-ups from the team.