Debugging AI agent failures at scale is hard Akhilesh Krishnan  ·  Oodle AI
Workato speaker card: Akhilesh Krishnan, Founding Engineer, Oodle AI
Enterprise MCP / lightning talk

Debugging AI agent failures at scale is hard.

Oodle AI All-in-One
AI-Native Observability
Akhilesh Krishnan  ·  Founding Engineer, Oodle AI
In production at Fello

100,000 phone calls a day.
Millions of spans.

Fello builds Felix, an AI twin for a real estate agent.

Challenges at scale.

  1. 01Storing all agent traces
  2. 02Detecting agent failures
  3. 03Fixing agent failures

Storing agent traces at scale.

$ COST 100,000 calls a day TRACES →

At this volume, keeping everything is the expensive part.

Getting too costly.
So let’s start sampling.

I am broken, but my developer is sampling traces.

You cannot debug what you can’t see.

Store everything.

OBJECT STORE every trace, no sampling λλ λλ serverless query, in parallel

Agents fail differently.

Agents lie
Done — I emailed the seller. TOOL CALLS ( none )
Agents overthink
get_price() get_price() get_price() ×14 12,480 TOKENS BURNED

Using LLM evals to detect issues isn’t cheap.

3,000,000 conversations judged / month $15,000 on Haiku 4.5

Back to the basics.

100% SENTIMENT ANOMALIES 25% LLM AGENT AS JUDGE only what looks wrong SILENT FAILURES

Traditional classification and anomaly detection can take you a long way.

Save up to 75% in eval inference cost.

Debug your agents from anywhere with Oodle MCP.

MCP

“You can’t sample your way to reliable agents.”

Oodle AI All-in-One
AI-Native Observability
CARD ← →  ·  F present

Takeaways

  1. Challenges at scale.
  2. Storing all traces at scale is very expensive.
  3. With Oodle’s architecture you can store all your traces cost efficiently.
  4. Running LLM as judge on all your traces can get very expensive, making it hard to detect issues.
  5. Use Oodle’s hierarchical eval architecture to detect silent failures at scale.
CARD