FIND US ONProduct HuntAgent ObservabilitySee how we help you debug AI agents, and share your feedback.

Oodle integration with TrueFoundry AI Gateway: AI observability for every LLM call

TrueFoundry AI Gateway sending OpenTelemetry traces and metrics to Oodle asynchronously
On this pageWhat the TrueFoundry AI Gateway doesWhat Oodle addsWhy trace at the gateway, not in each appHow the integration worksHow to set it upWhat you can do with gateway traces in OodleLet agents query Oodle through the MCP GatewayNext steps and related readingFrequently asked questions

An AI agent gives a customer a wrong answer. The same week, the model bill comes in much higher than last month. In both cases the first question is the same: which request caused it?

Most teams can't answer that quickly. Every app logs its model calls a little differently, if it logs them at all. And finance wants spend by team, in a format nobody planned for.

So TrueFoundry and Oodle built an answer at the one place every model call already passes through: the AI gateway.

That is what AI observability means in practice: a record of what each model and agent call did, how long it took, what it cost and where it failed.

What the TrueFoundry AI Gateway does

The TrueFoundry AI Gateway sits between your applications and the models they call. Teams reach 1,600+ models through one OpenAI-compatible API. The platform team sets routing, fallbacks, guardrails, budgets and rate limits in one place. The TrueFoundry MCP Gateway does the same for the tools agents call: access control and guardrails.

Because every request passes through it, the gateway knows things no single app does:

  • which team sent a request
  • which model served it
  • whether a guardrail stepped in or a budget blocked it
  • what it cost

TrueFoundry's guide to AI gateway observability covers these signals in more depth.

What Oodle adds

Oodle is observability built for agents. Next to metrics, logs, traces, APM, real user monitoring and infrastructure monitoring, it has Agent Observability: traces, evaluations, prompt management and automated insights. TrueFoundry runs alerting for its own gateway and platform on Oodle.

Oodle is built on OpenTelemetry, the open source standard for traces, metrics and logs. The gateway can label its spans with OpenTelemetry's semantic conventions for generative AI, the standard names for model calls, so Oodle recognizes them as GenAI traces.

Oodle also keeps every trace by default instead of sampling. The one request a user complains about is still there when you go looking for it. (More on why in You can't sample your way to reliable agents.)

Why trace at the gateway, not in each app

Most AI observability starts inside the application. Each team adds an SDK, and coverage depends on who remembered to.

Tracing at the gateway works differently. The gateway records every model call as a standard OpenTelemetry trace, so one export setting covers every app and agent that calls a model through TrueFoundry, including the ones nobody instrumented. It also captures events only the gateway sees, such as a fallback from one provider to another, or a request blocked by a budget or rate limit.

Here is how the work splits between the two:

AreaTrueFoundry AI GatewayOodle
Each requestRecords a timed span for each guardrail check, the model call and the provider callShows the request as a GenAI trace with model, latency and cost, plus the prompt and response
CostCalculates the cost of every requestPrices each call from model rates set in Oodle and breaks down spend for chargeback
Limits and alertsEnforces budgets and rate limits and records each blockRuns monitors with runbooks and routes alerts to on-call tools
QualityApplies guardrails and records each resultScores traffic with LLM-as-judge evaluators and turns failing traces into test datasets
Agent accessLists the Oodle MCP server and controls who can call which of its toolsAnswers agents' queries about logs, metrics, traces and alerts
Setup and dataAn admin turns on export once, and TrueFoundry keeps its own copyKeeps every trace by default instead of sampling

How the integration works

TrueFoundry AI Gateway sending OpenTelemetry traces and metrics to Oodle asynchronously

  1. An app or agent sends its request to the TrueFoundry AI Gateway, exactly as it does today.
  2. The gateway applies your policies, calls the model and records each step as an OpenTelemetry trace, along with gateway metrics.
  3. The gateway sends a copy to Oodle asynchronously, off the request path, so a slow or unreachable Oodle can't hold up a model call. TrueFoundry keeps its own copy, so its dashboards don't change.
  4. In Oodle, teams search and replay requests, track cost and latency, run evaluations and set alerts.
  5. Optionally, agents query Oodle back through the Oodle MCP server in the MCP Gateway.

Gateway traces cover every model call. For full agent sessions, per-user views and agent graphs, also add OpenTelemetry to the app itself. When the app passes its trace context along, the gateway continues the same trace, and the app's spans line up with the gateway's in Oodle.

How to set it up

TrueFoundry's Oodle setup guide and Oodle's TrueFoundry guide have the details for each step.

  1. In Oodle, open the TrueFoundry tile on the Integrations page. It lists the connection details for your Oodle account.
  2. In TrueFoundry, open the AI Gateway settings, turn on trace export and enter those details.
  3. If you want gateway metrics in Oodle as well, turn on metrics export the same way.
  4. Save, send a few requests through the gateway, and check that they appear in Oodle's trace explorer.

TrueFoundry AI Gateway OpenTelemetry trace and metrics export settings pointed at Oodle

A TrueFoundry AI Gateway request in Oodle with gateway, model and provider spans

In hybrid and self-hosted deployments, the gateway running in your own cluster sends the traces to Oodle itself, and the setup guide covers that path.

What you can do with gateway traces in Oodle

Debug a slow or wrong answer

When someone reports a bad answer, find the exact request behind it. Search on user, team, error type, guardrail result, request metadata, or text in the prompt or response. A gateway trace in Oodle contains:

  • the model that served the request, with its latency and cost
  • a timed span for each step: the incoming request, each guardrail check, the model call and the provider call
  • the result of each guardrail check and any error
  • the user, team and other metadata attached to the request
  • the prompt and the response

Oodle trace of a failed agent request with the slow tool call and its error chain highlighted

We checked the first two on gateway traffic in a test environment. If a fallback kicked in, Oodle shows each attempt as its own span under the same request. You can then open the trace in Oodle's playground to replay the conversation and test a fix.

Track LLM cost by team, user and model

In our tests, Oodle's per-request cost matched the gateway's own. On one test request, Oodle showed $0.030 and the gateway $0.029958: the same amount, rounded to a tenth of a cent.

Oodle prices each call from the model rates you set in Oodle, so keep those in line with your provider's pricing. From there you can split spend by model, team, user or any metadata label for chargeback, and get an alert when spending climbs. For the finance side, see TrueFoundry's guide to LLM cost tracking and Oodle's AI cost management.

Oodle dashboard splitting LLM spend by model, agent and provider

Watch latency, errors and rate limits

In our tests, these gateway metrics arrived in Oodle next to the traces:

  • requests, HTTP status codes and cost
  • latency, time to first token and the time between tokens
  • budget and rate-limit checks, and the requests they blocked
  • MCP request counts and latency

Oodle monitors can alert on any of these and send the alert, with a runbook attached, to the chat and on-call tools your team already uses.

Check answer quality on real traffic

Oodle can flag behavior as traces arrive, such as a refusal, or a user asking the same thing again without getting an answer. You can describe your own signal in plain English. It also runs LLM-as-judge evaluators for issues like hallucination and toxicity.

You can add failing traces to a dataset and test a new prompt or model against those real cases before it ships. We haven't tested these features on gateway traffic yet, so Oodle's docs are the reference for how they behave there.

Let agents query Oodle through the MCP Gateway

AI agents querying the Oodle MCP server through the TrueFoundry MCP Gateway with per-user OAuth sign-in

The Oodle MCP server is one of the official remote MCP servers in TrueFoundry's MCP Gateway. Agents behind the gateway can use it to query logs, metrics, traces and alerts in Oodle. An on-call agent, for example, can list active alerts and pull the traces behind them.

With OAuth sign-in, each user connects their own Oodle account, so an agent sees only what that person can see in Oodle. An Oodle API key works too. The gateway decides who can use which Oodle tools, applies guardrails to those calls, and traces each one with the caller's identity.

Oodle's Connect TrueFoundry page with steps to add the Oodle MCP server to the TrueFoundry MCP Gateway

Turning on this integration is a one-time change in the gateway settings, with no application code. From then on, every model call through the gateway reaches Oodle as a trace with its cost attached. Engineers can debug it, and finance can report on it.

To get started:

Related reading:

Frequently asked questions

What is the Oodle integration with TrueFoundry AI Gateway?

It connects the two products in two ways, and each works on its own. The gateway exports OpenTelemetry traces and metrics for every request to Oodle, where teams debug calls and track cost. Separately, Oodle's MCP server is available in TrueFoundry's MCP Gateway, so agents can query Oodle under the gateway's access controls.

What's the difference between LLM observability and agent observability?

LLM observability tracks single model calls: the prompt, the response, latency, errors and cost. That is enough to debug one slow or wrong answer. Agent observability follows a whole agent run across many model and tool calls, to explain why the agent did what it did. Gateway export covers every model call, and instrumenting the agent itself joins those calls into full agent traces in Oodle.

What latency does the TrueFoundry AI Gateway add?

Very little. The gateway adds under 3 ms of internal latency, and in TrueFoundry's published benchmark, run on a machine with 1 vCPU and 1 GB of memory, it handled 350+ requests per second. Export to Oodle runs off the request path, so an Oodle outage can't block a model call.

Does TrueFoundry support MCP and AI agents?

Yes. The MCP Gateway sits in front of the MCP servers your agents use, including Oodle's. It controls who can call which tool, applies guardrails to those calls, and traces each call with the caller's identity. Gateway metrics in Oodle include MCP request counts and latency, so tool traffic shows up next to model traffic.

Does TrueFoundry work with my existing observability stack?

Yes. The gateway exports standard OpenTelemetry traces and metrics, so it works with Oodle and with other OpenTelemetry-compatible backends. TrueFoundry's export docs explain how to connect them.

Written byGaurav Maheshwari"A body of clay, a mind full of play, a life of a moment - that's me" - Harivansh Rai Bachchan

See your own telemetry in Oodle

Start debugging in minutes.

Start freeBook a demo

Get new posts by email

Engineering write-ups from the team.