FIND US ONProduct HuntAgent ObservabilitySee how we help you debug AI agents, and share your feedback.

The best LLM observability tools, compared on sampling, retention and price

Langfuse, LangSmith, Arize Phoenix, Helicone, Datadog and Oodle each bill on a different unit. Compare what keeping every agent session costs.

Cover with the words "One session, four billing units". One agent session of 4 model calls, 8 tool calls and 1 score, measured by four meters: Langfuse about 14 units, LangSmith 1 trace, Datadog 4 LLM spans and Helicone 4 requests.
On this page
  1. The terms each tool uses
  2. What each tool traces
  3. How the billing unit decides whether you sample
  4. What keeping every session costs on each tool
  5. Why teams sample
  6. How far back each tool lets you debug an incident
  7. Running a tool yourself
  8. How the LLM observability tools compare
  9. The tools, one by one
  10. Other AI observability tools on the lists
  11. How each tool handles LLM cost tracking
  12. Which tool fits which team
  13. How to test two tools before you commit
  14. Common questions
  15. Sources

TL;DR

  • The shortlist is Langfuse, LangSmith, Arize Phoenix, Helicone, Datadog Agent Observability and Oodle, and each bills on a different unit.
  • Every tool here can sample, and teams turn sampling on to control cost, not because a tool cannot keep everything.
  • 1 million one-turn sessions a month cost $991 on Langfuse, $1,525 on Datadog and $4,989 on LangSmith.
  • A bill by the gigabyte grows with the size of each session, and a bill per model call does not, which matters for tool-heavy agents.
  • Default retention runs from seven days to forever, so check which events extend a trace's retention and what happens when it expires.
  • Before you self-host, read the license and count the services you would run, since a small team often pays less on a hosted plan.

Whether you keep every agent session comes down to what one session costs on each tool's bill, since every LLM observability tool here can keep all of them and lets you sample to spend less. Langfuse counts traces, steps and scores, LangSmith counts traces, Datadog counts model calls, Helicone counts requests, and Oodle, which publishes this guide, counts gigabytes. OpenTelemetry is the open standard for instrumenting code, and Langfuse, LangSmith, Phoenix, Datadog and Oodle all accept its data, so you can instrument once and compare them on your own traffic.

The terms each tool uses

The tools name the same things differently, so here is each term once.

The difference between a session and a trace matters when you decide what to keep. A one-turn session is one trace. A support chat with ten turns is ten traces in one session. A sampler that keeps a share of traces decides from the trace ID, so it can keep turn three of a chat and drop turn four. The conversation you need to read then has gaps.

What each tool traces

A tool sees the model request alone, the agent's steps, or the agent's steps inside the whole request. That decides what you can debug, because a tool cannot help with a step it never received. Some tools log the model request and little else. Most record the agent's steps: each model call, tool call and retrieval step as spans in one trace. A few keep that trace inside the rest of the request, down to the database query a tool call made, when your services pass trace context along.

The defaults matter here. Langfuse accepts OpenTelemetry traces, but by default its SDK exports only the LLM-relevant spans, and to send everything you set a permissive filter. Datadog records the whole agent workflow and bills only the LLM spans in it. To see the service request around the agent, Datadog customers add its APM.

To check a tool, send it one real agent session. Then look for the tool calls, the retrieval and the database query under them in a single trace.

How the billing unit decides whether you sample

A per-trace bill grows with traffic. A per-unit or per-gigabyte bill also grows when the agent takes more steps or sends longer prompts. When the bill grows faster than the value of old traces, teams turn on sampling.

Every hosted tool bills on a unit, and each one counts something different. So the same agent session costs a different number of units in each tool.

Take one session of one turn: four model calls, eight tool calls and one evaluation score. Langfuse counts every trace, observation and score as a unit, so the session is about 14 units if each call is one observation. LangSmith counts the session as one trace, however many steps it has. Datadog counts four LLM spans and does not bill the tool calls. Helicone counts requests, so it sees the four model requests.

Tools that bill by the gigabyte, such as Braintrust and Oodle, count the bytes of every span. Agent spans carry prompts and responses, so they are large, and a per-gigabyte bill grows with prompt size.

What keeping every session costs on each tool

For 1 million sessions a month, each the one-turn session above, keeping every session costs $9,610 or more on Arize AX, the most of the tools priced here. Each figure is from the tool's own pricing page, as of October 2026, on the plan a team at this volume would start with. Oodle bills by the gigabyte ingested, starting at $1 per GB, so its bill depends on the size of your sessions. Measure one of your own sessions to work it out.

Tool and plan One session counts as A month of sessions Cost a month Retention included
Oodle Agent Observability Pro Its size in GB Your sessions' total size Ingest-based, from $1 per GB 30 days
Helicone Pro 4 requests 4 million requests $294.50, plus storage 1 month
Langfuse Core 14 units 14 million units $991 90 days to view
Datadog Agent Observability Pro, annual 4 LLM spans 4 million LLM spans $1,525 15 days
LangSmith Plus, one seat 1 trace 1 million traces $4,989 14 days
Arize AX Pro 12 spans or more 12 million spans From $9,610, plus $3 a GB past 10 GB 30 days

The working, tool by tool:

A bill by the gigabyte grows when sessions are large. Take a tool-heavy agent whose eight tool calls each return a 200 KB document. Its session is about 2 MB, so 1 million sessions is 2,000 GB, and a per-gigabyte bill grows with every one of them.

Datadog's bill stays at $1,525, because it bills the four model calls and not the tool spans. Langfuse's stays at $991, because it counts steps, not bytes. A long conversation resent with each of the four model calls is billed by the byte each time. Datadog says you "Only pay based on the volume of LLM spans, even as context grows, reasoning gets heavier, and agents use more tools."

Why teams sample

Every tool here can sample, and teams turn it on to cut cost. LangSmith's sampling docs say sampling helps "balance observability needs with cost considerations." Langfuse's default rate is 1, so it collects every trace until you lower it. Oodle's pricing is set so that sampling is optional.

Most built-in samplers decide when the trace starts. That is called head sampling. In OpenTelemetry's words, head sampling makes the decision "as early as possible", from the trace ID and a percentage. Langfuse's SDK, for example, samples "on the trace level" in your code. So head sampling cannot make sure the failing sessions are kept. At a 10% rate, the session a customer complains about is gone nine times out of ten.

Tail sampling decides after the trace is complete, so it can keep every trace with an error and a share of the rest. The OpenTelemetry Collector does this with its tail sampling processor, which "keeps spans in memory while it waits to make a sampling decision". It also needs every span of a trace to reach the same Collector instance. Any tool here that accepts OpenTelemetry can receive its spans from such a Collector. Inside LangSmith, an automation rule can upgrade the traces it matches to extended retention, which keeps the failures longer at a higher trace price.

A session where the agent gave a wrong answer while every call succeeded has no error for a rule to catch. No sampler can promise to keep the session that failed, so the number to check for each tool is what keeping every session costs at your volume.

How far back each tool lets you debug an incident

Default retention runs from seven days to forever (the table below has each one), and what differs more is what changes a trace's retention and what happens when it runs out. Retention decides how old a trace can be when you open it to explain an incident, and whether the tool meets a retention rule you must follow.

  • LangSmith keeps base traces for 14 days and extended traces for 180 days, the SaaS maximum since September 14, 2026. An online evaluator or an automation rule can upgrade a trace to extended, which raises its cost. Matching a single run upgrades the whole trace.
  • Langfuse Cloud limits how far back you can view by plan. That window does not delete data, and older traces can be opened again after an upgrade to a longer window. Deletion needs a retention policy, set per project with a minimum of 3 days (on the Cloud Pro and Enterprise plans, and an Enterprise feature when self-hosted). Self-hosted Langfuse keeps everything by default.
  • Phoenix, self-hosted, keeps everything until you set PHOENIX_DEFAULT_RETENTION_POLICY_DAYS.
  • Datadog keeps traces for 15 days and sells 30, 60 and 90 days as paid add-ons, priced per 10,000 LLM spans a month.
  • Oodle Agent Observability Pro keeps traces 30 days, and keeping them longer is an added fee, up to multi-year retention.

Check which events upgrade a trace's retention, and what the tool does with a trace when its time is up.

Running a tool yourself

If hosted units or retention cost too much, the other route is to run the tool yourself. The licenses run from MIT and the Apache License to the source-available Elastic License, and what you operate runs from one service on SQLite to a set of databases or your own Kubernetes cluster.

Tools sold as open source on this list use three kinds of license. Langfuse's core is MIT, Helicone and Opik use the Apache License (Helicone, Opik), and Phoenix uses the Elastic License 2.0. The Elastic License 2.0 is source-available: Phoenix is free to self-host, but the license bars offering it to others as a hosted service.

The services differ more than the licenses.

For a small team where nobody wants to own observability infrastructure, the count of services is the real cost. A hosted plan, checked against your unit count, is often the lower-cost route.

How the LLM observability tools compare

The tools differ most in the unit they bill on, what keeping every session costs on that unit, and how long they keep traces by default. The figures below are as of October 2026, from each vendor's own pages, for the 1 million sessions priced above.

Tool What it traces by default Billed on Cost to keep 100% of 1 million sessions a month Can it keep sessions based on how they ended? Default retention Free tier Run it yourself
Oodle Agent session, and the request around it when your services pass trace context and send their traces to Oodle GB ingested Ingest-based, from $1 per GB Keeps every session, and sampling is optional 30 days on Pro None for agent observability Hosted by Oodle. Bring your own cloud (BYOC) and bring your own bucket (BYOB) are Enterprise options
Langfuse Agent session (LLM-relevant spans from its SDK) Units: traces, observations and scores $991 on Core Not with its SDK sampler, which decides per trace Cloud: 30 days to view (Hobby), 90 (Core), 3 years (Pro). Self-hosted: forever 50,000 units a month Yes, MIT core, free without usage limits
LangSmith Agent session Seats plus traces $4,989 on Plus Its sampler is a percentage, and an automation rule can keep matching traces longer 14 days base, 180 days extended One seat, 5,000 base traces a month Enterprise add-on only
Arize Phoenix and Arize AX Agent session over OpenTelemetry Phoenix: free. AX: spans and GB per plan Phoenix: your servers. AX: $9,610 or more on Pro, plus $3 a GB past 10 GB Head sampling is the OpenTelemetry SDK default Phoenix: forever. AX: 15 or 30 days AX: 25,000 spans and 1 GB a month Phoenix, under the Elastic License 2.0
Helicone Model requests Requests and GB stored $294.50 on Pro, plus storage Not stated on its pages 7 days to forever, by plan 10,000 requests and 1 GB Yes, Apache License
Datadog Agent Observability Agent session, and the request around it with Datadog APM LLM spans only $1,525 on Pro, annual Not stated on its pages 15 days 40,000 LLM spans a month No

The tools, one by one

Oodle

Oodle is agent observability for teams running agents at production volume. Its cost comes from how it is built. Traces go straight to object storage in Oodle's own columnar format, and queries run on serverless compute that spins up per query and shuts down after. So every session is kept in full, for your retention period (30 days on Pro, longer for an added fee).

Agent traces are kept on one platform with logs, metrics and application traces. When your services pass trace context, one trace runs from the API request through the agent and its tools to the database. Every session is analyzed at scale using statistics, ML classifiers and a small model for sentiment. Oodle's agent then checks each issue it finds against the evidence and removes the noise.

Signals tag each session as it arrives, and insights run once a day, as a short list each morning of what to fix. Fello, a real estate technology company, sends 6 million agent traces a day to Oodle with none sampled, at about a third of the cost of Langfuse.

Oodle has simple, ingest-based pricing, starting at $1 per GB, and is cost efficient. There are no seat or query fees. Agent Observability Pro includes 30 days of retention, keeping data longer is an added fee, and there is no fee to bring it back. There is no free tier for agent observability. Because Oodle bills by the gigabyte, a session with large tool results costs more on Oodle than on a tool that bills per model call.

Oodle runs as a hosted service. Bring your own cloud (BYOC) and bring your own bucket (BYOB) are Enterprise options, for agent observability as well as logs, metrics and traces. It is not open source. Setup is OpenTelemetry pointed at Oodle, and teams on the Langfuse SDK change only the endpoint. You can try Oodle on the playground without signing up.

Langfuse

Langfuse is for teams that want open source, self-hosting or a framework-neutral tool. Its core is MIT, with enterprise features under a separate license in its ee directories. The copyright is held by ClickHouse, Inc., since Langfuse is now part of ClickHouse. Self-hosting is free, and Langfuse says you can "Self-host all core Langfuse features for free without any limitations." On Langfuse Cloud, the Core plan is $29 a month with 100,000 units, then $8 per 100,000 units, falling with volume (as of October 2026).

The limit is the unit. A session with many steps uses many units, and every score an evaluator writes counts too. Before you choose Cloud, count the units in one of your real sessions.

LangSmith

LangSmith is for teams building on LangChain or LangGraph, though it does not require them: it accepts OpenTelemetry from any application. Because it bills per trace, one execution of the agent costs the same however many steps it takes. As of October 2026, the Developer plan is free for one seat and 5,000 base traces a month. Plus is $39 a seat a month for 10,000 base traces, then pay as you go.

The limits are seats, which count every invited user, and retention. Base traces last 14 days, and features that upgrade a trace to 180 days add to its cost. Self-hosting is for Enterprise customers only.

Arize Phoenix and Arize AX

Phoenix is for teams that want to start locally and trace over OpenTelemetry. Arize calls it "our open-source, local-first platform", and it is the lightest tool here to run: one service on SQLite to start. Its license is the Elastic License 2.0, as Arize's docs state, with self-hosting "free and fully permitted".

The limit is scale on your own database. For high-volume ingestion, Arize's architecture page points to Arize AX, its managed product on its own database. As of October 2026, AX Free includes 25,000 spans and 1 GB a month with 15 days of retention. AX Pro is $50 a month and raises those to 50,000 spans, 10 GB and 30 days. Past those, each span costs $0.0008 and each GB $3, as of November 2026.

Helicone

Helicone is for teams that want every model request, its latency and its cost logged with almost no code. You can change the base URL so requests go through Helicone's proxy, or log asynchronously so Helicone is not on the request's critical path. It is under the Apache License.

As of October 2026, the Hobby plan is free with 10,000 requests and 1 GB of storage, and Pro is $79 a month plus usage. The same page carries a banner: "Helicone is no longer part of Mintlify. Details to come soon." Its view starts from the model request, so check whether you also need the tool calls and the work around them in the same trace.

Datadog Agent Observability

Datadog Agent Observability, formerly LLM Observability, is for teams already on Datadog. It is a standalone product, and teams that also run Datadog APM can follow an agent span into the service trace on the same trace ID. Its docs also describe Insights, which looks for recurring cost and reliability problems in the traces it receives.

Agent Observability bills per LLM span, so it does not bill tool, retrieval or workflow spans. As of October 2026, Pro starts at $160 a month on an annual commitment for 100,000 LLM spans. Each extra 10,000 LLM spans costs $3.50 on an annual commitment, $4.20 month to month and $5 on demand. It is held back by the 15-day default retention and by the need for APM, a separate product, to see the request around the agent.

Other AI observability tools on the lists

Braintrust is often named by teams that build around evaluation. It bills by the gigabyte of data processed and by the score, with 1 GB and 10,000 scores free, then $4 a GB and $2.50 per 1,000 scores (as of October 2026). Opik, from Comet, is under the Apache License. LiteLLM and Portkey are gateways: they route your requests to the model APIs and log each request and its cost. Weights & Biases Weave, Grafana Cloud, Honeycomb and New Relic suit teams already on those platforms.

How each tool handles LLM cost tracking

You rarely need to build cost tracking yourself. LangSmith, Langfuse and Helicone price model calls for you from token counts and a price table, and Datadog's dashboards show cost next to latency and usage.

The OpenTelemetry standard leaves the price to the tool. Its GenAI semantic conventions, at "Status: Development", record token counts with no cost attribute.

A gateway such as LiteLLM or Helicone's proxy gives you accounting for every request without touching each call site, at the cost of one more service in the request path. For the coding agents your engineers use, Claude Code and Codex, Oodle's AI cost management tracks spend by user, model, team, repository and session. Whichever you choose, check a week of tracked cost against your model provider's invoice before you trust it.

Which tool fits which team

Your situation Start with
You want every agent session kept at production volume, billed by the gigabyte on object storage, next to your logs, metrics and traces Oodle
You build on LangChain or LangGraph and want a hosted tool LangSmith
You want open source, and can run PostgreSQL, ClickHouse, Redis and object storage Langfuse, self-hosted
You want a hosted tool, open source underneath, at moderate volume Langfuse Cloud
You want to start on a laptop with one service Arize Phoenix
You want model requests and their cost logged by changing a base URL Helicone
You already run Datadog APM Datadog Agent Observability

How to test two tools before you commit

Pick two tools from the table and run them side by side for a week of real traffic. Instrument with OpenTelemetry, and use the Collector to send the same spans to both.

  1. Count what one of your sessions becomes on each bill: traces, units, spans, requests or gigabytes.
  2. Work out whether either tool would push you to sample at your volume, and which sessions you would lose.
  3. Open a session from the start of the week, and find out when it would expire.
  4. Check each tool's tracked cost against your model provider's invoice for the same week.

See your own telemetry in Oodle

Start debugging in minutes.

Start freeBook a demo

Common questions

What's a low-cost or truly open source LLM observability tool that's easy to run?

Arize Phoenix is the lightest tool here to run, as one service on SQLite to start. But its Elastic License 2.0 is source-available. Langfuse's core is MIT, and Helicone and Opik use the Apache License. Self-hosting Langfuse or Helicone means running PostgreSQL and ClickHouse with a cache or object storage. If nobody on the team wants to run those, a hosted free tier checked against your session count is often the lower-cost choice.

Which agent tracing tool should we use: LangSmith, Arize Phoenix, Langfuse, or something else?

LangSmith if you build on LangChain or LangGraph, Langfuse for open source or self-hosting, and Phoenix to start locally on OpenTelemetry. If you already run Datadog, its Agent Observability links agent spans to your APM traces. If you need every session kept at production volume, Oodle stores traces in object storage and queries them on serverless compute, next to your other telemetry.

Which tool should I use for LLM cost tracking: Langfuse, Helicone, LangSmith, OpenTelemetry or something else?

Langfuse, LangSmith and Helicone all price model calls from token counts, so any of them works. Helicone does it from a base-URL change, while Langfuse and LangSmith do it inside your traces. OpenTelemetry is the standard that carries the token counts, and the tool you send them to prices them. For the coding agents your engineers use, Claude Code and Codex, Oodle tracks spend by user, model, team, repository and session. With any tool, you should check a week against the provider's invoice.

What are people using for LLM and agent observability in production?

Many teams pair an LLM tracing tool with the observability platform they already run. Common choices are Langfuse, LangSmith, Arize Phoenix, Helicone and Datadog Agent Observability, often instrumented with OpenTelemetry.

What are the best LLM observability tools?

There is no single best one. The shortlist is Langfuse, LangSmith, Arize Phoenix, Helicone, Datadog Agent Observability and Oodle. Compare them on what they trace, the unit they bill on, what keeping every session costs on that unit, how long they keep traces and whether you can run them yourself.

Sources

Share