FIND US ONProduct HuntAgent ObservabilitySee how we help you debug AI agents, and share your feedback.

Why LLM trace sampling misses failed agent sessions

Sampled LLM traces miss failed agent sessions because many failures, such as a wrong answer, report success at every step and are dropped at random.

Cover with the words "Sampling drops the session you need". A wall of session tiles, a few kept in blue and one failed session in coral among the dropped ones. A sessions table sampled at 10% keeps 1 of 10, the wrong-refund session is dropped, and a search for it finds no session.
On this page
  1. Why teams use LLM trace sampling
  2. What are the odds you kept the session you need?
  3. Where does sampling happen in agent tracing tools?
  4. Why tail-based sampling on errors alone misses agent failures
  5. How to keep the session you need
  6. Why keeping every session is affordable in Oodle
  7. Common questions
  8. Sources

TL;DR

  • Sampled traces miss failed agent sessions because many failures, such as a wrong answer, report success at every step and are dropped at random.
  • Your chance of still having the session a customer asks about is your sample rate: at 10%, it is gone nine times in ten.
  • A tail-sampling rule that keeps every reported error still misses agent failures that report success, such as wrong answers and loops.
  • The OpenTelemetry Collector's default 30-second wait can end before a long agent session does.
  • If you cannot keep every session, keep the ones your checks flag as each session finishes, with your SDK's sample rate left at 1.
  • Count every session as metrics before sampling, because keeping every failure but 10% of the rest makes a 1% failure rate look like about 9%.

To keep the tracing bill down, you use LLM trace sampling and keep every error with tail sampling. But that could result in missing the failed agent session a customer asks about. When an agent gives a wrong refund amount, every call it made can still return success, so nothing marks the session as an error. Your error rule passes over it, and the session falls into the random share with every normal one.

Why teams use LLM trace sampling

Teams sample because of the bill. Many LLM tracing tools charge by how much you send them, and agents send a lot, because every model call and tool call becomes a span. Tracing tools sell their sampling settings in those terms, as a way of helping you "balance observability needs with cost considerations". As traffic grows, many teams turn the rate down to keep the bill under control.

The cost of that shows up when a customer writes in to say your support agent gave them a wrong refund amount yesterday. You open your tracing tool to see what the agent did, and the session is not there, because your tool was set to keep only some of them.

A session is one conversation or task an agent handles from start to finish. Each step, such as a model call or a tool call, is a span, and the spans from one request make up a trace. A short session like this refund request is one trace. A longer session can be several. Sampling keeps a share of traces, the sample rate, and throws the rest away: a 10% rate keeps one trace in ten.

What are the odds you kept the session you need?

When you debug, you need one specific session, and the odds that you kept it equal your sample rate. That holds when every session has the same random chance of being kept and each decision is made independently. The odds on this page are for a session that is one trace, sampled on its own.

When you watch for problems, you only need some of the failures. Say 20 sessions fail in a day. At a 10% rate, the chance you kept at least one of the 20 is 88%. In general it is 1 minus (1 minus the sample rate) to the power of 20.

Sample rate Chance you kept a specific failed session Chance you kept at least one of 20 failed sessions
1% 1% 18%
5% 5% 64%
10% 10% 88%
25% 25% over 99%
At each sample rate, the chance you kept the one failed session a customer asks about, against the chance you kept at least one of 20 failed sessions that day. Source: worked example from the article, random sampling, one trace a session. At least one of 20 = 1 − (1 − rate)^20

The right column is good news if you want to know that failures are happening, as long as something flags the failed sessions you kept. But when a customer reports a problem and you need that exact session, the middle column is the one that applies. Even at 25%, three out of every four complaints point to a session you no longer have.

Users of the tracing tools see the same trade-off. A Langfuse user asking for a way to keep traces by outcome described it this way: "You effectively have to weigh cost of a high sampling rate against the probability of catching crucial errors."

Where does sampling happen in agent tracing tools?

In the most common agent tracing tools, and in the OpenTelemetry SDKs many of them build on, sampling happens inside the SDK, before the trace leaves your service.

All three samplers, set up this way, use head sampling: they decide at the start of the session, before anyone knows whether it will fail. This is why the odds above are pure chance.

Why tail-based sampling on errors alone misses agent failures

An error rule keeps only the failures something marked as an error, and many agent failures are never marked.

The alternative to head sampling is tail sampling, also called tail-based sampling. Instead of deciding at the start, tail sampling waits until it has seen the trace, and then keeps it if the trace matches a rule you set. A common rule is "always sampling traces that contain an error". The OpenTelemetry Collector, a service that receives traces and forwards them to your tools, can do this with its tail sampling processor.

That rule works well for failures that show up as errors, such as an exception or a 500 response. Set the span status to ERROR yourself for failures your code can detect, such as a tool call that failed or a model response that was not valid JSON.

But agents fail differently. An agent can call the wrong tool, go around the same loop five times, or give a confident answer that is wrong. In all of those cases, every model call and every tool call can still return success. Openlayer, which sells testing and monitoring for AI systems, describes it this way: "The status was 200. And the result was wrong." Since nothing reported an error, an error rule has nothing to catch. If that is your only rule, the session is dropped at random like any other.

Tail sampling also has two practical limits for agents.

  • It needs every span of a trace in one place. All the spans of a trace must reach the same Collector instance. If you run more than one Collector, you need a load-balancing layer in front that sends every span of a trace to the same one.
  • It decides on a timer. In the Collector's default mode, as of October 2026, it makes its decision 30 seconds after it receives a trace's first span. An agent session can take several minutes, so the decision may be made before the rest of the session has arrived. You can soften this by raising decision_wait above your longest normal session and setting a decision cache for both keep and drop decisions, so spans that arrive late follow the first decision. A longer wait also means more traces held at once, and a trace pushed out of the buffer before its wait ends is dropped without sampling, so size num_traces for it too. All of this costs memory.

How to keep the session you need

Keep every session if your tool's pricing allows it

Keeping every session is the only way to be sure sampling has not thrown away the one a customer complains about. So before you accept sampling as the price of tracing, work out what keeping everything would cost.

Start with how many spans or traces your agents produce in a month, whichever unit your tool bills by. Multiply that by your tool's price for that unit. Then add any separate charge for keeping data longer than the plan includes. That gives you the monthly cost of keeping every session.

At low volume, the answer is usually to keep everything. OpenTelemetry's own sampling guide says you may not need to sample if you generate "tens of small traces per second or lower".

If you must sample, check every session first

If keeping everything is too expensive, change the order of the decision. Instead of deciding at random when a session starts, check every session as it finishes, and then decide.

Simple rules can catch a lot of agent failures without calling a model. For example, you can flag a session when:

  • it called the same tool many times in a row
  • one of its tool calls failed
  • it used far more tokens than a normal session
  • the conversation stalled with no answer.

Keep every session that any of these four rules flags, plus a random share of the rest. For this to work, the rules have to see every session before anything is thrown away, so leave your SDK's own sample rate at 1. A trace dropped in the SDK never reaches the rules.

If you run the OpenTelemetry Collector, its tail sampling processor can apply simple versions of these rules. Langfuse's SDKs cannot tail-sample on their own (as of November 2025), and a Langfuse maintainer points users to a Collector for it.

This example keeps every trace with an error, every trace that took longer than 240 seconds (set above a normal session), and 10% of the rest. It covers only errors and slow sessions: the other rules (repeated tool calls, token outliers, stalled conversations) need attributes your code sets on the spans. The processor runs only once it is listed in the traces pipeline, which is the service block at the end.

yaml
processors:
  tail_sampling:
    decision_wait: 300s     # starting point: longer than your longest normal session
    num_traces: 200000      # traces held while they wait; size for your volume
    decision_cache:         # much larger than num_traces
      sampled_cache_size: 1000000       # remembers keep decisions
      non_sampled_cache_size: 1000000   # remembers drop decisions
    policies:
      - name: keep-errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: keep-slow-sessions
        type: latency
        latency: {threshold_ms: 240000}   # above a normal session, below decision_wait
      - name: sample-the-rest
        type: probabilistic
        probabilistic: {sampling_percentage: 10}

service:
  pipelines:
    traces:
      receivers: [otlp]       # your existing receiver
      processors: [tail_sampling]
      exporters: [otlphttp]   # your existing exporter

These rules will not catch everything. The wrong refund amount the customer reported can look like a normal session to all four. Catching it takes a check against your own data, such as comparing the refund to your policy, or a model that judges the answer. If the session was kept, you can still find it when the customer reports it.

Count every session, even the ones you drop

If you keep every failed session but only 10% of the successful ones, your traces will make failures look more common than they are, close to ten times more when failures are rare. A true failure rate of 1% shows up as about 9%.

Count every session before any sampling happens: how many sessions there were, how many failed, how long they took and what they cost. The failed count covers the sessions that errored and the ones your checks flagged, each only as complete as its check.

Keep those counts as metrics, separate from the traces. Datadog's APM, for example, calculates its trace metrics on all traffic, as long as sampling happens at or after its Agent and not in the SDK. Langfuse and LangSmith apply their sample rate in your code, so if you sample with them, you count sessions yourself. Then use the metrics for rates and trends, and the traces for looking into individual sessions.

Keep your own record of what the agent did

If you need to track cost, audit what an agent did, or recover from a failure, do not rely on sampled traces. As one answer on the LangChain forum puts it, you should "treat the trace as explanatory, not authoritative for recovery."

Instead, keep your own small record of every model call in your own database. It does not need the full prompt and response. One row per call, with the request ID, the model, the tokens used, the cost, the status and the time taken, is enough to track cost and errors reliably. If you do keep prompts or responses, redact personal data first, because both can hold what your users typed.

Write that record through a different path from your traces. If you see "fetch failed" errors when your service sends traces, your trace exports are failing, and some traces may never reach the tracing tool. In Node, "fetch failed" is a general network error, and the reason is in the error's cause, so check the connection and the exporter, not your sampling settings. One fix is to run an OpenTelemetry Collector next to your service, so your code sends traces to a local address and the Collector handles delivery. Either way, the record you count costs from should not depend on traces arriving.

Do not wait for an incident to turn the rate up

Raising the sample rate to 100% after something goes wrong helps you see the next failure. But it cannot bring back a session that was already dropped. If a session is worth debugging after an incident, it is worth keeping before one.

Why keeping every session is affordable in Oodle

Teams sample to control cost, and in Oodle keeping every session is affordable because of how Oodle is built. Traces go straight to object storage in Oodle's own columnar format, and queries run on serverless compute that spins up per query and shuts down after. With Oodle, every session is kept in full for your retention period (30 days on Pro, longer for an added fee), so you do not need tail sampling or your own session counts to keep the bill down.

Oodle is observability built for AI agents. Every session is analyzed at scale using statistics, ML classifiers and a small model for sentiment. Oodle's agent then checks each issue it finds against the evidence and removes the noise. As each session arrives, Oodle marks issues such as tool failures, loops, outages and stalled conversations.

Some failures, like a wrong answer, need an LLM judge, which is slower and costs money for every session it reads. So some teams run their judge on a small random share, such as 1%. In Oodle, the simpler checks pick the sessions the judge reads, so the judging goes where failures are most likely. And because every session is kept, a failure the checks missed is still there when a customer reports it.

Fello, a real estate technology company, keeps every agent session this way today. Ashish Gupta, Principal Engineer at Fello, says "We're already running 6M+ agent traces/day on Oodle with full visibility and zero sampling". Once every session is kept, the session behind tomorrow's complaint is there. You can open agent sessions like it in the Oodle playground without signing up.

See your own telemetry in Oodle

Start debugging in minutes.

Start freeBook a demo

Common questions

How do I sample LLM traces but keep every error?

Use tail sampling, such as the OpenTelemetry Collector's tail sampling processor with a rule that keeps error traces. Add a random share of the rest, and leave your SDK's own sample rate at 1. The error rule keeps only the failures something marked as an error. Many agent failures, such as a wrong answer, a wrong tool call or a loop, report success at every step, so add rules for tool failures, repeated tool calls, token outliers and stalled conversations.

I sample 10% of my agent traces. What are the chances the failed run I need was kept?

One in ten. Each session has a 10% chance of being kept, so the one you need is missing nine times out of ten. Across many failures you will probably keep some of them, but not necessarily the one a customer reported.

Should I log LLM requests and responses myself since traces get dropped or sampled?

Yes. Log your own small row of metadata for every model call in your own database, with the request ID, model, tokens, cost, status and time taken. Keep prompts and responses only where debugging or audit needs them, and redact personal data first. Write the rows through a different path from your traces, because "fetch failed" in Node is a general network error: your trace exports are failing, and some traces may never reach the tracing tool. Use the traces for debugging individual sessions, and your own rows for cost and error totals.

How do I set the sample rate in my tracing SDK?

Set your SDK's sample rate, usually an environment variable or a client option, to a value between 0 and 1. A rate of 1 keeps every trace, and it is the default in Langfuse's Python SDK and in LangSmith (as of October 2026). Some SDKs take an OpenTelemetry sampler instead, which you pass when you set up tracing.

Should I sample my LLM traces or keep all of them?

Keep all of them if you can afford to. At low volume you usually can, and OpenTelemetry's sampling guide says tens of small traces per second or lower may not need sampling. Counts taken before any sampling, or a plain random sample, work for watching trends, such as how error rates change from week to week. A sample that keeps every failure does not, because it makes failures look more common. Sampling is unreliable for debugging one specific session, because that session is kept only as often as your sample rate.

Sources

Share