FIND US ONProduct HuntAgent ObservabilitySee how we help you debug AI agents, and share your feedback.

Why self-hosted Langfuse traces go missing after ingestion succeeds

In Langfuse, Phoenix and Tempo, a success response can mean a batch was only accepted for processing. A database write that fails later can drop the trace.

Cover with the words "Accepted is not stored". A Langfuse batch timeline: the ingestion API answers the SDK with 207 accepted, then three ClickHouse writes fail and the worker logs Max attempts reached, dropping the record. The SDK is not told.
On this page
  1. What "Max attempts reached ... Dropping record" means in the self-hosted Langfuse worker
  2. Why Langfuse traces sent while ClickHouse was down are missing after it comes back
  3. Who is told when a self-hosted store drops a trace
  4. How to tell whether a missing trace is lost, late or only hidden
  5. How to get dropped traces back from S3 in Langfuse
  6. How to stop losing traces after ingestion succeeds
  7. Does a buffer in front of the store stop the loss?
  8. How Oodle takes the write path off your team
  9. Common questions
  10. Sources

TL;DR

  • In Langfuse, Phoenix and Tempo, a success response can mean only that the batch was accepted for processing.
  • A database write that fails after that response can then drop the trace.
  • In Langfuse, "Max attempts reached" means the worker gave up writing a batch to ClickHouse after 3 attempts, the default as of October 2026.
  • The sender is not told: the drop shows as a worker log line and the langfuse.queue.clickhouse_writer.rows_dropped metric, so alert on both.
  • Langfuse keeps the raw events in S3, but getting dropped traces back is a manual replay that needs S3 server access logging already on.
  • To make drops rarer, run ClickHouse highly available on persistent storage, as Langfuse recommends.

If your self-hosted Langfuse worker logged "Max attempts reached for observations record. Dropping record." or a trace your SDK sent now returns "Trace not found", look on the server for the missing Langfuse traces. The SDK had its answer before the background worker tried to write the trace to ClickHouse, so whatever went wrong after that is recorded, if at all, in a server log line or a counter.

In Langfuse, the dropped traces are often recoverable. Every incoming batch is written to S3 first, which Langfuse's docs say means "the events are not lost and can be processed later". That holds for the raw events. Getting them back into the database is a manual replay. Running that database highly available and alerting on the signals a drop leaves make drops rarer and make sure someone sees them, though they do not remove the retry limit.

What "Max attempts reached ... Dropping record" means in the self-hosted Langfuse worker

The message means the Langfuse worker tried to write a batch of rows to ClickHouse, failed every time it was allowed to try, and discarded the batch, which can leave one or more traces missing or incomplete. By then the ingestion API had already answered your SDK.

Self-hosted Langfuse splits ingestion across its containers. The web container receives each batch, writes it to S3 and queues a reference in Redis. The worker then picks up the reference, reads the events from S3 and writes them into ClickHouse, the database the UI reads from. Langfuse's FAQ says events are processed asynchronously, so the API does not wait for that last step.

The worker does not write each event on its own. It collects rows in an in-memory writer and flushes them to ClickHouse in batches. If a flush fails, the rows go back into the writer and are tried again, up to LANGFUSE_INGESTION_CLICKHOUSE_MAX_ATTEMPTS times, which defaults to 3 as of October 2026.

The path one batch of trace events takes through self-hosted Langfuse. The SDK gets its success response at the second stage. If all three ClickHouse writes fail, the record is dropped at the fourth. Source: Langfuse self-hosting docs, worker/src/env.ts defaults as of October 2026, and GitHub issue #13468 and discussion #10565, as cited in the article.

After the last attempt, the rows are gone from the worker. The reporter wrote, when the issue was filed: "At that point, only logs and the langfuse.queue.clickhouse_writer.error metric remain." The same report says this can mean "permanent loss of traces, observations, scores, and other ClickHouse-backed entities, even though upstream ingestion or processing jobs appear successful from the application's perspective."

Langfuse kept the design: its maintainer replied that the drop is "correct and expected", and pointed to a script that replays the events from S3. It did accept the reporter's change, which closed the issue as fixed and added the langfuse.queue.clickhouse_writer.rows_dropped metric, a count of dropped rows by entity type meant for alerting.

A second report, on Langfuse v4.3.1, described the same drop as data that is "acknowledged upstream but never lands". Its author added that "nothing signals the SDK or the user that spans went missing." A Langfuse contributor answered that "silently dropping records after retries is not ideal", but that most past drops were ClickHouse timeouts, which the retries handle correctly. The remaining failures, he wrote, are more likely "deterministic malformed or oversized records", which a replay would not help. He closed the issue without adding a dead-letter queue for this path and said he is reworking the ingestion area.

Why Langfuse traces sent while ClickHouse was down are missing after it comes back

Retries can cover a short ClickHouse outage. One that outlasts the attempts leaves gaps. Nothing re-ingests the missing traces when ClickHouse returns.

One team running Langfuse v3 on ECS stopped ClickHouse and found the traces sent during the downtime missing after it came back (issue #6380). Langfuse's maintainer explained that "Short outages should be covered via retries, but longer outages are not supported and may (/will) lead to gaps in the data." On recovery, he added, the traces "won't be ingested automatically once clickhouse recovers." Then: "However, you could manually replay them based on the data stored within S3." He also said: "It's up to you to detect which traces are relevant, though."

The reporter, who had expected the worker to reprocess the traces on its own once ClickHouse recovered, later wrote that "my understanding was incorrect."

The #15730 report reasons that the retry budget is "most likely to be exhausted during a ClickHouse incident or a sustained overload", and Langfuse's reply said that most past drops were ClickHouse timeouts. So most of the drops Langfuse has seen came when the database was in trouble, which is also when you are most likely to go looking for a trace.

Langfuse's maintainer recommended watching the length of the dead-letter queue. A dead-letter queue holds jobs that failed their retries, so they can be inspected and run again. But that queue covers an earlier step than the writer.

In discussion #10565, a team lost the traces sent during a ClickHouse redeploy while their failed-events queue stayed empty. They wrote that "There were no failed events during downtime", which they had confirmed through the admin API and CloudWatch metrics, and that the traces sent during downtime were lost. The only signs were worker log lines, among them "Max attempts reached". A Langfuse team member confirmed that "there is no dead letter queue for this writer queue available."

Who is told when a self-hosted store drops a trace

The sender is not told. In each case in the table, the store had already returned success, and the drop was recorded on the server, if anywhere.

Store and setup What drops the trace after success Where the drop shows
Langfuse self-hosted, v3 and v4 writer, as of October 2026 The worker's ClickHouse writer gives up after its attempts (#13468) A worker log line and the langfuse.queue.clickhouse_writer.rows_dropped metric (#13468)
Langfuse, after a Redis disruption The worker stops consuming its queues while the container stays up (Langfuse docs, #15509) Nowhere until someone notices, unless the opt-in health check is on
Arize Phoenix on PostgreSQL, two or more ingest replicas An open issue (as of October 2026) reports that two replicas deadlock and one span is rolled back, with no retry (#16113) A "Failed to insert span" line in the Phoenix server log
Grafana Tempo Per-trace size and live-trace limits are enforced after the distributor accepted the spans (Tempo docs) The tempo_discarded_spans_total metric. Some drops are logged only at debug level

The Redis case can go unnoticed for days, because the container and queue-length metrics still look healthy. Langfuse's opt-in health check on the worker's queue consumption is what catches it. Langfuse's docs say a Redis disruption, such as key eviction, a FLUSHALL or a failover, "can leave the worker container running while it silently stops consuming all queues". One team on ECS with ElastiCache Valkey hit it twice and did not notice "for 2 days and then 3 days". In their words, "every external signal (container state, queue-length metrics) says the worker is fine while data silently accumulates and is eventually lost."

The trigger was a Redis eviction setting that Langfuse's docs rule out, plus a manual FLUSHALL. But the same docs list a failover among the causes.

In Phoenix, when two ingest replicas deadlock on PostgreSQL, the losing span is rolled back, and the project's own contributor wrote that "nothing requeues it, after the SDK was already told 200". An earlier Phoenix report described the same deadlocks: "These kind of deadlocks are appearing frequently in the logs, and we are loosing tracing information." Issue #16113 is open as of October 2026.

Loss does reach your application when the ingestion endpoint itself fails. Langfuse's FAQ says that in that case "you should also see non-207 status codes within your application". A 207 (Multi-Status) is the response the ingestion endpoint returns when it has accepted the batch. One self-hosted Langfuse user saw the SDK log a failed export when requests timed out, and a single Phoenix replica refuses bursts with 503 errors that the exporter sees. Those errors at least show up in your application's logs. The drops in the table happen after your application stopped listening.

How to tell whether a missing trace is lost, late or only hidden

Check, in order, the status codes your application received, the S3 bucket, the worker logs and ClickHouse. That tells you whether the trace is lost, still on its way, or stored and not shown, which all look the same from the trace list.

Start with your own application, because a wrong endpoint or an SDK that exits before it flushes its last batch also leaves gaps. Then follow the order in Langfuse's own check for missing events:

  1. Look at the status codes your application received and at the web container's logs. If the web container failed, your application will have seen a status other than 207.
  2. Check the S3 bucket for the event. If it is there, it can be replayed.
  3. Search the worker logs for errors from the first minute after the event arrived. "Max attempts reached" here means the write to ClickHouse was given up.
  4. Query ClickHouse for the trace. If the worker says it processed the event and ClickHouse does not have it, Langfuse asks you to open an issue.

A trace that is not there yet may simply be late. Langfuse processes events asynchronously, so a new trace takes a moment to appear. In Tempo, a live-store that lags behind its Kafka partition can make recent traces look incomplete, though by default Tempo fails such a query rather than return partial results.

A trace can also be stored and only partly shown. A Phoenix user at high volume thought spans were being lost, and a Phoenix contributor found the spans in the database. The trace view had been fetching only the first 1,000 spans of the trace, a limit Phoenix has since fixed.

When traces appear and then disappear with "Trace not found"

A trace that shows up in Langfuse and later returns "Trace not found" can point to the ClickHouse cluster rather than to ingestion. One user with a large self-hosted install reported traces that "appear and disappear in the UI", with repeated "Trace not found" errors, and longer HTTP timeouts did not cure it.

The cause in that thread was never confirmed. But Langfuse's maintainer named one setup that often produces this pattern: the first migrations were run with CLICKHOUSE_CLUSTER_ENABLED=false before the cluster grew to several replicas. That leaves the ClickHouse nodes with tables that are not replicated. He added that "Changing this at a later point is not possible, unfortunately." So if you run more than one ClickHouse replica, make table replication one of your first checks.

How to get dropped traces back from S3 in Langfuse

To get dropped traces back, replay their events from S3 with Langfuse's script, using S3 server access logs to list them. Langfuse writes every incoming batch to S3 before the worker touches it, and its maintainer points operators to a replay script for events dropped during outages.

The replay is a manual procedure, and it needs one thing set up before the incident. The script's instructions list the events written during the incident window through S3 server access logging on the events bucket. Server access logging is an S3 setting that records the requests made to a bucket, including each event Langfuse writes, but AWS delivers those records on a best-effort basis, so the list can miss an event.

On AWS you query those logs with Athena for the incident window, export the event keys to a CSV file, and run the script with an admin API key. The script has a --dry-run flag, so you can check the list before anything is sent.

Turn access logging on now if it is off. The instructions note that the first logs can take some time to appear after you enable it. Without those logs, the documented script has no list of event keys to replay. The documented procedure is written for AWS. On another cloud you need your own way to list the event keys.

The same S3 copy also covers a lost database, as long as you kept the S3 events. One team ran ClickHouse on ephemeral storage, and a restart emptied the UI while the old events stayed in S3. Langfuse's maintainer said ephemeral storage is not a recommended setup, and that "you're also responsible for backups if you go with a self-hosted solution."

How to stop losing traces after ingestion succeeds

Run ClickHouse highly available on persistent storage, set Redis to noeviction, point the liveness probe at the worker's stuck-queue check, and alert on the signals a drop leaves. You cannot make the SDK hear about a drop that happens after the response, but these steps make the drops rarer and make sure a person hears about each one.

For self-hosted Langfuse:

Raising LANGFUSE_INGESTION_CLICKHOUSE_MAX_ATTEMPTS gives a short blip more chances, but it does not help when ClickHouse is down for longer. Most drops Langfuse has seen came from ClickHouse timeouts, so most of the fix is in ClickHouse. A malformed or oversized record fails every attempt, however many you allow.

For Phoenix on PostgreSQL, the deadlock needs two or more ingest replicas. One replica avoids it, but the Phoenix contributor who filed the issue found that a single replica then sheds bursts with 503s, and concluded that "neither configuration is safe under load". Until the issue is fixed, a single replica at least makes the failure visible to your exporter. Watch the server log for "Failed to insert span" either way.

For Tempo, alert on tempo_discarded_spans_total and size the per-trace and live-trace limits for your largest traces. Tempo's docs say what happens otherwise: "If limits are too low for your workload, spans are discarded and data is lost."

Does a buffer in front of the store stop the loss?

A buffer protects only what the store has not accepted yet. The usual advice for restarts is an OpenTelemetry Collector with a persistent queue in front of the store, and it does help when the store is unreachable.

The Collector retries for up to 5 minutes by default, and "Once the queue is full, incoming data is dropped to prevent the Collector from running out of memory." A Collector that crashes with an in-memory queue loses what it held. A persistent queue keeps queued data through a crash, but it does not protect against a disk that fails or fills. The Collector counts its own drops in otelcol_exporter_send_failed_spans and otelcol_exporter_enqueue_failed_spans, and those are worth an alert too.

But the drops in Langfuse, Phoenix and Tempo all happen after the store said yes, so the Collector has already let go of the batch.

A buffer can also hand the store more than it accepts at once. One Tempo user buffered an hour of spans in Kafka during downtime, and on restart Tempo "consumed all of them almost instantaneously, and dropped most of them" against its live-traces limit. The reply on the issue said the Tempo team never runs Tempo reading straight from a Kafka topic that way, so this is one report in one setup. Still, size or rate-limit the store's intake for the backlog a buffer will replay, not only for normal traffic.

How Oodle takes the write path off your team

With Oodle's hosted service, there is no trace database, queue or worker for your team to run, size, restart or replay from. What runs between the ingestion API and your trace list is the hosted service's to operate, not your team's.

Queries run on serverless compute that spins up per query and shuts down after. With the Langfuse SDK, only the endpoint changes, and OpenTelemetry instrumentation can point at Oodle's hosted service directly.

See your own telemetry in Oodle

Start debugging in minutes.

Start freeBook a demo

Common questions

How do I fix "Max attempts reached for observations record" in my self-hosted trace store?

The Langfuse worker failed to write a batch to ClickHouse within its attempts and discarded it. Your SDK had already been told the batch succeeded, and the worker makes 3 attempts by default (as of October 2026). Find the ClickHouse error just before it in the worker logs and fix ClickHouse, which Langfuse recommends running highly available. The raw events are still in S3, and you can replay them with Langfuse's script if S3 server access logging is on.

How do I stop losing traces when my self-hosted trace backend restarts or goes down?

Keep the database up through restarts, on persistent storage with backups, and alert on the server-side signals of a drop. In Langfuse those are the worker's "Max attempts reached" log line and the langfuse.queue.clickhouse_writer.rows_dropped metric. Watch the dead-letter queue length separately, for failures earlier in ingestion. Turn on S3 server access logging so you can replay what was dropped. A Collector queue in front helps only while the backend is unreachable, not once it has accepted the data.

Why do I get "Trace not found" under load when ingestion succeeded?

A success response means only that the store accepted the batch, and under sustained load ClickHouse timeouts can use up the worker's attempts and drop it. If the trace never appeared, it may still be processing or have been dropped: check S3, the worker logs and ClickHouse in that order. If it appeared and then disappeared on a multi-replica ClickHouse, check table replication, a cause one thread suggested but never confirmed.

Does self-hosted Langfuse lose data when ClickHouse is down?

The raw events are kept in S3. But traces sent during a longer outage do not reach ClickHouse, and nothing re-ingests them when it comes back. You have to find and replay them yourself.

Sources

Share