On Product HuntAgent ObservabilitySee how we help you debug AI agents, and share your feedback.

Has ClickHouse conquered metrics?

128 concurrent queries, Q1 rate query over 1K series and a 6 hour range, warm, p90: DenaliDB 3.2 seconds, ClickHouse 176.2 seconds
On this pageTL;DRWhy this benchmark?Benchmark setupResultsWhat about the new ClickHouse TimeSeries engine?Why does DenaliDB not degrade at scale?Has ClickHouse conquered metrics?

TL;DR

Not yet. ClickHouse slows down as series, time range and concurrency grow. These factors become even more important with agents becoming the primary consumer of observability data.

25M active series. ClickHouse on one 192 vCPU node with NVMe vs DenaliDB.

ClickHouseDenaliDB
1K series, 128 concurrent queries (p90)176 s3.2 s
250K series, 6 hours34 s3.7 s
250K series, 24 hours124 s3.7 s
Bytes per sample on disk, with HA233

Why this benchmark?

ClickHouse is an amazing real-time analytics database.

Oodle started as a metrics platform. There were great metrics databases already - Prometheus, M3DB and others - but none handled the scale we wanted at the price we wanted. So in 2024 we built our own metrics engine, DenaliDB. We evaluated ClickHouse for metrics back then too, but it was not suited to the shape of metrics data, which is very different from analytics and logs. Metrics arrive sideways: each scrape sends one sample for every series, but a query wants all the samples of a few series.

In 2025 we launched our logs product. We had to launch quickly, and ClickHouse was already well known for logs and events, so we launched on it. It worked great for our initial customers, but as we started onboarding larger customers, we ran into challenges and had to extend DenaliDB to support logs.

Tao talks about our learnings operating ClickHouse at scale in detail in his blog series:

ClickHouse recently launched ClickHouse Observability. One thing that caught my eye is this quote from the post about metrics on ClickHouse:

ClickHouse doesn't suffer with cardinality, so neither should you.

ClickHouse makes the same case for high cardinality in ClickHouse vs Prometheus for High Cardinality.

This made me rethink: would our metrics engine benefit if I changed its data layer to ClickHouse?

The question also looks different today than it did in 2024. Agents are now the main consumers of observability data: as of writing this, more than 93% of the metric queries at Oodle come from agents. And agents run parallel investigations when they debug - so performance under concurrent load is an important factor to test.

So I started on this benchmarking journey.

Benchmark setup

The setup, load and query scripts are in clickstack-vs-oodle-benchmark.

WorkloadInto each system
Active series25M
Metrics100 counters, 250K series each
Samples ingested1.67 M / sec
Data ingested1.62 GB / sec (uncompressed OTLP)
Labels per series24
Scrape interval15 sec

Load: 250 avalanche pods scraped by OpenTelemetry Collectors. Each pod had 100K series.

ClickHouse: ClickStack, ClickHouse's observability stack, from its official Helm chart 3.5.0 (HyperDX 2.44, ClickHouse 26.8). I tuned it per ClickHouse's docs to handle this ingest, and left the schema as it is.

Changes from ClickStack's defaults

At the defaults, ClickHouse refused inserts at 1.67M samples / sec. I changed only these, following ClickHouse's docs.

SettingDefaultBenchmark
CPU, memory0.5 cores, 2 GiB184 cores, 1.4 TiB (1,260 GiB for queries)
Disk10 GiB11.4 TB local NVMe (6 x 1,900 GB, RAID 0)
background_pool_size1664
Collectors114 (4 CPU, 8 GiB each)
Collector memory_limiter1.5 GiB6.4 GiB
HYPERDX_OTEL_BATCH_SEND_BATCH_SIZE10K rows100K rows
HYPERDX_OTEL_EXPORTER_TIMEOUT5 sec30 sec
max_concurrent_queries100500
concurrent_threads_soft_limit_ratio_to_cores24 (up to 30% speedup)
max_memory_usage10 GB1.2 TB
max_execution_time60 sec900 sec

Query: I benchmarked a common query - request rate per service, over 1 to 250K series - and two others at 100K and 250K series. A series is one metric with one set of label values, such as http_requests_total{service="checkout", pod="checkout-7f9c"}, and it gets a new sample every 15 seconds.

# Q1: API rate by service
sum by (service) (
  rate(avalanche_counter_metric_mmmmm_0_0_total{<selector>}[5m])
)

# Q2: Top 10 pods by request rate
topk(10,
  sum by (pod) (
    rate(avalanche_counter_metric_mmmmm_0_0_total{<selector>}[5m])
  )
)

# Q3: Error ratio by service
sum by (service) (
  rate(avalanche_counter_metric_mmmmm_0_0_total{<selector>, label_key_kkkkk_6="0"}[5m])
)
/
sum by (service) (
  rate(avalanche_counter_metric_mmmmm_0_0_total{<selector>}[5m])
)

Most results below use Q1. Q2 and Q3 are under Other queries.

Cold Query: before every ClickHouse run I dropped its caches and the OS page cache, close to how ClickBench measures cold.

Warm Query: the same query ran once first, untimed. Result caches were off on both sides. Median of 5 runs. ClickHouse does not read from disk in warm runs - it is all from page cache.

No Result Cache: Result cache is cleared before each run. Both ClickHouse and DenaliDB serve from result cache in < 5ms.

Verification: I verified that both ClickHouse and DenaliDB return the same data.

Scale: Given the 15 sec sampling interval, and 250K touched, a 6 hour interval would touch 6 * 60 * 4 * 250000 = ~360M samples.

Hardware

The next question was what kind of hardware to give ClickHouse to make this benchmark comparable. The better the hardware, the better ClickHouse would perform - but also increases your costs.

Since I wanted to benchmark with 25M metrics, I decided to start with list price of Oodle for 25M metrics. This comes to $19,500/month. I chose ClickHouse hardware that costs about the same.

Yes, that compares ClickHouse's raw hardware cost with Oodle's list price, which includes our margins. That tilts the comparison in ClickHouse's favor: take out a healthy SaaS margin and you get our actual infrastructure cost. What I went with was:

ClickHouse, single nodePer month
r8gd.48xlarge (192 vCPU, 1.5 TiB, 11.4 TB local NVMe, 50 Gbps network, 7 day retention)$10,299
Collectors, HyperDX, MongoDB (3 x c7g.8xlarge)$2,540
Total$12,840
With HA$25,680
With HA, 90 day retention$47,600

I used AWS on-demand rates for ClickHouse hardware costs. Oodle price includes HA and a 90d retention, and HA will be needed for any production observability setup. The local NVMe holds about 7 days. For 90 days I added the older data on gp3 at $0.08/GB-month, at 3.3 TB/day across both replicas (see Storage). Queries over that data would be slower than NVMe. S3 would cost less but be even slower.

Per month, 25M seriesClickHouse (hardware only)Oodle (list price)
With HA, 7 day retention$25,680$19,500
With HA, 90 day retention$47,600$19,500

I initially started with gp3 for benchmarking, but realized disk throughput was limiting ClickHouse performance. So I switched to local NVMe disks. ClickHouse cold was 2-8x slower than ClickHouse warm with gp3. With NVMe, cold adds 0.3 to 1.4 s to small queries and under 5% from 100K series up, since disk was no longer a bottleneck.

Diskgp3NVMe
Sequential read1.05 GB/s23.3 GB/s
Sequential write1.05 GB/s*11.0 GB/s
Random 4K read10K IOPS3.25M IOPS

* gp3's 1.05 GB/s is shared between reads and writes.

Would a sharded ClickHouse Cluster help?

No. ClickStack computes rate() with a SQL window function, and in a distributed query ClickHouse runs window functions only on the node that received the query. The other shards just read and filter their rows and send them all to that node. So in this case a single large node is better than splitting it into shards - assuming the same total capacity.

Would separating storage and compute help ClickHouse?

Yes, but harder to do for ClickHouse with disk backed data. It can be done with S3 backed storage, but performance would be worse. But the total hardware spec I chose was for Ingestion + Queries, to keep resources comparable with Oodle list price.

ClickHouse Cloud has separation of storage and compute with comparable performance, but also costs more :)

Would materialized views help?

Yes! But has to be manually maintained for each kind of query you want to perform, and adds extra cost at ingestion. I did not test with materialized views, and used the default ClickStack schema. I am planning to run with materialized views in my next benchmark.

Results

Concurrent workloads

ClickHouse performance degrades with concurrent workloads because all queries compete for the fixed available compute. With 128 concurrent queries, ClickHouse ingestion also fell behind (100M to 66M rows per minute) and the collectors backed up.

Concurrent query performance is actually even more important in today's world where agents are the primary consumers of observability data.

All concurrency results (warm)

Q1 over 6 hours, 5 rounds per level. No query failed on either side.

SeriesAt onceDenaliDBClickHouse
Medianp90Medianp90
1K11.9 s2.4 s2.1 s2.1 s
1K82.2 s3.0 s11.0 s11.2 s
1K162.4 s3.1 s21.5 s21.8 s
1K322.4 s3.1 s42.0 s42.4 s
1K642.4 s3.1 s87.0 s88.0 s
1K1282.3 s3.2 s173.9 s176.2 s
25K12.2 s2.8 s7.2 s7.3 s
25K22.1 s3.2 s10.6 s10.9 s
25K42.3 s2.9 s18.7 s18.9 s
100K12.7 s3.1 s16.9 s17.7 s
100K22.6 s4.0 s24.8 s25.6 s
100K42.8 s3.8 s42.6 s48.5 s
250K13.9 s4.8 s33.7 s34.1 s
250K23.9 s4.8 s49.2 s50.4 s
250K43.9 s4.5 s90.4 s93.3 s

More series per query

ClickHouse gets significantly slower as the number of series touched increases. ClickHouse cold and warm are pretty close now, but cold was much slower than warm when I was running the benchmark with gp3 instead of NVMe disks.

ClickHouse details (cold)

What ClickHouse reported in its own query log. Medians of 5 runs. In the Limit column, CPU means ClickHouse reached all 184 of its cores during the query (query plus ingest), not that it was CPU-bound throughout.

SeriesRowsCPU-sMemoryTimeLimit
ReturnedRead
173205M76<1 GiB1.8 s-
25073211M1332 GiB2.1 s-
1K73210M2023 GiB2.5 s-
5K73358M36911 GiB3.9 s-
10K73374M44819 GiB4.9 s-
25K73379M62139 GiB7.3 s-
50K146380M94372 GiB10.8 s-
100K292380M1,481124 GiB17.3 s-
150K438375M2,040173 GiB24.3 s-
200K584375M2,516217 GiB29.9 sCPU
250K730377M2,846259 GiB34.4 sCPU
ClickHouse details (warm)

The same query run once first, untimed, so the data was already in memory and nothing came off the disk. Medians of 5 runs. In the Limit column, CPU means ClickHouse reached all 184 of its cores during the query (query plus ingest), not that it was CPU-bound throughout.

SeriesRowsCPU-sMemoryTimeLimit
ReturnedRead
173205M70<1 GiB1.1 s-
25073210M1242 GiB1.5 s-
1K73210M1863 GiB2.1 s-
5K73358M36310 GiB3.5 s-
10K73374M44519 GiB4.4 s-
25K73379M61340 GiB6.8 s-
50K146380M94572 GiB10.8 s-
100K292380M1,478124 GiB16.9 s-
150K438375M2,010173 GiB23.6 s-
200K584375M2,455216 GiB29.5 sCPU
250K730377M2,778258 GiB33.8 sCPU

What limits ClickHouse?

When I was trying gp3 disks, disk was also a limit, but after NVMe disks, compute was the only bottleneck.

Other queries

Q2 and Q3 over 6 hours, medians of 5 runs.

QuerySeriesDenaliDBClickHouse
ColdColdWarm
Q2: top 10 pods100K2.3 s13.4 s12.8 s
Q2: top 10 pods250K3.7 s26.2 s25.3 s
Q3: error ratio250K5.1 s38.3 s37.5 s

Longer time ranges

Apart from number of series being touched by a query, it's also important to benchmark how time range of a query impacts performance.

All time ranges (warm)

Q1 at each time range, with the step the ClickStack UI picks for it: 1 minute for 1 hour, 5 minutes for 3 and 6 hours, 10 minutes for 12 hours and 30 minutes for 24 hours. PromQL used the same steps. Answers matched on every query. DenaliDB is flat from 3 to 24 hours; differences under 0.2 s are run-to-run noise. In the Limit column, CPU means ClickHouse reached all 184 of its cores during the query (query plus ingest), not that it was CPU-bound throughout. Warm runs read nothing from disk.

SeriesRangeTimeClickHouse
DenaliDBClickHouseRows readMemoryLimit
11h0.4 s0.5 s59M<1 GiB-
13h1.4 s0.8 s132M<1 GiB-
16h1.8 s1.1 s205M<1 GiB-
112h1.7 s1.5 s294M<1 GiB-
124h1.8 s2.0 s433M<1 GiB-
1K1h0.4 s0.9 s59M1 GiB-
1K3h1.7 s1.3 s125M2 GiB-
1K6h1.8 s2.1 s210M3 GiB-
1K12h1.8 s2.5 s298M4 GiB-
1K24h1.7 s3.7 s435M6 GiB-
25K1h0.5 s2.8 s63M10 GiB-
25K3h1.6 s4.6 s205M27 GiB-
25K6h1.9 s6.8 s379M40 GiB-
25K12h1.9 s10.1 s746M66 GiBCPU
25K24h1.9 s16.3 s1.45B101 GiBCPU
100K1h1.0 s7.9 s64M29 GiB-
100K3h2.4 s10.4 s205M74 GiB-
100K6h2.3 s16.9 s380M124 GiB-
100K12h2.3 s24.3 s764M200 GiBCPU
100K24h2.4 s35.5 s1.50B327 GiBCPU
250K1h2.7 s15.3 s63M69 GiB-
250K3h3.2 s18.7 s192M134 GiB-
250K6h3.7 s33.8 s377M258 GiBCPU
250K12h3.7 s47.4 s766M437 GiBCPU
250K24h3.7 s123.7 s1.52B703 GiBCPU

Cold over 24 hours, the 250K-series query took 123 seconds on ClickHouse and peaked at 703 GiB of memory. ClickHouse was allowed 1,260 GiB, so two of them at once don't fit.

Storage

For this benchmark, ClickHouse (with HA - 2 replicas) would store 3.3 TB/day and DenaliDB would store 0.43 TB/day.

What about the new ClickHouse TimeSeries engine?

In September ClickHouse announced a TimeSeries table engine with PromQL support. It's still in private preview, and I could not complete a benchmark of it. My notes so far are in ClickHouse TimeSeries engine for metrics.

Why does DenaliDB not degrade at scale?

DenaliDB stores all data in S3, and uses lambda functions for queries. This helps each query run independently of other queries, and allows massive parallel execution of heavy queries.

There is a catch for us though. Lambda functions, if constantly run, will increase our own infrastructure costs at Oodle. But how we solve that is by our result cache. In our production, 73% of all human + agent metric queries are served from the result cache. More details about our architecture is in how Oodle keeps observability fast at scale.

I believe with agents becoming the primary consumer of observability data, having this property is extremely important.

Has ClickHouse conquered metrics?

Not yet. In my benchmark, ClickHouse did not scale well with queries that run concurrently or read more data. ClickHouse was built for analytics and logs first. The new TimeSeries engine made for metrics is still in private preview and has some issues.

This is not the fastest ClickHouse can be. I ran ClickStack's default schema. Materialized views would likely be faster, but each view has to be built and kept up for the queries you expect, and adds work at ingest. It's not just dashboards any more and it is hard to predict what dimensions agents will query to build the materialized views. I plan to test with materialized views in my next benchmark.

So we are not changing our metrics engine to ClickHouse at this time. As ClickHouse improves its metrics engine, I will continue benchmarking against it and see where things go!

As a thought exercise, ask Claude or ChatGPT how much this benchmark - 25M series at 15s scrape would cost on ClickHouse Observability :)

Running metrics at this scale?

Point your Prometheus or OpenTelemetry collector at Oodle and compare on your own data.

Get Started for Free Try the Playground →

Written byVijay KarthikDistributed systems enthusiast!

See your own telemetry in Oodle

Start debugging in minutes.

Start freeBook a demo

Get new posts by email

Engineering write-ups from the team.