FIND US ONProduct HuntAgent ObservabilitySee how we help you debug AI agents, and share your feedback.

Cutting the cost of LLM-as-a-judge evals by running simple checks first

Cut LLM-as-a-judge cost by judging fewer sessions. Run code checks on every session, and send the judge only what they flag plus a small random audit.

Cover with the words "Simple checks first, judge second". A checklist of code checks that run on every session and spend no tokens, beside an example eval pipeline: code checks on 100% of sessions, 3% flagged plus a 1% random audit, so the LLM judge reads 4%.
On this page
  1. Why LLM-as-a-judge costs more than the agent it scores
  2. Why a lower sampling rate is the wrong lever
  3. Check every session with code before a judge reads it
  4. Send the judge the flagged sessions plus a random audit
  5. Make each LLM judge call cost less
  6. A scoring call that failed is not a score of 0
  7. Test a cheaper LLM-as-a-judge evaluation setup before release
  8. How to tell the cheaper eval still catches failures in production
  9. Why the judge's short list has to come from every session
  10. Common questions
  11. Sources

TL;DR

  • Judge fewer sessions: run checks written as code on every session, and send the judge only what they flag plus a small random audit.
  • A judge can cost more than the agent because it rereads the whole transcript once per rubric.
  • In our example, six rubrics cost $0.126 a session, about 25 times the agent's $0.005 a session.
  • A lower random sampling rate is the wrong lever, because it cuts the bill and the judged failures by the same share.
  • In our example, flagging 3% of sessions and auditing a random 1% cuts the judge's cost to about $0.005 a session, roughly the agent's cost.
  • Record a failed check or judge call as an error with no score, never as 0.

LLM-as-a-judge is an eval method in which a language model scores another model's output against a rubric, such as correctness or tone. If your judge evaluators now cost more than the agent they score, look at what each judge call reads. A judge usually reads a session's whole transcript once per rubric, often on a larger model than the one the agent uses, so every rubric you add multiplies the bill. Start with the failures a function can answer, such as invalid output or a tool call repeated in a row, and run those checks on every session before any judge reads it.

Why LLM-as-a-judge costs more than the agent it scores

A judge rereads the finished transcript once for every rubric, and teams often pick a larger model to judge with. An agent re-sends its growing context on every turn, but it usually runs on a small model.

The worked example below is our arithmetic from one provider's published prices, as of October 2026. Every token count in it is an assumption, so put your own in.

  • The agent. A session of eight turns sends 40,000 input tokens and gets 2,000 output tokens back, in total. On a small model at $0.10 per million input tokens and $0.50 per million output tokens, that is $0.004 plus $0.001, or $0.005 a session.
  • The judge. The finished transcript is 9,000 tokens, and the judge writes a 300-token verdict. On a mid-size model at $2 per million input tokens and $10 per million output tokens, one rubric costs $0.018 plus $0.003, or $0.021.
  • Six rubrics. Six judge calls, for rubrics such as helpfulness, correctness, tone and policy, cost $0.126 a session. That is about 25 times the agent's $0.005 a session.

This LLM-as-a-judge cost of $0.021 a call is in line with what one tracing vendor calls a typical cost of $0.01 to $0.10 per assessment.

The judge's model is not the only driver, though. Put the same six rubrics on the agent's small model and each call costs about $0.001, so six of them come to $0.0063, still a quarter more than the agent's $0.005 a session. The rubric count and the full transcript still put the judge above the agent.

One cloud's built-in evaluators bill the same way, by the input and output tokens they process, so a judge that reads the whole session pays for the whole session on every rubric.

Why a lower sampling rate is the wrong lever

A lower random rate cuts the bill and the judged failures by the same share. Even so, a random sampling rate is a common cost control. LangSmith's docs suggest that "to control costs, you may want to set a filter to only apply the evaluator to 10% of traces". Langfuse lists sampling a percentage of traces first among its cost levers, and AWS sells percentage-based and conditional sampling for the same job.

If one session in 200 has a wrong answer, a 10% sample judges about one of every ten such sessions and never sees the other nine. The failures you most want judged are rare, so a random sample spends nearly all of its budget on sessions that went fine.

Check every session with code before a judge reads it

Many of the failures you pay a judge to find are yes or no questions a function can answer. Write those as checks and run them on every session, so the judge needs no rubric for them. Each check reads the trace and returns a flag:

  • The output is not valid JSON, or fails your schema, or is missing a required field.
  • The agent called a tool it should not have, or called the right tool with arguments that fail validation.
  • A tool call returned an error, or the agent sent an empty reply.
  • The same tool call repeats several times in a row, which usually means a loop.
  • The session's duration, token count or number of tool calls is far outside what is normal for that agent.
  • The user asks the same thing twice, or says the answer is wrong.

Checks like these are code reading data you already have, so they spend no tokens, against $0.021 for each judge call in the example above. They also give the judge a smaller job. A judge no longer needs a rubric for whether the agent returned valid JSON, so the rubric count falls along with the session count.

Send the judge the flagged sessions plus a random audit

The judge then reads two sets of sessions. The first is every session a check flagged. The second is a random audit: a small random share of the sessions no check flagged, judged so you can measure what the checks miss.

The judge's cost per session becomes the full cost times the share of sessions it reads. In the example above, say your checks flag 3% of sessions and you audit a random 1%. The judge reads 4% of sessions, and $0.126 times 0.04 is about $0.005, roughly what the agent costs. For judging that can wait, a batch interface cuts the token price by 50 percent at this provider, which halves that again. The table puts the same assumptions at 100,000 sessions a day.

What the judge reads Judge cost a day, at 100,000 sessions
Every session, six rubrics $12,600
A random 10% of sessions $1,260
The 3% the checks flag, plus a random 1% $504
The same 4%, at batch prices $252
What an LLM judge costs a day on the same 100,000 sessions, by which sessions it reads, every approach priced on the same model and token counts. Source: worked example from the article, one provider's published prices as of October 2026. Token counts are assumed.

The agent itself costs $500 a day in this example. Your flag rate is your own, and you learn it by running the checks over a week of your own sessions.

Keep the two sets apart in every report. The flagged set tells you what kind of failures you have, but its failure rate says nothing about your traffic as a whole, because it was chosen for looking bad. The random audit gives you how often unflagged sessions fail. Scaled up to every unflagged session and set beside the failures the judge found in the flagged set, it gives the checks' miss rate: the share of all failures that no check flagged.

Your overall failure rate combines the two sets, each weighted by its share of sessions. Say 3% of sessions are flagged and 40% of those fail, and 1% of the other 97% fail. The flagged set holds 1.2 failures per 100 sessions and the unflagged set 0.97, so the overall rate is about 2.2%, and the checks' miss rate is 0.97 of 2.17, about 45%. When the audit keeps finding one kind of failure, write a check for it.

Make each LLM judge call cost less

Once the judge reads fewer sessions, make each call cheaper: a smaller judge that escalates, a trimmed input, related rubrics in one call and batch pricing.

A cascade starts with a smaller judge and sends a session to a larger one only when the small judge's confidence, calibrated on labeled examples, falls below a threshold. In one study it kept a set level of agreement with human ratings while reducing the API cost by up to 40% compared with GPT-4 without abstention, and with weaker models it used 12.6% of GPT-4's cost. That was measured on benchmarks of pairs of chatbot answers, not on agent sessions.

A cascade saves money, but it does not make the judge more accurate. A 2026 preprint found that cascades on rubric-grading tasks gained "a mere 2.7 points at best over the strongest single judge", because the cheap first-stage model and the LLM judge it deferred to were mostly wrong on the same items.

The other savings come from the input and the call:

  • Send only what the rubric needs. A rubric about the final answer needs the last turns and the tool results it used, not the whole session. Some tools let you target specific observations instead of full traces for this reason.
  • Score related rubrics in one call when they read the same context, and ask for a label and a short reason rather than a long critique, because output tokens cost more than input tokens.
  • Batch what can wait. Judging that does not gate a release can usually run an hour later, at the 50 percent batch discount where your provider offers one.
  • Cache repeated judgments. Key each result on the input, the output, the rubric version, the judge prompt and the exact judge model version, and clear it when any of them changes.

A scoring call that failed is not a score of 0

Record whether a scoring call worked apart from its score, and leave the score empty when it failed. A cheaper eval has more steps, and every step is a call that can fail. A judge call can time out, hit a rate limit or return text that does not parse. A check can crash on a trace it did not expect. If a failure comes back as 0, it reads exactly like a session that really failed, and your scores cannot tell the difference.

Eval tools have turned failed scoring calls into scores, or dropped them, both by default and through bugs.

OpenTelemetry's generative AI semantic conventions already keep the two apart, in an evaluation result event, gen_ai.evaluation.result. The event carries error.type, required if the evaluation ended in an error, and a score that is required only "if applicable", so a failed evaluation can carry an error and no score at all. The event is still in development status as of October 2026, but you can use the same shape in your own tables.

  • Leave the score empty (null), never 0, when a check or judge call fails, and record the error class next to it.
  • Retry timeouts, rate limits and server errors with a backoff before you record a failure.
  • Leave failed rows out of averages, and count them. Report "scored 9,812 of 10,000" next to every average, and alert when the share that failed passes a threshold you set.
  • Send a session whose check crashed to the judge. A crashed check is not a clean result, so the session must not drop out of the flagged set without a judgment.

Test a cheaper LLM-as-a-judge evaluation setup before release

A cheaper judge only keeps your signal if it still agrees with the people who know what a good answer is. Keep a few hundred sessions that people on your team have labeled, and run any new judge model, prompt or set of checks against them before you switch. Compare where the new setup disagrees with the labels, not only its overall agreement. Run the comparison again whenever the agent, its prompt or its traffic changes.

Send a known bad session through on purpose. The checks should flag it, and the judge should mark it as failed.

Give the CI gate three outcomes: pass, quality regression and evaluation error. An evaluation error fails the build with its own message, or reruns, so your team does not chase a regression that is not there.

How to tell the cheaper eval still catches failures in production

You know the new setup works in production when you can answer these from your own reports:

  • The judge's cost per session, tracked week by week.
  • The flagged share and the audit share, reported apart, with the overall failure rate the two sets combine to.
  • The checks' miss rate, the share of all failures no check flagged, and a new check for each failure the audit keeps finding.
  • The count of scoring failures, with its alert.

Why the judge's short list has to come from every session

Judging the flagged sessions works only if the checks saw every session. A session you sampled away to save cost can never be flagged or audited.

Every session is analyzed at scale using statistics, ML classifiers and a small model for sentiment. Oodle's agent then checks each issue it finds against the evidence and removes the noise.

Oodle can run that on every session because of how it stores them. Traces go to object storage, and queries run on serverless compute that starts for each query and stops after. With Oodle, every session is kept in full for your retention period (30 days on Pro, longer for an added fee), and sampling is optional, not required to control cost.

Oodle identifies issues in your agents out of the box: tool failures, loops, outages, stalled conversations. An LLM judge comes in after simple checks and reads only the sessions they flag, the ones most likely to have failed. So the judge's short list is drawn from every session, kept and checked. Your random audit can be drawn from all the sessions no check flagged, and any session the judge did not read is still there to open in Oodle's agent observability, within your retention period.

Oodle's built-in code checks run on every plan. Custom code checks, which run your own Python against spans, are on the Enterprise plan. Fello runs its evals this way. In its product manager's words, "Code-based evals filter traces before we run LLM-based evals." Chaining its evals that way cut Fello's eval inference cost by 80 to 85%.

See your own telemetry in Oodle

Start debugging in minutes.

Start freeBook a demo

Common questions

How do I cut LLM-as-a-judge costs without losing eval signal?

Judge fewer sessions. Run checks written as code on every session, and send the judge only the flagged sessions plus a small random audit, 1% in our worked example, where the two together cost less than judging a random 10%. Then make each call cheaper with a smaller judge that escalates unclear cases, a trimmed input and batch pricing. Before release, test the cheaper setup against sessions people have labeled.

Running an LLM judge on every trace costs more than the agent itself. Should I just lower the sampling rate?

The better lever is which sessions the judge reads. A lower random rate cuts the cost and the judged failures by the same share, and the failures you want judged are rare. Keep a small random share as an audit, and choose the rest of the judge's sessions by what your checks flag.

How do I tell a real zero eval score from a scoring call that errored?

Record the status separately from the score. A check or judge call that failed gets an error class and no score, the shape OpenTelemetry's gen_ai.evaluation.result event uses, so only a call that worked can return 0. Retry transient errors, leave failures out of averages, and count them with an alert.

My CI gate can't tell an eval that failed from a real bad result. What should it do?

Give the gate three outcomes: pass, quality regression and evaluation error, and fail or rerun on an evaluation error with its own message. Check what your framework does with an exception first, because some turn it into a score of 0.0 by default.

Sources

Share