Why LLM-as-a-judge costs more than the agent it scores
A judge rereads the finished transcript once for every rubric, and teams often pick a larger model to judge with. An agent re-sends its growing context on every turn, but it usually runs on a small model.
The worked example below is our arithmetic from one provider's published prices, as of October 2026. Every token count in it is an assumption, so put your own in.
- The agent. A session of eight turns sends 40,000 input tokens and gets 2,000 output tokens back, in total. On a small model at $0.10 per million input tokens and $0.50 per million output tokens, that is $0.004 plus $0.001, or $0.005 a session.
- The judge. The finished transcript is 9,000 tokens, and the judge writes a 300-token verdict. On a mid-size model at $2 per million input tokens and $10 per million output tokens, one rubric costs $0.018 plus $0.003, or $0.021.
- Six rubrics. Six judge calls, for rubrics such as helpfulness, correctness, tone and policy, cost $0.126 a session. That is about 25 times the agent's $0.005 a session.
This LLM-as-a-judge cost of $0.021 a call is in line with what one tracing vendor calls a typical cost of $0.01 to $0.10 per assessment.
The judge's model is not the only driver, though. Put the same six rubrics on the agent's small model and each call costs about $0.001, so six of them come to $0.0063, still a quarter more than the agent's $0.005 a session. The rubric count and the full transcript still put the judge above the agent.
One cloud's built-in evaluators bill the same way, by the input and output tokens they process, so a judge that reads the whole session pays for the whole session on every rubric.
Why a lower sampling rate is the wrong lever
A lower random rate cuts the bill and the judged failures by the same share. Even so, a random sampling rate is a common cost control. LangSmith's docs suggest that "to control costs, you may want to set a filter to only apply the evaluator to 10% of traces". Langfuse lists sampling a percentage of traces first among its cost levers, and AWS sells percentage-based and conditional sampling for the same job.
If one session in 200 has a wrong answer, a 10% sample judges about one of every ten such sessions and never sees the other nine. The failures you most want judged are rare, so a random sample spends nearly all of its budget on sessions that went fine.
Check every session with code before a judge reads it
Many of the failures you pay a judge to find are yes or no questions a function can answer. Write those as checks and run them on every session, so the judge needs no rubric for them. Each check reads the trace and returns a flag:
- The output is not valid JSON, or fails your schema, or is missing a required field.
- The agent called a tool it should not have, or called the right tool with arguments that fail validation.
- A tool call returned an error, or the agent sent an empty reply.
- The same tool call repeats several times in a row, which usually means a loop.
- The session's duration, token count or number of tool calls is far outside what is normal for that agent.
- The user asks the same thing twice, or says the answer is wrong.
Checks like these are code reading data you already have, so they spend no tokens, against $0.021 for each judge call in the example above. They also give the judge a smaller job. A judge no longer needs a rubric for whether the agent returned valid JSON, so the rubric count falls along with the session count.
Send the judge the flagged sessions plus a random audit
The judge then reads two sets of sessions. The first is every session a check flagged. The second is a random audit: a small random share of the sessions no check flagged, judged so you can measure what the checks miss.
The judge's cost per session becomes the full cost times the share of sessions it reads. In the example above, say your checks flag 3% of sessions and you audit a random 1%. The judge reads 4% of sessions, and $0.126 times 0.04 is about $0.005, roughly what the agent costs. For judging that can wait, a batch interface cuts the token price by 50 percent at this provider, which halves that again. The table puts the same assumptions at 100,000 sessions a day.
| What the judge reads | Judge cost a day, at 100,000 sessions |
|---|---|
| Every session, six rubrics | $12,600 |
| A random 10% of sessions | $1,260 |
| The 3% the checks flag, plus a random 1% | $504 |
| The same 4%, at batch prices | $252 |
The agent itself costs $500 a day in this example. Your flag rate is your own, and you learn it by running the checks over a week of your own sessions.
Keep the two sets apart in every report. The flagged set tells you what kind of failures you have, but its failure rate says nothing about your traffic as a whole, because it was chosen for looking bad. The random audit gives you how often unflagged sessions fail. Scaled up to every unflagged session and set beside the failures the judge found in the flagged set, it gives the checks' miss rate: the share of all failures that no check flagged.
Your overall failure rate combines the two sets, each weighted by its share of sessions. Say 3% of sessions are flagged and 40% of those fail, and 1% of the other 97% fail. The flagged set holds 1.2 failures per 100 sessions and the unflagged set 0.97, so the overall rate is about 2.2%, and the checks' miss rate is 0.97 of 2.17, about 45%. When the audit keeps finding one kind of failure, write a check for it.
Make each LLM judge call cost less
Once the judge reads fewer sessions, make each call cheaper: a smaller judge that escalates, a trimmed input, related rubrics in one call and batch pricing.
A cascade starts with a smaller judge and sends a session to a larger one only when the small judge's confidence, calibrated on labeled examples, falls below a threshold. In one study it kept a set level of agreement with human ratings while reducing the API cost by up to 40% compared with GPT-4 without abstention, and with weaker models it used 12.6% of GPT-4's cost. That was measured on benchmarks of pairs of chatbot answers, not on agent sessions.
A cascade saves money, but it does not make the judge more accurate. A 2026 preprint found that cascades on rubric-grading tasks gained "a mere 2.7 points at best over the strongest single judge", because the cheap first-stage model and the LLM judge it deferred to were mostly wrong on the same items.
The other savings come from the input and the call:
- Send only what the rubric needs. A rubric about the final answer needs the last turns and the tool results it used, not the whole session. Some tools let you target specific observations instead of full traces for this reason.
- Score related rubrics in one call when they read the same context, and ask for a label and a short reason rather than a long critique, because output tokens cost more than input tokens.
- Batch what can wait. Judging that does not gate a release can usually run an hour later, at the 50 percent batch discount where your provider offers one.
- Cache repeated judgments. Key each result on the input, the output, the rubric version, the judge prompt and the exact judge model version, and clear it when any of them changes.
A scoring call that failed is not a score of 0
Record whether a scoring call worked apart from its score, and leave the score empty when it failed. A cheaper eval has more steps, and every step is a call that can fail. A judge call can time out, hit a rate limit or return text that does not parse. A check can crash on a trace it did not expect. If a failure comes back as 0, it reads exactly like a session that really failed, and your scores cannot tell the difference.
Eval tools have turned failed scoring calls into scores, or dropped them, both by default and through bugs.
- An example whose evaluation raises an exception in
DSPy's
Evaluategets afailure_scorethat defaults to 0.0, as of October 2026. - A bug in Braintrust's Python SDK, fixed in
version 0.35.0, returned an empty score summary when fetching it failed. A CI gate
"sees the same shape (
scores: {}) whether the eval genuinely bombed or Braintrust just had a blip". - An open report against Opik's evaluation pipeline says failed metrics are left out of every average without notice, so "a crashing-judge run reports hallucination mean 0.0 over 5 survivors while an honest judge reports 0.5 over 10 on the same outputs."
OpenTelemetry's generative AI semantic conventions already keep the two apart, in an evaluation result event,
gen_ai.evaluation.result. The event carries
error.type, required if the evaluation ended in an error,
and a score that is required only "if applicable",
so a failed evaluation can carry an error and no score at all. The event is still in development status as of October
2026, but you can use the same shape in your own tables.
- Leave the score empty (null), never 0, when a check or judge call fails, and record the error class next to it.
- Retry timeouts, rate limits and server errors with a backoff before you record a failure.
- Leave failed rows out of averages, and count them. Report "scored 9,812 of 10,000" next to every average, and alert when the share that failed passes a threshold you set.
- Send a session whose check crashed to the judge. A crashed check is not a clean result, so the session must not drop out of the flagged set without a judgment.
Test a cheaper LLM-as-a-judge evaluation setup before release
A cheaper judge only keeps your signal if it still agrees with the people who know what a good answer is. Keep a few hundred sessions that people on your team have labeled, and run any new judge model, prompt or set of checks against them before you switch. Compare where the new setup disagrees with the labels, not only its overall agreement. Run the comparison again whenever the agent, its prompt or its traffic changes.
Send a known bad session through on purpose. The checks should flag it, and the judge should mark it as failed.
Give the CI gate three outcomes: pass, quality regression and evaluation error. An evaluation error fails the build with its own message, or reruns, so your team does not chase a regression that is not there.
How to tell the cheaper eval still catches failures in production
You know the new setup works in production when you can answer these from your own reports:
- The judge's cost per session, tracked week by week.
- The flagged share and the audit share, reported apart, with the overall failure rate the two sets combine to.
- The checks' miss rate, the share of all failures no check flagged, and a new check for each failure the audit keeps finding.
- The count of scoring failures, with its alert.
Why the judge's short list has to come from every session
Judging the flagged sessions works only if the checks saw every session. A session you sampled away to save cost can never be flagged or audited.
Every session is analyzed at scale using statistics, ML classifiers and a small model for sentiment. Oodle's agent then checks each issue it finds against the evidence and removes the noise.
Oodle can run that on every session because of how it stores them. Traces go to object storage, and queries run on serverless compute that starts for each query and stops after. With Oodle, every session is kept in full for your retention period (30 days on Pro, longer for an added fee), and sampling is optional, not required to control cost.
Oodle identifies issues in your agents out of the box: tool failures, loops, outages, stalled conversations. An LLM judge comes in after simple checks and reads only the sessions they flag, the ones most likely to have failed. So the judge's short list is drawn from every session, kept and checked. Your random audit can be drawn from all the sessions no check flagged, and any session the judge did not read is still there to open in Oodle's agent observability, within your retention period.
Oodle's built-in code checks run on every plan. Custom code checks, which run your own Python against spans, are on the Enterprise plan. Fello runs its evals this way. In its product manager's words, "Code-based evals filter traces before we run LLM-based evals." Chaining its evals that way cut Fello's eval inference cost by 80 to 85%.




