How to set it up
How evals work in Oodle
Pick an evaluator, chain it, watch the score, build a test set, run experiments, score the past.
1 · Pick
Start from the library.
Hallucination, task failure, helpfulness, jailbreak attempts, toxicity, forgetting, relevance, correctness. Eighteen ready to use, or write your own as a prompt or as code.
- HallucinationLLM
- HelpfulnessLLM
- Task failureLLM
- Jailbreak attemptLLM
- NSFW contentLLM
- ForgettingLLM
- User praiseLLM
- RelevanceLLM
- ToxicityLLM
- CorrectnessLLM
- ConcisenessLLM
- Context relevanceLLM
+ 6 more · or write your own as a prompt, as code, or as a comparer
2 · Chain
Cost-efficient first, expensive last.
Sentiment on every session, a domain classifier on the unhappy ones, and a code check or an LLM judge only where it applies. Hover a box to see what it does.
AISentiment
LLMActivesmall model
100% of sessions
Scores every user turn from −1 to +1 with a small model. A fraction of a cent per session, so it applies to everything.
100% of sessions · Scores every user turn from −1 to +1 with a small model. A fraction of a cent per session, so it applies to everything.
</>Runtime checks
CodeBuilt in
100% of sessions
Tool errors, timeouts, loops and empty replies, read straight from the trace. No model, no cost, always on.
100% of sessions · Tool errors, timeouts, loops and empty replies, read straight from the trace. No model, no cost, always on.
AIDomain classifier
LLMActivegpt-4o-mini
18% · negative sentiment only
Labels the conversation: refund, shipping, account, other. A cost-efficient call, made only when sentiment is negative.
18% · negative sentiment only · Labels the conversation: refund, shipping, account, other. A cost-efficient call, made only when sentiment is negative.
</>Refund policy check
CodeActive
6% · refunds with negative sentiment
Plain code: does the amount the agent promised match the policy table? Deterministic, so it only makes sense once the classifier says this is a refund.
6% · refunds with negative sentiment · Plain code: does the amount the agent promised match the policy table? Deterministic, so it only makes sense once the classifier says this is a refund.
AITask failure
LLMActiveclaude-sonnet-4
6% · refunds with negative sentiment
Reads the whole session and decides whether the customer got what they asked for. The expensive one, so it goes last, on the sessions the cost-efficient tiers flagged.
6% · refunds with negative sentiment · Reads the whole session and decides whether the customer got what they asked for. The expensive one, so it goes last, on the sessions the cost-efficient tiers flagged.
3 · Watch
Watch the score over time.
Every evaluator is a line on a chart, per service, per model, per prompt version. When a change moves it, you get a drift alert before customers tell you.
4 · Dataset
Turn real conversations into a test set.
Pick the sessions that went wrong, or a slice of the ones that went well, and save them as a dataset. Real inputs and real expected outputs, not made-up examples. Add to it from any trace.
5 · Experiment
Try a change in the playground. Score it as an experiment.
Open the playground with a session’s inputs already filled in. Change the prompt or swap the model and try again. When it looks right, run it against the whole dataset as an experiment. The same evaluators score every result, so you compare versions by number, not by eye.
6 · Backfill
Score the past, too.
Wrote a new evaluator today? Apply it to last quarter’s sessions. You still have every one of them, so your baseline is real.
Forgetting · last 90 days · order_support_agent
0 runs
✓ complete · baseline 2.1% · scores written back to every run