This is stage 4 of 5 in a guide to monitoring when AI coding agents do the engineering work. This page covers agent observability and evals: what changes, what to measure, where your team is on three levels, and which tools cover it.
What changes when agents do this work
The three stages before this one watch software that agents produced. This stage watches the agents themselves. It is a different data shape: a trace here is a chain of prompts, tool calls and model responses, and the failure modes are semantic rather than operational. Nothing crashed. The answer was wrong.
Agent traces almost always sit in a different product from the rest of your telemetry. So when a slow database makes an agent slow, the symptom is in one tool and the cause is in another, and the person joining them up is doing it by timestamp and memory.
What to measure
Eval score over time, per prompt version
A score without a version attached tells you something changed but not what. Attach the prompt version and the number tells you what changed.
Turn count and context growth per session
Both correlate with quality decline, and both are cheap to record.
Tool-call failure and retry rates
An agent that retries a failing tool six times is describing a broken integration, not a hard problem.
Whether the trace joins to your infrastructure trace
This is the most useful property here, and the one most often missing. Shared trace IDs across both are what turn two tools into one investigation.
Three levels, from doing it by hand to letting the system do it
Find the level your team is at today. The next one is the step to take.
- 01Manual
You read the transcripts
Someone opens the session log when a user complains. This finds the bug you were told about and none of the ones you were not.
- 02Instrumented
Sessions are traced and scored
Prompts, tool calls and responses are captured as spans with a stable schema. Automated evaluation runs against them. Be aware that some vendors bill for the objects their own evaluation features create.
- 03Autonomous
Regressions are caught before release
Evals run in the pipeline against a held-out dataset, and a score drop blocks the deploy. At this point the code stage, the CI stage and this stage are the same loop.
Tools that cover this stage
These are the tools that cover this stage. Each label says whether the tool has an MCP server of its own. The ones with a directory page link to it.