This is stage 1 of 5 in a guide to monitoring when AI coding agents do the engineering work. This page covers code, tests and review: what changes, what to measure, where your team is on three levels, and which tools cover it.
What changes when agents do this work
Writing code, writing tests for it and reviewing the result used to be three jobs done by three people at three different times. An agent does all three inside a single session, and the artifacts it produces arrive faster than any human review queue was built to absorb.
The unit of work stops being the commit and becomes the session. A reviewer looking at a diff sees the output; the thing that explains the output is the transcript, the tools the agent called and the tests it chose to write. None of that lands in your version control system by default.
One published data point gives the scale: a single organization recorded 48,095 merge requests and 131,246 review runs over thirty days, at a median cost of $0.98 per review run.
What to measure
Review runs, not just pull requests
A pull request may be reviewed several times as the agent iterates. Counting merge requests understates the work by roughly 2.7x in the published figure above.
Cost per review run
The median is small and the tail is not, so track the distribution rather than the average. A runaway retry loop hides inside a healthy-looking mean.
Test coverage the agent added versus the coverage it claimed
Generated tests can assert on the implementation rather than the behavior. Coverage that rises without defect escape falling is a signal, not a win.
Turn count per session
An ICLR paper covering 15 models and roughly 200,000 conversations found quality degrading by 39% in multi-turn settings compared to equivalent single-turn ones. Quality declines in long sessions.
Three levels, from doing it by hand to letting the system do it
Find the level your team is at today. The next one is the step to take.
- 01Manual
You read the diff
Agent output is reviewed the way human output was: someone opens the pull request and reads it. It works until the volume of pull requests exceeds the hours available to read them.
- 02Instrumented
Sessions are recorded and costed
Every agent run emits a trace: which tools it called, which files it touched, how many turns it took and what it cost. Review runs and merge requests are counted separately. You can now answer why a change looks the way it does without asking the person who prompted it.
- 03Autonomous
The pipeline gates on the signals
Sessions above a turn threshold, above a cost threshold or with a failing eval are held for human review automatically. The gate is set by what the session did, and nobody has to be on call to catch it.
Tools that cover this stage
These are the tools that cover this stage. Each label says whether the tool has an MCP server of its own. The ones with a directory page link to it.