This is stage 2 of 5 in a guide to monitoring when AI coding agents do the engineering work. This page covers CI and release: what changes, what to measure, where your team is on three levels, and which tools cover it.
What changes when agents do this work
When code arrives continuously, the pipeline runs continuously. Continuous integration was designed around a human cadence: a developer pushes, waits, and looks at the result. Agents push far more often and do not wait, so the pipeline stops being a checkpoint and becomes a stream.
Release windows compress with it. A change that took two weeks to reach production can reach it in hours, which removes the soak time that used to catch problems for free.
What to measure
Pipeline runs per merged change
Watch the ratio. A rising ratio means agents are retrying rather than fixing, and it moves before your bill does.
Time from merge to production
Track the distribution. The interesting number is how much soak time you have removed, because that is the cover you no longer have.
Flake rate
Agents retry failing tests, so a flaky test that a human would have raised gets silently absorbed.
Rollback and revert rate
This is the clearest measure that releases are moving faster than you can verify them.
Three levels, from doing it by hand to letting the system do it
Find the level your team is at today. The next one is the step to take.
- 01Manual
The pipeline reports pass or fail
Someone looks at a red build and investigates. That is fine at human push rates.
- 02Instrumented
The pipeline emits telemetry
Build steps are spans. Flake rate, queue time, retry count and cost per pipeline are tracked over time rather than inspected per run, so you can see the trend that no single red build reveals.
- 03Autonomous
Release decisions read the telemetry
Progressive delivery is gated on live signals rather than elapsed time. If error rate or latency moves on the new version, the rollout stops without anyone being paged.
Tools that cover this stage
These are the tools that cover this stage. Each label says whether the tool has an MCP server of its own. The ones with a directory page link to it.