What to measure when the machine writes the code

The first time we changed something significant in our harness, the team said it felt faster. I believed them, because it felt faster to me too.

Then I tried to prove it and could not. That sprint had two unusually easy tickets in it. One engineer was on holiday, so fewer people were competing for review. We had also merged a test speedup the same week. Any of those could have produced the feeling, and I had no way to separate them.

That is the actual risk with agentic development. Not that it fails, but that it works erratically and you cannot tell which parts, so you keep changing things based on vibes and slowly make it worse while everyone agrees it is going well.

So this is the measurement setup we ended up with: which numbers, how they are instrumented, and what the same design looks like whether you have 8 engineers or 800.

Three layers, and one rule

Metrics sprawl kills more measurement programs than bad metrics do. So organize them in three layers and apply one rule.

The outcome layer is whether the business gets software faster and more safely. This is the four delivery measures that have survived a decade of scrutiny: how often you deploy, how long a change takes to reach production, what share of changes cause a failure, and how quickly you recover. These are the truth. Nothing below is allowed to replace them.

The flow layer is where work queues up: time from pull request opened to merged, time spent waiting in CI, and how much work gets thrown away.

The harness layer is whether the agent system itself is healthy.

Now the rule. You may add a harness metric only if you can name which outcome metric it is supposed to explain. If it explains nothing above it, it is trivia, and it will end up on a dashboard nobody reads. That rule alone will cut your proposed metric list in half.

The harness metrics, each with a guard

Every one of these becomes a lie the moment it becomes a target, so each is paired with a metric that catches the obvious gaming. Report them in pairs or do not report them.

Autonomy rate. The share of tasks that reach a reviewable pull request with no human intervention along the way. This is the headline number for whether the harness is actually carrying work. Guard it with first-pass acceptance, because the easy way to raise autonomy is to hand the agent smaller and smaller tasks until it cannot fail.

First-pass acceptance. The share of agent-authored pull requests merged without a substantive change request. Substantive means logic, structure, or approach, not a naming nit. Guard it with change failure rate, because the other easy way to make this number beautiful is to stop reviewing properly.

Human review minutes per merged change. This should fall as the harness improves, since the mechanical pass is already done. Guard it with escaped defect rate, meaning defects found after merge, because review time falling while defects rise is not efficiency; it is abdication.

Eval pass rate, per harness version. Your fixed set of real cases, scored each run identically. Guard it with the size and age of the eval set, because the simplest way to improve a score is to quietly delete the cases that fail. Track how many cases exist and when one was last added.

Cost per merged change. Model spend plus CI spend, divided by merged changes. Not total token spend, which tells you nothing except that you are busy. Guard it with first-pass acceptance, because dropping to a cheaper model reduces cost per attempt and increases attempts.

Specification coverage. The share of merged changes that had a written spec before code started. This is a leading indicator: when it drops, first-pass acceptance drops about two weeks later, reliably enough that I now treat it as an early warning rather than a hygiene stat.

Rework rate. The proportion of lines changed within 21 days of being merged. Sustained high churn means the system is generating confident code that solves the wrong problem, which no acceptance metric will reveal on its own.

Instrumenting it without inventing a schema

OpenTelemetry is the right substrate, for the boring reason that it lets the agent runtime, the pipeline, and the production system speak the same language and end up in one place.

Model it as one trace per unit of work. The root span is the task. Under it sit the phases: specification, build, review, pipeline, deploy. Under build sit the agent's own spans.

The GenAI semantic conventions give you names for most of this so you are not inventing your own. Agent invocations appear as invoke_agent spans, tool calls as execute_tool, and token accounting as gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. Provider is gen_ai.provider.name. Most usefully, evaluation results have their own event,gen_ai.evaluation.result, carrying a name, a score, and an explanation, which means your evals stop living in a separate spreadsheet and become telemetry like everything else.

One warning, from experience rather than documentation. As of now, every GenAI convention is still at Development status; they now live in their own repository, which has no tagged releases, and the names have already moved twice. Token attributes were renamed once, and the provider attribute replaced an earlier one. So do not scatter these strings across your codebase. Emit through one thin module that owns the attribute names, and a rename becomes a single file rather than a migration.

The pipeline side is better behaved. CI/CD conventions give you cicd.pipeline.run.duration as a histogram with cicd.pipeline.name, a run state of pending, executing, or finalizing, and a result of success, failure, timeout, cancellation, error, or skip. That result attribute is what lets you separate real failures from infrastructure noise, which is the number that told us our flakiness problem was worse than anyone believed. cicd.pipeline.run.active Broken down by state, it gives you queue depth, which is your early warning that CI has become the constraint.

The decision that matters more than the tooling

One identifier has to travel the whole way: ticket, specification, branch, pull request, pipeline run, deployment, and any incident that follows.

Put it in the state file, in the branch name, in a commit trailer, in the pipeline as an environment variable, and on the deploy marker in your observability data. Everything else in this post is replaceable. This is not, because without it you have five disconnected datasets and every interesting question becomes a manual join that nobody performs twice.

The questions this makes answerable are the ones you actually want. Do changes with a proper specification have a lower failure rate? Did the incident on Tuesday come from a change the agent wrote unsupervised? Did review time fall for the changes produced under the new harness version, or only for small ones?

Retrofitting this identifier costs roughly ten times what adding it now does, and I have paid both prices.

The same design at three sizes

Under about 20 engineers, do not build a platform. A scheduled job that pulls your git host's API, joins it to your model provider's usage export, and writes to a single table gets you most of this. Look at one number a week. The only thing you must not skip is the correlation identifier, because that is the part that is expensive to add later and free to add now.

In the middle, run a collector and use the backend you already pay for. An OpenTelemetry Collector in front of whatever you use for production monitoring, traces for debugging individual weird tasks, metrics for the weekly review.

Here you hit the trap that catches most teams: observability retention is typically 15 to 30 days, and every question worth asking about a harness spans quarters. So aggregate daily into a warehouse table you keep for years. Traces are for diagnosis, the warehouse is for trends, and confusing the two is why so many teams cannot answer what changed since March.

At enterprise scale, the conventions become a data contract. Attribute names, required fields, and ownership get written down and reviewed, because 40 teams emitting slightly different attributes produces a dataset nobody can aggregate. Add per-team scorecards rather than one company number, since a platform team and a mobile team have genuinely different profiles and a blended average hides both. And make harness versions first-class, tagged on every span, because at that size something changes every week and unattributable metric movement is worse than no metric.

Treat every harness change as an experiment

This is the discipline that separates measurement from decoration.

Before you change the harness, write down which metric you expect to move, in which direction, and by roughly how much. Tag all telemetry with the harness version. Compare a fixed window before and after, at least two weeks each, and be suspicious of anything that improves within days.

Watch for mix shift, because it is the most common false positive. A fortnight of straightforward CRUD work will make any harness look brilliant. Track the distribution of task types alongside the metrics, or you will keep congratulating yourself for easy sprints.

Cadence that has worked for me: evals on every harness change, flow metrics weekly, outcome metrics monthly, and a quarterly look at whether the harness metrics still predict the outcome ones. When a harness metric stops predicting anything, delete it. Metric sets should shrink over time, not grow.

What not to measure

Lines of code, now genuinely meaningless. Pull requests merged, which measures how fast your generator runs, and your generator is not your constraint. Percentage of code written by AI, which is a vanity number with no decision attached to it: nobody has ever changed anything sensible in response to it moving. Total token spend as a headline, as opposed to cost per merged change. And story points, which never survived contact with reality and definitely do not survive this.

Each of those is worse than useless, because they occupy the space on a dashboard where a real metric should be, and they give an executive something confident to say in a meeting about a thing they have not measured.

The one rule again

Measure whether the business gets software faster and more safely. Then measure where work queues up. Then, and only then, measure the harness, and only the parts that explain the two layers above.

A harness that cannot be shown to move the top layer is a very expensive way of feeling modern. The good news is that the instrumentation to prove it either way is now about a fortnight of work, and most of the standards you need already exist.

If you want a hand designing this for your setup, that is the work I do.