How to measure AI agent ROI

Part 1 of 2 in the "Measuring AI agent ROI" series

McKinsey calls the result the Gen AI paradox: roughly 80% of companies report widespread AI use, and almost none see real bottom-line impact. It isn't a model problem — the models are fine. It's a measurement problem. Most teams count activity — hours saved, uptime, pull requests, "80% of the company uses it" — and none of that is value.

Gen AI paradox — companies adopt AI everywhere yet see no profit from it, because using a tool is not the same as getting value out of it.

This is a two-part series. Part 1 covers what to measure: why agent ROI breaks the old math, which single number decides it, how to spot an agent going bad, and the fork that changes every answer — whether your team builds its own product or sells its time to clients. Part 2 covers the machinery — self-measurement, the PM dashboard, and the role-by-role setup. Get the concepts straight first; the tools only work once you know which number they're chasing.

Why agents break the old ROI math

Traditional software ROI is linear: fixed license, predictable seat cost, the same behavior on every call. Agents break all three, because they reason and adapt — three steps to solve a problem today, ten tomorrow. That changes the cost side before you measure anything.

Variable cost. You pay per token, per call, per reasoning step, and the meter runs faster on hard queries. Spend scales with difficulty, not with a monthly line item. A poorly bounded agent can get stuck in a planning loop and burn a month of budget in an afternoon, and nothing about that trips an error.

A permanent trust tax. Because an agent is probabilistic, you pay just to keep checking it — human review, AI-graded checks, safety audits. Bain puts this at 10–20% on top of base compute, and it never drops to zero. What you supervise never becomes fully predictable, so the overhead is permanent, not a start-up phase.

Trust tax — the ongoing cost of checking an AI's work (human review, automated checks, audits). It never hits zero, because the output is never guaranteed.

LLM-as-judge — using one AI model to grade another AI's output at scale, instead of a human reviewing every result by hand.

An integration tax up front. Messy data and legacy systems — Accenture calls it "reasoning debt," where bad data quietly inflates token use — can cut up to 29% off projected ROI before the agent earns a dollar. The cost side of any ROI sum is bigger than the license. Once you accept that, the harder question is what goes on the value side, and that's where most teams go wrong.

Measure outcomes, not activity

The most expensive mistake in 2026 is measuring whether the AI produced something instead of whether it solved the problem.

Picture a tool with a 99% completion rate — it always returns an answer — while its outcome rate is 60%, meaning it actually resolves the task three times in five. ROI lands entirely on the second number. The first just confirms the machine is switched on.

For a dev team the same trap looks different. A tool that writes a thousand lines, opens ten pull requests, and closes a ticket that then bounces back from QA has produced a mountain of activity and almost no outcome. So measure the ticket that shipped and stayed shipped, not the code that got typed. That one shift — outcome over activity — is what the rest of this post applies, to a team and to a person.

It fixes the cost side too. Normalize spend on finished work, not attempts: Agent Cost per Completed Task is total AI spend divided by tickets actually resolved. A tool that looks cheap per token but needs three retries and a human rescue on every ticket is not cheap at all. Which brings us to the hardest measurement of the lot — the gap between what developers feel and what they actually ship.

Felt speed is not real speed

Lines of code and pull-request counts are vanity metrics — numbers that look like progress but track effort, not value. Agents inflate both without touching what you care about. One agentic task can open dozens of pull requests to rename a method across 500 files: false velocity, activity with no result.

Self-reports are worse than useless. Developers feel about four hours a week saved. In a METR-style study they felt 20% faster while measurably running 20% slower on work where they weren't already experts. Measure the real thing — median pull-request throughput across 400+ companies — and the honest gain is 5–15%. Not zero, but not the 10x on the vendor slide either.

METR study — Model Evaluation & Threat Research, a controlled test where experienced developers expected AI to speed them up ~20%, but the clock showed it slowed them ~20% on unfamiliar work. How fast you feel is not how fast you are.

To get the number right, compare each engineer against their own pre-AI baseline, never against the person at the next desk — a same-person baseline (also called longitudinal analysis). One study at a financial firm used this method and found a 30% year-on-year throughput gain for AI-assisted developers, while a matched group without AI moved only 5%. That control group is what makes the number believable. Expect a J-curve first: a ~15% dip while people learn to steer the agent instead of writing everything themselves, before output climbs. Teams that panic at the dip kill the tool right before it pays off.

Same-person baseline — measure one person against their own past output, never against teammates, so differences in talent and experience cancel out.

J-curve — the dip-then-climb shape of adoption. Output drops ~15% while people learn the new way of working, then rises past where it started. Quit during the dip and you never see the climb.

Big companies measure this differently, and the disagreement is useful. Uber names four eras — pre-AI, AI-assisted, agentic, and "software factory" — and dropped its "developer time saved" metric, because it worried employees and the baseline moved too fast to mean anything. It now tracks Feature Velocity: features shipped, a deliberately "agent-proof" North Star. Uber's own caveat is the honest part — defining what counts as a "feature" at scale is harder than it sounds. Meta and Gradle go more granular: Feature DDM (diffs per developer per month, with an AI sorting each diff into feature versus toil) and character-level telemetry — tracking which characters were AI-written versus later human-corrected, so they can reason about long-term maintenance cost, not just today's speed.

Feature Velocity — count of shipped features, not lines or PRs. Chosen because an agent can't inflate it with busywork.

DDM (diffs per developer per month) — how many separate code changes a person ships in a month, split into real feature work versus routine toil.

Toil — necessary but low-value work: migrations, dependency bumps, boilerplate. Real, but it doesn't move the product forward.

Two of the most advanced engineering teams in the industry can't agree on the unit of work. That's the signal: there's no settled metric yet, so instrument for outcomes and stay skeptical of any single number.

Four forces decide whether the gain is real or fake, and they matter for everything below:

• The assembly-line bottleneck — faster code just piles up if review, QA, and security sign-off don't scale with it. Some teams shipped 50% more defects after adoption, because code outran checking.
• Workflow debt — bolt an agent onto a broken process and you only speed up the drafting of work that still sits in the same slow queue.
• The coding ceiling — writing code is only 14–25% of a developer's day, so even a big coding speed-up moves a small lever.
• Phantom productivity — saved hours are worth zero unless someone spends them on real work instead of letting them dissolve into meetings.

Almost every practical decision below is about beating one of these four.

Catch an agent that's getting worse

That was the productivity question: is the person faster. There's a second question underneath it: is the agent itself getting better or worse over time? Normal software makes this easy. When it breaks, it tells you — a red error, a crash, a failing test. An agent doesn't. It hands you a clean, confident, wrong answer and marks the task done. You can't wait for a bad agent to break, because not breaking is how it fails you.

It goes bad in two quiet ways. It can be right for the wrong reasons — the steps look sensible but the answer's wrong, or the answer's fine and it got there by luck. Look at the final answer alone and you'll miss it. Or it can go stale: someone updates the model, or the code and docs it reads drift out of date, and the agent slowly gets worse while nobody touches a line. No error fires. It just costs more and delivers less.

Cheap signals any team can track

You don't need a monitoring platform. The useful signals are things you can count from what the agent already does day to day:

• How often it gives up and asks a human to take over. Creeping past roughly 1 in 10 tasks means it's losing the thread.
• How many steps it takes to finish the same kind of task. A job that used to take three steps and now takes eight is drifting, even if it still gets there.
• How often it repeats itself — same search, same tool, over and over. That's looping, and looping is what quietly burns a week of budget in an afternoon.
• How often it stops to ask you to clarify. A jump here means you're not giving it enough context — a context problem, not a model problem.

Each one is an early warning. They rise before the work visibly gets worse, and long before the bill does. You don't need thresholds to the decimal — you need to notice when a steady number starts climbing.

Track one number: completed and correct

Under all those counts sits one question: of the tasks you handed the agent, how many did it actually finish and get right — not "produce something," but solve? Track that number over time. If it slides, the agent is getting worse, no matter how busy it looks or how good the demo felt. It's the same "outcome, not activity" rule from earlier, pointed at the agent instead of the team.

If your volume gets too big for a human to check every result, the usual move is to have a second AI grade the work. That's fine — but the grader drifts too. It gets fooled by longer, confident-sounding answers, and a sharp agent can learn to write for the grader instead of getting better. So spot-check the grader against a handful of results a human scored by hand, and redo that check every few weeks as the models change. A grader nobody checks isn't watching quality; it's just stopped paying attention.

Can you prove it made you faster?

The honest answer to the productivity question splits in two.

For one person, the only fair test is you against your own past, not you against the developer at the next desk. The upside: it's free, it's fair, and it needs no spreadsheet. The catch: "I feel faster" is not evidence. People routinely feel 20% quicker while the clock says they got slower, so you need a few weeks of real before-and-after before the number means anything.

For a team, you stack those personal numbers and add a control — a few people doing similar work without AI. The upside: "we feel faster" becomes "we're measurably X% faster than the group that isn't using it," which survives a skeptical boss or client. The catch: it needs a recorded "before" and a few months of patience. Measure too early and you capture only the slow learning dip, never the payoff.

What you're allowed to claim depends on one thing above all: whether your team keeps the speed-up or hands it back. That's the fork the second half of this post turns on.

Two teams, two ROI problems

Everything so far applies to any team. Here's the split that changes the answer: does your team own the product it builds, or sell its time to build someone else's?

It comes down to who keeps the speed-up, and it maps almost exactly onto how the work is billed.

• You own the product. Finish sooner and the saved time is yours — you start the next feature early. Same logic as a fixed-price job: the gain stays with you.
• You sell your time. Bill by the hour, get 30% faster, and you bill 30% fewer hours. The speed-up flows straight back to the client as a smaller invoice. This is the time-and-materials trap.

Time-and-materials (T&M) — billing a client for hours worked, rather than a fixed price for a finished result. Faster work means fewer billable hours — which is the whole problem for an outsourcing team.

The next two sections take each case on its own terms. Read the one you live in first, then read the other, because most real teams are a blend of both.

If your team owns the product

For a product team the math is friendly: the speed-up is yours to keep. Finish a feature in two days instead of three and you don't invoice less — you start the next feature a day early. Over a quarter that compounds into more shipped product on the same headcount. Your job isn't to defend a billing model; it's to make sure the saved time turns into real product and doesn't quietly evaporate.

Measure features shipped, not code written. This is Uber's move — Feature Velocity as the North Star, because an agent can't inflate it. Rising pull requests with flat feature output is false velocity: the team looks busier and ships the same amount. Count the things a user can actually feel.

Redeploy the saved hours on purpose. This is the trap that turns a real gain into a zero. If a developer reclaims eight hours a week and those hours dissolve into meetings, your ROI is nothing, no matter how fast the coding felt. Decide up front where the time goes — new features (revenue), paying down tech debt, or learning the next framework — and check that it landed there.

The drift risk is yours too. Because you keep the upside, you also absorb the downside. A drifting agent that loops and takes ten steps where it used to take three burns your roadmap time, not a client's budget — the same as a fixed-price overrun. So the cheap signals from the last section aren't optional for a product team. Watch them.

The upside of owning the product is that ROI is honest here: fewer moving parts, no billing games, and a clean line from "faster" to "more shipped." The work is discipline — keep counting outcomes, keep redeploying the time, keep an eye on drift.

If your team sells its time

Here the generic advice falls apart, because an outsourcing team has a problem a product team doesn't: it sells time. Get 20–40% faster on time-and-materials and your revenue drops by the same amount. You made your people more productive and shrank your own top line.

So for an outsourcing firm, measuring AI ROI isn't an efficiency exercise — it's a defense of the business model. The industry is already shifting from selling time to guaranteeing results, and the measurement has to follow.

The billing problem, and how to price around it

You can't capture AI gains under pure time-and-materials. There are three ways out, and most firms blend them.

Charge for outcomes, not hours. Set a fixed fee for a defined result, or take 10–25% of the financial impact you deliver. Now the AI gain accrues to you, not silently back to the client as a smaller invoice.

Sell capacity, not cost-cutting. The most valuable ROI lever for an outsourcing shop is running more client projects on the same headcount. Reframe the internal story from "cutting hours" to "creating capacity" — it's true, and it's the number that grows margin.

Sell speed as an asset, not a discount. Pitch that AI lets the team deliver at 3x speed, not at 10% lower cost. The client's own ROI case should show a faster brief-to-launch cycle — 9–55% faster is the kind of number that wins deals.

Track two returns, not one

An outsourcing firm has to watch two returns at once. Mix them up and margin leaks.

Internal ROI is efficiency and margin, for you. Use same-person baselines to prove the lift is real, Agent Cost per Completed Task to normalize spend, and always fold in the trust tax. This tells you whether the tooling pays for itself.

Client ROI is speed and quality, the thing you put in front of the client — brief-to-launch cycle time, features shipped, and stable defect rates. This is what renews the contract.

A scorecard for the whole engagement

The real win is instrumenting the whole job, not just the coding. Each phase gets a metric and a target:

• Bidding — proposal turnaround. Aim to cut RFP response time 60–80%.
• Onboarding — time to 10th pull request. New hires ramp ~50% faster; one shop went from 86 days to 33.
• Execution — assisted commit share. Aim for more than 27% of production code AI-written or assisted.
• Quality — change failure rate. Raise a flag if defects climb more than 2 points as speed goes up.
• Post-launch — escalation rate. Review the setup if human handoffs pass 10%.

RFP (request for proposal) — the document a client sends asking firms to bid on a job. Writing the response is slow, manual work, so AI cuts it sharply.

Build a skills asset, not a pile of licenses

One reframe for outsourcing leadership. The first agent you deploy is the most expensive — it carries the integration cost, the prompt work, the data cleanup. Every deployment after it is cheaper, because you're not buying a tool each time; you're building a skills asset the whole firm reuses. The migration prompt one team gets working becomes a tool every other team gets for free. A pile of licenses loses value as it ages. A shared skills library gains value as it grows, which is why firms that build one pull ahead of the ones running scattered pilots.

Watch the outsourcing-specific trap: phantom productivity with a billing twist. If a developer reclaims eight hours a week and that time vanishes into Slack, you've lost billable capacity and gained nothing. The saved hours have to be redeployed to client work on purpose — which leads to the harder problem both kinds of team share: getting people to actually work this way.

Getting the team to actually use it

A team lead's real problem isn't "does AI help." It's that fifteen engineers will use fifteen tools fifteen ways — some burning tokens on trivial autocomplete, some ignoring it, some quietly shipping unreviewed agent output. Consistency is the job. And the finding across the sources is blunt: mandates backfire. Top-down "you must use AI" orders create resistance and gaming. What works is enablement plus social proof.

Show the power users, don't order the rest

Roughly 8% of your engineers will find the big gains first. Don't average them into a team stat — put them in front of the team. Have your top AI-assisted contributor run a short live demo each week, showing exactly how they used a tool to clear a specific bottleneck. Peers copy a colleague solving their codebase's problem far faster than they adopt a policy. This is habit formation through visible wins, not compliance.

Point AI at the boring work first

Adoption sticks when the first win is obvious. Steer people toward the tedious, high-friction work AI is clearly good at — stack-trace analysis, debugging, refactoring, migrations. Developers rank these above mid-task code generation, because the time saved is visible and the risk is low. Trust built on boring work transfers later to riskier uses. Trust destroyed by a bad early autocomplete doesn't come back.

Tell the team about the early slowdown

One line prevents most adoption anxiety: tell the team, out loud, that velocity will drop ~15% for the first 30–60 days. When leadership names the J-curve dip in advance, engineers read a slow week as expected learning instead of personal failure — and they don't quietly abandon the tool during the window when it feels worst.

Turn scattered use into shared assets

Consistency comes from a shared base, not from surveillance. Two rituals do most of the work:

• Weekly office hours where anyone can bring a stuck task and watch it solved with AI live.
• A team skills registry — a shared, version-controlled library of prompts and agent "skills" tested on real projects. When someone works out the prompt that reliably upgrades an API version, it goes in the registry and stops being tribal knowledge.

That registry is really a context engineering problem in disguise — the shared base an agent reads from is what makes fifteen engineers produce consistent output instead of fifteen dialects. There's a team-level version of this idea in Context Engineering Is a Team Sport. If it's new to you, start the series at the beginning — The Goldfish Problem is Part 1 and builds the full picture from the individual up to the team.

Keep token spend under control

Consistency has a cost side the team lead owns directly: fifteen people using agents freely is fifteen ways to quietly burn budget. The worst offender is the recursive loop — a poorly bounded agent stuck in a planning or tool-calling cycle can eat months of projected token budget in hours, and it never throws an error. The second is subtler: eager developers reaching for a full agentic run on tasks a one-line edit would fix.

The fix isn't a spend cap that punishes usage — that just teaches people to under-use the tool you paid for. It's Agent Cost per Completed Task as a health signal. Watch cost-per-resolved-task by workflow and by seat, and when one spikes, coach it: is the agent looping, is the context bloated, is someone using a heavy tool for a tiny job. Boring-work-first adoption helps here too — pointing people at debugging and migrations, where the value is obvious, crowds out the low-value autocomplete that runs up cost for nothing. The goal is a team that spends tokens where they turn into outcomes and stops where they don't. That's cost discipline as coaching, not surveillance.

That's how you get a pattern instead of a free-for-all: not by policing tool use, but by making the good path the easy path. Which raises the question every engineer is quietly asking: am I actually getting faster, or does it just feel that way?

Feeling versus fact is where the real measuring starts. Part 2 is the machinery: the self-test one developer runs against their own past, the one-command dashboard a PM refreshes each sprint, and the role-by-role setup that makes it run.

Next up: The AI agent ROI playbook: dashboards, roles, and a baseline that holds up