---
title: "The AI agent ROI playbook"
slug: "the-ai-agent-roi-playbook"
publishedAt: "2026-07-28T17:28:32.097Z"
dateModified: "2026-07-28T17:28:32.097Z"
canonical: "https://www.notesof.dev/the-ai-agent-roi-playbook"
author: "Hieu (Chris) Pham"
tags: ["ai-agents", "ai-workflow", "developer-productivity", "engineering-metrics", "ROI"]
excerpt: "The measurement machinery for AI agents: a self-test against your own past, a PM dashboard you refresh in one command, and the role-by-role setup that makes ROI real."
---

# The AI agent ROI playbook

***Part 2 of 2 in the “Measuring AI agent ROI” series***

**TL;DR:** Part 1 covered *what* to measure; Part 2 is how to prove it. It walks through the machinery: **a same-person baseline, a one-command PM dashboard, and role-by-role ownership**. None of it holds up without **a recorded “before” baseline** — skip that, and every “after” is just a feeling.

[Part 1](/posts/how-to-measure-ai-agent-roi) covered *what* to measure — outcome over activity, real speed over felt speed, how your team's business model decides who keeps the gain, and how to get people using AI at all. This post is the machinery: how one developer proves the gain to themselves, how a PM rolls fifteen of those personal results into a dashboard that holds up over a sprint, and what each role — tech lead, PM, developer, tester — actually sets up. Every number below assumes you already recorded a "before." If you didn't, that's the first step here.

## From a feeling to a number

A developer wants to sharpen their own workflow and prove to themselves the gain is real. A PM wants to roll fifteen of those personal results into one team number that holds up sprint over sprint. Those are two different measurements, and they need two different tools.

## The individual: test AI against yourself, ticket by ticket

The most honest test an engineer can run costs nothing and takes six weeks. It's the [METR method - Model Evaluation and Threat Research](https://metr.org/research/) at personal scale — run it daily:

> **1. Pick a ticket.**
>
> **2. Before you start, write down how long you think it'd take *without* AI.**
>
> **3. Flip a coin. Heads, use AI. Tails, do it by hand.**
>
> **4. Do the task, and record the *actual* time.**
>
> **5. Log the gap between the actual time and your estimate.**

After six weeks you have a chart of your personal AI edge — the stacks and task types where AI genuinely makes you 2x, and the ones where it's a net negative. That last part matters most. The point isn't to prove AI always wins; it's to learn where to use it and where to skip it.

The rule is the same one the org uses: **compare yourself to your own past, never to the person next to you.** Different client, different stack, different experience — cross-engineer rankings are noise and breed anxiety. Your pre-AI baseline is the only fair comparison.

And run a private **redeployment check**: of the hours AI saved you this sprint, where did they go — feature work, refactoring, admin, or meetings? If the honest answer is "meetings," your personal ROI is zero, no matter how fast the coding felt.

Your real growth comes down to one thing: **how much of the work you've learned to hand off to the agent.** Are you using it only to write code mid-task, or have you pushed it into planning, review, testing, and deploys too? Widening that is how you improve without breaking the shared rules — the team's skills library and quality gates stay fixed, but inside them you keep handing off more of the boring work. The gains also grow with seniority, which surprises people: senior engineers often get the **bigger** lift, because knowing the domain is what lets you steer an agent and catch it when it's confidently wrong. AI doesn't flatten the skill curve; it rewards the people who can check its output. That's a reason to go deeper, not coast, and a big part of [where developers actually pivot in the AI era](/posts/where-developers-pivot-in-ai-era)

![A lone rock spire on a wide golden plain, casting a long shadow across sunlit grass.](https://a.storyblok.com/f/288495199849781/1024x576/947d111b4d/ai-agent-roi-playbook-baseline-landmark.webp)

### The PM: a dashboard for the whole team

A PM can't watch fifteen individual logs. They need a rolled-up scorecard with three layers — are people using it, is it helping, is it getting worse — and fixed thresholds so a bad trend shows up before it costs a quarter. Group it like this:

- **Use — Weekly Active Users.** Target more than 74% of the team. Under 50% is an adoption problem, not a tooling one.
- **Impact — PR cycle time.** Expect a 10–15% year-on-year gain. A flatline means the bottleneck is code review, not coding.
- **Value — feature velocity.** Features shipped should rise. Rising PRs but flat features is false velocity.
- **Stability — change failure rate.** Hold it under ~4%. A rise of more than 2 points means shift the team to validation.
- **Drift — steps per task.** Stay within ±5% of baseline. Up more than 15% means the agent's logic is drifting or looping.
- **Drift — retry rate.** Under 5% per session. Over 5% points to model or API trouble.
- **Drift — escalation rate.** Under 10% of tasks. Over 10% means the context you feed it is too thin.

Read it top to bottom and it tells a clear story. Use shows whether adoption took. Impact and value show whether the gain is real or vanity. The drift rows are cheap counters that need no AI to compute, and they catch an agent going bad *before* the invoice does. Track it bi-weekly and the sprint-to-sprint trend becomes the ROI story you take to leadership.

For the team-level number, the same-person method scales up: run a **holdout group.** A matched set of people doing the same work without AI turns "we feel faster" into "we are measurably X% faster than the control," which is the only version of the claim that survives scrutiny. Prove it over a **6–12 month window**, not 90 days — measure too early and you capture all the J-curve cost and none of the later payoff.

> **Holdout group** — a comparison team doing the same work *without* AI, so you can prove the difference came from the tool and not from luck, easier tickets, or the season.
>
> **J-curve** — the dip-then-climb shape of adoption: output drops ~15% while people learn the tool, then rises past where it started. Measure during the dip and you undersell a tool that's about to pay off.

None of the rollup means anything without a "before," so the first thing a PM produces isn't the dashboard — it's the **pre-deployment baseline.** This isn't about anyone's salary or contracted hours. It's about how long work actually takes, measured the way the team already plans.

The most practical way to capture it costs one meeting. In sprint planning, the team goes through every ticket together, breaks down the requirement, agrees an estimate for each (story points or ideal days), and assigns it to a person. That shared estimate — recorded *before* AI is in the picture — is your baseline. Alongside it, log a few things per ticket as the sprint runs:

- **Estimate vs. actual** — the agreed estimate, and how long the ticket really took to reach Done. This is the core signal.
- **Complexity bucket** — tag each ticket small / medium / large, so later you compare like with like instead of averaging a typo fix against a new feature.
- **Quality** — defect rate and PR revert rate per sprint.
- **Capacity** — how many tickets (or concurrent projects) the team clears in a sprint.

Do that for a sprint or two before rollout and you have an honest "before" grounded in the team's own estimates — no timesheets, no salary math.

With the baseline recorded, the ROI math stops being guesswork. Three formulas are worth putting in front of leadership:

- **Net Time Gain (per dev)** — `hours saved per week − AI cost in hours`. You want this positive and growing.
- **Efficiency ROI** — `((time saved per task × tasks automated × blended rate) − TCO) ÷ TCO × 100`. This is the money view for leadership only; "blended rate" is a single business-level number, not per-person salary tracking.
- **Agent Value Multiple (AVM)** — `(cost savings + new revenue + margin gain) ÷ TCO`. Top performers hit about 3.7x, meaning $3.70 back for every $1 spent.

> **TCO (total cost of ownership)** — the full cost of the tool, not just the license. It has to include integration and the trust tax (the ongoing cost of checking the AI's work), or the ROI looks better than it is.

The AVM is the one that reframes the conversation: you're not reporting a percentage saved, you're reporting a multiple on money spent — which is the language the board already speaks.

The cadence keeps the dashboard honest over time. Different metrics move at different speeds, so measure them on different clocks:

- **Weekly — use review.** Weekly Active Users against the 74% target.
- **Per sprint — quality check.** Change failure rate; flag a rise of more than 2 points.
- **Bi-weekly — productivity sync.** Feature velocity (features, not PRs).
- **Quarterly — sentiment audit.** The Developer Experience Index — a survey score of how much day-to-day friction developers hit.
- **Quarterly — ROI assessment.** Net Time Gain and the Agent Value Multiple.

Checking use weekly catches an adoption stall fast. Checking ROI quarterly resists the urge to call a program dead during its J-curve dip. Match the cadence to the metric's real speed and the sprint-to-sprint line stops misleading you in either direction.

### Can the PM see per-person token use?

Short answer: partly, and be careful with it. Provider and platform dashboards show token spend, and combined with Agent Cost per Completed Task (total AI spend ÷ tickets actually resolved) you can attribute cost to seats and workflows — that's how you catch the person whose agent loops for an hour on every ticket, or the workflow that quietly eats the budget. Use telemetry (Weekly Active Users) to see who's actually using the approved tools.

But per-person token counts are a **cost check, not a scoreboard.** The moment "tokens burned" becomes a ranking, people hold back to look cheap and you lose the gains you paid for. Use the spend data to find runaway agents and coach the expensive workflows. Judge people on outcomes and their own baseline. Never rank a team by token count — you'll push people toward the wrong thing and create the exact fear that makes them stop using the tool.

### How to collect the numbers

You need three streams, and none of them needs a new vendor.

**Use — who's using it.** Weekly Active Users comes straight from the tool's admin console. Copilot, Claude, and Cursor all show seat activity out of the box. Target more than 74% of the team each week; under 50% means an adoption problem, not a tooling one.

**Tokens — what it costs.** If the team is on accounts you control, the provider dashboard shows token spend per seat. Honest caveat: on client work the account is usually the customer's, and you may not get per-person numbers at all. When you can't, fall back to Agent Cost per Completed Task at the team level, or ask the client's admin for a monthly export. Don't hold up the whole measurement waiting for token data you're never going to get.

**Code metrics and velocity — what came out.** This one is just git: cycle time, PR throughput, revert rate, and the feature-vs-toil split all live in the history. As the tech-lead section below shows, an agent can pull and label all of it for you.

### A prompt to build the dashboard

You don't need a BI tool to start. Point an agent at the three streams and have it draft the dashboard. A template you can paste and adapt:

```text
You are building a sprint dashboard for an AI-assisted dev team.

Inputs (CSV):
- usage.csv     : Weekly Active Users per dev, per week
- cost.csv      : token spend per seat, per week (may be team-level only)
- git-stats.csv : per PR — author, feature/toil/chore, cycle time, reverted?
- tickets.csv   : per ticket — estimate, actual time, complexity (S/M/L)
- baseline.csv  : pre-AI estimate-vs-actual and revert rate, by complexity

Produce a one-page view with:
1. Use       : WAU trend vs. the 74% target
2. Value     : feature velocity this sprint vs. the last three
3. Impact    : estimate vs. actual by complexity bucket (% delta), vs baseline
4. Stability : change failure + revert rate; flag if up more than 2 points
5. Drift     : any workflow with steps/task up >15% or retries >5%
End with plain language: what got better, what got worse, and the ONE
thing to look at this sprint.
```

Build it with the tech lead. They own whether the git labelling can be trusted; you own what the story means for the client.

### Make the refresh one command

Running that prompt by hand every sprint gets old fast, and anything that's a chore quietly stops happening. So make the refresh push-button. Three ways, cheapest first:

- **A saved script.** Put the pull-and-classify step in a script the PM runs on demand — say `npm run roi-report`. It grabs the git stats, feeds them to the model with the dashboard prompt, and writes `reports/roi-<sprint>.md`. One command, same shape every time.
- **A schedule.** Wrap that script in a scheduled job — a cron entry, a CI job on the sprint boundary, or a scheduled agent — so the report lands in your inbox every second Friday without anyone lifting a finger.
- **A skill.** Best of all, package it as a Claude Code skill or a Copilot command — `/roi-dashboard` — that already knows where the inputs live and how to format the output. Now the PM doesn't store the prompt anywhere; they type the command and read the result. It's the same move the tech lead made with the team's shared skills, pointed at measurement instead of code.

The rule of thumb: whoever owns the dashboard should be able to refresh it in one step. If it takes ten, it'll be three sprints stale by the time anyone looks.

### Reading the change: up or down, and by how much

Here's the part everyone actually wants — the percentage. The rule behind it is one line: **compare the team to its own estimates and its own past, never to other people.**

The core comparison is estimate versus actual. For a ticket the team estimated, once it's done with AI and the shared workflow, the change is: `delta = (actual − estimate) ÷ estimate × 100`. Negative means faster than planned, positive means slower.

Worked example: the team sizes a medium ticket at 3 days and used to land it in about 3. With AI plus the shared workflow, it now closes in 2. That's `(2 − 3) ÷ 3 = −33%` — roughly **a third faster on the same complexity.** Do that across the sprint, grouped by bucket, and you get a real read: small tickets maybe flat, medium ones 30% faster, large ones still slow because review is the bottleneck.

Three things keep that percentage honest:

- **Group by complexity.** Only compare small-to-small and large-to-large. A sprint full of easy tickets will look like a productivity win when it's just easier work.
- **Use the median, not the average.** One monster ticket drags the average around and makes a bad sprint look good, or a good one look bad. The median ignores the outlier.
- **Watch estimate drift.** If the team quietly pads estimates once AI arrives, every ticket looks "faster than planned" while nothing changed. Anchor against the pre-AI estimates and actuals, not fresh ones.

Then ask *why* the number moved, because the direction alone doesn't tell you much:

- **Improved** — is it really the tool and the workflow, or just a batch of greenfield work this sprint?
- **On-track** — steady is fine; it often means the coding got faster but review or QA absorbed the slack (the assembly-line effect).
- **Down** — expected during the J-curve dip, or a real regression? Check whether quality moved with it.

> **Greenfield vs. brownfield** — greenfield is fresh code with no baggage; brownfield is existing, messy, legacy code. AI helps a lot more on greenfield, so never compare across the two.

And always read the speed number next to the quality number. Tickets closing a third faster looks great — but if change failure rate is up 3 points the same sprint, you didn't get faster, you got sloppier. **A productivity gain is only real when quality holds flat or improves.**

### The setup work, by role

Everything above resolves into a small set of reusable patterns and a clear division of labor. Here's the pattern language, then who owns what.

![Four differently shaped sea stacks rising together from bright turquoise water under a clear sky.](https://a.storyblok.com/f/288495199849781/1024x576/08f7089369/ai-agent-roi-playbook-four-stacks-roles.webp)

### The patterns worth naming

**Measurement patterns.** *Same-person baseline* — compare a person to their own pre-AI history; the gold standard, because it kills experience and talent bias. *Feature-vs-toil classification* — use AI to sort diffs into feature work versus toil versus chore, so activity metrics stop lying. *Trajectory review* — inspect the agent's full decision path, not just its answer, to catch shaky reasoning that'll break under load. *Holdout A/B* — a matched manual control group during pilots, so the gain is provably real.

**Adoption patterns.** *Power-user demos* — let the top ~8% sell it through live sessions. *Boring-work-first* — deploy against the tedious work no one wants. *J-curve warning* — name the 15% dip before it happens.

**Anti-patterns to avoid.** *Vanity metrics* — lines of code, acceptance rate, raw PR volume; they track activity, and agents game them. *Phantom productivity* — counting saved hours you never redeployed. *Automating a broken process* — layering AI on a bad workflow and multiplying the mess. *Measuring too early* — calling ROI inside 90 days, before any compounding shows up.

### What the tech lead sets up first

Before anyone writes a line of code with AI, the tech lead (or solution architect on a client project) builds the ground the whole team stands on. Skip this and every developer invents their own context — and you're back to fifteen dialects. Do it in this order.

**1. Build one shared knowledge base.** Pull the scattered stuff into a single place the AI can read: architecture and conventions from Confluence, the real patterns from the codebase itself, the gotchas people keep tripping on. Give it **one exact path** everyone uses — say `docs/kb/` at the repo root — so the instruction file can point at it and it works on every machine. Better still, if the team lives in Claude or Copilot, publish the knowledge base and common workflows as **skills** so people install them instead of copying files.

**2. Check the client's rules before writing your own.** On client work you often inherit a coding standard or an existing instruction file. Read it first. If it's there, ask the client one question: *are we allowed to add a local layer?* If yes, create a `claude.local.md` (or the Copilot equivalent) that **combines their rules with yours**, and keep it out of their repo if they'd prefer. Never quietly override a customer standard — you layer on top of it.

**3. Write the harness — the exact steps the AI must follow.** This is the heart of it. A good instruction file doesn't say "write good code." It spells out the pipeline, with a human checkpoint at every risky step:

1. Brainstorm and explore — no code yet.
2. Q&A with a human, then a human review.
3. Write a brainstorm file; the human reviews it.
4. Turn it into a plan file; the human reviews that too.
5. The human clears the context to free the window.
6. Implement the plan in a fresh session.
7. Code review against the checklist, then Playwright tests.
8. The human reviews and approves.
9. Commit, push, and write a summary.

> **Harness** — the fixed set of steps and guardrails you wrap around an AI so it works the same safe way every time, instead of improvising.
>
> **Playwright** — a browser-automation tool that clicks through your app like a user, to confirm a change actually works end to end.

**4. Put the plan-file rules in writing.** The plan the AI writes before it codes must name, up front: the **unit tests** it will add, the **Playwright tests** it will run, and the **review bar** — linter clean, no obvious XSS or injection on the frontend, and a look at load and performance. If it's not in the plan, it won't happen.

**5. Capture what breaks.** When the AI or the developer hits a wall, write it down in a `lessons-learned/` folder — one short file per problem, so the next person (and the next agent) skips it. At the end of a ticket, have the AI write a short report into a `summary/` folder. Give it a few months and those two folders become the most useful context you own.

This is the compressed version; the full build order — knowledge base, instruction file, workflow, skills, and capture loop, in both Claude Code and Copilot — is its own post: [Give your AI coding agent a manual](/posts/ai-coding-agent-setup-guide)

Here's a stripped-down instruction file that puts it together. Drop it at the repo root as `CLAUDE.md` (or `.github/copilot-instructions.md`):

```markdown
### How this project works with AI

#### Always read first
- Read `docs/kb/` before planning — that's our knowledge base
  (architecture, conventions, known gotchas).
- If `claude.local.md` exists, read it too. It merges the client's
  standard with ours.

#### The workflow — do NOT skip steps
1. Brainstorm and explore. Ask me questions. Do not write code yet.
2. I answer. Wait for my review.
3. Write a brainstorm file to `plans/<ticket>/brainstorm.md`. I review it.
4. Turn it into `plans/<ticket>/plan.md`. The plan MUST list:
   - the unit tests you'll add
   - the Playwright tests you'll run
   - a review checklist: linter clean, XSS/input checks on
     the frontend, a note on load/performance
5. I review the plan. Then I clear the context to free the window.
6. In a fresh session, implement ONLY the plan file.
7. Run code review against the checklist. Run the Playwright tests.
8. I review and approve.
9. Commit, push, and write a summary to `summary/<ticket>.md`.

#### When something breaks
Save it to `lessons-learned/<short-name>.md` so we don't hit it twice.
```

### A first-day checklist for the tech lead

Say a new client project just landed. Here's the first day, in order:

1. **Make the knowledge base.** Point an agent at the Confluence space and the repo: *"Read this space and this codebase. Draft `docs/kb/architecture.md`, `docs/kb/conventions.md`, and `docs/kb/gotchas.md`."* Then review and trim what it writes — you own the truth, not the AI.
2. **Set the path.** Tell the team the knowledge base lives at `docs/kb/`, same path for everyone. Or push it as a Copilot/Claude skill so nobody copies files.
3. **Check the client's rules.** Find their standard. Ask if you can add `claude.local.md`. Merge the two.
4. **Drop in the instruction file** above.
5. **Dry-run one real ticket yourself,** end to end. Fix the friction you hit, then show the team in Friday's demo. Now it's a pattern, not a memo.

### Working with the PM on git analytics

The tech lead and PM turn raw git history into the numbers on the dashboard. AI makes this cheap now — you point an agent at the log and have it label and count. The tech lead owns *"is this number real and safe to trust,"* the PM owns *"what it means for the sprint and the client."*

The tech lead's half is one prompt:

```text
Read the git log for <sprint dates> on this repo.
For each merged PR:
- classify it: feature / toil / chore
- note the author, and whether the commit trailer says AI-assisted
- give cycle time (first commit → merge) and whether it was later reverted
Then summarize per author against their 30-day pre-AI baseline:
feature count, median cycle time, revert rate. Flag anything that got worse.
```

That output drops straight into the dashboard inputs from the last section. One rule: the tech lead sanity-checks the labelling before it ships — an agent that miscounts toil as features makes every later number wrong.

### What the PM owns

Own the scorecard and the story. Record the **30–60 day human baseline before deployment** — without the "before," every "after" is a feeling. Run the **bi-weekly scorecard** and the **holdout group.** Guard the **redeployment**: decide, before the agent goes live, which bucket the saved hours flow into — revenue (more features), capability (new frameworks), or debt (refactoring, documentation for the AI). And frame the whole thing upward as a money request, not a tech project: payback typically runs 4–18 months, crossing from red to black around month six.

### What a dev or tester owns

Own the personal practice and the honest signal. Run the **randomized self-test** to find your own edge. Use AI **boring-work-first** and contribute the prompts that work back to the registry. Answer the post-merge ping honestly — it takes ten seconds:

- Did you use AI on this PR? (yes / no)
- Flow or friction while doing it? (1 = friction, 5 = flow)
- More or less maintainable because of AI? (easier ↔ harder)
- Confident it won't cause an incident? (1–5)
- Roughly how many hours did you get back today? (a number)
- Where did that time go? (feature / refactor / admin / meeting)

> **Experience sampling** — a quick ping right after real work, instead of a long survey weeks later. You catch honest signal while it's fresh.

For the tester specifically: you stand where the inventory piles up. Watch **change failure rate** and **PR revert rate** closely — you're the early warning for the "50% more defects" failure mode, and your signal is what keeps velocity from becoming a liability. The defects that slip through aren't always loud, either; sometimes they're a quietly deprecated library or tokens stored in the wrong place, which is the whole argument of [Your AI Just Introduced a CVE — And Nobody Noticed](/posts/your-ai-just-introduced-a-cve-and-nobody-noticed)

### A worked example: automating a migration

One concrete drill ties it together, because it turns boring work into something the whole team keeps. Say you're facing a tedious API-version migration across many files. Once a sprint, on a refactor or migration:

1. Migrate one file entirely by hand.
2. Give the AI the before-and-after of that file.
3. Ask it: *"Write a prompt that reproduces this exact change for any file with this structure."*
4. Test that prompt on two more files.
5. Add it to the team skills library.

The developer learns to turn boring, repetitive work into a reusable skill. The tech lead gets a registry entry an agent can run on its own next time. The PM gets feature velocity freed up because the toil got automated once instead of ground through by hand fifty times. Same drill, three roles, one compounding asset. That's what "a pattern" actually means in practice.

### One thing to tell leadership

When the numbers go up the chain, protect one point above all: **give it time before you judge it.** Payback on this kind of adoption typically runs 4–18 months, usually crossing from red to black around month six, precisely because the first months are the J-curve dip. Report the trend over a 6–12 month window, not a one-sprint snapshot, or you'll get a program killed in month three that was on track to pay back well by month twelve. The estimate-vs-actual trend, read by complexity and next to the quality line, is the honest story — everything else is a slide.

### What separates the 20% from the 80%

Firms that industrialize AI this way — patterns, baselines, redeployment, re-engineered gates — see a **70% higher return** than those running scattered pilots. The top performers hit an Agent Value Multiple around **3.7x**: $3.70 of value per dollar spent. The 80% stuck in the paradox aren't using worse models. They're measuring activity, skipping the baseline, and letting saved hours evaporate.

And keep one humbling number in view: only about **7% of companies run fully autonomous agents in production.** Bain's frame is "Humans-Above-the-Loop" — most real work still routes to a person, so any ROI model that assumes full automation is pricing a fantasy. What separates a defensible case from a vanity one is small and stubborn: outcome rate over completion rate, redeployed hours over phantom ones, same-person baselines over cross-team rankings, and a recorded "before" you can point to.

---

The through-line under every source is that AI ROI can't be measured in isolation. You measure it after you re-engineer the workflow around it — the agent is the cheap part; the pattern, the baseline, and the redeployment are the work. So before you argue about which model to buy, go record your team's real baseline this sprint. Right now, most of you are measuring the feeling.

---

*This was Part 2 of the "Measuring AI agent ROI" series. Part 1 — [why the obvious numbers lie, and how your team's model decides who keeps the gain](/posts/how-to-measure-ai-agent-roi) — is the foundation this playbook stands on.*