How do you measure whether automation worked?

Automation worked when a step of the work is gone from the process and the number you singled out beforehand has moved. Four measurement points settle the question: the share of cases that ran the whole way without a person, the share of cases that came back, the cost per case, and the time from the customer making contact to the case being closed.

Apps AS has been building its own products since 2008 and now runs an agent that writes this blog. The figures in this article were counted in Apps AS's own systems.

What does it mean that automation worked?

Automation worked when a step that used to need a person no longer needs a person, and the process still produces a result at least as correct as before. Both conditions have to hold at once. A step moved from a paper form to an agent, where someone still reads every single answer before it goes out, has been digitized rather than automated.

The distinction between digitized and automated is worth keeping because the two states look alike in a status meeting and different in a set of accounts. A digitized step moves the work onto a screen. An automated step removes the work. The operations side of the same distinction is in Personalization is an operations task, not a campaign.

Which numbers measure something other than what you think?

An activity number tells you how much the system has produced. An outcome number tells you how much work is gone. Most automation reports open with an activity number, because the activity number is the only one that is easy to pull straight out of the system itself.

The bottom row is the most expensive one to skip. A system that does not give the same answer twice cannot be approved once and counted as approved from then on. How risk moves when software acts on its own is covered in What changes when software acts on its own.

Which four measurement points show whether automation worked?

Four measurement points are enough for most automated processes. All four share the same denominator: the number of cases that entered the process in the period.

1. Share of cases finished without a person. The share of cases that went from start to a finished outcome without a person doing anything. This is the only number that says directly whether a step of the work has been removed.
2. Return rate. The share of finished cases that come back as a complaint, a correction or a fresh enquiry about the same thing, within a window you set yourself. The return rate is the protection against making mistakes faster than before.
3. Cost per case. Model usage, integration, operations and the human review that still exists, divided by the number of cases. The review belongs in the calculation, and why the review does not disappear is covered in Manual review of AI agents is not a cost you can remove.
4. Time to a closed case. The time from the customer making contact to the case being finished, measured across the whole distribution rather than on the average alone. The median and the ninetieth percentile say more than the mean, because the tail is what the customer remembers.

Measure only the first point and the share finished without a person can rise while the return rate rises just as much. The process has then become faster at producing cases it has to do over.

Why does the baseline have to be measured before automation goes live?

The baseline has to be measured before automation goes live because there is no way to reconstruct the baseline afterwards. After launch you have one number and nothing to compare it with, and any movement in the number then gets read as an effect of the automation.

Three things make a baseline hard to recover after the fact. Seasonality: a support service in January and the same service in July are two different services. Case mix: automation usually takes the easiest cases first, so the remaining cases are heavier and the average looks worse even when each individual case goes better than before. Other changes made at the same time: a new form, a new price list or a new integration in the same quarter makes it impossible to attribute the movement to the agent.

Measure the four points for at least four weeks before you start, and write down which other changes were made in the same period.

What do you do when the number does not move?

When the number does not move after a quarter, the reason is usually that the process was not mapped well enough to be automated. Go through the three checks below before you build more.

Three checks before you build further

Who owns the process now? Without a named owner with the authority to change the rules, the process stands still however good the agent is.

How many exceptions are there? A process with 20 exceptions is 20 processes. Count the exceptions before you automate more of them.

Which step has actually been removed? If nobody can point to a step that is gone, the process has been digitized. The measure is not the thing that is wrong.

A project that has not moved the number after two quarters has reached a stopping point and does not need more development. The figures on where agentic AI actually pays off are in How Will Agentic AI Improve My Business?. What a supplier can guarantee when the outcome is what is being sold is in What a consultant delivers when agents do the work.

Count the four numbers for one process this week

Pick one process you have automated this year. Count the four measurement points for that process: share of cases finished without a person, return rate, cost per case, and time to a closed case. Send the four numbers to hello@apps.no. We will reply with which of the four we would make the steering measure, and which one we would stop reporting.