home/journal/Measuring the return of an AI project, honestly

Measuring the return of an AI project, honestly

How to measure what an AI automation gives back in a small company: hours counted from the system’s log, not estimated; time to first reply; errors caught; what the team did with the hours. Why we count monthly and what the first month usually shows.

Ben Sylvester Strautmann · 30 September 2026 · 5 min

The AI industry measures itself in projections. A Monaco company can do better: count what actually happened, every month, from the log.

Count, do not estimate

An automation that runs leaves a log: how many emails read, how many drafts produced, how many approved as-is, corrected or rejected, how many invoices booked, how many matched. Multiply each by the minutes it used to take, measured before the project started, and you have hours given back that nobody had to believe. We report that number monthly, with the log behind it.

The four numbers worth tracking

Hours given back, from the log. Time to first reply to a client, before and after, because in Monaco the first reply often decides the deal. Errors caught: duplicates, mismatches, missing documents the system flagged that a person would have missed. And the one number that is not in any log: what the team did with the hours. Ask them; the answer (called clients back the same day, closed the month on time, left at six) is the real return.

What the first month usually shows

A high correction rate in week one, falling fast as the rules are adjusted. Hours given back that are real but below the estimate, because the team still reads every draft; that is right, and it settles. By the third month the number is usually above the estimate, because the team has found uses nobody planned. We do not quote averages from other companies; your note has your estimate and your log has your truth.

When to stop

If the log shows the hours are not there after two months, stop the run and keep the code; it costs nothing to keep. An honest count protects you from the sunk-cost story as much as from the rosy one. In practice we have stopped very few, because the prototype step catches the bad ideas before the build.

Two questions.

Can you show us a real monthly report?

Yes, anonymised, on the quick-check. The work page describes five systems with their first-month figures.

Who measures the “before”?

We do, during the quick-check and the prototype, from the people who do the task. It is written in the note so the comparison is honest.

Related.

Start with a 30-minute call. Keep what it finds.

You tell us where the hours go. We tell you which two or three tasks to automate first, in a short written note. Free, in English or French.

ben@bss.mc · a reply within one working day