Essay 06

How to tell in 30 days whether it worked

Someone asks whether the AI trial is working, and you realize you are not sure how you would even tell.

Most AI pilots at midsize companies do not fail. They fade. Nobody says it did not work. The seats renew, a few people use it, and nobody can tell you what changed.

The fix is not a dashboard. It is deciding, before anything starts, what worked means and when you will look.

Pick one measure

Choose a single measure tied to the process: time from request to quote, items one person handles in a day, how often a document has to be redone, how long a weekly report takes to build, how many requests wait more than a day for an answer. One measure. If you track six, you will find one that went up and call it a win.

Good measures have three things in common. They already exist or are easy to count. The owner of the process cares about them. They would move if the step actually changed.

Take a baseline first

Before the tool touches anything, count the measure for two normal weeks. Not a guess from memory. An actual count, even if it is a tally on a sheet of paper. Without a baseline, every result is a feeling.

Watch for a normal week that is not normal: a holiday, an unusual order, someone out sick. Note it next to the number. You will want it later.

Write down what would count

Before you start, write one sentence: if after 30 days the measure is clearly better than the baseline and the old step is gone or shrinking, we keep it; if not, we stop or change course. Decide with the owner what clearly better means for this particular process. This heads off the argument afterward about whether it was really a success.

What to look at during the month

Once a week, briefly:

• Is the new step being used for real work, or only for tests?
• Is the old step shrinking, or running alongside it?
• What broke this week, and was it the tool or the process around it?
• What did the people doing the work complain about?

Complaints are useful. "It takes longer to check its output than to do it myself" is a real finding. So is "it works, but I have to copy everything into the other system." Both tell you where the next fix is.

Quality, not just speed

Speed is easy to measure and easy to fool yourself with. Pair it with a quality check. Have the owner review a handful of outputs each week against what a person would have produced. If speed goes up and rework goes up with it, you moved the work. You did not remove it.

Making the call

On the date, sit down with the owner and the numbers. There are three honest outcomes.

Keep it. The measure moved, the old step is gone or nearly gone, and the people doing the work would object if you took the tool away. Write down what changed and move to the next process.

Fix and rerun. The measure moved a little, but something specific got in the way: a missing connection between systems, unclear instructions, a step nobody owned. Fix that one thing and give it a few more weeks. Do this once, not indefinitely.

Stop. Nothing clear changed, or the team works around it. Cancel the seats. Write down why. That page is worth more than the license, because it keeps the same idea from coming back next quarter under a different name.

What not to count

Logins. Number of prompts. Seats assigned. Surveys about how people feel about AI. Those tell you whether a product is being opened, not whether the work changed.

On Monday

Pick the measure with the owner of the process. Start the two week count. Put the decision date on the calendar now, before anyone has a stake in the answer.