Is Your AI Feature Actually Working? Measure It From the Customer's Side

I'm Tom Lang — I've built and scaled engineering organizations for three decades, through multiple acquisitions, and I work as a fractional and interim CTO for growth-stage companies, with particular depth in regulated and integration-heavy environments.

Search "KPIs for AI applications" and you'll get a four-layer measurement framework: model metrics (accuracy, F1, AUC-ROC), system metrics, user-experience metrics, business metrics — a dashboard with a dozen numbers on it. None of it is wrong. It's just built for a company with a team to run it, and it buries the one question that actually matters.

For a growth-stage company, the question isn't "what's our model's F1 score." It's simpler and harder: is this AI feature actually working for the people paying us?

The only verdict that counts is the customer's

Here's the thing I keep coming back to, across three decades of shipping software: whether a feature or a product "works" is always, in the end, measured from the customer's side — not the model's side. The questions that decide it are customer questions:

• Is this meeting the customer's actual need?
• Is the customer struggling with it?
• What are they telling us about it — and what is their behavior telling us?

Everything else, every technical metric, is instrumentation in service of those three questions. If a number doesn't help you answer them, it's noise on a dashboard.

This isn't abstract for me. I've spent my career as the person accountable for whether what the team shipped actually landed. When we re-architected the platform at Levels Beyond, the number that mattered wasn't an internal engineering metric — it was customer delivery satisfaction, which we lifted more than 250%. Being accountable means being measured on the customer's experience, not on how clever the build was. AI doesn't change that. It just adds a new way to fool yourself with an impressive-looking internal number.

What to actually measure

At growth stage, translate those customer questions into a handful of signals you'll genuinely watch:

• Are they using it? Adoption and repeat usage — but read it as a proxy for "does this meet a need," not as a vanity number. A feature people try once and abandon is failing, however good the demo was.
• Are they struggling? This is the one teams skip. Watch the friction: task-completion rates, drop-off, retries, and how often people bail out to a human or just do it the manual way. Struggle is a louder signal than satisfaction.
• What do they say and do? Direct feedback — a thumbs-up/down on outputs (the same feedback loop you'd build for governance anyway), support tickets, the occasional CSAT — plus what their behavior tells you when they're not being asked.
• Does the AI actually beat the alternative? The honest test almost nobody runs: is the AI version genuinely better for the customer than the non-AI way — or than nothing at all? If it isn't, it's not a feature, it's a liability with a nice icon.
• Is it worth what it costs? Net of inference cost. A feature customers mildly like that costs a fortune per call isn't a win. (I get into the cost side in keeping your cloud bill from eating your runway.)

And the model metrics — accuracy, latency? They matter exactly where they show up in the customer's experience, and nowhere else. A model that scores 95% accurate but feels wrong to users is failing. A slightly less accurate model that users trust and rely on is winning. Serve the customer question; don't worship the metric.

Be willing to turn it off

Which leads to the part most "AI success" writing won't say: be willing to kill it. If the AI feature doesn't meet the need better than the alternative and justify its cost, it isn't a success just because it's AI. The willingness to remove a feature that isn't earning its keep — from the customer's point of view, net of what it costs you — is exactly what keeps you honest. I've never regretted removing something that wasn't working. I've regretted keeping things because they were impressive.

The honest goal

Measuring whether your AI works isn't a four-layer dashboard exercise. It's staying relentlessly close to one simple question — is this actually helping the customer, net of what it costs? — and being willing to act on the answer, including when the answer is "no." Do that and you'll build AI features that earn their place. Skip it and you'll have a beautiful metrics dashboard for a feature nobody needed.

If you're trying to figure out whether an AI feature you've shipped — or are about to — is actually worth it, that's a good thing to pressure-test with someone who's been measured on real customer outcomes for a long time. Book a call and we'll look at what your customers are actually telling you.