The Dashboard Is Green. Now What?

AI products can look healthy long before you know whether they actually helped anyone.

One of the strangest periods in product work begins right after you ship.

The thing is live. Usage starts coming in. Errors stabilize. Latency looks fine. People click. People return. The dashboard turns green.

And then somebody asks the obvious question:

Is it working?

Sometimes the only defensible answer is: I don't know yet.

I've lived this problem.

On seller decision-support work at a very large company (you've certainly heard of it), we could ship recommendations faster than we could learn whether those recommendations actually improved a seller's decision. Feedback instrumentation kept competing with launch pressure, and the launch usually won.

So we fell back heavily on interviews. Those interviews were useful, but they were slow, frequently cancelled, retrospective, and difficult to get from some of the users we most wanted to learn from.

We could see that the product was being used.

We could not always see whether the recommendation helped someone make a better decision.

That distinction has been bothering me again lately because AI makes the gap much more consequential.

The uncomfortable gap after launch

Product teams have always used proxies.

Usage, clicks, conversion, adoption, repeat visits, completion rates, latency, error rates. None of these were ever perfect representations of value. Good product teams have always known that.

But for a lot of traditional software, the interaction itself was relatively close to the intended value. If I built a checkout flow and more people successfully completed checkout, I learned something fairly direct. If I reduced the time required to complete a task, I could often observe that inside the product.

AI changes the shape of the problem.

The product may generate a recommendation, summarize a situation, rank a set of options, write something, predict an outcome, or increasingly take an action itself.

The important result may happen much later and somewhere else entirely.

• A seller clicks the recommended account. Did the outreach produce a useful conversation?
• A manager accepts an AI-generated recommendation. Was it the right call?
• A clinician reads a generated summary. Did it improve the next decision?
• An agent completes a task. Did it accomplish the intended objective, or did it just complete the workflow successfully?

The telemetry can be completely accurate and still answer the wrong question.

A click tells you that someone clicked. Adoption tells you that people adopted. An approval tells you that someone approved.

Those are useful facts. But they are not the same thing as evidence that the AI helped.

What is actually new in the age of AI

I don't think the lesson is that traditional product metrics suddenly became useless. They didn't.

The change is that the distance between product activity and real-world value can get much larger.

Two AI interactions can produce the exact same event in your analytics system while being completely different in quality. Two users can both click "accept." One may be accepting a genuinely strong recommendation. The other may be accepting because the answer looks plausible, the deadline is tight, or disagreeing takes more effort than going along with the machine.

And as systems become more agentic, the problem gets sharper. The AI is no longer just producing something for a person to consider. It may be initiating work, routing cases, changing records, contacting people, or triggering the next system.

Action gets cheap.

That means organizations can get dramatically better at doing things before they get any better at learning whether those things were good.

I keep coming back to that distinction: action velocity versus learning velocity.

If the first accelerates and the second does not, you have not necessarily built a better system. You may have built a faster way to operationalize a bad assumption.

AI didn't invent this problem. It just made it harder to ignore.

Some of my instincts here are much older than AI.

As an FDIC examiner, I learned quickly that the official representation of a process was not the same thing as the operating reality. A policy described what management intended to happen. Transactions, exceptions, approvals, records, and control activity showed what actually happened.

Later, in Army after-action reviews, the useful question was never just whether the outcome was good or bad. The point was to reconstruct what people knew, what they believed, what they decided, what happened next, and what should change because of it.

A dashboard is another representation.

It can be accurate and still be incomplete.

The question is what reality has a path back into it.

The missing return path

This week, a few conversations on LinkedIn helped sharpen the idea for me.

Guillaume Belisle made the point that outcome measurement belongs in the product architecture, not as an analytics layer you bolt on later.

Suzanne Chartier pushed it one step further: intent matters. You need to understand what the user was actually trying to accomplish, what happened along the journey, and whether the intended result occurred.

That is the piece I think many AI products are missing.

There needs to be some return path from what happened in the real world back into the product.

AI output → user or system action → downstream outcome → correction / override / exception / evidence → evaluation → future behavior

Most product dashboards are very good at the left side of that chain.

The harder work sits on the right.

• Did the recommendation lead to a useful action?
• Did the action produce the intended outcome?
• Where did the human disagree?
• What exceptions appeared?
• What evidence would tell us that the underlying assumption was wrong?
• And most importantly: if we learn something, where does that learning go?

A correction that never changes the next recommendation is not much of a learning loop.

What I would instrument now

I don't think the answer is to throw away the old dashboard and invent seventeen new AI metrics. (Please don't.)

I'd keep the normal product-health measures. I'd just stop pretending they answer the whole question.

Then I'd force five things into the design:

1. What outcome are we actually trying to improve?

Define the user or business result before choosing the metric. Otherwise the easiest observable behavior tends to become the objective by accident.

2. What did the AI influence?

Capture enough to connect the AI output to the recommendation, decision, or action that followed. "The model returned a response" is often too far upstream.

3. What happened afterward?

The answer will not always be immediate or perfectly measurable. But some downstream signal, even an imperfect one, needs to exist if the product is going to learn whether it helped.

4. Where did reality disagree with the system?

Overrides, corrections, exceptions, escalations, abstentions, and user disagreement are not merely friction to eliminate. They are evidence.

5. What changes when the evidence comes back?

Does it change evaluation? Ranking? Routing? Product design? A policy? A threshold? The human review process? Nothing?

If nothing changes, the system may be collecting feedback without actually learning from it.

This is why I've started thinking of instrumentation as part of the product itself.

An MVP that can act but cannot reliably teach you whether the action helped is partly a guess.

Green is not the same as good

There is a seductive comfort in a healthy dashboard.

Green latency. Green uptime. Growing usage. Good adoption.

Those things matter. I would be nervous if they were red.

But the more AI moves from retrieving information toward influencing decisions and taking action, the less comfortable I am treating them as evidence of success.

The important question is increasingly downstream:

Did the system help accomplish what we intended it to help accomplish?

And if the answer is no, or maybe, can that evidence find its way back into the system quickly enough to matter?

AI is making execution faster almost everywhere I look.

The organizations that benefit most may not be the ones that act fastest.

They may be the ones whose learning can keep up.

A green dashboard tells me the system is running.

I'm increasingly interested in whether it tells me the system is getting better.