Measuring Soft Outcomes
We've previously touched on the importance of objective evaluations when looking at an AI model's outputs. It's just as important to be objective about the project outcome itself. Otherwise, we risk going purely by gut feel.
Going all the way to the initative's inception, what was the needle we wanted to move?
Maybe it's an eminently quantifiable goal. Task X used to take 2 hours. Now it only takes 10 minutes.
Or it's an objective quality goal: Manual review was missing, on average, 5% of issues. Now we're only missing 1%.
However, goals can be softer: "Enhancing employee satisfaction" is excellent. Those can be harder to measure, but it's not impossible. For even the softest goals, you have a picture in your mind of what success would look like, or at least a sense of what's bothering you about the status quo. If it weren't the case, you wouldn't have a problem: If an outcome can't be measured, you might as well declare it achieved.
Sticking with the "employee satisfaction" example, let's assume you've noticed low employee satisfaction. So what? Well, maybe it leads to high turnover. And that's certainly something we can measure. Or it leads to lots of employees coming to their manager with complaints. That, too, can be measured. Whatever it is about employee satisfaction that's bothering you would have to manifest itself in some observable way. And if it can be observed, it can be measured.
So if you've determined that an annoying but necessary task leads to low employee satisfaction to the point that you want to do something about it, and you suspect that automating that task should help, you can then put the correct measures and objectives into place: The overall objective becomes, say, "reduce employee turnover by x%" or "x% fewer complaints to managers" (but be careful with the latter one... an easy way to achieve that metric is for managers to punish those who complain)
In any case, identifying the real goal of any project or initiative and tying it to a measurable outcome immensely clarifies what success looks like to anyone involved. It also moves the conversation to a more helpful place: If I know the ultimate goal, I can confidently make many subordinate decisions and trade-offs. How accurate does the model have to be? How fast? How much should we invest in a snappy, polished user interface?
Conversely, if there is no real goal other than a top-down mandate to "do something with AI", it's easy to see how none of the stakeholders would ever be able to align. Such an initiative cannot succeed. It'd be like playing golf on a course with no holes.
