Is AI Living Up to Its Promise? The Measurement Side (Part 2)
Hey there! This is part two about the journey of getting actual value from Generative AI investments. In part one, we dug into the human side of AI implementation. Now let's tackle the million-dollar question: How do we measure if this stuff is working?
Remember that statistic I shared in Part 1? Only about 20% of firms believe they are seeing a return on their GenAI investments. That means 4 out 5 companies are not seeing value. Yikes!
Want to join the 20% club? It's not only about investing in people (though that's crucial, as we saw in part one), you also need concrete ways to measure value.
Quantifying Value is Hard (But Not Impossible!)
"Value" has become one of those corporate buzzwords that's almost lost its meaning.
I like this down-to-earth definition from an interview with the author of Lean Marketing. Real value is one or all of these:
It saves people time (aka money)
It reduces risk
It reduces effort required
It reduces negative effects or consequences
What makes this definition strong? Because you can actually measure these things!
With AI, the promise is to:
Improve productivity (save time and money)
Reduce the "drudgery of work" (reduce effort)
Automate processes (reduce effort, save time)
But here's the million-dollar question: How do you prove it's working? You've got to put some numbers behind these claims.
Boring but true.
GenAI Implementation Framework
To get real about measuring AI's impact, follow these steps:
Get a baseline: Break down repetitive processes into steps and measure the current costs (time, money, effort)
Map AI capabilities: Determine where AI can reduce friction, speed up service, or produce better outcomes
Set clear metrics: Define specific, measurable outcomes you expect from implementation
Pilot and iterate: Start small, learn, and expand based on results
Measure and compare: Rigorously compare post-implementation metrics with your baseline
Share and scale: Document successes and replicate the approach in other areas
I get it; it seems like a lot of work. It is, but start small and grow from there. It will become easier.
Let's use a typical sales process for a B2B firm as an example. The steps might be:
Get an inbound lead
Get that meeting on the calendar
Frantically research before the call
Run the meeting where the prospect asks questions and the sales lead delivers a pitch
Summarize what the prospect needs
Decide if they're a good fit or just tire-kickers
Craft a proposal
Negotiate with the prospect
(Hopefully) close the deal
Hand it off to your delivery team without dropping the ball
The key to defining the ROI is to identify the repetitive, high-cost activities and focus AI on those.
For instance, if your team is spending a full week deciding on fit and crafting proposals (steps 6 and 7), that's your baseline starting point. What if AI could help slash that to just one day?
Imagine: AI takes meeting notes in real-time, analyzes them against your past wins, recommends fit with detailed reasoning, and then—after a human gives the thumbs up—drafts a killer proposal. What used to take five days now takes one. That's not just incremental improvement—it's transformation.
Success Stories: The 20% Getting It Right
So what's the difference between companies just playing with AI toys and those actually seeing returns?
They're obsessive about baselines, measurements, and constant tweaking.
Check out these examples (they're fictional but based on patterns I've seen work):
Example 1: Customer Service Implementation
This is a fictional composite example based on common implementation patterns
A mid-sized financial services company implemented GenAI for customer support and saw a 40% reduction in response time and a 25% increase in customer satisfaction scores. Their approach:
Started with a clear problem: slow response times for routine customer inquiries
Established baseline metrics before implementation
Invested heavily in training customer service representatives on working with AI
Redesigned job roles to focus support reps on complex issues while AI handled routine matters
ROI Timeframe: Initial results in 3 months, full ROI achieved in 9 months
Example 2: Repetitive Documents Implementation
This is a fictional composite example based on common implementation patterns
An energy company applied GenAI to creating repetitive documents such as letters and status reports. Results included a 15% reduction in time spent managing these documents and 30% fewer mistakes. Key factors:
Focused on a high-value, data-rich process with clear KPIs
Created a hybrid implementation team of AI experts and experienced managers
Implemented a gradual rollout with continuous feedback loops
ROI Timeframe: Began seeing measurable benefits in 2 months, full ROI at 12 months
Addressing Common Objections
If you're facing resistance to AI investments, you're not alone. Here are responses to common objections:
"It's just a fad"
AI has been developing for decades, and while there's certainly hype, the underlying technology and business applications are real. The organizations that learn to harness it now will have significant competitive advantages.
"Our industry is different/special/unique"
Every industry has knowledge work, documentation, customer interactions, and data analysis. While applications may differ, the fundamental value drivers apply across sectors.
"We've seen this before with [past technology]"
The difference with GenAI is its ability to understand context and generate human-like content and analysis. This isn't just automation; it's augmentation of human cognitive work.
"It will take too long to see returns"
While full ROI might take 6-12 months, many organizations see quick wins within weeks when they focus on the right use cases. Don't boil the ocean—start with something concrete.
Setting Realistic Timeframe Expectations
Understanding when to expect returns is crucial for setting appropriate expectations with stakeholders:
1-3 months: Quick wins in individual productivity and simple process improvements
3-6 months: Team-level efficiency gains and initial measurable business outcomes
6-12 months: Full ROI realization for well-executed implementation projects
12+ months: Strategic competitive advantages and new business opportunities enabled by AI
Organizations that expect overnight transformation will inevitably be disappointed. Setting a realistic timeline with concrete milestones helps maintain momentum and executive support.
Wrapping Up
When we combine the people-focused approach from Part 1 with the measurement frameworks outlined here, we create the conditions for successful GenAI implementation. The key is balancing the tech with organizational change while being relentlessly focused on measuring what matters.
By investing in your people, tracking the right metrics, and learning from what works, you can absolutely join that elite 20% getting real returns.
The most successful organizations recognize that GenAI isn't just a new tool—it's a complete rethink of how work gets done. Get your measurement right, and you'll not only prove the value to stakeholders but build momentum for even bigger transformation.
So what are you waiting for? Time to measure what matters and move beyond the hype!