AI-Driven Organizational Value

Host Intro (Gene Kim)

All right. Welcome back to the last segment of programming.

So yesterday, we heard from Randy Shoup and Michelle Gill, who shared the work of one of the forum teams that they did on code reviews for agentic code. And I love that presentation because throughout the last two days, we've heard about a lot of practices, and it was great to hear them talk about the theory of why these practices should work and what are the practices we can derive from first principles.

So the next two speakers are here with another piece of the puzzle. Specifically, when AI helps us do more work, how do we actually turn that into better outcomes for the organization?

So their forum paper was "AI-Driven Organizational Value," and I love the way that they framed the problem: that the AI age is everywhere except for in the profit and loss statement. So we can see people producing more, but how does it actually show up in numbers that CFOs, CEOs, and boards care about?

So many of you know my buddy Jeff Gallimore because for a better part of a decade, he stood on the stage helping me open and close this conference. He's CTIO and co-founder of Excella, and he's helped shepherd our annual forum events, bringing people that I admire most to help generate the guidance that you've seen featured throughout the two days.

And he is joined by another one of his co-authors, Dr. Topo Pal from M&T Bank, who you heard from this morning, and they'll be talking about what they learned about how we measure AI value and specifically what we need from leadership to make value happen. Here's Jeff and Topo.

Jeff Gallimore

All right. Think back over the last year and the settings that you've been in where AI has been discussed, and I will bet you that you've been in a situation where you've heard a leader say something to the effect of, "We used AI on some really important process, and we saved a really big number of hours from that."

Can I get an acknowledgement there? Have we been in that situation? Think back over that. Okay. It was impressive. There was a lot of enthusiasm, maybe some cheering.

Did anybody ask this question: what did your organization do with the hours you saved?

And if that question makes you go, "Hmm," this talk might be for you, this paper might be for you, and we're going to maybe demystify where that question is coming from and why many leaders are having such a hard time answering that question.

So as she mentioned, the title of this is "AI-Driven Organizational Value." It's about explaining why the productivity metrics are going up — the usage, the adoption, the costs — but they're not translating over into the business metrics that matter, like profit and loss and revenue and things like that. We're going to explain that. We're going to show you where maybe it does come from, share some perspectives on that, and then also what you might be able to do about it.

So we're riffing off of something called Solow's paradox. Solow coined this phrase back in the early '80s to explain what was happening with the computers, the rise of computers. He said that the computer age is everywhere, except in the P&L. And I think we're seeing the same phenomenon happen now with AI.

If you've watched over the last year, there's been this flood of reports, studies from research organizations and industry analysts talking about this exact phenomenon, where they're interviewing and surveying industry leaders, corporate executives, and all of these investments are going up. They're starting lots of AI work and pilots and projects, but they're not translating into those metrics that matter for the business, the outcomes that they want.

And there's a leader that we talked to, and I'll share the quote here: "We know that there's something there. I can see the value. I can feel the value. I just don't have a number."

And why that is — or at least one explanation for this, from again all the people that we've talked to, the research that we've done — is that measuring capability is really comfortable. Measuring whether it actually created an outcome is not comfortable at all, and we as humans, and especially as leaders sometimes, default into the stuff that's comfortable.

So this phenomenon isn't new. There was an article that came out almost 20 years ago called "The Seven Deadly Sins of Performance Measurement." Michael Hammer wrote that — from Reengineering the Corporation fame. And somebody that we talked to, Paul Gaffney, was interviewed in that article too, and he's saying AI is giving us new ways to commit the same sins today. And he says, "Stop scorecarding your progress against the roadmap and start scorecarding whether you actually moved the outcome, whether you moved sales, whether you moved profit and loss."

And we asked Paul, "Well, how would we know if we've got a leading indicator for doing that? How do we know if we're on the right track?" And he said, "Are you making more, better, faster, more consequential decisions than you were before?" It's like, okay, interesting take on that.

Which led us to a graphic and a concept from Laura Taoco, who used to be the CTO at DX — now she's at Amazon. And she framed this as: many organizations today are using AI to support automation and individual action and productivity. That's that bottom left quadrant. She said the organizations that are winning with AI and are really generating value are using it differently. They're taking a different view of that. They're actually using AI to make decisions at the team or the organizational level. And so this was the framing for what Paul said, "Are you making more decisions?" The framing of this is, how much are you living in that bottom left quadrant with automation? That would be things like PRs, lines of code, delivery frequency, features pushed — versus the top right quadrant, which is what are the business decisions that we're actually using AI to help us make, or to accelerate, or to make better?

Tapabrata "Topo" Pal

So that value isn't just the cost that it saves. Yes, it does, but it's just not that itself. It's the other options that it creates.

So to frame that, we took a few examples. We interviewed many people, and we can all relate to all these that we have on this slide — which is from eight months to seven weeks — and that's the value, or the options, that it opens up. Okay, now that you have finished the work that was assigned for eight months in seven weeks, what are other things that you can do in that space?

Another example is from IKEA. The last point to make here is the cost to the bets that you can make with that cost saving. So essentially, what we are trying to present here is that it's not just "don't look at the cost savings." Think about, with that cost saving, what else you can do, what it can open up for your team, for your company, for your organization. So that's one piece of idea that we got by interviewing many people.

Jeff Gallimore

And then AI value is a capital allocation choice. So Topo just mentioned that the value of AI is in the options that it creates. And those options come in the form of free capacity. I apply AI, I raise my productivity, I free up capacity. Things that used to take me X amount of time are now taking me less time, or resources, or dollars, or whatever it happens to be.

Now that you've got that free capacity within the system, the choice is what to do with it. And that's where the real value of AI comes in.

And there are three shapes, or three types of decisions, that we saw or heard about during our research and the conversations. I'm going to skip over the first one and come back to it.

One of them is learning. I can take those savings in the form of time and dollars and I can reinvest that in capability building for my organization. That could be skills, it could be my workforce, it could be experimentation, it could be new capabilities from a technology or process perspective.

Another way to reinvest those savings is in the form of growth activities. So being able to open up new lines of business or launch new products that I previously couldn't do, that might create new revenue streams. That's a way to reinvest that.

Now I'm going to come back to the first one. The first one is the one that unfortunately I think we see a lot of in the world today, the world of industry, and that's harvest. And harvest is: if I can save costs, if I can do the work with less money or fewer people, then I'm going to remove those costs from the system and I'm going to harvest them. And so that unfortunately looks like RIFs or layoffs or people getting fired or their roles being eliminated.

Now, we wanted to put that out there. We had a lot of discussions within the team about whether we try to tackle that because it is a sensitive topic, and we decided that it is better to acknowledge the reality of that because that is what organizations are doing today, and they are wrestling with that, and there are legitimate reasons for taking costs out of the system in the context of an individual business.

What we wanted to address, though, was the harvest-by-default choice. That's the problem. Harvesting — taking money out of the system, taking costs out of the system — it's easy, it's straightforward, it's immediate, it shows up on the financial statement immediately, it's very clear, which again makes it a likely default for many organizations.

The problem that we saw with that — if it's not already obvious, in addition to the human effects of that — is that it also has some potential downsides from a business perspective and the outcomes that you're looking for. And one particular case we saw was with Klarna. They took call center agents out of the system — seven hundred agents out of their organization — thinking that the AI was going to be able to give them the service that they wanted. And what they found over time was that it was not the case. The quality outcomes that they were wanting from those interactions with their customers were not up to what they expected. And so they started adding those service agents back into the system. So they recalibrated their expectations.

Tapabrata "Topo" Pal

So here's something very interesting that we found. Everyone is a builder now. You have a thought, you have an idea, you have a product in your mind, you just go and build it. The whole bottleneck — call it — of having engineers who can code it up for you, that's totally gone.

The builders are not only engineers. In fact, we found many examples where actually the product owners are taking the ownership, and instead of asking the engineers to build it for them, they are basically coming back and showing it to the engineers: "Look what I did." It happened to me, it happened to you. I am sure it's happened to each of you that some product owner from some business unit is saying, "I built it yesterday. Can you put it in production tomorrow?" That's one pattern that is happening on a regular basis.

The other one is that we interviewed one company, and they say that their non-technical people actually created 115 in-scope requests that they want to build on without needing any engineering help. That itself shows that non-technical, non-engineering people — the business people — actually now realize that they can do it without the engineers' help.

And essentially, it's the same concept: since you're a builder now, you look for help, you don't get the help, so you just did it yourself. And actually, it's a real procurement business person who has access to a SaaS vendor product, and he hated that product and realized that he doesn't use the whole product by itself — it's a fraction of the things that he does on a day-to-day basis, and the flow is bad. So he basically sat down and coded it up. And that's fascinating. And I think it relates back to some of the stories we heard at these conferences about the SaaS days are over. People are building the stuff that they need, and if they need something different, they'll just throw it away and build it again. So that's one pattern.

Scarcity used to say no for you. Now you have to on purpose.

Essentially, the story here is that a lot of requests used to come to the engineering teams in terms of business-to-agile flow stories and all that. And there was always a discussion about not having enough people in the team, not having enough engineering resources or whatever the case may be. The scarcity of resources basically forced people to say no to some requester. And we see that every day, even now.

But now that scarcity is gone, the actual business people are coming back to you with a solution, and all they're asking is, "You don't have to code it. It's already there. Just run it for me." So there's a question of how much stuff can you run, what are the duplications, is it the same thing, and if I run it, is that the correct business solution that we're looking for? So the whole equation has changed.

In fact, if you think about it, the whole excuse of "I don't have enough people" is now gone. Now you have to stand up and say, "No, I'm not going to do it," if you really want not to do it. So that's another angle to that.

And then the 10X function inside a 1X business is just a faster bottleneck, and we are seeing that. We all know that. There are ten different demos happening in multiple places in the organization, but none of them are showing up in production. And that's about: once you produce 10X, and if you have 1X of pipeline, then of course the 10X teams should not go — or will not go — through that 1X pipeline in a given time frame. Your expectation will not be met there.

Essentially, what we are saying is: enlarge the pipeline by some means so that it can actually take that throughput and have your 10X function go to production, or customer's hands, or whatever the case may be. Now, there are other bottlenecks that may not help you in increasing the pipeline in a technical sense, and that points to all the existing processes. Change approval board is probably one of them — that should have gone many moons ago, and they are still there. We all know that. And there are similar processes that can actually constrict the pipeline, but I think it will be an engineering effort and an emphasis on how to increase the width or the diameter of the pipeline to actually handle 10X functions.

And one of the other bottlenecks in the value stream — and we've heard about it over the last two days — is actually in your customer's ability to adopt the changes or the features of the product or the services that you're putting out there. And that's where we really get value in the most obvious sense, in a business context.

Jeff Gallimore

So for organizations — every organization is on their AI journey somewhere at this point, and getting value from that is on a similar journey. What we saw was this pattern emerging from organizations: that they were going through really three stages.

The first one is this adoption stage, where we're seeing usage go up, seats go up, token spend go up, token usage go up. All those productivity metrics are going up. The problem is with many of those organizations, they're stopping there. They're declaring victory with, "We've got all of our developers on it, we've got this many daily active users," all of this — in the high 90-percentage points. Wicked showed a slide with 100%: all engineers are using AI for development. Organizations stop there. That's okay as a starting point, but we have to put that into the context of the destination of the journey, which is actually value.

So moving through from adoption, we're starting to see organizations now putting token limits on. They're trying to optimize their token spend. They're trying to think about which workloads go with different capabilities, model capabilities and providers, so that they can become more efficient.

And then that ultimately leads to, we hope, realization of that value. We need to instrument that journey appropriately across the entire journey. We can't forget about one stage. Adoption will always be a thing, but we have to be able to put that into the context of the rest of the journey and then instrument that journey as we move through it.

We also learned that there is no single number for the ROI of AI. A lot of leaders are seeking that — "What's the ROI for all my investment in AI?" We found that was probably a fool's errand and an anti-pattern. What's better is to measure the business outcomes that AI is serving, and to do that in a very narrow way — to find the specific use case. If you can, do some baselining so you can get some before and after. And that is going to give you the actual business value in a meaningful way from your investment, or the organization's investment, in AI.

And there are three shapes to that that we found. I can get multiples of the outcome that I had been getting for the same investment. I can take the same outcome and I can accelerate that or compress the timeline. Or I can do things that were previously impossible. And so now we're seeing organizations starting to measure AI in the currency of impacts — whatever that impact metric means to you.

Tapabrata "Topo" Pal

The dip is not necessarily a failure. Fund the dip, reap the sky.

And we have seen this before — cloud migration. We knew that we need to invest in cloud migration to reap the benefit of it X years later. We have seen the same thing in any technology enhancement. The problem here is that in AI spend, it's so real, so fast, that it's really hard to see what's happening and what is going to happen two years down the line. But I think what we heard by talking to people is just that — 18 to 26 months — regular reaping the benefit of our investment, that needs to continue in the AI spend too. So it's not just spending; spend it carefully, but at the same time, don't get scared of spending.

Jeff Gallimore

The hard part about this J-curve that Topo just mentioned is that there's no evidence for the value yet. It requires conviction and leadership and air cover and belief that it's eventually going to come, and that's a real leadership challenge. And if you are a leader, that's the one that your teams and your organizations are either implicitly or explicitly asking you to do.

So with all that, the conclusion that we're left with: we can feel the value, and we at least believe that we've got some perspectives on better where to find that value. We can measure what matters, when it matters. We can work through the J-curve and fund the dip. We can back the belief and have the conviction that this is eventually going to pay off. We know it's going to happen, and we can stay for the proof — because when the evidence arrives, we can celebrate that.

These are the paper's authors. They were an amazing team. They were awesome. I learned so much from them, including Topo. All the people who contributed — we talked to a lot of industry leaders, executives — they contributed all their expertise. It was an amazing experience.

Here's the paper. It's available for download on the IT Revolution site. We'd love for you to read it, share it, if it resonates with you. Feedback is a gift, sharing is caring. If anything that we said resonated with you, fantastic — let us know that. If we missed the mark on something, we'd love to know that too. If you have stories to share with us that resonate with the paper, tell us that too. And we're also considering writing a book out of this because we have a lot more pile of insights that didn't make it in. We'd love to see if there's an interest there. So if you have an interest in maybe reading the book or sharing the book, let us know that. That would be really helpful.

All right. Thanks, everybody.

Thank you, Jeff. Thank you, Topo.