The 5 Fatal Mistakes of Enterprise AI Deployment

Host Intro (Gene Kim)

All right. Our final speaker of the conference is Joe Beutler, Head of Solutions Engineering, Strategics at OpenAI, focusing on financial services and private equity. Joe is one of my favorite people to work with.

Earlier this year, we worked together on something I'm just super proud of — our soon-to-be-completed forum paper on the AI-native company, which was meant to be written for CEOs and board members. And so obviously, CEOs will only read guidance written by other CEOs. And so we had just a killer team: four CEOs, a rear admiral who worked a lot of time in the special forces community that we heard about this morning.

And as I said in our readout, Joe has seen more transformations than a good chunk of us combined. So in April, at the conference, he shared this amazing stat from PricewaterhouseCoopers, who had benchmarked OpenAI's finance department. And their surprising finding was that it was only 20% of the size it should be, given entity count, revenue, complexity, and so forth. And their hypothesis was that the CFO was able to have an engineering manager inside of their organization working for her. In fact, for that matter, the Chief Revenue Officer also has an engineering manager working for them.

And so the natural trio that Jitesh talked about — the top decile business matter expert, the developer, and the AI expert — that trio that Jitesh was talking about, that came from Joe. So I think it just really gives us a glimpse of the future to come. And I think the fact that people like Jitesh Ghai, CEO of Hyland Software, is running with his advice shows the value of Joe's experience and his intuitions.

So I can't think of a better person to close the final two days about things that can go wrong in AI deployment, so you can avoid them. Here's Joe Beutler.

Joe Beutler

Thank you, Gene, and thank you all for staying to the bitter end and being here for this. So I want to try to have some fun.

As Gene mentioned, I spend most of my time working with executives, and my team is focused on helping them figure out how to execute on their AI transformation, how to put together their strategy, evaluating our products, evaluating different use cases that they could build. And along the way, I sometimes get very frustrated when I see people making the same mistakes over and over again.

I think one of the interesting things that we touched on a little bit earlier with Topo and Jeff was this idea of focusing everything around business value. And so some of them are reading from the same playbook that I must have been as I put this together. But effectively, I think we've reached a point now where a lot of your business stakeholders and the CEO may start to realize that a lot of the IT practices or the technology teams are actually slowing them down on the business side. And so that's where these five fatal mistakes come from. And I want to be able to share this candidly and directly with you all.

So you're going to get a little bit of Joe after dark. I think it works pretty well that I'm going last. But this is effectively the conversation that I would have with your CEO or your business stakeholders at dinner after a couple drinks, where they're asking for candid feedback, and I provide it.

And these mistakes are based on five conversations that I've had in just the last month.

---

01Mistake #1: Copying Someone Else's AI Strategy

So the first mistake is copying someone else's AI strategy. And this has been one of the hardest things for me to articulate.

The story behind this is I was meeting with the CIO and the executive team for a leading insurer — the kind of company you would know, the kind of company that has the funny commercials during a football game. And they came in and said, "Hey, we really admire BNY. We know that they built their own AI platform, Eliza, so obviously we should build that too, right?"

And this is one of those frustrating things where everyone wants to build their own platform. Engineers want to build — I'm an engineer, I want to build too. And so you have this build-first mindset, but that isn't always the best strategy for the business. But I was having a really hard time explaining to people why they shouldn't build their own platform. And being on the vendor side, when I say, "No, don't build your own platform, use ours," then I just sound like a sales guy. But really, the goal is to figure out how can you spend your time building what differentiates your business and drives value for you and your customers. Being able to communicate that in a way that doesn't feel self-serving has been really challenging.

So when I was sitting on a plane on Saturday going through my third redo of the slides for this presentation, I realized that there are just fundamental differences between BNY and an insurance company. You may not know BNY, or may not know them well, because they are a financial infrastructure company. They're basically a bank for banks, or an infrastructure provider for banks. They make their money through custody and settlement, handling payments and reconciliation around different financial transactions, and they spend their time building this banking infrastructure.

Then the insurance company is focused on underwriting and pricing — they're about claims and servicing, customer acquisition, and retention. And so those are the core drivers for their business. And so when I started doing this business breakdown, it made it more clear why it makes sense for somebody like BNY to go build all this infrastructure and then build their agents on top of it, where other companies should be using more market-driven solutions.

And so if you focus on pointing AI at where your business creates value for your customers, then it makes sense for somebody like BNY — whose job is to be ahead of the market, to anticipate the needs of what their customers are going to have, and build those products and services for them — and that is largely around infrastructure and operations. So it made a lot of sense for them to go ahead and build their own infrastructure and build their own agent-building platform.

Where that insurer is concerned, the advice I should have given them in the moment was that they need to focus on improving underwriting decisions, on making better risk decisions and having better risk models, and then also serving their customers better and focusing on customer acquisition.

So the first mistake that people make in this area is around initiatives not being tied to their business strategy, and the second part of that is the timing of those decisions.

BNY built Eliza, which was effectively a ChatGPT clone plus an API gateway that they built so that all their engineers and employees could start using and accessing the models. But they built that and launched it in 2024 — so they were probably building that in 2023. And then they launched Eliza 2.0 around January 2025. I met with their CIO in January 2025, and she was telling me about this agent-building platform that they had built that was allowing employees to start to build what you would now call skills and automations, or different agents, on top of that. I was really impressed, because we had been building one internally at OpenAI and we had access to it for about six months, but we were having a hard time figuring out how to get it to enterprise grade for us to launch it at the scale that we operate.

And so they were actually able to move faster in building these types of platforms internally for their own employees than we were able to move building a market solution for everyone else. And so it makes sense if you are a company like that — an infrastructure company that has that market insight early on and then picks that strategy — to go and build those tools because they don't exist yet on the market. But if it's 2026 and you haven't built that platform and you're not ahead of the market, then you're going to be better served by moving forward with market-provided solutions and focusing your time and investment on the areas that are going to differentiate your business.

So another part of the mistake that people make in this area is there are all of these different layers of the AI stack that have emerged. And because AI is the most horizontal technology that any of us have ever seen, you can literally build anything and everything, and people feel like they can win or that they can participate at every layer of this stack, and they want to own as much of it as possible.

I had a conversation with a large bank that is talking about building their own data centers. But what they don't realize is that these data centers are different than data centers of ten years ago, and in order to build a data center that's going to power AI compute, you're going to need to be spending a lot of time figuring out how you're going to get the power, and also going to town halls to do community relations to convince a community that they should allow you to build a data center in their backyard. So that is a fundamentally different business, and if you are a bank, you should probably be focused more at the top of the stack on building better products and services and doing AI-native services for your customers, rather than trying to own this entire stack when entire businesses are going to emerge focused on each of these different levels.

So before you build another platform, here are the four questions I would ask you to ask. One is: why own this layer? Is there a strategic advantage for you to build at that layer? The second is timing — will the edge still exist at launch? By the time you actually build the platform, will that still be a competitive advantage? And then upkeep — who will need to maintain this? Who is going to own that, and why would you make that investment? And then the last one, and most important, is reassessment — when should you consider replacing it?

I've seen companies like even BNY, as market solutions have emerged, now realize that coding agents provided by some of the labs or different providers are really powerful. Maybe we shouldn't be building our own coding agent anymore, and we should transition to a market solution. So while they started out with a really broad platform that provided tools to all of their employees, they're starting to narrow in on the things that are core to their business operations and starting to rely on market solutions when they see them improving faster than they're able to improve their tools internally.

---

02Mistake #2: Promising Value Without a Plan to Realize and Measure It

So the second mistake is promising value without a plan to realize and measure it. And I'm going to move through this one quickly because Jeff and Topo did a great job on this.

This came up in a fireside chat last week with the CIO of a large bank, and he asked, "Why do AI initiatives fail?" And the answer I had was similar to what we just heard — most people don't spend any time figuring out what success looks like before they set out on building that agent or building that initiative. And so if you aren't grounding it in understanding what does good look like, how do we measure success for this — you're going to end up with an outcome that looks like a failure, because it didn't drive the success you wanted, even though the technology or the solution actually did its job.

And so when people come out and they say something like, "We're going to go save $50 million, we're going to build an automated loan underwriting or insurance claims review" — they have this big goal of where they can drive a bunch of value in the business using AI, but they don't stop and think first: what work changes? Is this from a faster and better workflow? What are the actual actions that the business is going to take to redeploy capacity or avoid an expense? And then how is that value appearing — as more output revenue or lower spending?

So again, before you fund this new project or go after building one of these agents with one of these big multimillion-dollar numbers, you need to understand what is the baseline. What's happening today? How is that job done today? Then what changes? What work will change? Who will run that program? This is one of the biggest gaps we see — people will go out on an initiative and build it out, but then there's nobody who's accountable or responsible for the outcome and the value for the business. So you need to have that program management component, or what I like to call an agent manager, who's actually responsible for making sure that it delivers on the value that it needs to for the business. And then evidence — how will you measure it? And we've gotten a lot of great advice on this today.

---

03Mistake #3: Insisting on Doing Everything Yourself

So the third fatal mistake is insisting on doing everything yourself. This one's the most painful one for me.

I was working with the CIO of a major — but not leading — bank, caveat there, and he said, "We never choose one provider." Okay, I understand. You never choose one provider, we get it. You're not going to give us all of your business, but let's figure out how we can work together. And then he was kind of pushing back on us and saying, "We want you to engage with us more, but we don't want to give you access." Basically, they were refusing to give us access to the teams that were working on these things. And then he even went so far as to say that his engineers are better than my engineers. And I'm like — surely he means better than my solutions engineers.

So I kind of pushed back. I said, "Okay, well, we can engage directly with our engineering and research." And he's like, "No, our engineers are better than your engineers." Okay, I'm just going to leave that there. And so at the same time, he wouldn't allow us to go in and evaluate what they had built or what they were doing versus our solutions and kind of do this kind of bake-off. And then at the end of the call, he said, "Why aren't you guys doing more for us?"

And so this really got me thinking, because this is a really challenging conversation to have when people are so set on building everything themselves and think that they have the best engineers and that they can't even get value from another team of engineers that spends all their time doing this.

And so what I've realized is that this is really just the next technology upgrade cycle. We're really trying to re-engineer things or reinvent things because AI — again, since it's so horizontal — we're not following the same principles and decision-making that we've done with previous technology transformations. And so I started thinking, what can we learn from previous cycles?

I came up with a thought experiment. What would you say if I was an engineer and I came in to interview at your company, and you were a Microsoft shop, and I said, "I really just can't use Outlook and SharePoint. I need Gmail and Google Drive to do my job." What would you say? I did this exercise with one CIO, and she said, "Pound dirt." Just like, "No. Pound dirt. We're not going to work with you." So I think that makes sense.

And the question is, why would you say that if you're in the Microsoft suite? And it's really because Microsoft is an ecosystem. When you have something like Microsoft 365, you have file sharing, you have permissions, you have all of the different collaboration tools that have been built together, and they have this network effect that brings everyone value and creates that ecosystem. And so in this case, you do pick one provider. So don't tell me you don't pick one provider. And I did this with that CIO — and yes, they are a Microsoft shop — but I'm just not making friends with people.

So the second example I came up with was: what if I'm a software engineer and I come in and I know you're a Windows shop, and I say, "I want a MacBook. I prefer a MacBook." What would you all say? Yeah, probably reasonable. This CIO said, "No, we are only on these PCs." I'm like, okay, well, everyone else that I've asked this question has had some amount of employee choice available for something like your primary work device, because that's the tool that you're using to do your job. And so it's easier to justify this when you're thinking about how do we maximize productivity, or even from a recruiting standpoint — we want to provide the tools that engineers want to use if they're going to come work with us. And so the decision changes when you're talking about that specific job.

And then we talk about specialist tools. So say I say, "Hey, can I choose my coding agent?" Maybe I prefer one over the other — trying not to say product names. In that case, in most situations, people right now are evaluating and making Codex and Claude Code, maybe other agents, available to their employees and kind of letting the market do its job, letting competition win. Let us compete for your business and build the best tool possible.

And so that kind of shows that there's this need for employee choice, especially when it comes to specialist tools. And I started to put together a little bit of this framework, and I realized that people too often are treating this as one decision — one build-versus-buy decision for all of their AI tooling — when really it's a whole bunch of different decisions that need to be made at different layers of the stack or in different parts of the company.

And so it makes sense for an employee workspace that's going to be powering all of your work that you would choose a primary collaboration system like Microsoft 365 or Google Workspace. That's what we're now trying to build with ChatGPT. Anthropic is doing the same with Claude. If you have people building agents and sharing them with coworkers and automating workflows and building that in that ecosystem, there's going to start to be that network effect, and it's going to become more important that more people are in that one ecosystem.

Then the second would be specialist tools — you want to be able to approve these based on the results that you see and the preference of the people using them, where they feel that they can do their job most effectively. And then you have internal automation, where maybe you shouldn't build your own platform, but if you do build your own platform or use a third-party platform, most of the time you're going to want to have one primary platform where people are building their agents so that you're able to govern them and track them and share them internally.

And then the fourth piece is around customer products. This is more about building where you differentiate. And so this is definitely an area where people should be building — because you understand your business best, you understand your customers, and trying to solve their needs and grow the business is going to be a key piece around building AI-native products and services.

I wanted to touch on this quickly — people went into a little bit more depth earlier — but this idea of the agent harness. Right now, harness engineering is very popular. This is one thing that people love building. And I just wanted to touch very quickly on why I think people shouldn't be building their own harness.

There are two main points when you look at what a harness is doing. The first is this idea of managing the working context. We came out with an innovation last year called Compaction, which is the idea of how you manage the context window. All these models have a context window, and it used to be that if you filled it up, it would just discard the oldest information at the top. And so we invented Compaction, which allows the model over time to basically summarize the information, pull out the most important things, and discard the rest. That compaction is really important because it's not just for the performance of the agent, but also for the efficiency and the token efficiency — because it figures out what information it can get rid of so that you're not passing back those tokens every time you go through a multi-step process.

The second piece is memory. We brought memory into ChatGPT a couple of years ago, so ChatGPT started to feel like it knows you. And so the way that we manage memory and the way that we manage compaction have become really core engineering challenges, and there's a lot of complexity in there. And so if you're trying to build your own harness, your teams may not be spending as much time on things like compaction and memory — and you get those out of the box when you use our harness.

And I know you don't want vendor lock-in. So what I'm asking is that if you're going to use our models, use them with our harness — and you should do the same for other providers. You should use their models with their harness. And then move the routing layer up a level, so you have your routing layer that allows people to use different tools or different models, but it's actually built on top of the harnesses. The reason for that is we are training our models on our harness so the models know how to use the harness, and then we're optimizing the harness for the models. And so you can build your own harness, but you're going to lose out on all the engineering that we're putting into things like compaction and memory and optimizing the harness so that it knows how to use those different tools.

So that's my one pitch for you here — just understand the value that you're getting with these harnesses. And please don't build a Franken-harness that tries to be a layer across every single model and optimize for all the different models, because I think that over the long term, that's going to be a losing battle.

So what makes a good partnership? We want to be good partners. When we work with different customers, we don't want to be treated like a vendor — we want to be treated like a partner, and we want to treat you like a partner instead of just a customer. And so this is a key part of the conversation: please, bring us in when you're working with companies like us or our competitors, and treat us like a partner. Help us bring in our product and deployment expertise, help you solve the problems that you're working on, and follow through on this.

---

04Mistake #4: 3X Product — Why Hasn't Revenue Tripled?

So I'm going to move through this one quickly. We're only at number four. Four is: we 3X'd product — why hasn't revenue tripled?

I worked with a private equity firm that built a team of internal FEs that they deployed across their 60 portfolio companies. And they helped them optimize their SDLC to accelerate their product development. And they said they have confidently 3X'd product throughput — so not just three times as much code, but building three times as much product. But they didn't see a comparable lift in revenue or growth.

And so in that conversation, I was like, "Well, have you guys tried to 3X sales and marketing?" And they're like, "Oh, that's a good idea." And so we talked about this earlier with bottlenecks — I like to use the word constraints. One of the things I spend my time on is helping businesses understand where do you have constraints in your business where you can point AI today, and then once you resolve that constraint, what is the next bottleneck or next constraint where you should be focused on?

I think too often people start either on the left side with product development — if we're technology and engineering leaders, there's obviously a ton of benefit there — or they go after operations and support, and they make that mistake of freeing up a ton of capacity and then saying, "Great, we automated 75% of this process." And it's like, well, what do you do with those people?

I think the best way to think about this would be to go back to Justin's demand factory — totally going to steal that, it was a great point. If you focus on generating more demand, then you're going to need more product, more engineers, more customer support reps, more ops people, more people on onboarding. And so if you create the demand, you'll run into bottlenecks in the next part of the business, and you can start to resolve those bottlenecks with AI and actually see growth in the business.

And the way that you do this for sales and marketing — I actually touched on this a bit in San Jose in April — is this idea of delegation. We've finally seen people starting to understand this on the software engineering side: how do you package up your work and delegate it to an agent so it goes off and does it for you? Justin mentioned like no one looks at code — that's the type of mentality that we need to bring to the rest of the organization. And so sales and marketing orgs aren't operating that way today. We're mostly still working in an AI-assisted mode, where you're asking the AI to kind of help you with your job. But we need to figure out how do we teach people to package up work, delegate it to the AI, review the output, coach it, and tell it when it's doing things incorrectly, and then help it go through that cycle until you have an output that you're happy with.

And I think one of the challenges is that delegation is really an executive skill. Most people don't know how to do this by default — they've been responsible for writing every line of code, writing every line of email to the customer. And so you have to start thinking about how do you teach people how to delegate, the way that you do for managers or executives?

So how does the rest of the company start to get there? You need to provide more time for them to practice, and move to an AI-first mindset. That's one of the biggest things I would take away — everybody should be thinking, how do I ask AI to do this job for me first, and then I review the output? Maybe it's not ready yet, and you still need to do that thing yourself. But the more you can move toward thinking AI-first, the more people will start to get in the habit of delegating work and seeing that impact.

---

05Mistake #5: Making Every Team Wait for Your Platform

So the last fatal mistake, which is a quick one, is making every team wait for your platform.

For the companies that do have platforms — and everyone has some form of platform — this is a similar conversation I had with a different bank where they said, "We have a governed platform for all AI use cases." And I think that is right for a lot of reasons. You need to have governance — you need to understand, especially in regulated industries. But too often, we will release products or a new feature capability, and your teams are not able to adopt them because they're waiting for the platform to support it. And so that's now becoming a constraint or bottleneck in the business, and slowing people down from reaching the business outcome that they're trying to build on top of your platform.

And so I think we need to move away from a focus on building the entire platform that runs the entire company, because AI is really becoming a new operating system for how businesses operate — and there's just a lot there if you try to do it all yourself. Where companies should focus more is how do you build the connectors or the MCPs into all of your internal systems and data sources, so you can plug those into whatever third-party tooling you're using in different parts of the business. And not enough time is being spent here because people feel like they need to build the platform first, instead of trying to use a third-party platform and then owning all those connectors into the business context to make sure that that's as valuable as possible.

So the insight that came out of a lunch with this executive team is they realized that they needed to give their strategic team an approved way to move faster. And one of the executives said, "We basically need to be able to create internal startups that can run a little bit faster and aren't reliant on this platform." And so the approach there is you basically need one platform where all of your employees can build agents — and that's very governed and controlled — but then if you're going to build a net new product or service for your customers, maybe they can operate more like a startup and be able to move faster than the product team with that specific kind of goal.

---

06Summary

So here are the five decisions to revisit. We'll share these slides out. But: don't copy a peer. Don't try to go after ROI without having a plan for how that will actually be measured and realized. Don't go it alone — there's so much work to do, you're going to need different partners. You'll need to work with us, you'll need to work with our competitors. We understand that, but be a good partner in that process. And don't just stop at engineering — think about the other parts of the business that you need to accelerate with AI. And don't just have one mandatory platform.

And so hopefully this — I didn't offend everybody here — but it is really important that we start to invest differently today. And the reason why — sorry, getting one minute — is you need to invest while it can still win you market share.

If you look back at the last couple of years: the first window, 2023 to 2025, was about building the capability, understanding how models work, automation, and starting to build these assistants. Right now, we're in a window — I think now through 2027 — where you can really start to differentiate your business by building AI-native products and services and focus on growing your business. But I think by 2028, this is just going to be table stakes. Every business is going to be AI-native. Every business is going to be AI-first. And so you will effectively be forced to make the investments by that time.

And I know you all, being here, are probably ahead of the curve and hopefully in that business growth focus. But this is why we need to move past just focusing on efficiency and move toward focusing on how do we have AI drive growth for our business before it becomes table stakes.

So for what I'm looking for — I'm a huge nerd, a huge technology nerd, but also a huge business nerd. So I'm really interested in the different operating models that you all are looking at, and different problems or mistakes that you're seeing in your organizations and how you're solving those with your different strategies. I'd love to hear from you on those different things. I added my links on here, but thank you so much for your time, and thanks, Gene, for having me.

Host Outro (Gene Kim)

Thank you, Joe.

Thank you so much for dishing out the tough love. All right. By the way, I like this new Joe that's dishing out the tough love — this Joe that doesn't care whether he has friends or not.

By the way, if you're in financial services and you're not talking to Joe, you're missing out on something amazing.