Who Gets to Fly First Class? Value per Token in the AI Factory

Host Intro (Gene Kim)

All right. So it was brought to my attention by Marguerite, my boss and wife, that I seem hyper-caffeinated. My only excuse is that nothing's going wrong, and apparently that made me very happy, and apparently very hyper. So I will keep reminding myself to breathe and talk slower.

All right. So all day you've heard about how fast AI is spreading through our organizations. And we all know, I think, for long-term, medium-term planning purposes, coding is trending towards instantaneous and free. But realistically, AI tokens are anything but free. Which is pretty funny to me because 20 years ago, vendor management was one of those most boring of jobs. But anyone who's faced a surprise token bill — maybe involving hundreds of thousands of dollars or maybe more — will attest that vendor management is anything but boring.

And so our next speaker is going to give us some very sobering news. So I met Jeremy Eder in New York last fall, and I learn something every time I talk with him. He's a distinguished engineer at Red Hat, headquartered just up the road here in Raleigh. In 2019, IBM acquired it for $34 billion — at that point the largest software acquisition in history.

So for the past 17 years, Jeremy has been an absolute believer in the economic inevitability of software development being disrupted by AI. But in this presentation, he will make the case that as currently practiced by developers, AI is absolutely not tenable. He's going to present some amazing stats, experiments he's done, and give us some advice on how we can make AI more viable at an enterprise scale. Here's Jeremy.

Jeremy Eder

Hello, friends. How's everyone's afternoon getting going? Thanks for having me. Awesome. Sobering? I don't know about that. Let's just talk about it. I'm an engineer, so you're just going to hear real talk. Let's get started.

So value per token — I'll define that in a minute — and who gets to fly first class. I assume some of you snobs flew first class to get here. I drove.

All right. Okay, quick overview. Hopefully you know who Red Hat is, but if you don't, we've got some greatest hits I wanted you to know about. Things like the Linux kernel or Kubernetes. More recently, PyTorch — foundation for just about everything in this room — and vLLM, the leading open-source inferencing engine.

We've got about 5,000 engineers, even more depending on how you count these folks, that are working with AI right now, and several hundred in the group that I'm in who use an AI factory approach to deliver product as our day-to-day workflow. I'm pretty proud of getting to that by this point. It has not been easy. We hit every tree branch on the way down, like I think most of the speakers before me did.

And one thing I'm particularly proud of is, in the true nature of Red Hat, we're being as open as we possibly can with all this information, including what I'm about to talk with you today. Seventeen years from sweeping floors to where I'm at now — I'm pretty proud of that, too.

Other things I've done in the past — maybe because this is the first time I'm talking to this audience — things I've done: I was in charge of wiring GPUs into Kubernetes in the first place, going back almost 10 years now. And then I was on the PyTorch governing board to make sure that when they launched the PyTorch Foundation out of Meta, it went in a constructive direction for open source AI. And now I'm studying economics, which is interesting because it's actually been a hobby of mine in the past, and now is just weirdly part of my day-to-day job.

01Token Maxing and the Budget Constraint

All right. Provocative slide number one: we can't do it the way you all are doing it right now.

Token maxing is on its way out. Hopefully that's not a surprise to some, unless you've got unlimited budgets and crazy margins — maybe you work for Nvidia, I don't know. But the proof is with your peers, and you've seen case studies from Uber in particular, who made a lot of news in early Q2 of this year by saying, "We're done with our AI budget. We have to figure something else out." That's a very real constraint. If you've read The Goal — the Theory of Constraints book — you're going to hear a lot about constraints in this talk.

Microsoft, in addition, even though they serve half of OpenAI's models, they also have their own models, and they found pockets of people spending tens of thousands of dollars and running Claude everywhere on side projects and all this kind of — well, let's just say they exploited the gray areas there, some people.

In Red Hat's world, we sort of experienced the same thing. I'm going to share a slide with you in a moment of some of the actual technical data about how this theory of constraints is actually affecting us and what we're doing to deal with that. And I think that's hopefully the subject of this whole talk.

So we've got a couple of interesting things that have played out in the last couple of months. The first one is the introduction of token budgets. Someone mentioned a thousand flowers blooming. I thought that was an IBM saying, but I guess it's more general. But yeah, we were in that mode too — thousands of flowers blooming. Learned a lot of stuff. Also a lot of stuff to avoid. And the talk that was just preceding this break talked about systematizing all of those learnings. I'm a systems thinker, hopefully like many of you, and so this makes perfect sense.

Here's a slide. The X-axis is time, and the Y-axis is the number of PRs that landed in a repo that I care about that were AI-generated. Why is this going down? What do you guys think happened at the red arrow point?

Budgets? Bingo.

The good news is that by running wide open, we now know what the ceiling is for what is possible — what our engineers can do, or are willing to do, with their current environment before they have to learn more. For example, there are complex crown jewel-type systems that we're just not going to touch. Not for a while. And there's lots of stuff that can be completely commoditized. And actually, there were two talks earlier this morning — one from the folks at Morgan saying, "We agree with a completely industrialized software development process in the future." I am of that mindset, as is Red Hat, and I think the larger IBM is trending towards that as well.

And then the second one — Steve Yegge — was mentioning that it's interesting if you measure how models fail and how much it costs. And in fact, that is one of the studies that I've done here, and I'll show you some data.

So we introduced the token budgets around the red arrow, right? And there's this delta. One thing before that — someone on Slack, and I promise you I did not see this beforehand — they asked: "What other constraints are you putting in place?" Well, a budget constraint is a self-imposed limit. Essentially, we're not out of money, but we want to put a limit on this thing to see what's possible. I love the idea of putting constraints into the system because they drive hygiene. Most of that is like SLOs drive hygiene, KPIs and OKRs — they hopefully drive hygiene in the system and help develop innovation that helps us close the gap. And so this is where this discipline comes in.

This delta now — which is 80 down to like 40 — we are going to close with hard engineering effort. And some of that stuff is underway that I can share with you today.

02Coding Buddy and Agentic Memory

So a couple of angles. The first, or one of the primary asks from our group at large, is: I want a coding buddy. I want this thing to embody the organizational constraints, to know everything that I should know about how the org operates — like what the strategy is, what the current priorities are. And I want this thing to have private knowledge as well. So think about these agentic memory systems that can have tiered memory: organizational memory, team memory, and then personal memory — that kind of thing.

Agentic security — I want to talk about that. Has anybody heard about OpenShift Enterprise? Not enough hands — Google that. And then the second piece is the Open Agent Safety Platform. Two key announcements from not just Red Hat, but Nvidia, IBM, OpenAI, working together to deliver a persistent agent hosting platform that's open source, coming out of the OpenShift Foundation.

Third piece — and I think this is the meat of the talk — where I'm trying to say: you hopefully have some sense of what the most valuable work is. Not a lot of people know that. Right now, token maxing has led to some pretty weird behaviors where people are shipping a lot of stuff, but it can't really be tied back to value as directly as you might want, as directly as it has been in the past when things were at human speed. You had those gates that are just gone now or not happening. And then I think this is a key piece where I'd really love to talk to somebody — or many people — after: what are the new and emerging techniques and roles that we need to establish in order to make this industrial software development process and factory actually work?

03Defining Value per Token

So how do I define value per token? We have this job that needs to get done — this is potentially a new feature that needs to get developed or an RFP that we need to respond to. How many tokens, or what is the cost of producing that? The total cost, whether that's engineering employees, engineering headcount, and then of course the resources they need to do their job — whether that's office space or tokens in this scenario. And then we get value per token.

I will argue right now that token is the completely wrong unit to be thinking about value. It just happens to be the way that we're currently getting billed. And the prices of tokens are constantly changing, just the way that people get different prices at AWS or different prices at Azure depending on their business constraints and how much negotiation power they've got with those cloud providers. Tokens are a fluctuating unit, and it's very difficult to do cost estimation based on a fluctuating unit cost.

I'd like to call your attention to an article in The Wall Street Journal on October 4th, which looked across almost 400 businesses and calculated that less than 10% of them feel like they can accurately forecast AI spend. That's a pretty damning number, and it has played out in fact for me as well.

So determining value is kind of a moving target at the moment. But I think the idea is that you know what the most important work is — and not a lot of people know that. We'll go through all those topics.

Here's a screenshot of an enterprise agent mobile app — this is the UX that folks have been trending towards. We want to establish a platform that has all of the common criteria, controls, and data security. We have all of the same certifications that we need to maintain, at the same time trying to take advantage of AI. We don't have those substrates now, and that's where this OpenShift Enterprise work and open shell work that we're doing in the community is hopefully trending towards.

We tried retrofitting existing systems. You can imagine how that went — not great. In a lot of cases, the vendors that we're already working with are themselves adding the features that we need. So it's kind of almost like a timing impedance mismatch — if we had waited three months, it might've been better. Hopefully you'll have a better experience with vendor management than I have in that case. And arguably maybe we're trying to move a little faster than others.

In the enterprise assistant scenario, we would love to have a case where it is completely sovereign, where we're running hosted models locally with all of our local data, allowing you to port across providers and essentially do cost arbitrage against them eventually — which we are actually doing now.

04Model Routing and Agentic Memory Savings

Another way to save money — and actually this is an emerging technique — is where you take these powerful models that are very expensive to run (that's kind of what I meant about token maxing: we're just using Opus or the top-tier models for everything), meanwhile these are tasks that can be easily achieved by a mid-tier model.

In this case — and I don't know who's worked with agentic memory systems before — some of them use flat files, almost all of them end up using Postgres on the back end and front it with an MCP. That's what ours does. In this case, we take these benchmarks — there's an enterprise assistant benchmark that's in the HELM framework. If anyone's worked in the HELM framework community, this is a frontier LLM benchmarking community run by the Stanford CRFM, which is the same folks behind Alpaca. Those benchmarks for enterprise assistants are just now emerging. These are customer service workloads, managerial decisions, HR-type scenarios — you're attempting to build agents that can solve very common enterprise workloads that are not agentic development.

In these cases, some of the benchmarks that I've personally run allow us to take cheaper models, couple that with memory that's essentially been extracted by running with higher-end models, and achieve significant savings. So remember that 40% number I showed you before, trying to close that gap? Here's one technique by which we can close it: leveraging agentic memory — not necessarily RAG, but maybe RAG.

In this case: 27% cheaper, 16% faster, passing the benchmark at the same level.

In this case, the benchmark is like an eval. Earlier, folks were talking about spec-driven development. I'd argue there's almost a riff on that called eval-driven development, which is essentially taking advantage of the fact that we have semantic capabilities in an LLM versus static TDD, which was cast-fail in a lot of cases. So there's money to be saved here — plenty of it — but you have to be careful about this memory as well. I'll get to that in a minute.

05Security and Supply Chain

Let's talk about security. I was mentioning earlier OpenShift Enterprise — IBM and others, I think Chainguard's another one — have announced products that attempt to solve and ship fixes as quickly as they are found. This is the Mythic moment that we all saw a couple of months back. How do we address that? IBM is stepping into that space, and this is actually the first use case for our internal AI factory: Lightwell. They are generating — I think we just said yesterday — 400 previously undisclosed Java vulnerabilities patched by Lightwell. All that stuff goes through these pipelines. Pretty cool.

So how do we exist at this release speed? All of our policies that we had are now just broken. All of this "I'm going to patch it within 24 hours" — that's a joke now. We need to move certainly a lot faster. We also need to maintain supply chain security. That's kind of where Red Hat's enterprise approach is hopefully meeting all of your compliance checkmarks at the same time, giving them to you in a timeframe that is actually useful — not days or months. And hopefully your own internal processes evolve at the same time so that you can roll these fixes as quickly as we can give them to you.

06Agentic Authentication

Has anyone tackled agentic authentication successfully? Kind of a pregnant pause there for a moment. I think that's a space where hyperscalers are out in front, but we're working to develop standards on authentication and authorization as well. If you've ever gotten a ping on your phone saying "Claude needs permissions for this or that" — that's what I'm talking about, where it's delegating to a human. They've got these agents working in the background, and they're delegating to a human.

There's a paper that just came out in the Enterprise Technology Leadership Journal called "When Agents Decide." I was one of the co-authors of that, and I'm not sure if any of the other authors are in the room, but I was privileged to be able to write the developer tooling section of that document. The focus was on how we keep agents moving. How do we provide them with governable ground truth context across the entire enterprise concern and do it at machine speed?

There were many talks today talking about context engineering. I'm talking about doing that at machine speed, at scale, and fronting it essentially with MCP or some other kind of client-server paradigm, where agents can then consult these external systems, make decisions autonomously on your ground truth, and keep moving. So instead of having developers tied to keyboards getting that next hit of dopamine, these agents will hopefully run for longer and longer periods of time autonomously in our AI factory. That's one of the metrics we're looking at — I'm calling them agentic interrupts. These are essentially gaps in context.

07Value Per Token and Work Prioritization

Back to the value per token idea. I don't know if anyone's read the Outcomes book, but it's fantastic, and I wanted to connect that with my idea of value per token. You have business and organizational strategy — I mentioned this earlier about team strategies — but what about wiring in the operational context, like: "We've got a bug in the wild that 30 customers have hit in the last 24 hours," or "We've got a $100 million renewal on the table and we need X, Y, and Z features"? These are ways to determine value and stack rank a work queue in a factory. I think this is an evolution from where folks are today to where we are: working this backlog, grooming this backlog, and then putting it into these autonomous systems for execution in a much more disciplined way.

In fact, towards the end of this slide, there's a link to a GitHub repo which has the complete proof-of-concept implementation for something I'm calling first class tokens. You'll be able to generate your own synthetic data, put it into this system, use a decision model like JEVS to make decisions on the prioritization of those things based on the business context that you have fed into it.

All right. So I mentioned CRM and issue tracker. There's also urgency and blast radius of a change. For example, a security change in an encryption algorithm might have a massive amount of oversight related to it — aging out old algorithms, et cetera, tend to have those kinds of oversight. So that's what runs first.

The second decision that these systems have to make is: what runs it? I mentioned we're doing cost arbitrage across providers. That requires a centralized AI gateway that knows what my costs are per provider and can also map that to the capabilities of each individual model or provider. So far this has been a lot of manual learning on our own part — experimenting with what works, what doesn't, where models regress, where they're strong, and developing that kind of map.

As a short term, this is helping with the fat middle of the prompts that come through — not the high-end cases, but the middle.

What I expect to happen — and what our most forward-looking teams are doing — is, if your teams are really good at writing evals already, I expect that to be the closed loop for these programmable AI gateway routing tiers. If anyone's got an AI gateway in production, they have systems like this that you have to build around. Most AI gateways will have these features if they don't already.

The idea, though, about what runs it: I mentioned earlier about agentic memory. Why are we buying the same tokens twice? If you accrue memory and you find a way to curate that ground truth, you can totally avoid provider calls. I think that's the ideal outcome — you're essentially building this proprietary learned knowledge cache in this memory that now needs its own lifecycle, but at the same time helps you shield GPUs or avoid SaaS-based API calls to models.

Something that I found is that in prioritization meetings for different features, everyone thinks their problem is the most important thing. Boiling that down to more of the business value — where individual concerns might be more localized — having that more global vision is something a system like I've put together will actually do. It will stack rank things like a reporting redesign versus a customer renewal type of scenario and prioritize them in the most business-impactful way.

In this case, we put a tighter budget in place, and that means we need to find more money. So an effort around cloud cost reduction that used to be buried in a backlog somewhere might come to the forefront because that frees up more budget for other activities, including AI.

08Emerging Roles

Now, a couple more slides left. Let's talk about which roles are emerging.

I mentioned the ground truth curator. I was talking to someone over breakfast this morning and they called it a librarian. I think ground truth enterprise knowledge base curation is something many people have in place already. But now we're having agents consult those sources of truth, so they can make very expensive errors very quickly — things that are potentially even difficult to unwind. So that's one role.

The second role is hill-climbing harnesses, or self-improving agents. If anyone's looked at LangChain's LangSmith product that they released in June, that's a good first step in the direction of self-improving agent loops and having the scaffolding for engineers to actually become the factory workers that work on the machines rather than type the code themselves.

And then the last one — at least the last recent one — I'm just calling a trajectory miner. If you have teams that are application performance management (APM) type specialists or observability specialists in the SRE space, this is what I'm talking about. If you've ever looked at a corpus of agentic trajectories — which are the record of what the agent did along the way — you'll find an incredible amount of value and learnings you can extract from there. So that's another person who used to be doing maybe APM and now needs to look at a different type of trace: an agent trajectory, to try and squeeze some value or performance out of those. I'm calling it exceptional trace mining.

I looked across almost 7,000 of our own agentic AI factory traces. I found almost 30% of them had a very specific class of issue that was introduced by our own harness. Very easy, basic Jupyter Notebook machine learning analysis across these traces very quickly identified clusters of similar issues.

I bet in this room there are many people who have those things buried, and there's plenty of cost savings there.

Something from earlier — I'm proud that Red Hat is being as public as we can about some of the stuff we've learned. There are at least five blogs going from product security to my own manager, my own boss, about how we do it in our team, as well as the whole business operations aspect, which I think is just a completely untapped area. But lots of value potentially there.

I sprinkled this in along the way — help that I'm definitely looking for. If you have a different way of thinking about value, I would love to learn. I have a certain view of the world and it's certainly incomplete. I'd love for people to connect with me on LinkedIn. If you're already working on evals and you think you've got a good handle on it, I'd love to see those in-work examples. And then the agentic identity delegation scenarios.

The codebase is at the end — first class tokens. Take a look, let me know if you get a chance to play with it. Thank you.