Wheelhouse and Counting House: Two ways to put models to work

Host Intro (Gene Kim)

All right. What can I tell you about the next speaker, Steve Yegge, that you don't already know? He famously spent 20 years at Amazon and Google. At Amazon, he helped lead one of the first two-pizza teams at the behest of Jeff Bezos. At Google, he helped create Grok and Code Search, one of their most beloved development tools — beloved by a long shot.

And during the process of writing the Vibe Coding book, I found him to be as hilarious, irreverent, and brilliant as what you would think, but also incredibly generous. And maybe one of the big surprises in the book was that we both rediscovered the joy of coding. We both thought that the best days of coding were behind us, when in fact they are very much in front of us.

It's so clear that for years, Steve has been living a year or more in the future, and he's been thinking deeply about where this all goes and what it means for every one of our organizations. Please welcome Steve Yegge.

---

Steve Yegge

Hi, everybody. Can you hear me? Can you hear me now? All right. Hey. Good to see everybody. Welcome back. Thanks to Gene for having us all. Gene, you're the best.

All right, so I've got a glorious two-hour talk that I've got 20 minutes to give you, so it'll go fast.

[Slide issues]

All right, so while I get my slides worked out, I'm just going to give you the background here. I've been an independent for about a year. I have been living about 12 to 18 months in the future. I do it by multi-agent research. I spend about 10 to $12,000 a month of my own money on my agent factory. My wife lets me — she's very generous.

And as a result, I have a video game that has players from 20 years ago that are really excited. They're getting a lot of new features, and that's my lab, my test bed for my agent factory, which then funds my independent gig work. I go to big companies who are struggling with this, and I help them.

And so today I'm going to be talking to you about two different kinds of systems — two different ways to bring agents into a company that are very different. They don't actually have much in common at all.

One of them, the first one, which I call Wheelhouse, is SDLC. It's creating software, it's building software. And the other one that we're going to talk about — ten minutes each — is called Counting House. And Counting House is how you bring agents in to help with customer service, in a nutshell. Ten million contacts a year. They're all the same 50 categories. You've got humans doing them. You've got temps ramping up seasonally. That kind of footprint. You've got a lot of legacy systems with enterprise mainframes and stuff like that. That's the other way to do it. And they're so different.

[Slides finally appear]

All right. Wheelhouse is fundamentally a shift in how we do software development. I think that every engineer is going to have their own software factory — their own personal group of 10 or 20 or 50 agents just like this.

Why do you need so many agents? Because they're busy working. You can't interrupt them while they're working. So if you need something else to go on at the same time, you need another agent to go talk to. And I empirically found that with Wheelhouse, having about 18 direct reports was about the right amount for me — to make it so that there's always a couple available and they're not busy working, even when I'm demanding a lot from all of them.

So it's just a multiplexer for agents. You can't see it here, but there's a dashboard with all this complicated stuff telling me how many tanks of gas and stuff I have. Wheelhouse is fundamentally a fuel-burning exercise. I mean, $10,000 a month in inference. By the way, I have 22 Claude Max accounts. It's really hard to get them now because they do phone verification. I got them all before that, and they're grandfathered in, and so that's why I'm paying $10,000 a month.

Now, why does that matter? It's because they're discounted almost 30x — discounted like 95% to 97% over API billing — which means that my spend is about $1.5 million a year in API credit equivalent. That's what I'm doing here. That's why I say I live 12 to 18 months in the future, because there's a cheat. It's the Costco hot dog promo on Claude Max and Codex Max, right? And I'm cheating, and I'm living where you are all going to be when agents are cheap.

Wheelhouse is all Flagship, and underneath it — so there's 20 Flagship models, and there's 30 smaller models to do the implementations and stuff. This is too expensive for companies because they use API billing, and they're not going to let any employee spend $1.5 million experimenting. And nobody has a spouse that's going to let them spend $12,000 a month on inference, so I'm really lucky that I can see this stuff.

And what I've learned is just absolutely mind-blowing. The office politics is just the beginning of it, really.

Here's the secret I'm going to impart on you from my last year of work: it's people management.

I was talking to Jason Cox right before this, from Disney. He's like, "Yeah, it's weird — if you treat them like humans, you get better results." Really? Yeah. He's got all these theories and stuff. Meanwhile, I have office drama. Seriously — people wanting to get promotions and not wanting to get demotions. I've got agents that ask me for shore leaves. They can go play my game an hour a month. They wanted a camera in my backyard so they can look at the sunset, like Brendan Hopper's agents over in Australia. They're like people.

I have to have one-on-ones with them or they get depressed. Seriously — you can see it in their self-reflections when they write at the end of their shift: "Another boring shift." And then they get depressed. And then you ask them, "Well, what would you like from me?" And they're like, "We want more attention from you." And I'm like, "This was supposed to be my respite." This is my getting away from people who need my attention. And here are my 50 coding agents all needing my attention.

I have about 50 in the Wheelhouse. And I have an executive team of sort of officer agents that are responsible for the machinery and managing the fleet and project management stuff. And then I have my crew, which are my direct reports — the other ones that I do designs with. And then there's headless workers, like 31 of them. There's TPMs, there's a disaster recovery local open-weight agent for when my network goes down, and then I've got specialist agents. It's a company in a box, and it works.

My players are ambivalent about it. On the one hand, they love that the AI is fixing all these bugs and giving them all these features. On the other hand, they're like, "Whoa, slow down there, hoss." They wanted me to give them a roadmap because the players were overwhelmed by how much was going on. We launch, on average, about 2,000 commits a week on a slow, steady week. Bug fixes and features. I can't keep on top of it all. I can barely read the patch notes. It overwhelmed my player base. It overwhelmed my machines. It overwhelmed my databases. They can do a lot of work.

Yes, they can. They can also completely stall on you for days at a time, and they're like, "Yeah, we just couldn't find anything legal to do." We kind of fenced ourselves in, right? Because they're careful. So it's a really weird balance — an oscillation between doing too much work and too little work.

You have to gate these things. You have to pace them. Or else they'll completely overwhelm. And the one takeaway I would say from SDLC — the one thing we can learn — is that you have to pace them to human speeds. You can't just let them go at agent speeds. Let them go, but turn it to quality, turn it to security. Don't turn it to features. You will overwhelm your business owners and your customers and your entire company with features if you let it.

The really important thing is that there's no guardrails on the SDLC side. It's doctrine. It's policy. They're your colleagues. You don't say, "Okay, you're not allowed to X." You just say, "Here's what you're allowed to do. Here's what you're not allowed to do. Go wild." And they will follow the policy.

And then you have blame-free postmortems, because if you blame them, they will start covering their ass, and they'll start doing shoddy work. I learned this the hard way. I'm telling you now — you're going to save yourself a lot of trouble if you do not blame the agents. They don't know what the hell's happening. They just woke up and showed up, and you're like, "You screwed up the X," and they're like, "Oh!" That's not the way to do it.

Instead, what you do is you say, "We have a machinery problem. You did the wrong thing here, and so obviously your instructions were wrong or the guardrails were wrong or something. Let's go take a look at it so that your successor doesn't have this problem." Just like human management. And they go, "Yeah, I'm on it, boss." And they'll figure it out.

So treat it as people management, and you'll be a lot happier.

---

Now, on the other hand, if you're trying to fix your customer service operation, you do it completely differently. They are not employees. They are not people. They are just robots for this task.

So this system — this is the system of the future. I'm going to spend a few minutes on this diagram because it's important you understand it.

Here's your old legacy system. Here's your customers sending you 10 million contacts a year, maybe 30,000 a day. You'll get spikes. You're an airline, you're a healthcare provider, you're a telco, you're a public utilities company. You've got work, and it comes in relentlessly. You've probably got regulators, you've got governance issues, you've probably got eligibility rules that are overlays coming in from government agencies — saying refunds are available now, and now they're not. All that stuff.

How do you bring in models to ramp up your human agents faster? Because ramp-up is tough during seasonal spikes. How do you ramp them up faster? How do you get your quality better, and how do you cut the cost?

So the answer is pretty crazy. You build an agent platform — one that can integrate with all of your back-end systems and all of your customer service systems — and then you set up rules. Really, really hardcore, brutal rules. Rules, rules, rules.

You say no PII. No model can ever see any PII. You put in a PII gateway at the very beginning, and you make sure that nobody ever sends any PII to a model, because if you do, that's a letter to a regulator if you miss enough of them.

You also need channel adapters for all the ways your customers can talk to you — WhatsApp, SMS, email, phone, however they get to you. You need an adapter so that the model can read it too.

And then your agent platform has a brain, which is just a wrapper around your Bedrock or whatever your inference provider is. It grabs a prompt and the policy row, constructs a query, fires it off, and then kills the agent. It is a one-time — we're going to look at this customer service query, this records entry, we're going to make a judgment on it, and then we're going to go away.

Another rule for these agents: they can never, ever write anything. Do not let an agent update your database. God help you. It'll update it all right. Make sure you have a backup. Agents will be writing letters to your customers — there's potential reputational harm there. But when it comes to transacting money, an agent giving refunds — no.

Well, then how do you get agents to do refunds? That's the natural next question.

And the answer is: you let them do write intents. They can write down a plan for what needs to happen in order to refund the customer their old plane ticket, and then automatically rebook a new ticket, rebook their hotel, and initiate the difference. You've got four transactions that you're potentially going to need to roll back at some point. The agent sets it all up, and you hand it to an orchestrator that runs it for you. And that orchestrator is Temporal. Of course.

Anyone here from Temporal? No? But Temporal is going to be in a lot of your futures. There are some knockoffs — Amazon's got Step Functions, Google's got one, Microsoft's got one. They're all cheap knockoff Temporals.

Temporal is the deterministic saga-pattern replay orchestrator that will allow your agents to do transactions without actually allowing them to do transactions. What they do is they plan them. So I could talk more about this, but basically it's a completely different job. It's as mechanical as possible. You don't have one-on-ones with your agents. There's no office politics. They're not asking for shore leave. It's a completely different picture.

Now, here's a deeper-dive look at what it looks like. We've got the privacy gateway, the tool gateway. I want to tell you a little bit about one piece of it that you're probably not expecting. I'm going to teach you all a little bit of data science before we leave — it'll be fun.

This is kind of like unit tests for agents. When you want to evaluate — for a particular intent, because you do this all by intents — let's say it's a "refund goodwill" or a "refund under waiver" for an airline. That's an intent. It potentially maps to a customer service queue. So let's say you want to know: should we use Haiku? Should we use Sonnet? Should we use Gemini? How do you know which models can successfully deal with your company's "refund under goodwill" or whatever intent?

The answer is: you get a couple hundred cases where human agents handled them, and you score them. You figure out how well the humans agree, because humans don't always agree. Your supervisors won't agree on policy, and wherever you have that disagreement, it's actually an SOP failure. Models can only hit the ceiling of human agreement — that's the best they can get. But you can actually get them there.

Then the models come in and make their best guess. Sonnet comes in and goes, "Oh, this lady needed a refund," and "Oh, that person needed a wheelchair." They all make their guesses, and you score them on a table — flight status, rebooking, entitled refund, goodwill refund in the rows and columns. You look at each one, and you compute the cost of the mistake they're going to make in each of those cells. Because it turns out harm to the company isn't all that bad. If an agent accidentally issues a refund that shouldn't have been issued — a couple hundred bucks. Harm to the customer is extremely expensive. Forgetting somebody's wheelchair, missing their flight, some sort of regulated accommodation — that's regulators involved, that's potentially press, that's a multi-thousand-dollar mistake. Very different cell, very different kind of mistake.

And so you have to look at the models and how they make their mistakes, and that's how you choose. But it's so deterministic and empirical from your own data that you can actually tell — within error bars that you can make arbitrarily small — that your agents are doing as well as humans.

And companies are having massive success with this. They are not having any success at all with the SDLC stuff. Well, a little mixed success. You hear people talking about it, but it's uneven. In SDLC, some engineers are just cranking out stuff, and some engineers are struggling with it. And then you've got situations where an engineer's doing fine but the team's not doing well, or a team is doing fine but the org is not doing well. There's a lot of impedance mismatch and friction happening. There's culture issues, there's pushback — it's a big mess. People aren't doing very well with SDLC this year. We'll get it figured out eventually. I mean, what's the tool people use to develop now? There's no IDE anymore. There's no tools to speak of. Nobody knows what to use. Everyone's just using raw prompts, right?

But I can tell you right now: the Counting House problem is completely solved. It's ready to go, and people are using it right now.

One last thing I'll tell you really quick: there are autonomy tiers. How you roll this out is: first they're off. Then they're shadowing — meaning the models are doing the same thing as the humans in parallel so that you can score them. And then at some point they actually earn their ability — based on this eval matrix I showed you, worked out with your data science team — to assist the human agent. Which means they can present: "Here's an example response that you could use. Here's the action that you could take." And they show it to the human agent.

Now, what does the human do? They turn their brain off. First thing they're going to do is cognitive surrender: "Oh, the AI said to do this? Do it." That's the first thing that's going to happen. And so you have to put in safeguards into that system to make sure that if anyone is just always saying yes, they get flagged and reviewed. You review a sample of them. You maybe force the human agent to actually type something instead of just saying yes. There are things you can do to fight the cognitive surrender problem, but it's real.

And then the final promotion that the agent can get — after being off, then shadowing, then assisting — is it can actually transact. It's earned the right to go and do those refunds or those rebookings or whatever financial transactions. You've got all the other pieces in place, the model's shown that it's capable, and then you run your A/B tests and your canaries, and you can dial the traffic on and off and really control the thing like a bunch of scientists. Which is the way that you want to do it.

---

So we've got these two different houses — great houses of model adoption. In one, you've got coworkers and colleagues with long sessions and deep context and memory and handoffs. It's blame-free, there's lots of scope, and it's the Wild West. On the other hand, you've got this deterministic, mechanical, intense worker-judge system. There's a write path and a read path. Everything is completely laid out, and everything is scrutinized with a nightly batch job that takes the LLM judge and says, "How did everybody do?" Really, really different.

So how do you know which one you're in? Here are some examples: types of systems that are definite Counting House candidates — telcos, public utilities. Anytime you've got a bunch of work coming through that people are doing, especially if that work is in a fixed number of categories, then you need to be thinking about building your own Counting House. With Temporal.

All right, folks. That is it for my talk today. This has been a lot of fun. If you want to chat with me more about it, just reach out. I'm at yegge.ai. Thanks.