It's a Feature, Not a Bug: Making Sure Code You Didn't Write Does What You Meant

Host Intro (Gene Kim)

All right. If you ever learned a programming language from a book with a picture of a puzzled-looking person on the cover, there's a good chance you've read one of our next speaker's works. I met Andrew Stellman at one of Tim O'Reilly's events. He's written a number of books over the last 20 years with Jennifer Greene. He wrote the very first Head First O'Reilly book in the series, which was a departure from their standard animal cover books. He's covered C#, Agile, PMP, and Beautiful Teams.

And so like many of us, his area of passion is AI. In fact, he's building this amazing course called "Be the Model," where you learn about how large language models work by actually being part of one — a tiny model with a 62-word vocabulary and about 2,000 parameters, all done by hand. He's currently a SVP and Principal Engineer at Bank of America, whose headquarters is right around the block.

Like many of the speakers here, he's been deeply studying how to help engineers use AI to do consequential work that matters without screwing up. Here's Andrew.

Andrew Stellman

Hey, folks. All right. Man, I was teed up perfectly.

James just gave an amazing talk, and before I jump in, I wanted to call your attention to something that he was talking about — some of the challenges of reviews, how you actually get an AI to figure out what's wrong with what you built. And some of those challenges, he kept saying it over and over again: giving it the context it needs. I'm going to start there.

So yeah, my name's Andrew Stellman. I am a SVP and Principal Engineer at Bank of America. I'm not going to really talk about what I do there. What I am going to talk about is sort of building on what you've seen so far already — about how do you get your AI to build the thing you want it to build, which turns out not to be easy.

And I'm going to go back to a story from the '90s, where a guy called up the help desk and said, "Hey, the coffee cup holder on my computer is broken." And the help desk was like, "What?" And if you're about my age, you'll remember that you used to press a button on the front of your computer, and a little CD drive would come out, and this guy was using it as a coffee cup holder.

Now, we developers can be kind of jerks, and we'll have names for things. We'll say, "This is an IBM error" — an Idiot Behind the Machine — or a PEBCAC, Problem Exists Between Keyboard and Chair.

But was he really wrong? Was that really a stupid thing to do? Let's say some enterprising CD-ROM manufacturer said, "Hey, you know what? This thing actually works perfectly good as a coffee cup holder. Let's reinforce it and advertise it as a feature." Then he's not an idiot.

We love to say when something goes wrong, "It's a feature, not a bug." How do we know if it's a feature? How do we know if it's a bug?

So I'm from Brooklyn, and that is an intersection just a couple blocks from where I live. And sometimes I like to take the bus when I have to go someplace on the other side of the neighborhood. And what I did was I vibe-coded a cute little bus tracker app that hit up the New York Transit Authority's open API, figured out my bus, the bus routes. I told it what my stop was, how long it would take me to get there, and it gave me a nice little webpage that refreshed itself and told me exactly when I had to leave. And it's perfect. I had an errand to run clear on the other side of the neighborhood.

And luckily, it was a nice day, because I got to the bus stop — perfect timing. The bus was coming exactly when I needed it, on the other side of the street. It was going the wrong way. And you can probably figure out what went wrong. Clearly, it was looking at the wrong route. It was looking at the going-to-Downtown-Brooklyn route rather than the going-to-Kensington route. So was I the idiot behind the machine? What went wrong? Is it my fault? It's not really the AI's fault. So what happened here?

And this is something which I will freely admit I totally stole from Gene Kim — the idea of specification and delegation. And this is really, really important for when you're trying to get an AI to build stuff. As we do respective development, where you build a specification and feed it into your AI and tell it what you need to do, or just explicitly tell it all the things you need to do.

And I've learned a lot about this not from working with AIs, but years ago — I spent about 10 years of my career managing teams, from small teams like five-person, 12-person teams, up to really medium, large-sized teams, 80-something, 90-something people across multiple countries. And when you've got that many people, I have a very high standard of quality for my code. And I wanted to make sure my whole organization had that same standard of quality. But I didn't have time to code review everybody's work. And AIs didn't exist to do all that code reviewing for us because it was 10, 15 years ago.

So I had to learn about specification. Luckily, I spent a few years managing teams of brilliant requirements engineers and business analysts. I learned a lot about how you write good specs, how you really build something that a team can build. And then delegation — you have to actually trust your team to build the stuff.

And then there's another piece, which is verification. And I like to sort of compare this to a shopping list. Let's say you're doing an Instacart order. You've got a specification — that's a list of stuff to buy — and then delegation, the shopper's out in the store shopping for you. But then you're going to check what's in your bag when you get it, because if you're using it to make dinner tonight, you need to know now so that you can run out to the store and get the thing that they got wrong, or that was missing, or whatever.

Now, here's the funny thing. None of these problems are new. This is a picture from the 1968 NATO conference on software engineering, and at that conference they were talking about something called the software crisis. And the software crisis was all about companies who had these teams — this is the late '60s by now. Most of them had these big rooms with mainframes, like a Control Data 6600 or a big IBM mainframe or a PDP. And they had these guys programming with punch cards. And they were thinking, "Hey, wait a second, our projects — they're going over budget. These guys aren't building what we need them to build. Is this like a skill issue? Are they just bad at writing code?" And it turns out that no, they weren't. And in addition to actually coining the term "software engineering" at this conference, they also came up with another really important term, which is "requirements engineering," which really borrowed from stuff that had been in the industry — and outside the software industry, but broader industry — for years.

Here's why requirements are a problem. Because writing them down isn't always enough.

And this is the Mars Climate Orbiter. NASA launched it in 1999, and it was supposed to enter orbit. And you know what happens when a spacecraft enters orbit — we've all seen the space shuttle on re-entry, and it gets all bright and glowy because there's a lot of heat. And so they had to engineer it very carefully. NASA's super, super good at it. I actually did an interview with someone at NASA for my book Beautiful Teams, specifically talking about the way they validate code and the quality practices they have, and they're really stringent.

But something slipped through. The code was built to use newton-seconds. And if you took physics in college, you probably know that pound-force seconds are not the same units as newton-seconds. And unfortunately, whoever put in the data for that put them in in pound-force seconds, and the thing burned up on entry. And that really sucked.

I want to talk a little bit about what's going on there, because it's not immediately obvious how we deal with problems like that. That's kind of "stuff happens" territory, right? Or is it?

This is called a Norman door, named after Don Norman, who wrote an amazing book called The Design of Everyday Things, which has a cute little teapot on the cover with a spout pointing the wrong direction — so if you tried to pour it, it'll spill all over yourself. And there was a door like that. That picture of the buses from earlier — about two blocks from there, there was a bodega, a little store that had a door just like this, where the handle was that kind of curved handle that really you had to pull. It was clearly a pull handle, but it was a push door. And it was every freaking time I went to that store, it got me. I'm like, "Ugh" — I felt like an idiot, right?

Here's the thing: contractors aren't idiots. They know what they should need to install. And the contractor who put that door in probably did exactly what he was told. So was it his problem? Or did the store owner — they probably just asked for a door and assumed that a door would get installed that worked well. And eventually, they came up with a clever solution of just removing the outside handle entirely, so you had no choice but to push it. That's a great solution. A little bit of a kludge, a little bit of a hack, but again — whose fault was it? What happened there? And more importantly, how do we prevent this? And what does it have to do with AI?

This is a problem I've been working on for a while now — well over a year. Well, I've been working on quality engineering for 20 years. Our first book, I wrote with Jenny Greene — Applied Software Project Management — was really a book about quality aimed at project managers.

And what we did was, some of the things we talked about were the history of quality engineering, and these three guys are really, really important.

And this is where Justin from AI Fusion had that amazing slide where he showed the mold — the plastic injection mold. Deming talked about exactly that. You've got an assembly line, just like the machine that he was showing, and somewhere along the line, something happens, and at the tail end of the line you've got the widget that comes out of it that's supposed to be painted, and something like every fifth one there's a paint scratch.

Now, one thing you can do is try to test the quality in at the end. And the way you do that is put a bunch of quality testers — the same people who stick a little sticker in your pants that said "inspected by number 78" — and they find the ones that have that scratch on them, pull it out and put it in a separate bucket. And then you've got a painter who's got a little brush to paint over the scratch. That's what it looks like to test quality in. It's really expensive. And what Deming pointed out was, what you really need to do is actually look at the whole line and see that, oh, these two parts bump into each other five stages earlier.

And he said, "Maybe what you need to do is look at the whole process." And I know Justin was talking about process too, talking about Kanban. That's exactly what this is pointing at. In fact, the term Kanban came from the Toyota Production System that Deming learned from and used to basically revolutionize automotive engineering. Catch that problem where those two — the burr that was on one part scrapes the paint off the other part — put a standing process in earlier, and you don't have to spend nearly as much on testing and repair later.

Then Joseph Juran came up with this idea. And notice the dates on this — these predate computers, right? At least not computers entirely, but certainly computers widespread across industry. And Juran said, "Quality means being fit for use." And that's another thing that Justin pointed out — he had a whole slide about intent. What is our intent for this project? How are we actually going to use it? These are really, really important. There's a really important reason he's talking about AI but using these terms from traditional quality engineering.

And finally, Philip Crosby — he said, "Well, we've got to write it all down, requirements, and quality means conformance to requirements" — how well we meet those specs. And this kind of forms a chain from requirements to implementation to test, where you build the quality in, you make sure the requirements capture not just what you think you need to build, but the actual intent of what you want to build.

And I failed at that when building my little bus tracker app, right? I showed up just in time for the bus going to Kensington, but the wrong bus was coming. So the app worked perfectly — it did what it was designed to do. It told me the bus schedule, except that I intended to go down to the other end of Park Slope and not Downtown Brooklyn. So it met the requirements I gave it when I was vibe coding the app, but it failed to meet my intent because I didn't tell it what I actually wanted.

And this basic idea is the inspiration for an open source skill that you could download today, called Quality Playbook, that I've been working on for about a year now. And it's pretty good.

What it is, is a skill that first reconstructs intent. And again, I've got to give big kudos to those amazing requirements analysts and requirements engineers and business analysts I worked with about 25 years ago for teaching me so much about this stuff.

It actually tries to figure out the intent that you're trying to meet, in addition to the other specs. And then if you think back to the slide that James was showing, where he was talking about the challenges of reviews — especially adversarial reviews — one of the things he said over and over again is the context, because we have to give the reviews context. Well, the purpose of the Quality Playbook is to actually review your code against the actual intent of the software, against the requirements, because the whole point is filling in that context first. Because this is the core of quality engineering — the real basic idea here is that quality engineering is built around that quality chain, and the quality chain starts with intent.

And I tried to capture that. The basic hypothesis was: if we take these basic quality engineering principles from the pioneers like Deming, Juran, and Crosby — and you can see they're literally talking about things that, when you apply them at scale and talk about the AI, it'll do a better job of finding those bugs where the software did exactly what it was meant to do, but it didn't actually capture the intent.

And honestly, it's working pretty well. I've run it against — probably hundreds at this point — of open source projects. And I've submitted pull requests to a lot of them, so it's found bugs in almost all of them. I highlighted a couple of ones that I really like that are prominent, that got merged in by maintainers.

I really like the two Linux ones, because if you've ever worked with Linux development, you know that they have an extremely high quality standard and a very stringent process, especially for some repos. In fact, Virtio was one of the ones I picked from the very beginning because they've got a maintainer who is really, really well known for having extremely high testing standards. And to get this bug submitted, I actually had to reproduce it myself in Linux — spin it up in QEMU and actually make sure I understand the bug, make sure that I can reproduce it correctly, and do real red-green testing before submitting it, because he needed that evidence. That's the way quality is supposed to work.

But the interesting thing about that Virtio bug is it was in the code base for 17 years. And it has to do with how interrupt handling happens — like where the driver handled an interrupt, and then it told the kernel it had an interrupt. And the fix was like a two-line fix. But the interesting thing was that it hadn't been found yet. And you know that there's been a ton of AI code review, because AI can only look for its keys under the streetlight. It can only look at the code and see what the code is expected to do. And maybe there's docs, and it'll look at the docs and compare it with the code, but it can't find anything that isn't in the code.

And that JSON bug is an especially good example of it. Now, for those of you — I use JSON all the time.

Just to tell you a little bit more about myself and why I do this: my North Star is helping people, helping teams. This is why I write books. I've written a bunch of them. And very few people write books to get wealthy — we do this because we want to share knowledge. I spend all day, every day trying to help people improve their code, improve their careers.

And a big part of my day job is principal engineer. And while I can't really talk much about that part of it, what I can say is I spend a lot of time working with code, and I've used JSON many, many times in my projects when I'm writing code, going back long before AI was a thing.

And what this particular bug was, was in a part of the code that validates JSON maps — where what you're supposed to do is, if the map... It's perfectly valid JSON to have a map with the same key, but JSON has a way of finding those maps and saying, "Oh, this isn't valid — you've got the same key mapped to two different values." And I was floored that this bug was still in the code: that it found if you mapped a key to null and then later mapped that same key to another value, it just silently accepted it. Which seems like such a basic case.

Now, here's the interesting part. Quality Playbook found this bug not because it was in the official documentation, but because it cast a wide net. And what it does — the way it works — is it casts a really wide net, tries to find documentation everywhere it can: Slack channels, Teams chats, Discord, and in this case, issues from Jira or GitHub Issues. And there were several GitHub issues where people were complaining about this behavior. And it basically said, "Oh, well, this is a thing that people wanted." And this is an important lesson for AI-generated code: people aren't always going to use your code the way you intend it to. You might not realize that you have a use case that you didn't think of — not just an edge case, but an actual intent issue.

This is exactly the intent problem that my bus tracker app missed. And quality engineering means listening to your users, making sure it's fit for purpose. And one of the best places to find that is issues. And so what the playbook skill does is it actually finds all those bugs by first gathering all that formal and informal documentation, and then turning it into a triple — like a traditional IEEE 830 specification — and then just doing the traditional quality engineering validation, making sure that everything it built meets that specification.

And it's finding a lot of bugs. I'm really careful about submitting them — I have to make sure I read, understand, and validate the code myself before submitting a PR, because maintainers are drowning in AI slop, and I refuse to contribute to that. But I've found a ton of bugs.

And this brings me to the help I'm looking for. This is a completely free open source project because, like I said, my goal is to help you find ways to make your AI code better. I would love it if you tried running this on a project you know well and tell me if it actually found something. Help me evaluate against real requirements, because my ongoing hypothesis here is that the problems that you heard James talk about and you heard Justin talk about — the specifics of how we get the AI to build stuff that we want it to build — my ongoing hypothesis, which is increasingly validated, is that we can roll back to traditional 30-, 40-year-old software engineering techniques, quality engineering techniques that predate software, going back to the '50s. Like you saw James' slide with Kanban — going back to these ideas that have been proven over and over again across the entire industry, not just in software, but globally.

Because we're specifying and delegating and not getting our hands on the code, because we don't have our own human loop that looks at all that code all the time, we need this — or something like this — to help us find those bugs that we don't really realize are there, because we didn't fully capture the intent when we told the AI what to do. I really hope this is helpful for you.

Anyway, just things to leave with: when an agent writes the code, you still have to check the intent — make sure it made it into the software, and that our tests actually verify it, because we can't test quality in at the end. We have to understand what quality engineering is all about, because it works really, really well. And finally, before we talk about the idiot behind the machine, before we say it's user error, we owe our users that investigation.

Anyway, thanks so much for listening.