The System Pokes Back

Host Intro (Gene Kim)

All right. Up next is a hero of mine for decades. I first encountered her work more than 20 years ago when I went to a Software Process Improvement Network meeting called SPIN, where we were discussing her paper, "Better Testing, Worse Quality." So in short, she was describing a roundtable of software test managers who compared their best projects and their worst projects. And here's a curious thing. People's worst projects had the most testers, and often, people's best projects had no dedicated testers at all.

So obviously, it's not the testers that were causing problems, but the conclusion was that when developers know that there's someone else out there who will catch all their mistakes, something changes in the system that allows more failures to happen. And so I think that's what's behind the shocking principle that Netflix came out with 20 years ago, which is saying, "You build it, you test it, you secure it, and when things break at 2:00 a.m., you wake up and fix it." There's no ops team that's going to wake up and fix it for you.

So Elisabeth is a phenomenal engineering leader. She was a VP of R&D at Pivotal for their data products, and I remember seeing a picture of her on the floor of the New York Stock Exchange when they went public. And I was like, "Man, that's awesome." So now she's written an amazing book with Joel Tosi called Signals and Levers. And I was honored to write the foreword, and just to describe why — for decades, I've studied systems thinking. It shows up prominently in The Phoenix Project. I've loved The Beer Game, Conway's Law, lean feedback loops. But the quality of books in that domain are not great. And so when I read her book, it was like — I think it's going to be widely considered to be the best entry point for that incredible domain.

So here to tell us about how that body of knowledge can help us make better decisions when AI is changing everything from first principles, here's Elisabeth.

Elisabeth Hendrickson

We're almost done. Okay. So, yes, this is the book, Signals and Levers. It came out September 22nd. I want to be clear with you that this is not a book about AI. In fact, we only mention AI in the very beginning and the very end. However, my coauthor and I, we believe that this is a book that helps you reason about what's happening in AI, because the systems thinking tools that we're talking about are frankly evergreen. And to demonstrate that, this is a talk that is both inspired by and applies the concepts in the book.

To start, I want to tell you about my own AI journey.

In March 2025, I posted something on my blog about my, at that time, recent experiments with Cursor. I felt like I was very late to the game. This is classic for me. It actually takes me a while to get excited about new stuff. It's not that I don't care, it's that I'm pulled in so many directions. And so I had been hearing about, "Copilot's amazing, and this is amazing. Cursor's amazing." Okay, yeah, whatever. But once I hear enough signal that it's actually amazing, okay, now I will try it. And so I tried it, and it was amazing. And I could do stuff that I hadn't been able to do.

And one of the things that sold me and that told me that this is, in fact, changing literally everything — I had been working on a simulation of work flowing through a software organization. It had a JavaScript component to it. My local live JavaScript reload had gotten borked up. I didn't have the chops to fix it myself. I tried. I had tried error-driven development a lot, and I had not gotten anywhere. Thank you. And so I had then gotten people who were better at Rails and the asset pipeline than I was to try to help me, and they couldn't help me. And so eventually I just gave up, which meant that every time I wanted to make a JavaScript change, I would make the JavaScript change, and then I would recompile the JavaScript, and it would take three minutes. And three minutes is excruciating in this day and age.

I know I say that now, and in 1998 or whatever, when I was going through the pain that led to the paper that Gene referenced, I was excited about a test cycle that only took two weeks. So three minutes — I should be thrilled, right? No, that's not how our world works. So three minutes was excruciating, which meant that I avoided making JavaScript changes as much as I possibly could, and that was harming my ability to get stuff done.

So Cursor and I were getting to know each other. I'd picked a model. I'd done a few small things, and then I thought, "Hey, wait a minute. I have a borked up JavaScript reload. Hey, Cursor. What can you do about this?" And Cursor came back with some ideas, and we went back and forth. Forty-five minutes later, the thing that I hadn't been able to fix for months was fixed. And that was when I realized that AI — this thing — this is amazing.

And when I wrote about it, again, this is March 2025, so this is, whatever that is, 19 months ago. The analogy I used is one that I'm sure you've heard before. It was: this is a mech suit. Now, the thing about a mech suit is that the human is very much controlling every aspect of it. It's just adding muscle to it.

Yeah. Well, today, AI is powering agentic software factories. And as we've already heard today, if you try to work at human speeds in this world to correct or to check the output from the models, it is a losing game. You're not in for a good time. So the world has fundamentally changed, and that simulation — it's now rewritten entirely in TypeScript. I don't speak TypeScript. I have written zero lines of code, and it has far surpassed my wildest dreams for where I would be able to take it.

So what AI is capable of is amazing.

But here's where things get weird.

So CircleCI, their 2026 State of Software Delivery Report, which was based on data analyzed from 28.7 million CI workflows — they reported that a small group of top performers is shipping faster and more reliably than ever, but most teams are experiencing the opposite. So if this is so amazing, how is it that only a small percentage of companies are actually seeing the outcome from that?

Hmm. Well, let's use some systems thinking tools to analyze this.

One of the things that we talk about in the book is illusions — maybe illusions are so comforting we want to believe them. Maybe we're just fooling ourselves. And then there's Pharos. Pharos is a token engineering platform. They analyzed data from 22,000 developers, 4,000 teams, and they wrote this report that — if you haven't read it yet — it is fascinating. It's called "The Acceleration Whiplash," and they talk about how AI made it easy to start but not easy to finish. And so it feels like we're going so fast, woohoo, and we're not actually getting anywhere. So is it that we're suffering from an illusion of progress, and that's what is showing up in this CircleCI data? Well, maybe.

But I also want to consider the possibility that we're looking at unintended consequences. Now, let me explain what you're about to see. This is a causal diagram. You may have seen these before. Joel and I adopted a style of diagramming that borrows heavily from both Donella Meadows' style of diagramming and also Jerry Weinberg. I studied with Jerry Weinberg, and that's where I really, really started getting into systems thinking.

And so let's talk about merge congestion. You've got a merge queue. Merge congestion — it's a problem. That means that you've got a change that's theoretically ready, but it takes some amount of time: minutes, hours, days. I've actually seen a circumstance where it took over a week to get that thing all the way through the pipeline so that it can go out. So as merge congestion goes up, the cycle time also goes up. That's what that plus sign means — these things are moving together. So cycle time is going up. If cycle time is going up, you've got this thing that's stuck in the merge queue, and as one head of developer experience told me, "It's not a problem that the merge queue is deep because that means the developer's not blocked. They can go work on something else."

Yeah. I didn't think that was quite working out the way that he thought it was working out. So your work in progress is going up, and if the work in progress is going up, that means more branches are happening. And as more branches are happening, you have more merge congestion. So this is a reinforcing loop — kind of a doom loop. Anybody seen this before? Seem familiar?

Okay, that's enough laughs. All right.

Well, hey — so now also let's consider the possibility of PR size, the pull request size. As that goes up, it takes longer and longer for things to get through the whole queue, right? And guess what AI does? Tends to create larger PRs. So maybe what we're seeing in this data is that it is the result of a system. It's a system effect. It's an unintended consequence.

Let's consider another one. Let's talk about shadow IT. Now here, I want to compliment one of my favorite talks. There are so many favorite talks, but one of my favorites was Michael and Marios' talk from Thrive, where they talked about the work that she had done to build a tool that she needed. That is not what I mean by shadow IT. That's the good kind. That's empowerment.

What I mean here is the kind where they say, "Hey, I built this thing," and then when it breaks, they hand it over to engineering and say, "But now you fix it." That kind of shadow IT. That's not so good.

So using the same diagramming method — as the speed of delivery goes up, the minus sign means these things move in opposite directions. As the speed of delivery goes up then —

Oh, I did it wrong, didn't I? I've got a bug.

Oh, no, I just can't talk. It's late in the day.

As the speed goes down — so cycle time is going up, speed goes down — then the amount of shadow IT goes up. Yes? Because life will find a way. Okay, so as the shadow IT goes up, guess what happens? The speed of delivery goes down, because now there's all these things that engineering gets handed. Well, guess what happens? AI theoretically can speed up IT delivery, but it also enables a whole lot of shadow IT. So maybe that's happening — and I'll bet in some organizations it is. Anybody getting handed some AI-coded stuff from somebody somewhere else in the business? Yeah. Hi. And told, "Hey, run this." What's that going to do to the speed of the roadmap delivery?

So part of what we're seeing — and you've heard it before, I am sure — AI is an amplifier. It multiplies everything. It multiplies the benefits. It also multiplies all that chaos and the bottlenecks and the noise. So any organization that was already in a world of hurt is even more in a world of hurt now.

But I want to reframe that a little bit. The way I see it: you poke the system, and the system pokes back, because that is what an adaptive system does.

Software delivery is an adaptive sociotechnical system. Now let's unpack that, because there's a lot there. So a system — Donella Meadows would say that a system is a structure of things that is organized for some purpose, and Stafford Beer said that a system does what a system does. Joel and I decided to say that a system is a set of things that does something, because trying to put an intention on it — well, it may not do what you intended it to do. So it's a set of things that does something.

In this case, "sociotechnical" means it's a combination of people and technologies all together — many, many moving parts in our delivery systems.

The thing about being a system is that it has three key properties. One is that it is indivisible — it is a whole. As soon as you try to slice it in half, or whack out 800 customer support engineers or whatever, as soon as you try to whack it in pieces, it is no longer the same system. It does not have the same properties. All of the parts within the system are interdependent, which means as soon as you change one part — like, say, increasing or decreasing the amount of emphasis on independent testing, independent QA — then you affect other parts in the system, and they're going to adapt. That's the adaptive part. So the outputs from one area end up becoming inputs into another.

And furthermore, they are fractal. So if you're on a team, the team is a system. That team exists within a department. The department is a system. The department exists within a company. The company is a system. But it plays within an ecosystem with customers and competitors — that is also a system. That plays within an economy — that is a system. It is systems all the way up and down, just like turtles.

And what Russell Ackoff, brilliant systems thinker, said is that in any system, when one improves the performance of the parts taken separately — like, say, speeding up the delivery of code — the performance of the whole does not necessarily improve, and frequently gets worse. So from my point of view, systems thinking explains the CircleCI results.

All right, so let's look even more deeply at this.

This is a report from DX. 2026 — they looked at data from 500 companies, and what they found is that while some organizations are seeing clear improvements to quality as AI usage increases, others are seeing serious degradation, with many swinging wildly.

I did find it fascinating that so many 2026 reports essentially echoed the same findings. So once again, some organizations — a small number of organizations — are seeing improvements. A larger number of organizations are seeing either wild volatility or degradation.

So let's look at a case study. I love this one, because — first of all — Derek Curran with Intercom published their experience report of doubling productivity at their company. It's a great read. If you haven't read it yet, I highly recommend it. It's not the breathless, enthusiastic, effusive endorsement of AI as the magic 1000x kind of thing. It's a very down-to-earth, "this is how we did it, you can do it too." But one of the things that they did say in the report is that quality got worse before it got better, and I found that really interesting. And this is one of the things that I think we're all challenged with.

And you heard Justin's talk earlier today, and I've worked with Justin. I very much appreciate his methodology and also his metaphor — that the key thing is the injection mold, that lines of code are now the little plastic pellets, and the injection mold is your validation framework.

So what I think we're seeing in both the DX report and in the Intercom case study is an instance where the volume of output goes up, but then also so does the risk of compounding slop. Now, if you look at control systems, this is running open loop, right? So in a control system, you set an intention — like your thermostat. You set an intention: this is what I want to be true. And then you've got sensors that are reading the current state, and you've got a controller of some kind that's taking action in order to reconcile. And Kubernetes brought us the words "reconciliation loop." Well, when you're running open loop, you're not doing that, so there's this increasing risk of compounding slop.

And the countermeasure — the intervention that can address this — is by investing in your guardrails. The better and more guardrails you have — and we heard from James as he talked about this from a security standpoint — that is an example of a set of guardrails that are critically important in this day and age. As your guardrails go up, then that's going to help bring down the risk. So it is so important to be looking at your guardrails.

Another case study that seemed to echo this is from a very small company. I was not previously familiar with the company, but I very much appreciated the blog where they reported out their experience with setting up their own agentic software factory — pull requests up 24 times. But also, the amount of rework was going up. So if you look at — I know it's not legible, it's not intended to be on this slide — but if you look at the red lines, the red lines are getting bigger. The red and the pink — those are the rework. And I read that first, and I thought, "Whoa, well, actually, the productivity — maybe the pull requests went up 24 times, but the actual value being delivered — did that really go up?"

And then I realized something. I was falling into a trap.

Here's the thing. We called our book Signals and Levers. You are surrounded by signals. Signals are everywhere. And then using the signals, you can figure out which levers to pull — interventions, like adding guardrails. But we knew what the signals meant in the old world. We don't know what the signals mean in the new world. So is that signal, or is there some possibility that that is noise, and that it doesn't matter, and that if we hyper-fixate on things like the number of turns or the number of rework cycles or whatever — as long as it doesn't impact the outcomes that we care about, maybe it's okay, because maybe that's the new normal. And that's the thing that we all have to be questioning. Is this a signal that we should be reacting to? Or if we react to it, are we actually going to be overreacting, and in the way of systems, actually causing even more volatility by trying to control something that we cannot control? Or is this noise?

We kind of have to question everything, because right now, we don't know. Anybody who tells you that they know for sure — well, they're either delusional, or they're really trying to sell you something.

Okay, so now — what do we do about this?

I've just said we don't know anything anymore. What do we do? Well, there are some things that I think we can pretty much say do stay the same. One is that if we look at what we're actually trying to achieve — the business outcomes that we're trying to achieve — you're probably trying to increase productivity, because I've never met an organization yet in my entire career that said, "No, we want productivity to go down." So chances are you want people to be able to accomplish more with the time that they have available. You probably want to see at least steady quality, preferably improved quality, depending on your context, and you want the financials to be good and solid, right? Anybody not want these things? Maybe some of them are more important to you than others, and maybe there are other things that are on your list. The key thing is to be looking at the outcomes and not putting too much stock in unproven proxies or vanity metrics.

So for the financials, you want to be looking at revenue — because if you're churning customers, like I think it was the Klarna case study, if you're churning customers, that is a really strong and very bad signal, and you need to do something about it. You also want to be looking at the all-in cost. So you've got your revenue minus your cost — profit. So you want to be looking at the all-in cost. But if you get too hyper-fixated on the token cost — so now you've got people spending an enormous amount of time trying to save as much money as they can on tokens, so they're actually now not able to get nearly as much done because they can't use the tokens, but they're spending all of their time trying to optimize for tokens — you're probably not getting the results that you wanted. So look at the financials that actually matter to you from a holistic perspective.

For quality, you want to be looking at things like product failures and incidents, and not the interim rework or internally found bugs. Bugs still matter, but in terms of figuring out how we can tell better or worse — I don't know that we know what that means anymore. So you have to look at the things that really, really matter.

And then for productivity, look at value delivered. I come from the Extreme Programming community, where we talk about running, tested features being the only measure of productivity — as opposed to productivity being pull requests, or lines of code generated, or any other output measure. In the age of AI, if you're measuring output, your numbers are just going to keep going up. But I don't know that they're necessarily going up in a way that is meaningful for your business.

Which is why it's so important that you design experiments. Have a hypothesis, design your experiment, and then observe and measure before you try your next hypothesis. The scientific method — kind of evergreen — going to work even in the age of AI.

And yes, I don't need to hammer this home because Justin did such an amazing job of talking about your validation structure. So important to invest in that. I have a hypothesis that I would love somebody to test, and that is that if you actually invested there first, before you tried to invest in delivering more features, that you would then be able to deliver more features. So many organizations are stuck with validation systems that are simply tooled for human speed, and they're not keeping up in the age of AI.

So ultimately, do remember as you do all of this: you're poking a system, the system is going to poke back. And I have some help that I would like — I want to hear your stories, especially if you have any really good, juicy stories of ironic, unintended consequences. Thank you so much.