00:00:05

I am an enterprise strategist at AWS. And what that means is that I'm part of a small team of ex CIOs and CTOs of large enterprises. And what we do is we, we go talk to our customers who are large enterprises who, uh, are in the midst of a transformation, moving to the cloud, adopting DevOps, whatever it is, and are hitting impediments of some sort. And, uh, usually it's non-technical impediments. It's, uh, cultural change and organizational change and, uh, financial issues and, um, bureaucracy that they have to overcome. Uh, and in fact, the, uh, what I just said describes pretty much every large enterprise customer that, that we talk to, and we try to give them some strategies, uh, based on our experience and based on what we see other customers do. So I'm gonna talk a little bit about what I think is the, the key impediment that everybody faces.

00:01:05

Um, it's Luke. Hey, um, I can actually see a little bit. Um, so, uh, before I joined a WSI was the CIO at US Citizenship and Immigration Services, which is part of the Department of Homeland Security. And, uh, before that I was CIO of a company in San Francisco, CEO of a, a software company in Oakland. And I, I think what happened to me is, uh, one day I was reading a newspaper article about all of the IT problems in Homeland Security. And being the problem solver that I am, my reaction immediately was, okay, I can fix that. Let me add it. You know, we can do this. Uh, so somehow every, all the stars aligned. I wound up at U-S-C-I-S and about a year ago, uh, it suddenly hit me. Uh, we were done, you know, everything was fixed, the government was running like clockwork, and it was probably time to move on to the next opportunity.

00:02:06

So I wound up at AWS uh, along the way, I wrote a couple of books, published a couple of books, uh, that you might have seen. And I am just finishing off the third book. We're gonna have some preliminary copies available tomorrow. And, uh, it's called War and Peace in it. And, uh, as the title would suggest, it's a, it's a sequel to told stories novel. And, uh, you know, I'm trying to bring out, uh, his critical messages on DevOps, uh, at which I thought, you know, maybe required a, a sequel to, to really articulate. Um, but, uh, let me, uh, let me tell you a little bit of a story from Warren Peace to set the stage for what I want to say about some of the impediments that large enterprises face when they're adopting DevOps. So in this novel, what's happening is Napoleon is invading Russia, right?

00:03:03

That's, that's kind of the through line through the whole thing. And Napoleon great, you know, military leader, and his army is coming into Russia, and the Russians keep withdrawing, and his army pushes deeper in the Russians withdraw a little bit more. And then finally, the, the big climactic scene comes when they're finally gonna have their, their big battle. It's the Battle of Borodino is the name of the place. And, uh, what happens if Borodino goes something like this? You have Napoleon the day before the battle, and he's sort of surveying the ground with his, with his key reports, direct reports, and, uh, he's giving some instructions. You know, you guys go over there and do this, and you guys do go over there and do that. And he's setting the, the Satan for the battle. And of course, each of the, uh, generals or, or whatever their, their rank is, they, they think they know better than Napoleon.

00:03:59

So they're mostly not listening to 'em, and they decide to do some other things that don't really fit together very well. But then when the battle starts, Napoleon withdraws to a fort that he had taken the day before, about a mile away, ch Bernardino. And he's at this, at this fort a mile away while the battle is going on, and he's issuing commands. And the way that this works is messengers ride up from the battle and they tell Napoleon what's going on, and he makes a decision, gives an order, and the messengers go back to the front with the order. Napoleon can't actually see what's going on in the battle because he's a mile away and it's a misty day. There's lots of mist, and there's smoke all over the battlefield once the battle starts. And there are actually gullies in between. So he can't see anything.

00:04:47

He has to wait for the messengers to come. And so the messengers come with a message, and by the time they get to Napoleon, the news that they're bringing is completely outdated. And in one very memorable moment, they write up and they say, we've taken the bridge at Bodino. Do you order the Army to cross it? And Napoleon says, yes, have them cross it and form into ranks on the other side. And what he doesn't know is that not only have the Russians taken it back, but they burned it. <laugh>, there is no bridge anymore. And this is, this had actually happened, like the moment the messenger had left to go tell Napoleon, right? So Tolstoy's point, and, and this is, this is partly a novel and partly all about history, but Tolstoy's point is that Napoleon has nothing to do whatsoever with what's actually going on on the battlefield. He's issu, he thinks he's issuing orders, and none of it actually means anything.

00:05:46

So, uh, what is Tolstoy trying to tell us about DevOps? Uh, and I think the, the point here is you have a long decision cycle. Napoleon's decision cycle is really long because he has to wait for the messenger to, to come and go back. But the action that's actually happening has a very short cycle time. It's moving very quickly. And there's a sort of an impedance mismatch here, right? He, he has these long decision making cycles, but things are moving really quickly. And there's, there's no way you can actually control something that's moving quickly if you've got this, this big decision cycle. So if you look at DevOps in an enterprise, you have a business situation that's changing very rapidly, that's full of complexity, uncertainty, rapid change. You have a process, a technical process, DevOps that works religiously to, to shorten lead times, and it's embedded in a business process that is nothing of the sort that has a really long lead time.

00:06:50

DevOps, more or less, takes us from the point where we write code and, and commit it to the point where it gets deployed and gets into users' hands. And, uh, as, as the last presentation here mentioned, uh, monitoring and telemetry kick in and, and prove that it's actually working. But it's from code commit, essentially to use in production. But the process for making business changes is this big thing. Generally, it starts with something like, uh, realizing that you have business needs or mission needs, uh, figuring out what they are, putting a group of them together into a big project or a big program, building a business case for that big program, uh, passing it through an investment management process or a governance process, a steering committee, whatever, uh, finding the resources who are going to work on it. And eventually you get to the point where a code is written and deployed, and then you also have a bunch of downstream stuff to realize the value or harvest the value out of these changes that you've made.

00:07:58

So what you have is from concept to deployment or concept to, uh, concept to cash in some cases, but idea to impact, you have this really long lead time. This is Napoleon on his hill, you know, he's, he's making decisions about what needs to happen, what's gonna be valuable, and by the time it gets implemented, everything's changed already. No matter how much you speed up the part that DevOps covers, it doesn't, it doesn't make such a big difference, you know, for the big project work, at least when you're in sort of a maintenance mode and you have rapidly churning requirements, perhaps, but, uh, in a lot of cases in a big enterprise, you're dealing with big capital investments that need to go through this, this big drawn out process to get to the point where you can actually start writing some code. So that's the, that's the area that really interests me. What, what can we do about all of this? Because really to take full advantage of what DevOps can give us, we need to speed up the entire lead time. We, we need to somehow shrink it because that's what's going to let us maintain our nimbleness, our agility, uh, our innovation where we can try experiments and, uh, make decisions based on the results. And so my interest is in that, that big upfront, you know, fuzzy front end, and what can we do about it?

00:09:30

So first, uh, first idea in my train of thought here is the way we set up this big process is designed for a world that's predictable or at least reasonably predictable. And we happen to be in an environment as Napoleon was an environment that's filled with uncertainty, complexity, rapid change, and the mental model for how you make decisions, how you make investment decisions, in particular in an environment where things are reasonably predictable, is very different from the mental model you use when things are changing very rapidly and, and things are uncertain. So the old scheme was to put together a business case, and your business case involves projecting revenues and costs based on this new IT system you're building. And, uh, everyone understands that you can't exactly project your revenues and costs, but the assumption is that if you put a stake in the ground and you put a number there, you're gonna be somewhere close to reality, right?

00:10:37

There's gonna be some variance. What if the, uh, what, if any accurate projection was so uncertain that a confidence interval of, you know, 60% would require plus or minus 200% from your estimate. In other words, this is gonna make us $10 million of revenue, maybe, you know, plus or minus $40 million or something like that. You know, the, the more extreme your uncertainty is, the less meaningful a business case is. And the environment that we're in, I think is one of enough uncertainty. So that trying to project revenues and costs and make decisions based on them maybe is not the right mental model. Uh, when you are in that mental model, when you are making your decisions based on a business case, it's really important that when you actually execute the project, you execute it exactly as you planned it because the business case is based on your exact plans.

00:11:41

So if the business case says, this is gonna make us $10 million a year and it's gonna cost $1 million to build, well, when you execute that project, you'd better spend $1 million and you better get back $10 million. You know, execution according to plan is highly valued, but in a world of uncertainty where you know that things are not gonna go the way that you plan, it's a bad idea to pretend that they are right. That's, that's just, you know, willfully misleading yourself in making bad decisions. So we have this oversight process, governance process that is based on assumptions about the old world that don't quite tie to the new world. And if you think about why we have all of those processes, I, I would submit that there are really two reasons. Two, two end goals. One of them is to reduce risk of the actual execution, and the second is to make the best decisions about where to put your resources. You have an opportunity cost. If you're gonna invest in this project rather than this project, well, you better have a sense of how much return you're gonna get from both, and the return from this one has to be bigger. So we set up these processes of building business cases and reviewing them and trying to align with strategic objectives and all of those things in order to mitigate risk and to allocate capital to the, to the appropriate investments.

00:13:14

Well, in Napoleon's world, making those decisions in advance about exactly how you're gonna conduct the battle probably is not that effective. It's not effective at mitigating risk, it's not effective about, uh, using the right deployments of your resources to conduct the battle. It just doesn't work very well in an environment of uncertainty where unexpected events are going to happen. And you know that they are, this sounds at first like a highly risky environment, and the traditional way of thinking about risk would call it that, okay, things are going crazy, there's disruption in all the industries and geopolitical things and, and all of this stuff that changes just gonna be constant over the next few years. That sounds pretty risky. It sounds pretty risky if you're still thinking in terms of business case and trying to execute exactly on that business case. But the fact that unexpected things are going to happen is not necessarily a hazard.

00:14:17

It could be an opportunity as well. Something unexpected happens. Either it destroys you or it makes you better, essentially, right? The difference between those, whether the unexpected is gonna lead to something good or lead to something bad, is your ability to seize an opportunity out of that change. In other words, it depends on your agility, your speed, your flexibility, and your inventiveness when you see what actually happens. And the best way to manage risk in the kind of environment that we're in is to make sure that anything that unex anything unexpected that happens, we can turn it into an opportunity, right? We, we have the, uh, the agility and the inventiveness to be able to, to turn it into a positive thing. Risk in a sense, in, in today's environment, is the same as not being agile in that sense, right? And the agility I'm talking about is that entire value stream where you have to notice the opportunity, figure out what you're going to do, justify it, apply the capital resources and everything else, and do all of that quickly and agilely.

00:15:38

So how can we rework this long value stream in a way that promotes speed, agility, flexibility, and innovation or inventiveness? Well, I can think of three basic models and I, I'm gonna walk through the three models. This is not to say that these are the only three models, these are just the ones that come to mind when I think about it. The first model, actually, I'll tell you all three, uh, at least a name for each so that, so that we can correspond them. The first one is what some people would call the product model. Second is what I'm gonna call the budget model. And the third is the objective model. So product budget objective. The product model says we're gonna, we're gonna reduce the time that this takes by having fast decision making cycles because we've decentralized them into a product team. This is very much the way that AWS works, for example.

00:16:43

So I'm very familiar with, with the model. Now in AWS we offer 125 or so different services, and there's a team responsible for each of those services. And the team maintains its own roadmap. It is influenced by central, uh, input, let's say. But the team has control of its roadmap and it works with customers to figure out what customers need, and then it decides what it's gonna do about those customer needs. 90 to 95% of our product roadmap is drawn directly from requests from customers. So it's a process that's optimized for that. You know, let's listen to the customers make decisions. The decisions don't have to go to a, a central authority. The governance is all done locally. So that's, that's a, a product type model. Now, the central, uh, concerns do have some influence on it. For example, there's an imperative to all of the service teams to reduce their costs and pass the savings onto customers.

00:17:48

And each service team can interpret that however, however they want to or need to. They figure out how to reduce costs. But there's this, this high level objective that gets passed on to the teams. So that's what a, a product model might look like. A budget model, uh, to me, this is interesting. Let me, let me back up, uh, a tiny bit on this one. I think we've had this myth in the IT community for a while that we have two kinds of costs in our IT budget. We have keeping the lights on costs and we have innovation costs. And all of us, I think, say, innovation is where we wanna spend our money. Maintenance is not where we want to spend our money keeping the lights on. I disagree actually. I think a lot of what goes by the name of keeping the lights on a lot of that maintenance stuff is actually innovation work.

00:18:42

It's actually what's advancing the business. It's actually what's changing the IT systems to keep them consistent with what the business needs. You don't have to maintain software, by the way. You have to maintain a car to make it keep functioning the way it always did. Software keeps doing exactly what it, what it did when you bought it, right? You don't have to put money into it to make it keep doing that. The problem is you never want software to keep doing what it did when you bought it. You want to keep changing it as your business changes. So the maintenance spend or the keeping the lights on spend to a large extent is remaking the decision that this is the right software for you to use and making changes to it as you need to. So I, I say that, uh, as a backdrop to my budget model because usually the maintenance side of things is done out of sort of a budget, right?

00:19:33

We're, we're gonna put this much money in keeping the lights on and then we'll figure out how to spend it. As opposed to innovation money, which is usually a large capital expense that has to go through a governance process. So why not treat everything that we do as just ongoing maintenance of our IT assets? Sometimes it involves building or buying a new system. Sometimes it involves making changes to an existing system. More often today it's refactoring existing systems. Maybe a strangler pattern, you know, breaking off pieces and doing something with them. But it doesn't really matter which of those it is. And it doesn't matter to the business users how you're getting them, the capabilities it could be from an existing, an existing application or a new one. Uh, and in fact, if you're building on an existing application, it's a very effective way to spend your money.

00:20:25

'cause you've already got something there. Every dollar is probably getting you more. But either way, you've got this big legacy estate of it and you're constantly changing it to keep up with the bus, what the business needs. And you do that through a budget, and the budget is passed down through an organizational structure. And that means that the amount of money that makes its way to whoever's spending it, um, they have control over how it's spent in their budget. So you don't need these long cycles of, you know, asking, is it okay if we add this feature and this feature? You don't have to go to a a steering committee generally for that. So the budget model potentially is a way to speed up your, your time to decisions around your DevOps initiative. It's the third model that I think is really interesting and, uh, something that we tried out at U-S-C-I-S that I think was really effective.

00:21:19

And since then, um, I'm hearing about more organizations using this. I, I called it the objective model. And I'll give you an example of how it's used because it's easier to see that way at U-S-C-I-S, one of the systems that we were in charge of is E-Verify, which is the application that employers can use to make sure that their employees are eligible to work in the United States. And we anticipated that at some point soon for political reasons. That's gonna have to scale up like crazy. You know, at some point Congress is gonna say, all employers need to use this. Uh, right now it's just a very small set of employers that do. So we were gonna have to scale the thing like crazy. And we realized that it wouldn't scale, not because of technical reasons, because it, it's in the cloud now. You know, it's, we have elasticity in the cloud, it can scale.

00:22:12

The problem is the human part of the system. So E-Verify at, at the time we started this could, in an automated way, handle 98.6% of the cases. That's great, except the other 1.4% of the cases a human being has to look at. And if we suddenly got a lot more cases, we wouldn't have enough people to do that. So an objective we had was to increase this 98.6% to a higher number. A second objective was a human being who is adjudicating these cases, could do about 70 cases a day. And we said we need, we need that number to go way up. We need to, we need to develop software or something that's gonna let them adjudicate more cases every day. A third was when companies signed up to use E-Verify. We had a shopping cart abandonment problem. About 40% of them actually made it through the whole process and the rest did not.

00:23:08

They got stuck somewhere. So we said we want that number to go closer to a hundred percent. Altogether. We realized that we had five big business objectives in this, in this project. 98.6% make it go up 70 cases a day, make it go up 40%, finishing the registration process, make it go up. Uh, and then two others that I won't talk about. So instead of the way, normally, uh, you would do this in the traditional way of making investment decisions, you would translate those objectives into a bunch of requirements and then you would put all those requirements into a bundle business, build a business case for it, and then try to execute those requirements. This is a little strange if you think about it, because adding those requirements actually adds risk to the project. What I mean is, if you're just gonna take those objectives and try to execute them, that's one thing.

00:24:08

But if you've now said those objectives turn into these requirements and that's going to have the impact we want, you've now added the risk that your requirements are not the right ones, right? What you really want is to accomplish the objectives. The old project way of thinking said that what you want is to finish off these requirements, but you've added a risk in between. You've also, uh, you've also tied the hands of the innovative people who you want to be thinking of good solutions by adding these requirements. So what we chose to do in this case is instead of creating requirements, we just took those objectives and passed them to teams directly. So we would create a team that was a cross-functional team. This might sound familiar from DevOps. It included developers, operations and infrastructure people, testers, security people. And it also included business operations people.

00:25:06

So truly cross-functional team. It had business skills, it had technical skills. And then we said 70 cases a day make it go up. And that's the only instruction we gave. We didn't say, here's a bunch of requirements, execute the requirements. That team then owned making 70 go up. And they could do it in whatever way they could figure out that would make that number increase. In fact, we told them specifically, if you can do this without writing any software, just changing business processes, go right ahead and do that. If you, uh, wanna write some software, write some software, you know, whatever it is that's going to make that number go up, all we care about is the number. And since you've got the cloud and you've got DevOps tooling and, and a process all set up, you should be able to start producing functionality tomorrow.

00:25:58

So in two weeks, tell us what that number is now, 70, now in two weeks, tell us what number you've gotten it up to. And then every two weeks after that, we're gonna review it and see what number you've gotten it up to. And every time we're gonna make a remake the decision to keep investing in this based on what we see. So after a month, we could see that the number of cases per day was going up and we said to the oversight body that was responsible for the investment, look, the number is going up. We think we should continue investing in this objective. What do you think? And they said, yeah, sounds good. With the shopping cart abandonment rate problem, uh, I remember it was 40%. It went up, it went up, and then it started plateauing. And in our biweekly discussions with the team that owned that objective, we asked what kinds of things they were doing to make the number go up.

00:26:53

And what they told us made a lot of sense and it just didn't seem to be budging. So when we went back to the investment committee, we said, here's what we're seeing it, it did this, we're trying everything. It's not helping. We suggest that we stop investing in this objective and put the money on something else. And that they thought was pretty strange. 'cause no project ever returns money. Uh, but that was, that was, uh, the point of this process was that because we were going to constantly be, be reviewing it, the central body still had control over it in every relevant sense. In fact, more control than if we had started with a big set of requirements and could constantly remake the decision about whether the spending was going well and divert resources. So ultimately this project, uh, it, it was continued. I think they just officially ended the project.

00:27:47

It was planned to be about a four year project. After about two and a half years, the team said, okay, we're, we're done. You know, we've accomplished these five objectives to the extent that they can be accomplished, so let's just return the rest of the money. This, I think, is the perfect model for, uh, the age of Napoleon, right? Um, where you have all this complexity and uncertainty by decentralizing the decision making to the teams, but yet agreeing on an objective that could be used for control centrally. Everybody was aligned, all of the controls were in place around the investment. The team could be innovative and could be testing hypotheses and investing, uh, continuing to invest in the things that we're working. And their accountability was to accomplish the objectives, which is what the business case was built on in the first place. So the three models that I've given are always to stop this big long investment decision cycle, make it much shorter by decentralizing control over the decision so they don't have to go to Napoleon who's a mile away and come back again.

00:29:00

But yet the central authority still has control over it in every meaningful sense and can manage the risk of it. So the three models, again, were product team that owns its own roadmap, but has influence from the center, the budget model where funds are allocated through a hierarchy until they reach a team, which now has the funds are really in, in most cases, the number of teams, uh, you know, the production capacity and they're going to decide how to use it. And then the central authority can reallocate funds and make changes based on how well it's being spent. And then the third model where what's cascaded from the center is just an objective. And the team then has the freedom within that objective to do whatever it is able to find that will accomplish the objective. Uh, and that can, you know, essentially the requirements can vary, which is really what we want in the agile world.

00:30:04

So these aren't the only possible models, uh, but I wanted to throw them out there as ways to think about the problem. But the problem is still we have to shrink this long cycle if we're gonna take full advantage of the, the DevOps short cycle time and the flexibility that we get from it. Going back to Napoleon, I think the important thing to realize is shrinking cycle times is not about doing things faster. You know, his armies on the field, it's gonna take as long as it takes. It's not about the speed, it's about the quality of the decisions. Napoleon can't make good decisions from where he is because the cycle time for his decisions is out of sync with the cycle time that things are actually happening. And I think when you put DevOps into an enterprise, it's the same concern. It's not just about how quickly you can get stuff to market, although that's important, it's about how can your central organization still have good control and still make good decisions while the action is moving really fast, both in the business context and in the DevOps process. So fast cycle time here, gotta make this cycle time fast so that you can make really good decisions so that you can lead your army to success as, uh, Napoleon didn't really do in Russia in the end. Uh, and that I think is what Tol story has to teach us about DevOps. So thank you all.