Las Vegas 2019
Eating a Bureaucracy One Bite at a Time with DevOps
Eating a Bureaucracy One Bite at a Time with DevOps
MS
Mark Schwartz
Enterprise Strategist, Amazon Web Services
Transcript
My name is Mark Schwartz. I am an enterprise strategist for AWS. And what that means is I'm part of a small team of people who were senior leaders in it organizations before they joined AWS, uh, generally CIO, CIO, CTOs, uh, who pulled off some sort of a digital transformation, move their organizations to the cloud. And what we do is we work with AWS customers, large enterprises, generally to help them get over the impediments to transformation that are typically non-technical things, almost always cultural change, organizational structure, uh, getting the right skills for their people, figuring out the right investment models and, uh, overcoming bureaucracy sometimes. Um, one of the, one of the topics that comes up a lot when I'm working with customers very often, they'll ask me when they hear about our transformation. I tell them at, at USC I S where I was the CIO, before I joined AWS, uh, us citizenship and immigration services, part of the department of Homeland security.
And I tell them we moved from a release cycle time on the average of about 18 months to deploy code. In some cases, three times a day, four times a day, we, uh, moved into dev ops. We moved everything to the cloud or almost everything. Uh, we started refactoring everything into microservices. We made a lot of changes to how we did procurement and investment management. And, uh, the question seems to often be how long did that all take you? And, uh, I think when I hear that question, um, if the person who's asking it is trying to justify taking a long time and their organization, right. So I love to tell them that actually it took us two days now. Um, I know a lot of organizations can't do it in two days because they don't have the advantage of bureaucracy like we did, but if they did, then they would be able to do it as well.
And, uh, in order to, uh, explain what I mean by that, I'm going to have to give you a little bit of context first. Uh, and by the way, there are a bunch of people who worked with me in the audience here. So if I call them out by name or something, you'll understand. So, um, first, what was the impediment to transformation for us? And it was a document. It was a wonderful document called MD 1 0 2 management directive, 1 0 2, a DHS policy on how, uh, essentially all it investments were going to have to be overseeing and run. So it was essentially the, the SDLC that we had to follow. Uh, and last night I double-check the last version of it. Um, this document MD 1 0 2 called for us, for each system, we created to write 87 and to go through 11 gate reviews and, um, define 21 different oversight roles for people who could say no to things.
So, uh, it was an awesome thing. It called for us to do, um, a S uh, system definition, review that check to make sure that all of the requirements had been written down and fleshed out. And, uh, there was one great, was it like a test in integration review or something which, uh, where you had to get approval to integrate the code in the system and to test it? Um, so clearly this was set up to be a waterfall process. And as we started our agile transformation, we had the idea that maybe it wouldn't fit too well if this MD 1 0 2, which was officially the policy. Um, so, uh, we gave it a little bit of thought. Um, we had a lot of arguments around this. How can we be agile with this, uh, wonderful piece of bureaucracy? Uh, and, uh, finally we realized something which I think is true of a lot of pieces of bureaucracy.
There's usually some sort of exception process, right? Some, you know, the people who wrote it, uh, aren't, aren't dumb, they know things are gonna happen. So, um, there was something in this case called a project tailoring plan, and we could use the project tailoring plan for a particular project to, uh, explain why we were going to need to follow a different process for the special needs of that project. Uh, and of course the special needs for our project were that it was a project and project should be done in an agile way. Right. So that was, that was, uh, essentially the argument, but we, we created this tailoring of the official process, uh, that essentially said, we're going to do the opposite of everything that the process says. And, uh, it's justified because we shouldn't use the process. Right. Um, essentially. So, so you know, that that was, um, kind of a little bureaucratic move that you might want to learn.
Um, now there's a little snag with this, which is that, um, somebody had to actually sign off on our plan and, um, that was a little bit harder to master, but, uh, the way that we did it, another good learning is we took the project that was failing the most and that nobody could figure out how to fix. And we suggested that we use the tailoring for that. And, uh, since nobody else had any better ideas, uh, I think, um, we were able to get an official approval to try it out. So we chose in fact, the project, if you, if you listen to the U S CIS presentation the other day, our, uh, enterprise transformation project. And, uh, that was the project that had already spent about a billion dollars and gotten nothing for it. So, uh, the idea that we might try a different approach was something that people bought into.
Okay. Uh, so without approval, we started doing a scrum process and, you know, doing our a two week iterations, actually, I think we started with three week iterations and, uh, you know, standing up every day for 15 minutes and, uh, retrospecting and all of that, it was all going fine. An interesting conversation took place at that point, uh, between me and the people who were in charge of this MD 1 0 2 thing, uh, where I said, you know, uh, the agile way of running it projects is actually much better than the old waterfall way. So why don't we change MD 1 0 2 to say, do things in an agile way? And they said, well, there's no need, because if you want to do things in an agile way, you could do it under MD 1 0 2, you just need to do a tailoring plan. And I said, yes, but maybe it's better.
If the official policy, doesn't say you have to do 87 documents and 11 gate reviews, uh, and endorse that kind of behavior. So why don't we just change it? And they said, ha, we have to be responsible around here. No. Um, but, uh, we don't see what you're complaining about because you can do this in an agile way. Um, so I want you to remember this conversation because it's going to come back in a minute. Um, so here we are doing our little agile thing and, uh, suddenly another snag arises. And, uh, what that was, um, because of this project was so high profile because we had wasted so much money on it. We were of course, going to be audited by the inspector general and GAO the government accountability office. And, uh, of course we thought since we are now actually producing stuff every couple of weeks, that they're going to say glowing things about us, right. I'm looking at my colleagues in the audience. So, um, they, it turned out, they didn't say glowing things. Um, now, uh, what they did say though, um, was a bit of a surprise, uh, essentially what they, they said, we sat down for a debrief with them and the auditor said you're not agile enough.
And, um, that, uh, I thought required a little bit of explanation since I had been doing agile it initiatives since long before I joined the government. And, uh, as far as I knew, GAO didn't know anything about it, development let alone agile approaches. Um, so I wanted to hear their reasoning and it went sorta like this. They said, uh, you said on your project plan that you're going to do scrum. And, uh, we checked what the practices of scrum are and
Yup. There was a list of practices and they said, you're not doing some of those. And I said, well, number one, don't confuse scrum with agile. And number two, you're supposed to retrospect and do continuous improvement and change your process in ways that make it better. And, uh, we made a few changes around, especially around product ownership, which, which wasn't working well for us, the way it was visualized in the books on scrum. And, uh, and they said, well, you're, you're not doing what it says. And I said, well, agile means you change. And they said, well, you said, you're doing scrum. And we read in the book where Jeff Sutherlin says you can't change anything in the scrum process. So you're not doing scrum anymore, which he did, which he did. Um, and, and so we had to admit that they had us on that one. Right?
So it was a failure that, that we learned from, um, if you think about it for a minute though, I just want to point this out. We had been defeated by bureaucracy, but it was not government bureaucracy that defeated us. It was the bureaucracy of scrum actually in the end. So, all right. So, but this gave us, this gave us the perfect idea because we realized that GAO, when they audited us, they were going to compare us. They were going to compare what we said we were going to do to what we were actually doing. And so all we really had to do was to say, we were going to do the things that we wanted to do, uh, which is obvious if you've got bureaucracy behind you. So, um, we decided we would write a new policy that said, you have to, you have to be agile.
And, um, so, uh, we started to go down that route and we found out that actually, uh, writing policy is difficult in the government. You, uh, there's a, it's actually in law, how you have to go about writing policy and you have to put it out for public comment and all of this, it takes a really long time to do so. Our next idea was why don't we create a management directive, just like MD 1 0 2. And it turned out that we couldn't do that either because a management directive, I don't even know somebody in our group probably knows better than me, but I think it has to come from somebody in a management directorate and, uh, you know, the parent DHS department. So we couldn't do a management directive, but fortunately we had some good bureaucrats on our team and they said, you know what?
You can do a management instruction when, right. So, so we knew that for our, for anybody to take our management instructions, seriously, it was going to have to have a good title, um, sorta like , you know, a compelling title like that. And, um, since it was the first one we'd ever written, we were going to have to come up with it. So we invented M I for management instruction dash CIS, the name of our agency dash O I T uh, the office of information technology dash 0, 0 1, uh, and figured that would make people take it seriously. So we created MIS dash C dash CIS dash O I T dash oh oh one. And it said everybody has to be agile from now on. Um, and, uh, we, we publish it and there was an effective date. So as of that date, everybody was agile. So that part of our, that part of our transformation took one day.
All right, there's the first day. Um, and the, uh, the policy, actually, it was pretty clever. We didn't want to have that same fight with GAO again. So we listed the eight practices that we were going to call agile. It was things like, um, frequent delivery of value and retrospecting and things like that. And then five other optional practices that were sort of advanced agile. Uh, and so, you know, boom, we were agile, um, a little bit later on, we, we learned about dev ops and we decided we wanted some of that and it wasn't going to fit under, under that. So we had to write M I dash C I S dash O I T dash oh oh three, uh, which essentially said everybody's dev ops now. And, um, so we had, we had, uh, transformed to dev ops and that was the second day of our transformation when we issued that policy.
Um, now, uh, there are a couple of problems. Uh, maybe you, you probably didn't realize that there were any problems, but, um, the, the first problem we figured is it conflicted with MD 1 0 2 and MD 1 0 2 was written by somebody much higher in the organization. Um, so maybe that would be a problem. And so my team's worried, you know, uh, a boss, he would keep telling us new, do things this way, but somebody is gonna come in and say, you're not doing what MD 1 0 2 says. And so I said, right, that's an impediment. My job is to remove the impediments, you follow my policy and I'll deal with it. Uh, I was pretty sure I could deal with it because if you remember back to that earlier conversation, um, the, the folks who were responsible for MD 1 0 2 had told me there was no problem doing agile under MD 1 0 2, even though it says exactly the opposite, all you need to do is tailor it.
So I figured what are they going to do? Tell me I can't be agile. They just told me I could. All right. So that was an easy one. Uh, the second little challenge was as soon as I would meet with a bunch of people in the organization and talk about, uh, doing dev ops and being agile, they would then go out into the hall. Yes, he did. Uh, and say, oh, he's crazy. What are we supposed to do? What is this thing? Right. Um, but I figured, um, eventually if they were, uh, going to try to do it, they would figure it out. In fact, uh, what you need to know is that, um, my position is what's called a senior executive service and SES. Um, technically an SES is supposed to be the same rank as a two-star general, uh, cause I wanted to make, uh, an equivalence between military ranks and government civil service.
So, uh, you have to imagine what happens when, uh, essentially the chaos monkey becomes a two-star general, right? This is the situation here. So, uh, when I said, uh, everybody do dev ops, um, it was like, you know, general chaos monkey saying everybody do dev ops. And, and uh, of course the whole team tried their best to do what I said. Uh, we brought in some wonderful coaches, some of whom are here and, uh, over a little bit of time, people kind of figured out how to make this work. And, uh, they started to do these frequent deployments. They learn how to deploy things with no downtime and how to roll back, if anything went wrong. And they built a CI CD pipeline and started doing frequent deployments and all of that. Uh, and then it was time for the GAO audit again. And, um, uh, you know, we knew that, uh, we would pass with flying colors even though in the history of the universe.
GAO has never said anything nice about any thing that they've audited. Uh, but we figured this time we got it. You know, we're doing exactly what our policy said. Uh, and, uh, it turned out that they didn't agree. Um, they said the problem here is that even though, um, people are doing what your policy says, you don't have a mechanism to make sure they are okay. You know, auditors love that. You gotta be able to prove that you're actually compliant. So, um, so M I dash C dash OIT dash oh oh four is the policy that says we're going to have our independent verification and validation team look at what all the dev ops teams are doing and make sure they're doing dev ops. And, uh, that was pretty brilliant, right? Um, they expected us to have a policy that said, you know, like if somebody, if they find that somebody not doing dev ops that are going to take them out and shoot them or something.
And instead our policy said, if they find somebody who's not doing dev ops, then they'll work with them to try to help them be better at dev ops, uh, which was a little bit of a surprise, um, to them. But anyway, we now have this, uh, this policy. So we had a policy that said, everybody's gotta be agile. We had a policy saying, everybody's got to do dev ops. And then we had a policy saying, we're going to check to make sure that everybody's doing dev ops. All of were beautiful pieces of bureaucracy that had compelling titles to them. Uh, and we can do all of this because we had bureaucracy behind us. It was gorgeous. So, um, obviously then it was time for me to leave the agency because you know, my work was done. Right. So we were there.
So, um, uh, a couple of the things I want to sort of step back and take a few learnings out of this. Uh, and the first, the first point that I want to make is everybody's got bureaucracy. Uh, it's not just the government, every company that I talked to when I mentioned bureaucracy, they say, oh, come on, we need help with that. You know, uh, it's actually going to be the topic of my next book. Uh, and every time I tell people what I'm working on, they get all excited. We could use that. We need that one. Um, so I'm going to be writing about bureaucracy, but everybody has got it. And, uh, in fact, as I said, uh, it was the scrum bureaucracy that killed us. Uh, it organizations are great at creating bureaucracy. We make standards and then we make a process for making sure people are following the standards and an exception process for if they don't want to follow the standards, um, you know, requested in triplicate and send it here and send it there.
And then, uh, we've got this DevOps thing, which is, uh, in many ways, the extreme of bureaucracy, um, think about, think about this. Um, one great piece of bureaucracy we used to have is after you finish coding some, some new capability and you want to deploy it, you had to go to the cab or the change control board or something, and you had to submit all this stuff and they thought about it. Uh, it was always, you know, a group of people that really had no way of knowing whether you should deploy this or not. Um, but what they would do is they would look at the paperwork and they would say, uh, yeah, let's check the test reports. Oh, it looks like they ran all the tests and all the tests passed. So I guess that means they can deploy it, um, you know, beautiful bureaucracy, right?
That's, that's really a bureaucratic structure to it. Uh, and that was how we used to decide on whether something could deploy. What do we do in dev ops? Exactly the same thing, except it's automated, right? We, uh, set up the pipeline so that you can't actually deploy, unless all the tests have passed and it's gone green. What's the difference. It's, it's really the same concept. Concept being let's create rules to encapsulate good practices, and let's enforce those rules in every case, no exceptions allowed, unless you, you know, fill out the form in triplicate. So, uh, in a way dev ops is all about automating bureaucracy. Uh, this, uh, I'm not saying anything negative. When I think about what it is about bureaucracy, that drives us crazy. Um, I think it's not the nature of bureaucracy itself. It's two characteristics of bureaucracy. The first one is that often we see bureaucracy that is not lean it's wasteful.
So it says, if you want to deploy a piece of software, you have to write 87 documents and go through 11 gate reviews. What's the problem with that? The problem isn't that you have to have some structure. The problem is that 87 documents is ridiculous. It's just extra work that you shouldn't have to do when you have to fill out the forms and triplicate and take them from window one to window two and window three, it's just extra stuff. And it's really frustrating because it gets in your way and you keep feeling like, just let me deploy. You know? Uh, so the fact that bureaucracy often is not lean is the first thing I think that drives us crazy. The second thing is that bureaucracy tends not to learn. So you have a set of rules and that set of rules doesn't change very fast.
Uh, in fact, it changes slowly. So MD 1 0 2 perfect example, here, here are a thousand books. Let's say it's better to do things in this agile way. You're going to get better results. Uh, by the way, the waterfall way has never actually worked. And if you look at your portfolio of things that you're overseeing, you'll see that every single project is failing. Uh, so maybe we should change MD 1 0 2. Nope. Uh, bureaucracy tends not to change, but those two characteristics, the, the waste and the resistance to change. Those are not actually essential to the idea of bureaucracy in bureaucracy. You apply rules uniformly to everybody without exceptions. That doesn't mean that the set of rules that you're applying can't change. And there's no reason why you can't take a bureaucratic process like MD 1 0 2 or any other one, and look at it as a value stream and look for where the waste is in it and reduce that waste either shortening steps or taking waste down to them or taking steps out entirely.
Any bureaucratic process is subject to exactly that same approach as we do with any lean approach to a process. So there's no reason why a bureaucracy has to be wasteful or unchanging. You can have lean bureaucracy and you can have a learning bureaucracy. And in fact, when we set up this, uh, am I dash C I S dash oh dash oh three, we deliberately set it up to be learning bureaucracy. Um, in the sense that the, the policy itself did not define what anybody should do. It define the objectives that we were looking for, frequent delivery of value, et cetera, et cetera. And then it said, the practices we want you to follow are in appendix a and appendix a has what we think today are the best processes or best practices, but we expect that appendix a is going to change over time as we learn new things.
And so we deliberately separated these two concepts of the outcomes we were looking for and the best practices we were going to use to get there so that we could keep changing those best practices as we learn new things. W why can't you do that in a bureaucracy? Right? This was a, I mean, okay, it's a little bit arrogant, but I think this was like the best pure piece of bureaucracy ever written. This was art, right? So, um, I, I contend that you can actually take bureaucracy when you encounter it, and you can take advantage of the little loopholes and exceptions and so on. Uh, but what you can also do is take that same lean approach to bureaucracy that we take to all of our, all of it. These days, look at it as a value stream, yank waste out of it to reduce lead time and just keep doing that.
And at the same time, establish a feedback mechanism so that you can learn as you go, and you can alter the bureaucratic rules based on your learnings. And, uh, in a way, this is what we're doing in dev ops, where we take what could have been a slow wasteful bureaucracy and automate elements of it and eliminate elements of it. And, um, set up ways of establishing compliance through automation and through artifacts that are generated automatically to demonstrate compliance, uh, and things like that. So, um, bureaucracy to me is not, um, and, and, uh, immovable impediment, uh, it's something that sounds terrible. The words are terrible. Um, we use the words to describe anything that's frustrating to us, but the words, bureaucracy and the fear of having to deal with politics and, and controls that are coming from outside yourself. That's just a fear. It's not an actual impediment.
It can be removed as an impediment. It can be worked through, um, it can be altered and improved, and even used as a way to establish controls where you do want to establish controls in cases where you want to have standardization, let's say, or centralized control shared services. Uh, there are all these applications in the world of it, where we want to put some structure and some formal process around things. But when we do that, we have to be very careful to make sure that we're not creating bureaucracy. That's going to resist change, and that's going to add a lot of costs and a lead time as people try to follow it. So, uh, my concluding message is that bureaucracy painful as it might be. And as much as you might feel like it's getting in your way is actually something that you can tackle and you can work with, and you can use bureaucracy to get better impact just as we use bureaucracy to actually drive a transformation strangely. So good luck to everybody with your .