00:00:17

All right, the next, uh, one of the next speakers is, uh, John Ky. I met him in 2016 because something, uh, remarkable that he did. So he attended this conference in 2015 as one of, uh, uh, their SVPs and Director of Continuous Delivery and feedback at Key Bank. And after that, he did something so utterly jaw dropping and amazing that blew away the entire programming committee. So he shared that story in 2016 on the plenary stage. So I love those stories because it just shows how quickly ideas within this community can be shared and how quickly similar outcomes can be generated. So, I mean, this is what I think is one of the key marks of an incredibly productive seniors, uh, which I talked about yesterday. Uh, he later presented with in 2017 with Stephanie Gillespie, who led the digital programs of Supported Keys retail bank. Uh, and he also spoke in 2019 about this crazy idea about, uh, automating technical governance. So since then, he's moved on to P N C Bank serving as their SS v p of technology infrastructure. And I'm so delighted that he's here to speak about where he's taken those ideas, which is at the heart of the Investments Unlimited book, uh, which he co-authored, and he will be co-presenting with Mic Michael Edinson, one of his fellow co-authors. Here's RES and Michael.

00:01:36

Awesome.

00:01:39

Thank you guys. Okay. Um, so we're gonna go fast and furious because, uh, otherwise they're gonna turn off our slides like they did Jason yesterday, <laugh>. Um, so real quick, I'm John Rez, senior Vice President of Infrastructure at P N C. And this is, I'm Michael Leadings

00:01:55

And with FI Labs.

00:01:56

All right. So Mike and I have been working together the last three, four years at P N C, and we've done some amazing transformations, a lot of awesome work out there. But one thing that's incredibly close to our hearts is this, uh, uh, idea of automated governance culminated essentially into us being the co-authors with some awesome other co-authors in Investments Unlimited, which we're really excited about. So, the story we're gonna kind of tell you guys today a little bit is this, uh, what happens at the end of Investments Unlimited, right? So the, um, uh, uh, think about the fact that the transformation occurs, something big happens, but now you've got to adopt and you've got to onboard, and you've got to be able to get the rest of the organization to buy in. We're gonna use some of our own personal experiences. We're also gonna use a lot of the research essentially that we did in doing the book.

00:02:42

Quick disclaimer, we heard the Law of Mobility yesterday. This is an enterprise conference. So we're gonna quick create a quick addendum to that, which is with Words and Things. So, some of you guys might relate to this, but if you're in a meeting and you're asking yourselves, is this meeting about words? And if it is, you might want to consider to utilize that law, we also want to end every single meeting with Let's go do the things. So let's go do the things. So why now, right? Uh, we've been talking about 20 14, 20 15 enterprises have been doing this movement and have been carrying a lot of the DevOps, uh, capabilities across, uh, the industry. Uh, but when we go back to think about it, right, these, these original things came out of, uh, big tech companies that were really focused around building the big web, and they got a ton of awesome benefits about it.

00:03:23

10 deploys a day at Flickr, et cetera. One big problem was regulated industries. They got to take advantage of a lot of it, but they didn't get the value that these big tech companies were getting because they have millions of lines of defense, millions of problems that they have to solve, that there really is never, there's not really a great playbook for. So we also come across a lot of people that say, well, we do this already, right? We do secure coding standards, we do pipeline integration, we have automatic change record creation, and we have a million tools that, that go out and do this, right? And, you know, our argument is a little bit different, right? Um, this is a big transformation. This is gonna be reaching across the aisle, talking to a lot of different teams and a lot of different groups. You're gonna have to build Providence build for 10 different runtimes.

00:04:04

You're gonna have to work on distributed, non distributed systems. You're gonna have to deal with a ton of legacy code bases. This is not a cookie cutter solution. So whatever you do, don't go buy DevOps with a side of automated governance by any means. So, like I said, Mike and I have been working together for three, four years, and we essentially had this main goal to do continuous delivery at a large financial services institution. We want to do 10 deploys a day, and we, when we sit, when we step back and kind of looked at it, right? We've got a lot of procedures, a lot of policies, a lot of best practices, a lot of things that we have to follow. A lot of those turned into requirements for something that we needed to engineer. And so, if you think about it, basically we're sitting in a conference room and a voice calls out into the speakerphone, Hey, if you build it, they will come. 'cause engineers and developers absolutely want this. Like,

00:04:49

Yeah. So for those of you that have seen the movie Field of Dreams, Ray Kinsella hears the voice, uh, in the cornfield, but our voice came to us in a conference room and it said, if you build it, they will come. So we asked ourselves, well build what, right? What are we building? Um, and so we had this vision of a superhigh, and this super highway would allow a developer to go from new code to production without any human intervention, right? Without any manual work, fully automated. And I've, I've talked to some of you that are, that are at these cutting edge tech companies, and you say, well, we've been doing this for years, right? This is something that we do. Yeah. And that's true, that's true. But for those of you that are in regulated institutions, you know, like banks, defenses, contractors, uh, healthcare companies, I think you'll know that there, there's a lot of manual processes that have been built up over the years all related to regulatory requirements. And that makes it really difficult to do this in an automated fashion. So when, you know, when we had this idea, we had this vision, we asked ourselves, well, how are we gonna do it?

00:05:46

So where do all great stories originate a paper from the DevOps Enterprise Forum. Uh, in 2019, John Willis got to pull some of us together. We got to write essentially a ton of things around the controls. We got to talk about exactly what, uh, uh, uh, evidence that cannot change ephemeral evidence, what that really meant, and how you needed to quantify it. And we got to talk about a lot of best practices associated to what we needed to build. So this really gave us our blueprint.

00:06:11

Yeah. And so when we were talking about, you know, the automated governance reference architecture and what it would mean in its final state after we've built it, I think John and I looked at each other and we said, wow, this is gonna have a huge impact on the developer experience, you know, at, at our company. And so I think we had to be really careful and say, well, we don't want this to fall into the wrong hands because, you know, this is gonna give a lot of power and it could improve developer experience, but if used in the wrong way, it could also make developer experience a lot worse. So in the beginning, we set forth on our beliefs, and these are beliefs that we derive from all of you here in this community. And I think they're beliefs that we all share, and just a few of them will list and, and, you know, and how they, uh, pertain to what we were doing.

00:06:52

But our first belief was developer autonomy. And the way that we saw that was developers should be able to have autonomy and the freedom to choose things like your i d e, your build pipeline, your builder image, even things like the ability and the freedom to prioritize their own work sets with the business free of any administrator oversight from someone that may not be on the team. Uh, secondly, we believed in carrots instead of sticks. And I don't think I need to explain this one, you know, too much, but we believed in positive incentives instead of punitive justice. And lastly, we believed in no downward pressure. We didn't wanna do anything that put an undue burden on the developer experience.

00:07:35

So our strategy was to create a simple shared language, right? We're gonna take the, the principles of the automated governance reference architecture and boil it down to a very simple shared language. So take a given control, you know, in the development process, it can have four states, right? Pass, warn, fail or not found. Passing means you're good to go fail. Obviously you're not, and not found means we don't have evidence for that control, so we're gonna deem it as as a failure. Now, when we, uh, we built this product and we rolled it out, right? And we were really excited about it. We said, we built it, we, we, we built this super highway, but the problem was, is that we built it and they didn't come. So we said, well, what's that all about? Right? And the reason is that onboarding failed. And when we, we dug deep, or we realized that onboarding failed because you can't onboard what you can't see.

00:08:26

And this challenged our first belief, and we had to make a choice. Do we promote autonomy or standardization? And I don't wanna make it seem like these are a binary decision. I'm not saying that you can't have standardization while still preserving autonomy, but in our case, we really had to look at putting in some serious guardrails because, you know, at this point in time, developers could have any permutation of build patterns. And it made it impossible for us to see what was going on, and we couldn't onboard what we couldn't see. So we standardized, we actually used our automated governance tool to standardize, and we could talk about that maybe in a different conversation, but we standardized and onboarding took off really quickly. And so John looked at, uh, you know, we looked at each other and we said, wow, all right. You know, we're, we're on the right path now, but then really quickly it fell flat.

00:09:11

Um, and that's because compliance didn't improve. So we had all this great observability we're we were producing these immutable attestations, and we were giving great continuous feedback to developers, but they weren't using it to get compliant. And if they weren't getting compliant, they couldn't do automated releases. And so the automated releases flatlined, and that was our entire goal. So we dug deeper and we realized that what we sold was really different than what the developers saw. We sold a superhighway that was driven by compliance. And what the developers saw was the reality of software development. And that is that their apps were not compliant. We had a long way to go as an institution to get these applications to the point where they could be released. And so <laugh>, there's a bit of a disconnect that we realized in this process. We had to decide, you know, is it feed 'em a fish or teach 'em to fish? And us, as the proprietors of the data that were given to the developers, they thought that we were responsible for remediating the compliance. And we had to teach them that, no, you know, you own your own compliance. Only you, the development teams can be responsible for your applications. So in, in summary, our first carrot didn't work, right? And we had to try something different. They weren't using the carrot of automated releases. So we said, well, what about scorecards? Right? It seems

00:10:32

Pretty simple. Horrible idea, horrible idea. Well,

00:10:34

<laugh>. So yeah, the scorecard would be pretty simple. We take controls and we bundle them together, and then we say, all right, a, B, C, D, F, you know, we're gonna aggregate and then we'll give you a score. And we talk about scorecards. We can't avoid showing re's favorite movie

00:10:48

<laugh>,

00:10:51

But the scorecard gave us this other question, which is, what should the data be used for? Do we, do we package this data into these scores and throw it over the fence and say, don't talk to us until you're compliant? Or do we give all the troves of data over at which point we risk someone misusing the data for a game of gotcha. Because all a scorecard really is, is an abstraction. You know, it's packaged information, it's in the aggregate, and it abstracts the details that you can make a directionally accurate decision as to how compliant something is. But the problem with directionally accurate is it's not sufficient for automated governance. We need granularity in our data. So when we provided the details, um, it presented a different problem. And that is, you know, with details, great power comes great responsibility. And there was a huge risk of the data being taken out of context and being misused. And I think John has a a good example of this.

00:11:46

Yeah. So you gotta remember, um, you're gonna be reaching across a lot of aisles with working on, on something, on automated governance. And a lot of the groups you're gonna be working with, they're not technologists, they're not developers, they're not engineers. And so when you do things, when you say things in procedures and policies, it can have a very, very ill and negative effect. Some of the research we found when we talked to a couple other organizations, things, something as simple as a unit turning off unit test during a build process essentially is an infraction of a policy. In a couple instances, we found essentially that's termination applicable. Um, so you really have to think very carefully about applying things like this. Uh, Jason said something yesterday, I really hit, uh, home with me, which is this idea of proximity. Um, and this is exactly, you need to stay very close to the risk units that are gonna be enforcing these procedures and policies that you're going to write. 'cause this is incredibly, could be incredibly detrimental to your culture and, uh, your development community.

00:12:41

But in short, you know, we tried the scorecard and it just didn't matter. It didn't make a difference, right? There was a nominal increase in compliance, but it, it got us nowhere near where we needed to be. So we had to try something different once again. And this challenged our third belief of no downward pressure, right? So <laugh>, when we asked ourselves, we have the mother of all downward pressure, which is, we've got this data and we can start blocking deployments and, and just interject ourselves in the release process. But, but then, and I'm sure some of you'll relate to this, you know, you go to release and someone uses that opportunity to tell you, oh, I found all these new vulnerabilities you can't release, right? So this downward pressure is the ability that someone has to interject themselves directly in the workflow of a developer, because that's the most convenient time to get the developer to do what you want 'em to do, right?

00:13:28

And when that adds up, it becomes a real bottleneck. So we said, are we sure that we want to do this just because we can? Does that mean that we should? And our belief came from Sidney Decker and just culture when he said that anything that puts downward pressure in your organization on honesty, openness and learning is bad for business. And we believed that, but we decided to do it anyway. So we put in our automated enforcement, we alert you throughout the build process, throughout the pipeline process, and then warn you when it came time for you to deploy to a lower environment, and if you still went forward with your production deployment, we'd block you. I think John's gonna talk about a little bit, you know, why we decided to do this.

00:14:11

Yeah. So I mean, this really hit our beliefs, right? 'cause this is not something we wanted to implement. This is not something we wanted to do. Um, we wanted to build a high trust organization with, uh, many, many types of things. And so we really had to convince ourselves, well, well, why is this the right way to go? Um, and, uh, and, uh, Andrew Clay Schaeffer use always uses this picture in a lot of his slides. And he talks about, uh, finding the thing that changes the game, right? So in the Gatling gun is, is the example he always gives. And, and it essentially, it it's very applicable and contextual to whatever situation you're working on. So we tried to kind of run this theory out about game changing. 'cause we thought that our game changer was automated governance within our organization. And when we look at, uh, players in this type of a game, we're gonna have, you know, product owners, developers, engineers, and then governance risk and compliance on one side.

00:14:53

And if we think about the optimal outcomes for these, uh, two different groups, if I'm a product engineer, if you read the book, bill Lucas is obviously the product owner that says, we wanna do features, they're the most important. They're wild, wild west. That's what we want to get done. Governance, risk and compliance. The optimal outcome is a very lengthy review process. We are gonna check this thing to death before we actually let it out there. Um, not surprisingly, the not optimal outcome is the exact opposite for these two players, which is why we're, we're, we're stuck in this game that we're playing. So what does the industry do? The industry does exactly what they, they fall back to, which is a very subjective change process, right? So I'm essentially going to fill out a mortgage application as my change record, uh, with all that documentation.

00:15:35

And then I'm going to, uh, you know, try to get through my approvals depending on who likes me that week. Hopefully I can get my change approved and it's gonna get into put some type of change window. And the ironic part that Michael and I were just so struggling with we're like, well, no, why isn't automated governance the optimal output for these two parties? It ended up being the not optimal outcome for both of these. Um, the reason for that is development and engineers. It was yet another thing they had to implement, just add that to their backlog. And on the G R C side, it's just looks small batch release. That's not an easy thing to swallow for, for a governance risk and security organization to be able to producing change that often and frequent. So, um, the prisoner's dilemma kind of teaches us essentially that the not optimal outcomes for the two different parties playing the game is actually what's best for the entire organization. So it really actually fit. And so now we gotta figure out, we need a game changer for our game changer. How do we change this game in order for us to be able to get buy-in for the not optimal outcomes for these two parties? And that ended up being automated enforcement because it absolutely removed the other outcomes that were even possible for these two players and forced everyone down. They're not optimal solution.

00:16:45

So real quick, we're just gonna hit some tactics. You know, we start with simple controls and soft gates, meaning that failing teams will get a warning, Hey, you know, coming, coming up soon, we're gonna start blocking you for this. And then really quickly we move to put in the hard gates. And we recommend that because we wanna send that shockwave through your organization so that people know, you know, this is now how things are gonna be, but then quickly move to more, uh, difficult gates all the way up into your most difficult, uh, controls. And I think you'll see that your developers are gonna make the adjustment very, very quickly. But I will warn you, and that is that we did not make any friends in this process. <laugh>,

00:17:24

Mike has to walk me to my car.

00:17:25

Yeah, exactly. Um, we're not trying to scare you, but we just want you to know that this is something that's not gonna be, uh, a favorite amongst, uh, your developers at first, right? But we think it's worth it because what we saw after we implemented this and after the developers started to change their behaviors was, was pretty extraordinary. We saw a dramatic uptick in engagement. And that was, was really encouraging to us because engagement to us meant that developers were starting to care and they were starting to take ownership of their compliance. They were trying to figure out how do I, uh, remediate this failed control, right? And then they started helping each other, which was another thing that we didn't expect. So it created a sense of community, and that engagement was encouraging, and that's when we knew we had it. But also, there were a lot of stakeholders in the beginning of this process that were really worried that we would cause production incidents by blocking deployments because teams would miss their change window and it would have a cascading effect.

00:18:22

And in our research and in our experience, we found zero instances of that happening. These teams took it very seriously and they know what they need to do to make sure that they don't cause these incidents. So we, we tell you all this to say, if you're gonna try this, go into it with a lot of confidence. Know that you're not gonna be a fan favorite, but that it will work. And most importantly, we reached our goal, which is that compliance shot up and automated releases followed soon thereafter. So we were able to realize our vision by pretty much abandoning all of the beliefs that we held in the very beginning.

00:18:59

So, um, the regulated industries are inherently consequence based models. Those consequence based models go all the way down into the application teams and it kind of permeates or, or, uh, essentially drives through the organization. Um, so we're gonna, we understand that a lot of people aren't gonna agree with some of this, right? In fact, we've had people tell us <laugh> very much to our face that this is the wrong model. You're doing the wrong thing. Um, so I think it's important to understand the contextual nature of what you're trying to do and, and what industry you're working on. But we would love to hear other stories associated to adoption and onboarding something like this in your industry and how that, and how that is accomplished. So hopefully you guys learned a few things and we can all go do the things. Thank you guys.

00:19:55

Thank you.