00:00:02

All right. So believe it or not. Uh, during the two thousands, I was, uh, due paying member of the Institute of internal auditors for almost a decade. I loved auditors because of the clarity and precision in how they spoke about controls, which is interesting because today, without a doubt, one of the biggest obstacles voiced by this community in terms of who's in the way of implementing better ways of working of which dev ops is certainly one of them are the auditors, whether it's internal auditors, external auditors, or the external regulators. So thanks to the tireless work of Sam Guckenheimer from Microsoft, Dr. Topo pal, after over four years of work, we have finally been able to get representatives from each of the big four firms here to help us bust some DevOps myths. Uh, as I mentioned yesterday, uh, these are people who support the audit teams.

00:00:50

I suspect, uh, that when you meet these people, um, that they're not going to be like any auditor, uh, that you've ever met. Instead, I suspect your reaction will be, wow. They're one of us, the boundary spanners. They all come from a technology background and they've all been working tirelessly within their own firms, um, to help their audit colleagues understand why dev ops is important, uh, and how dev ops could be used internally within their firms as well, as well as helping their clients use DevOps safely and educating the entire audit risk and community. So I am so grateful that they are willing to spend two days with us and help us learn how to better work with auditors. How do we engage them with them ideally and end up with FIBA outcomes, both for us as the auditor's audited, as well as for them as the auditors. So we're going to be playing a game called busting DevOps MIS on the, I'm going to describe on the slide

00:01:50

On the slide will be a title saying busting dev ops myths. And so we're going to state a myth and then we're going to ask each one of the panelist, is it busted or confirmed? And specifically we're going to try to, we're going to eliminate plausible, right? Because we don't want to hear as it dependence. We want to actually hear a specific guidance that you can take away to better arm yourselves in your next interaction, uh, with, uh, your own auditors. So with that, I would like to introduce our distinguished panel, please come on out. The auditors are here. All right. Uh, so if you could just, uh, briefly, uh, each introduce yourself and maybe say whether you're in the assurance or advisory side and maybe tell us why that matters.

00:02:53

Sure. You'll say Levine from a Deloitte, I've actually never been invited on stage and say said that I was an auditor and the people clap, but there's a first for everything, right? Absolutely. I am a Rouge, true financial auditor that, uh, kind of got into the technology space a bunch of years ago.

00:03:13

Hi, my name is Mike Wolf, uh, from KPMG managing director of our modern delivery practice. So I'm at the advisory side. Um, but I was a, a highly evolved developer who one day figured out if I really want to get stuff done, I need to make friends and, uh, with the audit side. And so I ended up being a translator internally and with our customers is how we tie these things together. Awesome.

00:03:33

Hi, Jeff Roberts with EOI and my background is as an it auditor and specifically for public companies, but over the last few years, spending time helping companies modernize their it compliance space to help as they, as their rapid pace of development in their organization.

00:03:49

Hey, I'm Matt bonds. I work with PWC. I started out actually as a financial auditor, um, and over the last, um, kind of 10 years have moved into the it audit space and then have been focused on, um, agile and dev ops and specifically the audit response to, to DevOps, both helping our audit teams understand what to do as well as helping our clients get ready for their auditor's

00:04:10

Awesome. And by the way, when I say that they all have a technology background, I've I sort of knew that, but I was actually a little bit surprised that we were all geeking out over Scott Haven's presentation, critiquing his CQRS event, sourcing patterns. That was really awesome. It was nice to know that within each one of these audit firms are technologists who are deeply steeped in technology. All right. So let's go to our first, first myth, which is audit will never allow dev ops to happen busted or confirmed

00:04:40

Busted fully busted.

00:04:42

Right? Can you maybe substantiate that claim?

00:04:46

Well, if you don't have dev ops, you don't have business transformation, so you got to figure it out one way or the other.

00:04:53

Yeah, I, and I would say that in general, why this is fully busted is our audit teams, our internal it audit teams are embedded in to these DevOps teams consulting. I mean, I seen risk, uh, team members submitting a compliance bug in JIRA, right. And then not being prioritized on the backlog of activities. Um, and I'm seeing that in customers all day long

00:05:19

And Jean, I think the big piece is trying to change the question as well from, you know, are we allowed to do this to how do we do this and how do we do it? Right. And I think that's a challenge to everybody in this room is stop asking permission, but start asking how, and, and how do you get that conversation started with your internal compliance folks, whether it's internal audit or your external auditors to really figure out how to do this and do it well,

00:05:42

This is going to be unanimous. Yeah. Yeah.

00:05:45

I'm going to expand on that. One of the things that I do a lot of is sitting walkthroughs that are early on in the design stage for pipelines, um, where they're getting me in with as part of the audit team or part of an internal audit team to say, Hey, this is what we're thinking. This is, this is where we see the control sitting. Um, can you give us some early feedback before we spend time building and actually getting that input right at the stop?

00:06:06

All right. I think we can say this is busted out. Awesome. And maybe I just want to acknowledge, you're helping, uh, doing a trial of this in the London event. And you specifically were very explicit that you are using DevOps principle practices to, uh, for applications that support the ATIs practice. Correct. So if clients rely on your judgment, they are interned, relying on applications that are using dev ops. So clearly it is consistent with all the internal controls,

00:06:31

Definitely consistent. I think our approach to manage our own internal systems that we use to serve our clients, where we're a data storage for our client's data. We're, you know, we're practicing what we preach. We're self imposing elements of COSO and some elements of Sarbanes-Oxley on ourselves to make sure we're doing things appropriately. And to the point that my colleague mentioned that you embed technology controls and security into your dev teams by design. And if you're an auditor compliance by design is a really, really cool concept. And you as dev ops aficionados have to really get with your auditors and explain to them how it can make their life easier.

00:07:07

Awesome. Any other comments on the sub or Astros or footnotes on this, or what are we ready to go onto the next myth? Next myth. Yep. Change approvals, segregation of duties and access to production controls, prevents code from ever being automatically deployed in their production busted or confirmed. Totally

00:07:25

Busted.

00:07:26

Totally.

00:07:27

Alright. Then when brave enough to substantiate that claim,

00:07:30

I'll take parts of it. There's I mean, segregation of duties as a really important part, but if you lay out for somebody, something like get ops and you, and you lay out exactly the access controls to do that. I mean, myself as a recovering developer, we used to talk

00:07:45

About,

00:07:47

Okay, it's, you know, taking it day by day. Um, so we could ask for it. Um, we used to talk about, oh, the importance of code reviews. I would say nothing has enforced that more than a pull request that they can fully trace and go, oh, X amount of people were involved in that. Right. Let alone rolling it down to something like documentation. Right. Which is also a massive thing. And I often say, which would you rather look at an encyclopedia Britannica from the sixties that said, man, one day we'll land on the moon, or do I want to look at Wikipedia? Right. And the speed of update and governance across that stuff. I mean, all of these things is really about a properly put together tool, chain and process. Um, it really actually is the glue or a guard guardrails that ties all this together. Yeah.

00:08:33

And then certainly the way we're seeing it is that secured pipeline, um, is going to allow ultimately for promotion into production. But there's a lot of caveats to that. It's about controlling access across the pipeline. It's about making sure that if I'm being controlled, quote unquote by the pipeline, that I can't go in and change the configuration within one of the tools or bypass a control or update a test script so that it, um, you know, so that it's going to pass my nefarious or just bad code. Um, it's about having processes in place to make sure the test scripts are being kept up to date. Is it introducing new features or, or, um, or working on bugs that all of those things are being changed across the pipeline. And then those in turn helped to create your segregation of duties because it's whilst the person may be doing a one-click commit it. Other people have controlled, um, and set up the configurations along that pipeline and the person doing the one key click commit, can't go and change those.

00:09:25

If you think about the power of the different resource groups, you can put into your, into your deploy capabilities, you could bring in security, the devs, you know, all the different competencies that you need, including the business users. If you pull them in, they are now a part of deployment. So you're bringing that whole lifecycle and everybody together. Yeah.

00:09:44

And from, from that audit point of view, the moving away from those historically manual controls to these configuration tooling enabled processes is so much more powerful, both for you from an engineering standpoint, but also from us as an auditor, it's much easier for us to test and rely and get comfort around that system design than simply these manual human, human driven processes that do have a lot of risk.

00:10:07

And I think there's a myth kind of hidden in this myth where you're saying, you know, automatically deployed into production, much like a previous speaker talked about data definitions and a type of governance over that data. There's certain things risk-wise that we may still want to go to a manual acceptance, but in the pipeline, I can see that that change was requested who approved it. And then after somebody approves it, then cool, let the machines take back over. And,

00:10:33

And that audit trail is exception monitoring. So that leads to, you know, really the new era of auditing where everything is hard-coded and you're managing exceptions. And that is the new audit trail.

00:10:44

And by that, I just want to confirm, just validate this. You're not saying this just to make new friends, whereas everybody, these are what you say to actual audit clients and the advice you give yep. A hundred percent. So I'm busted or confirmed. Plus, are we unanimous round of applause for our auditors, by the way, for all right. Next myth, developers aren't accountable for any portions of the compliance and quality objectives in an audit busted or confirmed.

00:11:15

It's less busted say more. So, you know, what we've taken upon ourselves is that every time there's a deficiency and there's all different types of deficiencies that turn it, turn it to PBS. There has to be somebody who's held responsible for getting that in the next pipeline and learning the next deployment. And obviously you rate them based on severity and where they're, they're going to go in, but you have to have accountability for all of these and monitor them and age them in a holistic risk management lens.

00:11:45

Yeah. I think as we've progressed to creating these full stack teams, right at first, the question was, is QA the responsibility of developer. And we changed it. We said, actually, unit testing, a behavior driven test driven development, this all of a sudden changes, we're all responsible for quality. And then it was security, right? We're like, oh, that's the security teams. And we're like, actually, it's all our responsibility. We knew. We now need to embed that. I think this is that last or one of the last silos there to break it down and saying, actually this compliance is a non-functional requirement. Right? If you think of a developer, I'm thinking about all the other things I have to do, this is one of the non-functional requirements that we now take on and we own as a full stack team.

00:12:27

Yeah. And the, just the, the prior view of compliance to that is so much as a business blocker, right? As a stage gates that you can't go forward and you can't get to the speed and velocity that you're looking for. And, and it's really, how do you change that conversation to build compliance into your thinking into your process, into your pipelines so that it becomes part of your everyday operations. So that it's, that it's an enabler for your speed, not a, not a blocker to that.

00:12:55

Yeah. And I mean, just to kind of pile on everyone, um, I'm all for shifting as much lift as possible, um, compliance and security by design the earlier you can be doing it in the process the better, um, and you know, to the point where what we're encouraging our clients to do is, is right at the start. Think about what those non-functional requirements are and build them out as user stories and have them go through the dev process, the same as everything else.

00:13:18

So I think the consensus is unanimous. Again, this myth is busted. All right. Another round of applause for all the way, is this helpful to you in your own mission? How many people here might be showing this video to your compliance specialists or your internal audit teams, maybe even, uh, your external audit teams. All right. Perfect. And we're going to have, talk about ways that, uh, these professionals can actually help you in further, uh, in that mission. Alright. Next myth devil processes can be made audible traceable and secure busted or confirmed confirmed Though. Could you say more and substantiate that crazy flame?

00:14:03

Th th the best thing about dev ops is everything is audible and traceable. The question is where's it hidden? Where's the data point and how do we go about creating an environment that we can get that evidence put to our fingertips when we need it? So, some of the steps that we've done is we've created, you know, free, you know, most companies have a level, one level, two level, three compliance for level two compliance. We've given, read only access to, you know, a lot of different environments. So the auditors could go in and actually see things that are rendered to them and do follow up.

00:14:33

Yeah. I would say that the, where this becomes a problem isn't that it inherently isn't auditable and traceable, actually, you have so much data in that system. They exhaust coming out of it is noise is noise being deliberate and creating golden pipelines and golden containers that have already been blessed through compliance. And then all of a sudden motivates other people to say, oh, I can skip some, some agreement and alignment because these exist. I think that deliberate nature of creating a truly connected ecosystem makes this by default the default action, right? And to your point, the exception is, okay, we need to get out of that for X, Y, or Z, but this becomes a default golden path, but you have to be deliberate using the exhaust, makes people question being deliberate and laying out the trusted path gains a lot of competence.

00:15:25

And so I think a little insight into the way your auditor's think is that your auditors generally are afraid of the unknown and in this space, there's a lot of unknowns and they've been trained to, to not think the way you guys do around what could happen w what, what positives could happen instead of the negative, right? What could actually go wrong? And it challenged back to you guys of how do you bring them along on this dev ops journey, right? Helping them understand that you've thought through your process, that you understand the risks that you've thought about what you're going to do to address those risks really will help that audit conversation, that risk and compliance conversation move forward. Because there, there is a lot of unknowns from them. And the more you can share, the more you can share your knowledge, right? The better those conversations.

00:16:11

Well, let me just throw something to the audience. How many people here engaged in dev ops have ever engaged in a risk discussion about a particular application released by a show of hands. Okay, Jean, I mean, I know I'm a bean counter, but probably 30%, right? Why not a hundred, 104 point. Yeah. About 33rd. We're going to deputize you as an auditor, but you know, one thing that we've done is for every minor, send me a major, major release. However, you classify your different releases, go into a holistic and a risk discussion with all the relevant stakeholders and understand what are the risks of this, of, of these PBAs. What do you have to think about? Is there any upstream or downstream impact any connectivity to other subsystems that, you know, may have a quote-unquote issue that we didn't think about going in? So having these discussions really helps, you know, from a risk based perspective to understand the impact of your different releases from PBI lines.

00:17:09

Yeah. Awesome. And, um, I just kind of add to that too. I would argue that the target end state for dev ops is going to be really auditable. Um, there's going to be automated controls. There's going to be a lot of great stuff in there, but in that transition phase, when you're moving from kind of the old fashioned manual processes into this dev ops phase, um, state, and you've got, um, you know, some automated tools to manual a manual pushes or that kind of thing, just really be really mindful of that. And that's going to freak your auditors out, um, because they're going to, they're going to hear, yeah. Okay. There's automated controls. Can I test them well, not quite yet. We've got some work arounds and we've got some manual approvals here and some automated approvals there. And so just really having a clear view of not just talking about that target end state, because the target end state is great, but being able to also say, here's the steps we're going to be working through. And, and here's the risks along the way, and here's how we're going to mitigate against those risks. But just being mindful that, you know, change those three quarters out a little bit, and that, that transition period where you've got lots of different things happening and tools coming in and out, and you're, your tools are maturing and the processes is maturing. Just being mindful of keeping them up to date as you're going through that.

00:18:14

Awesome. Well, I think we were unanimous, uh, that this is confirmed by the way I, we didn't rehearse this, but I mean, I think it was actually to take us out of the hypothetical into concrete. I mean, could you each one of you say that you have seen the demo process that you have personally, you know, uh, at tested or whatever, right. Said, this is audible traceable and secure. Uh,

00:18:32

We personally, me and my team do it every day.

00:18:36

Yeah. Our teams are working today with internal it audit, confirming those controls across a very add various, uh, uh, toolset.

00:18:46

Yeah. We have public companies that run pure dev ops models and they are requirements that they have for compliance and they're meeting them. So there is a way to do it.

00:18:55

Yep. And same with us. And we've also got a lot that we've done kind of design reviews of lately, and they're really starting to look good and we're starting to see more and more code go through them. Yeah.

00:19:04

And Jean, I think a key element here is, you know, some of them are highly regulated industries and the financial sector in the healthcare sector. They're going deep on dev ops, deep on cloud. They're a data centric organization. And if you're a data-centered organization in terms of how you operate from a dev ops lens, now you're just taking those same dated management concepts that are generally from a front end office operational lens, and you're applying it to dev ops. So, you know, one would think in a highly regulated environment or industry people would be scared of dev ops, scared of the cloud. It's actually the opposite when you're data rich, you actually know how to manage it better from a DNA lens. All right.

00:19:41

Uh, doubly confirmed. So Ronald flaws for another busted myth or confirmed myth, something to help you in your journey. All right. All auditors are ready for DevOps busted or confirmed.

00:19:56

I'm pretty. Yeah. So I'll go for it. Um, it is ready to be ready. Um, no, but as we said, auditors, uh, I don't, I'm particularly embracing of change, but in a,

00:20:09

Uh, exactly. Right. So that was sort of meant as a joke. Let's do the more generous one. Um, we can help auditors understand, busted or confirmed.

00:20:18

So that, that one is definitely confirmed. Yeah. Um, but, but as I was saying, audit is especially external auditors who are doing financial statements. Um, they only have to deal with what's on the docket for that year. Um, so if you're looking at the financial year 19, um, financial statements, if there's no DevOps processes that are in scope, then you're not looking at dev ops, you're not worried about DevOps, so they're ready to be ready, but it might, may not have actually come into their focus yet. And so that's where we go to, um, as you're making changes, you're bringing these processes in, um, get them, get them on board earlier, um, you know, chose to show some walkthroughs of, of the design. Um, get some, get some early input because, you know, folks will be happy to point out where they think there might be weakness and you're not asking them to a pine on it or a test or anything like that. Just, you know, get getting them in the room early and, and giving a point of view on where they think that design is heading is important because they may not have had, or have had a reason to re to focus on it up until now.

00:21:17

I think one thing that you're pulling out in, in what you're talking about, sharing the experiences of how an auditor thinks, right, and how the risk team, uh, works through things, as concerned as it is on us as a community to really push empathy as this massive skill that we need to, to grow and that empathy of understanding, wow, this is a massive change for them. How can I educate? How can I evangelize? How can I pull them into my team and structure? That's, that's the reality of enabling that team to be ready? Because this is a community that is hundreds of years old as a skill, right. It's little scary

00:21:55

Guys, a little scary audit, predates technology. Yeah.

00:21:57

And so you have to keep that in kind of your, your, your mind to say, what can I do to enable them, but at the same time, similarly, because it's been around that long, there's so much changes has had to happen that they're also used to change. And how do we though feel empathetic around us to go, what do you need? How do I help you? How do I embed you into me? How do I go from manual things to compliance as code and policy as code and, and build those teams. But it is, as you're talking about a very deliberate, we need to understand each other. We need to build this into our full stack.

00:22:29

I mean, if you could imagine a room, I got a story that may hit the point home. I love getting points across through stories. You know, when we, Deloitte, you know, from the audit business, when we moved to the cloud in about four or five years ago, you know, we took our whole it department, the risk compliance and security people into a room and said, okay, what do you do today? And everybody went around the room, kind of explained, explained it. We had some consultant guys there, and the guys from our cloud practice there to help facilitate a good dialogue. And then we said, okay, well, here's what the cloud is. Here's what compliance by code is. Here's the processes that you need for hardening all the different services for all your different clouds. Okay. And this is what we call creating dev ops capabilities that your operational dev ops individuals will now take.

00:23:14

How does your world now fit into this? Let's put it on the, on a whiteboard and figure out where the deficiencies in the gaps are from a skillset, from a knowledge, and how do we then, you know, fill those gaps with the right competency and skillset and retool to create this dev ops relationship. So we could go fast. And when we did that, you know, I kid you not, if there's some people from my team here, they were at that meeting. Everybody was a ghost. They know what new world I was talking about. You know, I'm looking around the room, who's in charge. They hit him like that. They had no idea. So, you know, on the other hand, auditor's embrace change. We're really, really good at change. We just have to be taken on the journey slowly and at the right pace. And most importantly, in the language that we can understand. So the translator common in terms of doing translation becomes really important and get into your own.

00:24:03

And I think it's helpful just to remember, as you go through these journeys, that your goals and your daughter's goals really are aligned. And so, right, your, your objectives is to drive to these new processes, to enable your business. Your auditors are also looking at that, right? They want to understand in the easiest way possible how you're addressing risk, how you've managed that within your organization, right? So it's, it's starting those conversations early, it's being open and upfront. It's asking for input. It's not being afraid to have those conversations. It's an organization that, that happens. They're the most successful for, through these transformations. Okay.

00:24:38

And I would just add, like, don't forget that there's different types of auditors. So you've got your external auditors who are focused on your financial statements and have requirements around independence and all that kind of stuff. And can only give you a bit, some input, but limited input as you're, as you're kind of going through the journey, but definitely bring them on the journey and showing them what you're doing. Then you've got your internal audit and internal auditors and compliance functions. They're part of your organization, even if they're staffed by one of us, but they are part of your organization, they don't have the same independence requirements. And so get them in the room and workshop with them, find out what they worried about. Show them a process and say,

00:25:12

Take them out to lunch. You can take

00:25:14

Them out to lunch. You can absolutely take your order. Does out to lunch.

00:25:20

I think to that point, look, we, as technologists are used to technical debt, right, we get it. We know how to deconstruct it. This is cultural debt is long seated, cultural debt, that to bring the auditors to, to now enable us to deploy at speed resiliency, all the stuff we all want. We need to purposely break down that.

00:25:39

I think there's another hidden question in here, how to make your auditors happy. So, one thing that I taught tell the coders that I work with is you have a choice on different technologies to utilize in your applications, because we all have choices. There's monetary considerations, there's speed to market considerations. The more you use a pass service from a cloud provider, the happier your auditor will be because that has service probably has controls coverage in the cloud providers, SOC two report. So that's something that you could do to help influence and educate the differences in the choices that you make from a development lens.

00:26:13

Awesome. Well, so do we have unanimous consent that this is confirmed? Awesome. And then another round of applause for, uh, friendly auditors, but let me tell you a little bit about what we have in store for the next two days. Uh, today, uh, we have an auditor's workshop and this is, we piloted this, uh, with, uh, Josephine team in London. Uh, it's what you've wanted to ask your auditor, but we're too afraid to ask. So the ground rules are, and you'll give a more instructions, but, uh, don't wear your name tag. Don't say what your, what organization you're from. Uh, the cameras will not capture on camera, but you'll be able to ask, uh, all of your, bring your toughest questions and, uh, Sam and Topo will moderate that panel. Um, and because these distinguished augers are so eager to help you overcome your obstacles are doing the follow-up workshop, uh, where you imagine the same room, but then each one of these folks with their teams will be in four corners. And you can talk to each one of them, uh, with a lot more confidence, uh, with a little more, um, in most public setting. Uh, so I think this was just a phenomenal, um, demonstration of how much you want to help this community. So I much, I want to thank you so much for that. Um, any partying, uh, piece of advice that you want to give to this audience in the last two minutes,

00:27:34

Those are not scary. We're actually fun. So, you know, the key is just gauge your early, okay. You could actually use the auditors to help you. If you're understaffed from a security lens, understand from a compliance lens and understand from an, from an automation engineer's lens that help you with compliance, your internal auditors could help escalate that need, because now they're, you're, you're working with them together to protect your business and your entity. So think about a very powerful, but it's a very powerful message they could really use they're your team. And if you don't view them as your team, you've got to take us, take a seat back and rethink of it. They are your team and a huge

00:28:12

Asset for you. And I think to me, to your point earlier about making auditors happy, if you've never seen the work that an auditor does, it's kind of exhausting, right? It is highly

00:28:24

Detailed. That's why we're all like on gray hair.

00:28:26

Yeah. And so what you have the opportunity to do is understand their pain and go, you know, what could I automate a lot of this stuff for you, and then allow you to now be more valuable and your efforts and strategic efforts. But again, that's deliberate. You have to reach out, you have to find those partnerships within your organization, pull them into your, um, into your teams when you're building out. Your OKR is maybe one of them is about establishing partnerships with the risk team and how you pull it in, right. Be very deliberate and action oriented.

00:29:00

Awesome. And that's critical to do just that, right? It's around the communication, right? Is, is staying out of your silos and, and having those conversations, whether internally or externally to drive these DevOps conversations forward, that's the only way you're going to get there.

00:29:14

And I'm going to take a slightly different tech, please don't forget controlling access and the configurations to the tools doing that and being very upfront about that will make your audit happier.

00:29:24

Everybody. Could you please give an incredible round of applause to these incredible waters? Uh, there's one more thing that we need to do, uh, in time, which is we're going to take a photo of everyone standing in front of this slide, auditors love dev ops. So maybe we'll see if we can get a picture. Um, maybe a thumbs up. Thank you. Thank you again. Thanks.