Auditors' Workshop: What You've Wanted to Ask an Auditor but Were Afraid to Ask
Director, Digital Risk Solutions, PwC
Advisory Services Senior Manager, Ernst & Young
Product Owner, Visual Studio Team Services, Microsoft
Managing Director, Global Technology Controls, Confidentiality & Privacy, Deloitte
Principal, Cybersecurity & Privacy, PwC
Director, Advisory Practice, KPMG
Senior Manager, Advisory Services, Ernst & Young
Audit Senior Manager, Deloitte
Managing Director Modern Delivery Lead, KPMG
So I'm Sam. Guckenheimer welcome. This is a, um, this is six years in the making that we have a chance to get all of the big four in one place for you to ask any question you want. Um, I'll be in the hall with a mic, uh, uh, so that you, we, we don't need to play the, repeat the question game TOPA. Why don't you go over the ground rules? Yes.
So welcome. And I think, uh, we want more people and I expect that more people will come in after John stuff is done in the other room. But in the meantime, I think we got enough to start off, uh, for Sam and I. This is like our dream moment. And we had been trying this for about four years now. Oh, expectations. Yes. Uh, so this is kind of, uh, in our view, the last hurdle in our DevOps transformation or DevOps journey, uh, we brought in business, we brought in security, we brought in ops and everybody involved in the development life cycle for our DevOps transformation. And I think this is the last one last hurdle that we need to cross, uh, to make it really happen. A couple of years ago, we, as a team in gene Kim's DevOps enterprise summit forum, we published a love letter, dear auditor.
And I think that got a good traction. It was kind of, uh, uh, uh, olive branch to the auditors. Please come and help us. We really want to work with you. So the result of that, dear auditor love letter movement. If you call, if you want to call it that way, is this that we have all the people who can actually help us on the same stage. And for us, it is again our dream moment. So before he started there some ground rules, uh, for the camera, uh, we know that they are not focused on, uh, the audience. They are focused on actually the stage here, and you're not going to introduce ourselves. Uh, all you need to know is that they are the actual people who can help you and help us. And we are just two developers, right? And we are going to ask some dumb questions, and then you are going to ask any questions that you may have in your mind in terms of audit and compliance and how the DevOps transformations taking place. So the ground rule is you raise your hand to ask a question, do not introduce yourself or where you are from, unless you're under, get into trouble,
Raise your hand ahead of time. And one of us will bring you a mic, right?
And then as the question, do not think about whether the question is relevant or not. If you have a question, please ask, and they are going to try to answer that way, uh, before we start, uh, we can do some icebreaking. Sure. Right? So we have some questions you want to ask the first one.
Yeah. So true or false, all that adds value to software delivery.
I think this is the question for you guys
Who thinks audit, adds value to software delivery.
Don't feel stressed out that all of us are staring at you.
So for those of you who didn't raise your hand, who wants to argue that it's of no value, No. The same thing opinion. So those of you that did raise your hands, who wants to argue that it's valuable.
No, thank you. Um, so I'm not saying who we are, but when the medical, you know, medical software, it's like our software as a medical device, a regulated, and the regulations basically just want to ensure that the software can be used safely and as intended, like that's basically all of that comes to that. Right. And, um, we find that in order to ensure that we need to have our own internal controls so we can keep our customers happy and we care about the patients at the end. And so we're totally aligned with what really the regulations want. Of course, there's a little bit more bureaucracy when you deal with like particular, um, regulations, but that's what we're struggling right now with. But, uh, we think definitely that, um, auditing helps us achieve the goal we're after. Anyway,
So I actually raised my hand to say that, no, I did not think it added value, but I want to make sure that I understand the question correctly adds value to software delivery. Was that the question? Yes. I think that auditing adds value. I just don't know that value is in the software delivery necessarily. That's a, a not very strong opinion, held very loosely and I'm willing to be convinced otherwise. So I'll take a first pass at convincing, like one of the largest churns and software delivery is rework, right? Like the amount of times as, as, and it said this early, I consider myself a recovering developer I'm on the advisory side and help translate between the, the auditors and technologists. Um, the amount of times I, as a dev rework, the same stuff was, it was astronomical. Now, if audit is part of that internal audit, it audit and compliance and risk controls are part of that. They're going to catch it, right? So if I make a mistake that impacts a medical device or EMR, and they can catch that compliance thing before it ever gets out of my user story into code, that really helps my software development life cycle, because all of a sudden I'm not wasting efforts, things like that, because I can only know so much as a developer. Uh, I'm not going to be a compliance expert, but we have compliance experts who can help with that. That's my shot, at least.
And I would expand it a little to say, it's smart auditing, auditing that, understanding the methodology, understanding what it is you're trying to do, and, um, kind of adapting the audit approach to that. And so not bringing old fashion controls and techniques, but actually working with the methodology, working with the way that you're working, um, to essentially not be part of the quality check cause obviously quality Nate needs to be part of the dev team, but where you've got a lot of overlap between what audit's doing and what quality is doing.
I have another concept on this that I talk to developers about. And that is what, if you, as a developer, your annual performance was based on the quality of the software that you delivered. Would you feel the same way? Would you care about quality more? Would you want to get all the help that you could possibly get in doing your job? You know, when you take it down to that level, you know, when you, all of compliance is based on quality, and if we all own quality equally, that really helps, you know, bring everything up to go quicker and faster. And that's what we all want.
I'm interested to ask the panel. What for the audit profession, uh, and compliance professionals, what do you think is the biggest behavioral change that is you're seeing is needed or you're seeing is happening or is not happening? I'm interested around the kind of the culture and the behavior.
Um, I'm not one of the auditors in the room and I want to address that. Maybe it'd be kind of powerful since I'm on the other side of the fence. I'm a security person. First used to be a developer, also a recovering developer. What I've seen at least from a, uh, cyber side. Is that kind of along the lines of what the folks are talking about here on the last question is we have a lot of people with the same skill sets, especially in internal audit, um, that are all kind of going, uh, trying to achieve the same means to an end yet throughout most of my 20 year career, they were kind of pitted against each other. Some, uh, you know, at places particularly where there's cost pressures, I'm seeing a lot of motivation to pool resources, um, on, you know, I would say probably activities that were a little further down the chain, uh, and they're realizing that as they kind of shift left together, there's a lot of economies of scale to be had. And especially when you're talking about some, you know, the things that are in the security space, whether it is a war for talent right now. Sure. We're always battling each other on that front, um, is there's a lot to be gained there. So I see the cultural, the culture shifting to be kind of more inclusive with both groups and realize a lot of those efficiencies, uh, for a variety of different drivers,
Tom internal audit. Uh, I know that a lot of people have questions around that. What is internal audit and external audit and how
Do they play to,
Yeah, sure. So, um, we're saying audit is very generically ordered as are a lot of different things. Um, the main distinction is internal audit and external audit, external audit of the people who are coming in and reviewing your financial statements and giving an opinion over those financial statements over the way that whether the information contained in those is fair and true. Um, and that is presented to the market to the sec, um, or whoever your investors are, um, to help give them some comfort about the accuracy of those financial statements. Those internal auditors are typically, or not typically are kept at arm's length. They need to be independent. They're, they're called external for a reason. Um, and so the role that they can play in this process is especially in the front end of the process is relatively limited. Um, however they will audit, um, and they will audit in particular systems that help generate those financial statements or feed into those financial statements, your internal auditors.
And I'm saying that very kind of loosely now to bring in your compliance groups, your ITA compliance, um, your security, Chames your I'm going to kind of anything that's internal to the company. They still may audit you. Um, however, they, they don't have those independence restrictions. They're not reporting out to the market. Um, and so they can give a lot more advice than they can be in the room a lot more with you as your creating and designing controls or, or setting things up. So they're, they're kind of the two main distinctions. There's a lot more distinctions, but, but that's kind of the two different paths and internal audit can play a lot more of a, kind of a helpful role along the way, whereas your external orders need to be kept at home
Just to kind of add to what Matt was saying that I think to expand the external too. I think a lot of, I mean, a lot of it's obviously external auditors versus financial statement auditors, but also a lot of I'm sure a lot of you issue SOC two reports or SOC one reports where you potentially, these products might not be in scope for financial audit purposes for your external auditor, but you might have it in scope for the product you deliver or the device that you're creating. So a lot of times those also are also some of your audience on the external audit, but just to kind of jump on to add to your point, like, how has the mindset changed from an audit from an auditor's perspective? I think where I've seen where it's really the whole dev ops, our models really pushed us. It's really audited to have to get away from a checklist checklist based audit, and really are forced to really understand the business and the flow of transactions through the systems. I think more than ever, we need to really understand how transactions flow and how things work through the whole life cycle of the system to understand where the key risks are and working closely with the it organization to understand how are they mitigating, how are they grading regression tests or unit tests that mitigate some of those risks? So we really care about,
And just to bring that home when we start in the Emmy could probably give some great soundbites on this. When we scope the audit for a product to do a level two type of internal review, the first thing is auditors, whether you're the business risk or the tech risk that we asked for as we go to the architects, we say, show us your, your PR, your proposed architecture and walk us through the tech stack and all the services that go into it. That's where the risk starts.
Let me jump in and say, the question was about culture. So does anyone want to want to do a good culture versus bad culture?
I can do a quick sound bite on that. So from my perspective, a good culture is how do we get to a place of, yes, there should be yes. On all sides, right? The business has an objective. It, you want to get us there, you know, some of the best ways to do that and compliance wants to enable that. So how do we get to our state of yes and bad culture would be you coming to me and me saying, no, I don't ever want to have to say, no, that's not a good feeling. It's not a good feeling for you. And it doesn't make sense for enabling the business and for what you guys do with technology. There are a lot of great ways to make things happen. There's a dozen different ways to perform a control and mitigate a risk. Let's do it together and figure out how,
Yeah. The other important piece about culture, and it's a gong gong I've been banging for the past several years. And very much today is about empathy and understanding, right? So, so who people, how many people in this room are considered as a developer. Okay. So if I asked all of you 10 years ago, what a UX person did, you would have said pretty pictures, pretty picture people. But now if I ask you, what's a journey map. Yeah. That's something, you know, right. We, we absorbed that into our culture. Now, if you go and ask the average internal auditor explained to me, dev ops and agile, they'll say developers have access to production, right? Segregation and desegregation duties. And no documentation. It is on our own is to have empathy for what their problem is and how we communicate it and inverse, right. That community opening up and going, tell me, as you're saying through the architecture, how does this work? And you can go through and talk about all, a lot of different things around culture about what is behavior we want when something goes wrong, right? What's the behavior we want when something goes, well, things like that. But to me, that, that uncommon understanding and not seeing them as the office of no right, which is often the perceived perception,
Looking at it as a partnership.
So I think I am the one that raised hand for both of it. Does that add value to software delivery? I raise hand saying yes and no. Right? So here is my experience. So when we started doing continuous delivery into production, right, the first thing is we go to the infrastructure team and say, look, we want to do this continuous delivery. They go, you need to go talk to your, your audit and compliance people. What are they say? What did they tell me? They told me, you know what? You cannot do that. You need to have segregation of duties or compliance. It goes on. And I asked them, why, why does it matter? Because all we are trying to do is make sure that wrong militias, whatever things does not get into production, I asked them to, why can't I just show you, what is the compliance route? How did we go from there to production? And the answer was right. So we have to go from that place to a place where we are today. And that was a very, very rough journey. So does it add value to production? I think it's in context, it's in context of the people that you are working with in context of how much they are aware of DevOps and this continuous delivery, right?
You guys have beer in your office. Maybe that could help. You got to take down these mental silos, right? If everybody has the common goal of delivery and providing value for the business. And if leadership on the business side stands up and says, we need this technological approach to drive value for the business and to grow revenue and to get us to where we want it to be. Strategically dev ops is now a strategic priority for, for, for the, for the business. So don't just think about the compliance people as having to the burden, being on your shoulders, to influence them, look who influences the compliance people. Other than the regulators have the leaders of the business and the CEO and whoever else you need, like CIO type of person make it their priority. It starts with
Tell me, let me just say, what do you say to the gentleman's experience where you're in a situation where your internal audit is saying, no, you can't. How do you,
If you happen to have an external auditor, they could be an ally, right? Cause when the external auditors come in and they're looking at the work that the internal auditor does as well, I spend a lot of time as all of our competitors up on a stage, just talking to clients from a sheer, from a mere educational element and explaining to them, what is dev ops work? What's this new type of process. How are organizations utilizing this approach to transform? So once you play that, a couple of times, the repetition makes us start to sink in and people ask intelligent questions. You'd be surprised. Hello.
Uh, I have a question here, but before that, uh, one last bit of the, I think the problem that you are saying is, is, is felt by many developers. Developers do not directly talk to
That's that exactly one point that I wanted to add is we don't know who the external auditors are. We have no ally.
So you got eight people right over here that you, that are here to help you now. Yeah. And I think that's also just finding those folks within your organization. We talk a
Lot at this conference about the engineering focus, change, and modernization, right. That same exercise has to happen within your organization on the compliance side as well. And so it's, it's finding those allies within your organization. And then it's, it's taking the time and energy to also educate them as well, because a lot, a lot of them, unfortunately, haven't in a sense to use Gene's words, you know, seeing the light as it, as it comes to this here. And, and, uh, unfortunately those first conversations can be rough, but it's, it's sticking with it. It's helping them understand the impact to you and your business and the need for them to be supportive of, of that end goal.
I think it's actually nice that you guys went and asked. I have clients where they tell us we are going dev ops you're the auditor get comfortable with it. Um, so we've sat down with the head of engineering, say, okay. And this is, comes back to my earlier comment where we really have to understand the business. And really what are the key KPIs that you guys are committing to your clients and also from your own products, like how you recognizing revenue, all those things. So we sat down with the head of engineering, say, okay, if you're going to go down this model, here are the key things you cannot get wrong. And so we sat down with them and I created unit tests or regression tests, specifically tests that functionality every time you push code, and that's how we get comfortable, we got comfortable with it.
Yeah. I think one of the ways that, uh, I'm also a security guy by trade and we have this experience that I think has gotten much better in recent years where security is considered. And we now have that DevSecOps name or whatever acronym you feel like using around that. But in the same way that we would say, Hey, um, dev team, you should probably talk to the security folks earlier in the SDLC to make sure that you bake in the right things. Um, and I say maybe that means cutting a ticket so that every six weeks or whatever period is periodicity, it makes sense you have a conversation. It can be that tactical. And I'm exaggerating for the sake of storytelling, but hard coding those conversations with the people in this, um, example IAA or whatever auditors we're talking about. Um, at least in the beginning of our journey can enforce the right behaviors and the muscle building between those two silos,
Change the culture from the top. Those of you that have influence start, you know, the way I always say it to people is, look, you need to change the mindset from it being any compliance function, whether that's kind of internal audit or otherwise change it from being a, a tax to being a competitive advantage. If you kind of live that mantra that brings together the culture, all of these points that all these great points people are saying, that kind of seems to resonate because nobody likes to pay taxes
So much of governance, or at least a lot of the conversations I wind up in, uh, at some point start to mention COBIT, um, from, I I'm wondering to what extent any of you are active in that, and in particular call even Colbert 2019 is still based on plan build run, which is essentially a statement of waterfall development. I'm developing the opinion that plan build run is no longer suitable as a basis for governance is. So the question is two parts is COBIT relevant. Should it be changed?
So we talked a little bit this morning about you have a control objective at a high level, and then how you implement that may vary depending on your delivery model, I'd say the holistic principles that you just nicely articulated. You could apply those to a dev ops world. It's just repeating itself. You know, in Microsoft's case, what is the 85,000 deployments a day you guys do? So, you know, it's about the application and how you implement the principle, even like accounting and basic auditing standards, their principles was with a loose guidance of how to apply them, the facts and circumstances that you apply them to will dictate if you met the objective.
Yeah, I would agree with that. I think the one thing it does is it really pushes the boundaries of the tools you're using, because you can still do that same waterfall approach, as long as you're using the tooling to kind of automating the whole process through that.
I will, I let's, let's talk about this straight up frameworks for a second, right? COBIT has its issues from a SharePoint that somewhat dated methodology. Now people have found ways to satisfy those aspects of it in a modern realm. And you can also see the progression. So like what, what is the latest so that I tell V fours what they're calling it, it's basically DevOps, right? You look at that framework and you're like, yup. Connect the dots. Cool. There is an industry shift going on. Right? I argue that right now, we're at a, we're at a key nexus point, right? I described it earlier as like we went from being a garage band punk band in the garage to now we're on a major label, right? We're at that sort of nexus point where I think a lot of the things are starting to change. You wouldn't have us up here having this conversation, unless this wasn't the moment. But when you look at that moment, there's this, this uncomfortable pivot, right? Where some of those things of like, oh, that framework that I said, we're going to follow it, doesn't match the thing. And I think you are at that point where it is, it is shifting. And because it's being more iterative, it's much more about these interrelationships and understandings.
I would look at that from the perspective of that's verbiage on, on paper. And it, it may function that way, but you have to look at it from the other side of what's the risk that is trying to mitigate all those frameworks are driving towards risks. The risks haven't changed. Our verbiage that we use around them, the processes and procedures, the controls we put in place to mitigate those risks. That's, what's changed. And so articulating that back and pushing back on them saying, here's the
Risk, and this is how we're mitigating it. And the best way I've explained this to coders before is I think most of us probably work at organizations that have data sovereignty or regulatory challenges when you're doing business at a global scale, and you have so many different countries around the EU, you know, around the world and they have their own data postures, the own data rules. If you go to certain jurisdictions, you know, specifically in Europe, they may have some archaic data rules that have been on the books since the sixties or something since world war two. Right. And how do you apply these old regulations that have never been updated to comply with the spirit of the law? So you have to think about how do you comply with the spirit. I think we've got a question. So hello. Um, I guess where my struggle is, um, is with the way that this engagement is.
Um, and let me explain that the, the developer community is expected to get nonfunctional requirements very clear. This is what you do. This is the chain of custody. This is the segregation of duties. These are all the requirements that you need to fulfill. And, and that's very clear, right? These are all the rules that you must follow. Um, in contrast, the auditors that we, we heard from this morning, four of you, um, said that you believed in DevOps values and you thought that DevOps was a good framework for, for audit after that. A lot of conversations that I had, um, people were, um, saying, you know, that would be an interesting video to play to my auditor because I can bet my auditor probably will not agree with that. So why internal audit? You mean, why is that a, a critical mass that needs to be achieved?
Why is there not a clarity there? Why is there not a standard body for auditors who could be convinced and who could say these are our standards, these are our rules. These are the rules that we live by just like the developers are supposed to live by a certain set of rules. These are the things that we will accept. These are the frameworks we will accept. These are the changes to the COVID rules that we're going to make. Uh, because we understand that it's outdated rather than achieving a critical mass or, um, talking to enough people and making enough change happen, or making sure that your auditor likes you or having to buy them beer. Why is that the conversation I should be buying you beer. I just want to be clear on that, but, but, but it, you're taking a, a binary view as, as a coder, as a programmer would to give me the NFR and I will implement them, but that's not how it works because your data sensitivities are different.
Your different services are different. You may get some levels of coverage from a SOC report from your cloud provider for some services, others, you may not. So there's all of these factors that come into play and depend depending on the type of functionality and the type of compliance mechanism that you have in place. Somebody mentioned medical devices. Somebody may have to comply with the confidentiality principle from a SOC report or a high trust or, or Phi, you know, how you comply with all those, depending on the tech stack that you, as a developer want to use. And every tech staff and technology is uniquely different. So the approach and how to comply is never going to be the same with all these different differences. And we want to empower you the developer to use the best approach that you believe from a technical lens is needed. And we have to partner with you to find the right answer to mitigate these risks. So it's never going to be give me the NFS and I'll implement them. It's very much an art and not a binary science.
Ultimately it comes down to the risks and, and that's, that's the shift in the audit world as well. It from really, as mentioned earlier, that that real checklist approach to what are the true risks in your organization. Some of those historical controls or checkpoints that you had may no longer apply because they're not providing any value. And so it's, it's really that interaction between the process owners in this case, the engineers and your auditors to have that open and Frank conversation of look, this is our environment. This is how we understand it. These are our risks, and this is how we've determined, how we're going to address those risks. And so your auditor may have an opinion of whether yes or no, that those risks are addressed, but now you're having a conversation around the substance of how you move forward to actually do this versus just the, the no, or the hesitation that you've gotten in the past.
I think, and I use this verb, um, on purpose. I, we empathize with the frustration. I think the eight of us on the stage here now are here because we want to be part of the solution with you and have our colleagues where we work, be part of that solution. Um, but it will take time, but as Sam and Topo opened up this conversation with, uh, we'd like to think that this, that DevOps enterprise summit is a signal and, um, hopefully an inflection point that to the outcomes that we're all hoping for together.
The other thing I'd say is let's be real if you're a developer and you're at this conference, you're the minority, right? You're the minority of developers. You're passionate about this, you're changing. This is then you on the advocacy to go the other way. So this year is, was a turning point for me. I presented at two I soccer conferences. I soccer is, uh, ISACA is a, a, a bounding organization around, um, it audit, right. I presented including guy Hubert, who is a, he is best title and world. He's a risk futurist at Atlassian. He's their chief head of risk. Right? And we were both presenting about agile and dev ops at an it audit conference. And then I go and I speak at dev ops world. And I go, you guys should talk to the auditors. We should bring this together. So just as much as, as, as it is them educating, it's also us as an advocacy group for tearing down the silo, go into those communities. Right. And, and have those conversations, because it is as much about that, that learning and understanding they do have groups like this, just as much as, as we have, you know, meetups and things like that. So maybe next time you should invite some developers to come from.
He needs to ask the question, Topo he's waiting patient lists.
All right. So my question is kind of two part, first of all, have you had success having conversations about modern DevOps practices with the United States government, particularly the OCC or FFIC on the financial side. And then secondly, um, if you have, what, what has made that successful and how have you reconciled their handbooks, which can be as much as 15 years old and very focused on waterfall methodologies and they come in and they have very prescriptive thoughts on what's your, your risk and compliance and audit focus needs to be based on those handbooks.
So I have an interesting story on this one, because, you know, again, I'm an, I'm a financial auditor. I went to the technical side and now I'm a risk manager, but we as an audit business. And if you think about how old the auditors are, like we have a several hundred year old profession, we went through a massive transformation in terms of how we do our work, changing things, using enabling technology to really shake up how we do audits, you know, from a qualitative standpoint, from a values standpoint, and that's a big transformation. So a lot of the government agencies are recognizing that the audit community has gone through a lot of transformations have reached out to us. And I've had found myself in multiple agencies having a discussion about transformation. What does transformation look like? How do you transform? So our consulting practices, everybody's trying to help the government transform.
And depending on the agency that they may have, you know, different commitments that they've made in terms of what they want to do, but that is being driven by the business. When your business drives your transformation, the risk side of the house comes along with that and has to look at themselves internally and come to a conclusion, say, well, if this is what our leadership wants, what do I have to do? Or what do is a government it organization have to do to modernize, to meet the needs of the business. So that should never be just on the law, you know, on the lap of the it audit and the compliance group, but it's really a business objective. And we have this concept internally that, you know, the cloud is not a technology strategy. The cloud is a business. So when you look at it through that lens, and we talked before about the tone of the top that opens the doors and the conversation just really changes.
And the bird specifically to those CC and FFIC, um, those handbooks are old, um, and they are definitely out of date. Um, and I think that's acknowledged, um, we've been through a couple of exercises where we actually take those principles or those requirements and map them through to a methodology like safe, um, where, and we say, Hey, this is what you're trying to achieve here. And this is where we achieve it over here. And it's not the same thing. It's not one for one, but we're achieving that objective. So it's a bit of a slow process. You have to go through, you have to do that mapping, and then you very much have to take your regulator on that journey with you. So whoever's in the trenches with you, um, whether it's, um, remediating a finding or, or doing an audit and actually work through and say, Hey, it's not that we've just ignored it. We just think it's got covered in a different way and just kind of go one for one, um, through, but yeah, they, they, it doesn't cover them off neatly. And so it's kind of on you. Um, either as the technology organization probably has the technology organization to actually say how you're achieving those objectives. So you kind of pull it back to rather than the specific activity, what's the objective that activity is trying to achieve. And here's where, where we achieve it, even though it looks and feels different.
And I'll just add to I'm, I'm definitely not in our federal practice, but at least once a month, I am pulled into a pursuit with our federal practice because they're asking for our approach for agile and dev ops, right? So I can't, who's the design arm that the federal government spun up at some point something 20, I can't remember, but they had built up 18 F right. They built up an agile standard methodology that they rolled out from design thinking through CIC D. And every single proposal I get from fed at this point is talk to us about your dev ops principles and your agile methodology. And what is the tools that you're using that is becoming standard expectations from fed. Now you brought up an interesting point, which is, look, we say, dev ops, like it's a single thing, right? It's got a single definition and what I'm going to do to implement dev ops an organization that is a mainframe based legacy application is inherently different than a functional programming, uh, go cloud native Kubernetes thing.
Right. But all of my tasks are going to be in a task management system, like, like Azure dev ops or JIRA, right. I'm going to follow a standard process process. And even when I'm doing a legacy waterfall thing, I can still use those tools, right. We've been referring to this as a, as, as a waterfall with modern tools, right. To enable those sort of scenarios. So it's, it's really, we need to be really conscious of what are the delivery models within dev ops, not just going, I put it on the dev ops. So now I'm the agiles. And I just want to share real quick that it's not just the civilian
Side that's asking for support and how to transform into, into this way of working. But we see requests coming in and the military side as well, which is
Exciting. We do a lot of labs where we'll bring in the folks from the specific government organizations and talk through it, white space, it do the mapping again, you're going to try to take a substance over form approach, like was articulated in terms of the objectives. And again, it's about getting all the people together, but that requires tone from the top.
I, so, um, I was wondering if the community, as a whole, the audit community, the external audit community to be precise has a community of practice developing as well around either dev ops, sand, or agile or lean. And I'm asking because to put a shout out to both KPMG and PWC quite a few years ago, when we started our first implementation of an application that came with a tool set that did implement continuous deployment capabilities. So our processes didn't allow for it. They were the ones that actually helped us teach her internal auditors. Right. So that was great. Um, and so now, as we're really adopting it wholeheartedly at the enterprise, our internal auditor, at least there was one small group that actually understood what we were trying to do. And they've been really helpful, but I'm wondering if there is some community of practice or some organization, um, informal organization or formal that we can point other people to, because it's great to say, yeah, go to your external auditor. But as other people pointed out, number one, not all the teams have visibility to their external auditors. I did. I was lucky. Um, also we're heavily regulated like that gentleman over there, various government bodies who I, I was really happy to hear you say, you've been mapping it to safe, but our arm of the fed clarity doesn't know that based on the fi the questions we've got this here. So it would be really awesome if we could sort of point them.
So I th I think my colleague over here Yosef has an answer, but I want to repeat, she said KPMG and PWC, just to be clear, that's the words I heard. Okay. Your turn buddy.
I heard Arthur Anderson. Oh, just kidding. We've got to have a sense of humor here, but you know, there is a consortium of internal auditors is called the IAA, and they do talk about cutting edge technology. Now, I know if January were here right now, he would stand up, grab them. I gotta want to enhance and say, Hey, when Sarbanes-Oxley came around, we actually helped write one of the chapters in the internal audit approach to testing controls under a COSO perspective. And, you know, Jean and, you know, a lot of his colleagues help create that for the Institute of internal auditors for compliance. You know, maybe there's another chapter that has to be added to evolve it. But, you know, I would say cloud is definitely talked about within the internal audit space. And with any industry,
I have a question over here.
Yes. Uh, he was raising hand. Uh, you go first and then you come,
What guidance do you have if there's an it team wanting to go to compliance for the first time, make an assumption that a couple of times it didn't go well, they just shut everything down. What would you recommend? Or a 90 day period to bring the compliance team on the journey? What should we do? We'll buy them Bayer. We'll do all of that stuff, but what else can we do? Can we bring them into a dojo setting? Can we, um, can we bring them to the development teams? What would your advice? Sure.
Um, this YouTube video, I mean, we're all internal auditors. We heard were external auditors or security professionals that deal with dev ops and compliance every day. So it works. So that's the first thing you have to make sure people understand that it's something that can be done, then it has to be a strategic priority. In other words, it's something that they must do for a business reason. Once you overcome those two obstacles, then you're into, okay, well, how do we get it done? You got to break through the barrier and then get into the implementation.
I don't mean to sound, um, maybe patronizing that this doesn't sound that way, but I would broker those conversations with an alignment to the broader mission, right? If both groups have the shared mission and we can at least agree on that, that's the starting point for how do we get it done together? Which Joseph was saying, um, it's oversimplifying it. But I find that that's helpful
W where we seem to be most successful is when those compliance folks are part of your liver and deployment team of, of rolling this out right there in the planning sessions, from the start that they're, you know, brainstorming with you around those risks and how you guys are gonna kind of define those risks right there. They're part of the process from the start. It w where it becomes more challenging is you've, you've developed your pipeline, you've dealt with your process, and now you bring the compliance folks in, and you say, certify this, tell me that it's good. So we can go forward. It's much easier to, to kind of guide, guide the path with them there. From the start,
We have a couple of examples where we've worked with them, kind of from those early stages, and you get the internal auditors or whomever in the room as early as possible. And probably our biggest success story is they gave multiple feedback points along the way. Um, and so when we first asked them in, they were like, really you're asking you syndrome or, okay. Um, and then it wasn't like, Hey, we've done this, give us feedback. It was like, Hey, we've, we've done a little bit. What do you think we've done a little bit more? What do you think we've done a little bit more? What do you think? And so we, weren't getting to the end and saying, Hey, isn't this great. Can you sign off on it? We were giving them multiple, um, kind of iterations along the way to, to give that feedback back and give their input.
I think to like very few places
For the military is truly a hierarchical organization, right? Like, raise your hands. Are you in a matrix organization? Like, I know that I am, I know that this guy is right. Finding those advocates. Right. And, and, and in interest, a person who can be your advocate that is aligned into that organization, and you can build a relationship with that may not be the lead of the organization, but isn't an influencer and have them bring it along. Right. It's kind of like bringing your friend to a party who's then going to spread the word to the other friends, to come to the party, as opposed to what we went to the head of internal audit. And they said, no, right. Finding those, those advocates.
And we're everything we just talked about. It's cultural and, and soft skills and change management. It's not technical in nature. It's how do you get buy-in, you know, to be an enthusiast and then go to the next level of being a, you know, a person of change and that not, not just a change advocate, but being a driver of change within your organization.
One thing that the audience may have gotten from us is that we keep repeating, you know, bring them in early, bring them in Austin. Um, that sounds, um, you know, a little bit blunt, um, there might be a strategy where you say, Hey, you know, we've got a few experimental products over here. We're going to just see what happens, you know, invite these guys to the three small things, very low impact, uh, you know, requests from the business, not the big one, you know, with massive strategic implications. You wouldn't for the first time, broke the conversation there, but, you know, build the muscle small, like it's just like going to the gym. You're not going to go bench three 50 your first time there, you know, work your way up to it on smaller gigs,
Just to engage them as, as partners. I mean, have that kind of attitude. I think the fact that we've got folks up here that are mix of auditors and non auditors, and we obviously work very closely together, speaks to that and speaks to the success of that.
And I, I would just add making sure that you can ask them, what are they really concerned about? You're going down this path, help us understand what are you guys real? Why is this a roadblock for you guys? What are the, some of the requirements you are stuck on and let us work together to figure out what those are.
So let's try to close this out with a great story, right? You have to be able to effectively drive change to be an unbelievable salesman or storyteller. So internal auditors don't always want to do a lot of work there. They tend to be, if it's the old fashioned way, their checklist space, they may be doing the same thing every year, unless the risk profile changes in the organization, paint a picture. What if internal auditor, we create a paradigm where you will see in a dashboard, all of the exceptions for whatever the specific control is. And you could actually see not just the exceptions, but how they've been resolved by clicking through a screen in front of you from your desk. Is that a future that you think would help protect the business and drive efficiencies? Everybody's going say, where do I sign? So paint a picture of what the future life of an internal auditor or an internal compliance specialist is going to look like in the data, heavy dev ops world, where you could drive compliance to automation.
Uh, my questions around PCI. So I know you guys might up the, having the segregation between dev and the production in dev ops, we're doing the opposite, right? We're having developers, giving access to production. Um, do we expect that PCI requirement to change? And then also, um, if we're being audited now, um, is having the controls around the deployments and then also having logs, uh, going to an external system sufficient to pass an external audit.
Yeah. So I'll take a stab at that. I'll start with the first, the segregation of duties. That thing no dev actually wants access to production. That's terrifying. It's horrible. Right? The actual value is can you put controls in place? So I've got a guard rails to work within, right? And then that system creates the ability to have tracking of all of those things, right? So if all of a sudden I have the guard rails in place, the system will control whether I have multiple people approving pieces, right. So I don't see that going away. I actually see the guardrails improving. I've told this story multiple times today, but it makes sense to me, like it wasn't the FBI that stopped people from stealing music. It was iTunes. It wasn't the FBI's compliance and governance and scaring everybody into it. It was Spotify right now, if you apply that to our dev ops model. Well, yeah, if I'm using a connected tool chain, that's enforcing my compliance policies about, you know, segregation of duties and, uh, tracking of that, uh, end to end piece of it. Then it's, I'm living the compliance because it's supported by my tool set. I don't see that going away. I actually think it gets more so because infrastructure's code should not be developed by the same people. We're writing software
Coming up quickly. Uh, you mentioned, uh, fully automated systemic change of production systems. And then you also said about multiple people approving it. So can you kind of,
Let me try to talk about this. So if you think about like your chain of command on a deployment, if security has to sign off on a deployment, if dev ops have to sign off on the deployment and all the different stakeholders involved have to systematically say, yes, this is really ready to deploy in multiple environments. You've just made your world better than it is in waterfall, by design, by definition. And all of that has completed audit trail. Cause it's all systematic. That's number one. Number two. When you think about fundamental access to production in the old world, there would be a few individuals that would always access to going, and maybe their access was logged. Hopefully, you know, and somebody was monitoring it and today's world, the individuals have the potentiality to have access to production where they will go and request a temporary, you know, 24 hour or six hour time span, you know, code that gives them access to it so that it's not standing access, it's potentiality to get access, but they have to open a service ticket in statewide. They're going in. And then it's removed. That is a new world, you know, on-prem, you don't necessarily have, unless you've invested in those scalable
And even better, right. It's actually nobody who has access to it. I'm using a good ops type system where I commit a thing. It gives me just in time tokens that it opens up for the automation only, and you govern the automation like a human right as who has access, et cetera. But the key is being deliberate about that. So like, one of your other questions was like, Hey, if I've got all this, what I call like data exhaust from the system that gives me the logs and the traceability. Yeah. But this is a human issue. I'm not probably going to trust that you have the data exhaust. So I'm going to want to trace bullet through that whole thing. And that's the gaining the trust of that, that organization
Fused about multiple people approving production deployment, and a fully automated, how do they line up?
So I'll give it, and then I'm, I'm, I'm my calling. Um, I think w there's a couple of things there. One, which is I had a customer is like, I want, I want my dog to be able to approve builds. And his rationale was that should be fully automated, right? The reality is that is not the case. For most things. There are many things that may be fully automated based off of a risk threshold, right? There's other ones which may be notification based. And someone let's just say in service now gets something that somebody has to approve and then kicks it back to another system, actually build it. There are still some hormone human controls you might decide to be in that system. Some of them might be automated controls that it passes all of the tests, right. I w I ran through sonar cube and it got a low enough threshold. It passed all a certain level threshold. My unit tests, I'm not impacting more than X systems lo it passes through the workflow and says, yeah, go ahead. But that was something that this guy approved the workflow of that this was an agreed upon set up.
That's what the guardrails, I think it's, what's important here in that conversation is organizations defining their process and their risks, right? Different organizations have different risks, different thresholds. And they're there very well could be cases that require no approval up to multiple levels of approval. But it's, it's how you define and understand your environment. How you lay that out and say, look, this is how we look at our world. This is what's critical. This is what's important. And these are things that are just kind of day-to-day operations that, that are, are lower risk for us, right? Each of defined kind of thresholds are going to have potentially different approval requirements. And so it's, it's really back to how do you understand your own environment? How do you determine those risks within your environment? And then how do you explain that thought process to your internal and external auditors so that they can weigh in on yes, we agree with this or we don't, but now you're having a conversation on risk on process versus no, you can't do this.
And I'm going to add to that. I mean, you really do understand the environment. What can a developer do in production? If the code's already compiled and sitting in get hub, I can make configuration changes in production, but you have a configuration management tool that runs every 15 minutes share for overwrites. Everything is a production anyway. So worst case there might be a configuration is out of date for 15 minutes and production. So again, it comes back to understanding the risk and understanding the environment.
I think that too, this is a mind shift scenario you heard, um, at UHG the other day on the main stage, they said they went from doing a quarterly release to two releases a day. That also means that my risk blast radius became so much smaller. And so, so on, on the, the risk and internal audit side, they're willing to accept a different risk because what's the big deal. I've got a blue-green deployment. I hit a certain threshold. I rolled it back. The risk of that, and the blast radius of that became much smaller than big bang release. We do every quarter that in fact, that that impacts big bang, five systems
Risk. If you have a release once a year, your risk profiles through the roof, the incremental releases, the more releases you have, the less risk you are.
Hey, so, uh, it's a hypothetical situation, right? Uh, so we, we heard about some of the questions were on Colbert. Some of the questions were on the higher level. What do we do about it? Right. If you think about, uh, let's say some, uh, large organizations, right? Where all the policies, standards controls were written in an era where they were waterfall, where there wasn't automation, right. And, uh, all of a sudden they're doing projects to product, right? The product based. Now they have water for two agile. They are now DevOps, right. They're adopting cloud. What would be your suggestion? Let's say we've been through your two steps and we're trying to figure out how to implement the controls. Where do you look at that individually? Or would you step back and say, do we just need to, because it changes your risk profile changes, your responsibility model changes, everything, right. So where do you step back and say, are those policies, standards and controls and implementations the right ones? Or would you do it one by one? And so,
Um, I mean, you really want to look at it as a, as a whole. Um, and you want to, it's kind of everything you just described, because if you just, if you look at things in silos or in isolation, you not be covering off that risk. So you've got to say, okay, we're moving to this new process. Here's what that new process looks like. Here's where the risks exist in that process. Here's old controls that we had, or any of those still good or not. If they, if there are some that are still good, pull them forward, and then look at the remaining risk exposure controls to, to close any gaps and then, um, going around and making sure you're updating those policies and procedures, because it's boring as it is when I ordered it comes in and policies and procedures, aren't updated, they're going to dig new on that straightaway. It doesn't matter how good the controls are. If the policies and procedures aren't updated, you've got to issue.
Yes. I have one more question. If I can add on who do you think in your mind is, uh, this approach who owns recommending this approach in terms of owning saying, this is how we should approach compliance. We are doing all those four things. So
The there's the spirit of the law, but somebody has to own compliance at the end of the day. So when we came out with COSO 2.0, which came out three years ago, four years ago, I forget already when it became effective, it was an update to the Sarbanes-Oxley initial framework. So what all public companies around the world had to go through and do is they dusted off all the initial controls documentation that somebody did five years before, and most likely hadn't changed unless there was a new process. And they had to now look at all of their control mappings and all their policies in light of COSO 2.0. And what people learned when they went through that exercise, is that why did we not go back and change this? And why were we forced into a way of operating? Just because at a point in time, this is how we did the mapping. There's nothing that says you cannot adapt and change how you comply with the control objective from a compliance lens. So somebody has to, you know, take it upon themselves to transform within the organization and challenge the way we've done things to make it better. And that doesn't mean that you're not complying. It means you're redefining how you're complying. There's nothing against that.
And like in terms of ownership, that's that that can be challenging. Um, it may be an it compliance group who kind of quarterbacks the process. Um, but it, it, there's no easy answer on that one. Um, but it's a case of finding someone and kind of putting them to it, um, especially in this transformation state and saying, Hey, you know, maybe it's your VP of business agility, or whoever's driving that. And now, now they're not going to do it obviously, but having someone who's on the hook for it, and then to your safe, subpoint looking at that in the ongoing, the ongoing process and making sure you don't just blindly follow these policies that were written down and keep iterating on those as you change. And as you grow,
All right, good evening. Um, so as an engineer, I practice failure, right? Deliberately break things in test, I practice major outages. How do I practice failing an audit? And,
You know, is there any kind of techniques or am I really stepping into a Hornets?
So there is a concept that we refer to as a defendable position. So if you want to take an approach for something, and it's not exactly how, you know, maybe your process is laid out, but you still believe that it complies with the spirit of the law hat made sure you could defend it. So w w if you want to go through an audit and having an approach that set up an I'm audited and, you know, and she's audited because of all the 54 applications, we support go through internal audit, go through ICER readiness audits. So we're audited and we're, we're sort of the auditors and we have audit applications. So we deal with literally off vote all the time. But if you take a position, you've got to have a story to tell. So when you go through that lifecycle, why are you willing to deploy code? If you know that there's known frailties in it, because nothing's perfect. There's always realities. We know that, you know, there could be tons of different things that fail from a functional perspective, but you take our risk-based approaches to why we're willing to go to production, you know, with all of that, that's a defendable position.
I think one way to look at it, as well as maybe building on your question as well, when you ask, who owns compliance in this space, right? Who owns the dev ops process, right? Those are the owners of those controls of that process. And just like, you're trying to constantly improve your product right. Similar from a compliance standpoint, right? It's another thing to regularly look at and say, how are we doing this right? Is this the right way? Is this the best way? Is there a better way that we can do this? And unfortunately from a compliance lens, you also mentioned it, right? Controls stay static until they're forced to change. Right. You look at an organization, they say, we've been doing this since we implemented it for the last 10 years. And that's the mind shift that also has to happen is how do we constantly improve the way we're, we're meeting our compliance objectives.
And so to take, to take the very end engineering response to that. So, so we created, uh, you know, chaos engineering and, and all of that stuff to kind of break our own stuff. Right? So the same rationale I've been trying to push as compliance as code, right? Not policy as code that's, you know, configuration everything, but compliance has code. So pairing a developer with a risk person to say, yeah, try and write a test, that commits code, and then approves your own pull request, right. Try and break the system. And then you can continually run that on every single build. Right. Um, and, and that really means just like a content expert, uh, would, would be helping the developer with the functional requirements around, uh, a user story. This would be a risk or compliance person who's helping you write the test that proves that the control works.
Typically, if you're bringing in these risk and compliance folks early on in the process, you can have them go ham on an, a QA environment on a lower environment. If you want to fail it on it and fail it fast, have them look at one of your lower environments and say, here, come in and take a look, tell me what I'm doing wrong.
If you can't hear me quick, follow-up to dark, uh, compliance has scored, right? If you think about securities, uh, sorry. It's related. So think of, think about security, right? You have secure coding, you have threat modeling that we're starting to teach, develop developers, right. Is there something like that from a compliance perspective that you guys, you said compliance as core, but is there something like that information
Audit perspective? You know, actually we, we try and get developers to push bad code because they're always pumped to these downstream monitoring controls that will catch anything. So we're saying, okay, well, if you can push this go, let's see if it falls out on the back end. So I mean, happy to talk to you, you know, I work in the ad tech space a lot. And so, you know, one of the things are, they'll never charge for duplicate clicks. I'm like, okay, now push code that all of a sudden, now you're going to start recognizing revenue on duplicate clicks. It's so hard to push code. And so some of these things, we definitely try and see if we can test that monitoring that they have on the back, it would detect some of these things that we really care about in the bad code.
So just following up into that, where what we're reliance on compensating controls can go wrong in a major way. If you always point to the same compensating controls, in addition to where those controls serve as a primary code, you have so much weight now and so much dependency on it. And if that breaks, you're failing a lot. So pushing things and being reliant on other compensating controls, you have to really be methodical in terms of the frequency that you do it. And think about what we call aggregation risk. What's the risk. If this control now breaks to everything else in the system,
Hello is the three lines of defense model dead. Cause I hear a lot talk about IAA being involved in consulting and doing things that you know, where I come from. You don't do as internal audit. What's your perspective on that?
I've got one at, I mean, I think the three lines of defense are working much more collaboratively with each other. Uh, it doesn't mean that it's necessarily dead. I mean, I, I, in fact, I think it's actually strengthening it. Uh, we're in, you know, there's, I'm seeing projects in the last 12 months, not just even with the pipeline, but just in the whole security portfolio where we've got projects that are chartered paid for by the security organization and the internal audit function, and they're working on it together. And I'll give you one specific example. We were doing, um, a security assessment where both parties are paying for it within that same enterprise. And we're, you know, we're doing a kind of a data-driven assessment. Uh, we're doing a traditional kind of subjective type of, of, uh, interviews and things on top of it. And then we're taking that same body of work, and we're going to go and have, write an audit report over here with it and a security over here for the security organization. They've saved a ton of money. They haven't wandered out of their own boundaries and there are three lines and everybody's benefiting because they're collaborating and they've adopted that partnership type approach that we've all kind of been intimating.
So the three lines, you know, for many, many years, if not decades, there were silos. Every single one was siloed. They didn't talk to each other. It was a checklist based approach. And that was it organizations today with transformation, with the speed of technology, the collaboration that's needed, the knowledge share that's needed the technical skills that are needed. You're still maintaining that objectivity in terms of being able to, you know, to, to apply those principles, but it's much more flat and it's, and those silos are really just from a budget lens.
It's about speed. I mean, you, if you, if you don't move faster, you die in this culture. Right. And so they're doing an out of survival and you don't need to break down kind of your fundamental role in the organization to move faster. You just have to move more collaboratively with each other.
Yeah. I have a question. So once a year we have external auditors who come in and audit us for, uh, ISO 9,001 and TL 9,000. And they come and audit various products that use different, you know, styles, whether it's waterfall agile and, and the, uh, the dev ops styles. Uh, and then they find, they find they have some findings and then through some kind of a magical formula, they come up with major nonconformances, minor nonconformances and sometimes, uh, things that we need to improve on my question to you is ends specifically on the dev ops style in your thoughts, what is that criteria that they use to say, oh, okay, this is a major non-conformance versus maybe a minor non-conformance.
So, so every firm has, has their own approach in terms of classifying errors and findings and translating them. So the language you just used, you know, it's probably specific to a particular audit firm, but in a nutshell, right, there's different criteria. That's typically risk-based that leads you to a level of severity. However, it happens to be defined and there's different factors. There's the frequency of, of an error. There's the severity of an error there's should there have been a compensating control? Was there bells and whistles going off? And you actually had the logging, you actually had the monitoring, but nobody did anything. So it really depends on the, really the facts and circumstances of the findings that potentially could lead to different buckets of severity. And especially with,
I saw they have the specific domains, I'm more familiar with ISO 27,001. So major nonconformity is you cannot meet one of the criteria, right? So that could potentially, they can pull your certificate. So there are some of those things where they're looking at, and then a minor nonconforming might not have not been in compliance with your own policy and then areas for improvement. Typically, it's just like, you know, we're adding value to your business.
You do have an internal audit department. They should come in before the auditors and do a readiness exercise.
So last question, go ahead, stat.
And to add onto that. So I understand the determination of risk is going to be very specific to the, to the company, to the application, to the environment. Uh, but are there any specific controls that would be common that will determine risk? I mean, we, we seem to be throwing some examples around separation of duty. You talked about, uh, uh, in, uh, vulnerabilities, you talked about test automation. Are there any, like if you do this, these 10 are the most common that everyone in their pipeline should include to judge the risk and then everything else will be kind of context.
So I got this standard answer. I give to this question, it's called background checks, right? Because background checks are in every single one of these rules. And typically there's a, you know, a standard approach or that all the background check agencies utilize. And it's really a joke, but it's facts and circumstances. The spirit is the same. The application is going to be different depending on the type of cloud you're using, depending on your data, depending on your application. So I'd love for there to be like a unicorn way that it's always going to be a checklist, but we talked a little bit before the checklist was sort of gone now and we take a risk-based approach and it may vary come find me after. And we can double click on that. Yeah.
But there, there are going to be some things that are always going to be common, so especially around access. Um, and so if you're being controlled by the tool, um, you can't have access to go in and change the configurations of that tool. Um, and so regardless of what those configurations are, that's going to remain, remain true across the secure pipeline. Similarly controls around who can change those configurations and making sure that changes to the tools themselves. So more on the ITGC side of the tools, um, there's definitely commonality in those controls. Now what exactly what they look like is going to change, but around security access, locking down those pipelines, that's going to be consistent.
We've got a little bit over one minute left. Um,
You can take a last word if you want. I was actually, okay. So for those of you in the room, we have a session tomorrow where you can go deep, privately and ask anyone here, specific questions, and you can say my goodness hours cases, so messed up. What should I do? And for those of you who aren't in the room, but are watching this on YouTube, if you're a situation is so messed up, that you're not doing compliant DevOps, and you think that audit is blocking you, we would like you to reach out to anyone on the panel to say, how do we fix it? Let's give a hand to our panel.
Unlimited users from organization
Clarissa Lucas’ Audit Playlist
Matt Bonser, PricewaterhouseCoopers LLP; Yosef Levine, Deloitte; Jeff Roberts, Ernst&Young; Michael Wolf, KPMG; Gene Kim, IT Revolution
Auditors' Workshop - What You've Wanted to Ask an Auditor but Were Afraid to Ask
Matt Bonser, PwC; Pierre Fourie, Ernst & Young; Sam Guckenheimer, Microsoft; Yosef Levine, Deloitte; Darren Orf, PwC; Topo Pal, ; Caleb Queern, KPMG; Jeff Roberts, Ernst & Young; Amy Sword, Deloitte; Michael Wolf, KPMG
DevOps and Internal Audit: A Great Partnership
Rusty Lewis, Nationwide Insurance; Clarissa Lucas, Nationwide Insurance
DevOps and Internal Audit: A Great Partnership (Part 2)
Clarissa Lucas, Nationwide Insurance; Rusty Lewis, Nationwide Insurance; Ethan Culp, Nationwide Insurance
From Your Auditor Friends: What We Wish Every Technology Leader Knew
Rusty Lewis, Nationwide Insurance; Clarissa Lucas, Nationwide Insurance