00:00:17

So I'm super excited about the next two sessions. So over the years, we asked the community the top obstacles of things they want to achieve. And almost every year when they talk about what the obstacles in their way are, it's audit that strikes the most fear, dread, and frustration. Because of special powers, they have to generate findings that are seen at the highest levels of the organization and their use of sometimes decades old audit practices. And so I am personally grateful to all the work that Clarissa Lucas has done. She is director of Technology audit at Nationwide Insurance based in the US and is one of the largest mutual insurance companies. And over the past two years, for reasons I'm just now starting to understand, she's presented at this conference with some very, very specific, uh, and useful advice about people who have to work with audit.

00:01:05

And amazingly, she and her team have come up with a very specific technique on overcoming issues, specifically around seg uh, separation of duties, change, approvals, and so forth. So in the past, uh, she's presented with fellow auditors on her team. But, uh, earlier this year, to my utter shock and surprise, she planned on co-presenting with someone in technology leadership, uh, at Nationwide, which to put it lightly, is just something that is not normally done, at least not in polite company. So she's presenting with Todd Bickley, associate VP of Information Risk Management, responsible for the identity and access management systems. It is a shared service, which is so important because so many major applications rely upon it, and they describe a startling audit engagement model. In fact, it's startling. Uh, and I say this based on, um, decades of seeing talks from ICA and the I, uh, the isaka and the Institute of Internal Audit, uh, communities. So here is Clarissa and Todd to talk about what they did, why they did it, and the value they created. Here's Clarissa and Todd.

00:02:17

Welcome everyone. Let's kick this off with a quick question. How many of you brought your auditor with you today?

00:02:26

Just me.

00:02:29

Oh my God. The Nationwides did. <laugh>. Oh,

00:02:31

He works with us. So,

00:02:32

All right. Let's try another question. How about this, um, raise your hand or make some noise if you've ever been dreading the auditors coming, or you've cringed when you heard the word audit or auditor. Okay. Okay. As much as my heart is breaking right now, it's all right. You're in the right place.

00:02:52

As Jean mentioned, I'm Clarissa Lucas, and I'm here with Todd Bickley. And we are going to walk you through Nationwide's journey to a better audit experience through auditing with agility. We'll explain the benefits that you can experience when an audit is performed in this way, and then we're gonna give you some actionable takeaways that you can take back to your organizations to influence a better audit experience. I'm the Technology Audit director at Nationwide. I've been there for 10 years. I've got about 15 years of audit and risk management experience. Helping management and auditors work together towards that better audit experience with more value for your organization is something that I'm incredibly excited about. It's something I'm passionate about. So really excited to be here with all of you live in Las Vegas. Todd.

00:03:31

Great, thanks, Clarissa. Uh, my name is Todd Bickley. I lead the identity and assets identity and access management capability at Nationwide. And when we say i a m, we're talking about authentication, authorization, single sign on, identity governance, M f A, all the things. Uh, I've been at Nationwide about 20 years. I actually started on the I a M team as an engineer, uh, in the early two thousands. Uh, after a couple years, I looked around at my fellow engineers and said, you know, I'm, I'm just not as smart as these people. Uh, so I became a manager and started leading different teams in infrastructure and the application teams. And then in 2019, uh, came back to the i a M team as its leader, and came back, uh, to, to several things that, that, that were going on. Um, we were having problems with flow, we were having problems with getting work through the system, uh, and that was my charge to help do that.

00:04:19

We also had some problems with audit issues that kept reoccurring, that we weren't getting our arms around, that kept coming up. So a little bit about Nationwide. Um, so Nationwide is a US-based company. Uh, I think we're 80 in Fortune 100 is what we saw. Um, we're really an interesting, unique company in terms of, we're a large property and casualty company, uh, and we're also a large financial services company. Um, and our financial services company, up until about 15 years ago, was publicly traded, and the property and casualty side was not. So the technology's really evolved differently over the years, which adds a lot of complexity. Um, so, uh, about 15 years ago, the companies came back together. Uh, we're now a mutual, we're privately held, we're owned by our members. Uh, so it's great. Um, so we're number one in 4 57 plans. We're number one in several property and casualty areas, and pretty much in every other place you're gonna see we're in the top 10.

00:05:13

So, uh, a pretty big company overall. Um, one thing that I think is important to mention about Nationwide Two is, uh, we're big in our communities. We are avid supporters of Nationwide Children's Hospital, uh, of Hungry Leaf, hungry Leaf of the United Way Feeding America. We take real pride in contributing to the communities, uh, that we have businesses in and that our associates live in. And that, that, that really makes me proud to work there. So let's back up to 2019, sort of the before times. Uh, when I came in and when they asked me to come back to Identity Access Management in, in late 19, I said, absolutely, because I really, I love I a m and I love it for two reasons. First, uh, you know, i a m and my subjective opinion is the foundation for Zero Trust architectures. And actually to be in a conference in Vegas and not hear to the word zero trust for a cyber guy in the last day and a half is shocking to me, because usually I hear it 700 times a day when I'm out here.

00:06:07

Foundation for Zero Trust. The second thing I love about it is it's a set of technologies that touches every associate, every member, every customer, every business partner, every day. And I love being involved with something that really has such an impact. So if we go back to 2019, when we, when we take the rollover, uh, a lot of your common things were, were, were going on in the team, uh, that was just causing issues. Uh, we had no flow of work. The teams weren't organized effectively. Uh, you know, there was no priority. Everybody was just chasing things around, uh, individually. Uh, so I came in and took the team over, and my boss at the time had just come from the, the software side of the house and had implemented Agile and product and, and, and DevOps methodologies all across the app teams, but we'd never done it in the infrastructure space.

00:06:53

And he said, Hey, it's a new team. Why don't we look at putting in product models here? And I said, well, it can't get any worse, so let's, let's give it a try. Uh, so we did really within the first few months, uh, we just started doing all the standard DevOps agile things that application development teams have been doing for years. We reorganized our teams and product teams. We had plan and build and run all on the same teams. We created product management and product owner models. Uh, we started, uh, getting all of our, uh, demand and backlogs in Jira and doing prior prioritization and unit costs and sprint deliveries and all those cool DevOpsy things that all the developers have been doing for years. Uh, and we were really, you know, doing well. Like the team was focused and, you know, engineers were actually, you know, only going to meetings in the morning and actually doing work all day instead of sitting in meetings figuring out what they're supposed to work on.

00:07:43

Uh, and we were also in the middle of a large technology transformation then. So, uh, in about, you know, the middle of the year, in the middle of this, I get a call from my auditor friend, Clarissa. She says, Hey, Todd, congratulations on your new role. Um, it's time for your biannual i a m audit. I said, awesome. I'm in the middle of a multi-year multi-million dollar technology transformation. Uh, we are in the middle of a huge product reorganization on how we do work. Really, the, the first thing I want to do on top of that is an audit <laugh>, but in all, seriously, I, you know, it, it was, it was fine because, uh, for a couple reasons. First, um, you know, audit work, when you look at it, it's just demand into your team. Like any other set of demand, it's typically lots of evidence asking when you're in the IM team, we own tons of controls.

00:08:27

So we're generating that evidence that that Clarissa's team has to validate it's meetings to set up, to go through evidence, it's meetings, to set up, to review results. Uh, it's, it's really, if you think about it, just like the demands you would get from any other infrastructure team or application team, I said, great, Clarissa, you, you know, love to have your team come in, uh, you know, let's, let's, uh, get this done. But we cannot do this in a waterfall method. Um, the team's all around Agile, we're delivering in sprints. They've got everybody going around it. So we need to really think, uh, agile, uh, in terms of this. And Cliff said, that's awesome because I've been thinking about Agile and reading about Agile, you know, for the last year or so, and would love to kick the tires on how we start to do agile deliveries with audits, uh, within your team.

00:09:12

So that's great. Uh, that's absolutely how it happened. Uh, but it wasn't quite that simple or quite that easy. So both of our teams, the auditors and the I a M team faced a couple challenges when we were figuring out how to change our ways of working and not bring our waterfall audit to the i a M team. So, uh, the first challenge that the auditors face was a fear that we would violate those professional auditing standards that were held to, to make a long story short, there's no violation of those standards. We were able to modify our ways of working and still comply with those. But walking into this, we, we didn't realize that we as the auditors also didn't really have any experience or knowledge, uh, in Agile or DevOps ways of working. So that, that also created some, some hiccups for us. And then there were some major cultural and procedural changes from an audit perspective that we needed to overcome in order to change our ways of working.

00:10:03

Yeah, and you know, we have some challenges too on our team. Um, you know, first the, uh, sprint approach, um, and aligning on how we were gonna deliver the results, uh, was new for the auditors. So we had to really work with them to help understand the agile meth methodologies. Um, next time, I think we're gonna put a bigger emphasis on upfront planning. Yep. And this was just a little bit of a result of a situation, right? The call, we're in a transformation. We didn't have a lot of time to do planning. I think next time, which is coming up here in about six months, we'll sit down for a day or so and say, Hey, how do we want to lay in these controls out across our sprints? Um, and the third was just, uh, you know, auditors didn't, auditors had knew about Agile, and they had read about Agile, but they had never done anything with it before. So, um, really just helping them through those agile processes. And one thing I've loved over the last couple days that I've heard here is just empathy for people around you. You can have empathy for your auditors too, because they are people and they, they really do want to help you and the company make sure that you're staying secure.

00:11:01

Definitely. So let's take a quick backup and set the stage with, with some context about, um, we, I know we've mentioned this waterfall audit. Let's figure out where we started so that we can see where we were going with this experiment. So for as long as I've been an auditor, and even well before that, we've used a waterfall approach to conduct our audits. So similar to the waterfall concept and software development, a waterfall audit is a very gated and phased approach that's got dependencies. So we start out, we plan the entire audit, identify all those key risks and key controls, write up our testing procedures, get a request list together, get that all approved. Then we move to field work. And field work is when we execute everything that we planned to do. So we test all of those controls, document our evidence, and our work papers get those approved, and then we move to reporting.

00:11:45

And in reporting is when we finally get to deliver those results to our clients. So based on the reactions I got in the beginning of this, I'm gonna guess you're pretty familiar with the challenges that this type of an approach, um, presents both in software development and during an audit. So I won't spend a ton of time on that, but I do wanna touch on a couple. The first is in regard to, um, applying that same rigid audit approach to every situation. Uh, this used to work, this waterfall approach worked really well in the past because risks were pretty static. They didn't change. We could go in and do the upfront planning and then take all of our time during all of the other phases and then deliver results at the end. And not much really would change. And there are still some situations where that makes sense, but thinking that we can apply that in every situation in today's environment where risks are changing with a velocity that I've never seen before, doesn't, doesn't really work.

00:12:35

The second major challenge with this approach and applying this in every situation is with regard to feedback. So we hold that feedback to the end. Uh, not only do we wait until that reporting phase to deliver feedback to Todd and his team, but we also wait to solicit feedback from him. Uh, and I do love that we get that feedback. Uh, waiting to the end isn't the best because when Todd says, Hey, it would've been better if you would've done this. We really missed that opportunity to, to pivot during the audit. And the best we can do is a few years later, when we come back to audit him again, is hope that we've still addressed that feedback. Um, so a number of challenges, but on this experiment, we had encouragement and coaching from Todd and his team, and we were able to modify our ways of working to address those.

00:13:18

It's not just that nationwide that we knew we needed to change from the waterfall approach to other approaches. This is something that the entire internal audit profession has been, uh, working towards. And a lot of organizations that are making these changes have moved to what's called agile auditing. Agile auditing is really a sprint based delivery model for an audit. So you divide the audit timeline into sprint time boxes, and you deliver iteratively at the end of each of those sprints. This is great because you're doing feedback more frequently, but it again, is applying that same rigid framework to every situation without taking into account the team and their delivery model and the environment and all of those other things. So, made some progress there, but, um, still applying that rigid framework and assuming that one size fits all. So what we did is we did a sprint based delivery model for this audit, because Todd and his team delivers in sprints, but we also applied the three ways of DevOps.

00:14:08

So created kinda this customized approach, the first way of DevOps the flow in systems thinking. We really came together as one team. So it wasn't the auditors and Todd's team. We weren't sitting on opposite sides of the table. We were together as one team collectively trying to provide assurance over those key controls that really mattered to the organization. Uh, that was great because we were no longer getting each other's way of Todd's trying to do this. The auditors are trying to do that. We were really focused on that one goal together. The second way of feedback loops, so we addressed this in a couple of ways. The first was delivering in sprints. We delivered every 30 days. So instead of waiting until the end of the audit, which was a few months later, and handing all of the findings and assurance to Todd, we delivered every 30 days.

00:14:51

And that enabled his team to start working on some of those findings and, and gaps in the beginning of Todd. These are the controls. These are the most important controls that you are absolutely your team has to get right. And either they're working right, so go get sleep tonight, or they're not working the way you want them to. So start working on those. So by the time we get to the final audit report at the end, a lot of progress was made. We also implemented feedback loops by intentionally soliciting feedback throughout the audit. At the end of each of those 30 day sprints, we had a retrospective review with the entire team. Uh, during that first retrospective review, we got feedback on the frequency of our meetings. Big surprise, right? So at that point, we were meeting twice a week. We would meet on Tuesdays as a working session where we would do control walkthroughs and document requests and questions and answers, and then we would meet on Friday to do some of the same things and talk through the results of that week.

00:15:45

And in that first retrospective review, we got feedback that, um, instead of meeting twice a week, they wanted to meet with us daily. <laugh>, yeah, let that sink in for a second. Management wanted to meet more frequently and have more meetings with their auditors. Frankly, I, I had to play it back in the retrospective to make sure I heard that correctly. And I wasn't just imagining and dreaming. Uh, but no, it, it was, it was an actual request. So obviously we, um, we did that. We moved to daily meetings. They turned into standups. They were really only 15 minutes, and we got clarity on requests. Todd's team's questions were answered, my team's questions were answered the same day instead of waiting a few days or even a couple of weeks. And then finally, the third way, the continuous learning and experimentation. This is something that we had built throughout the audit from day one when it was let's experiment and change from Waterfall to try adding agility, try adding, um, DevOps concepts to this audit.

00:16:40

We knew it was gonna be an experiment, and it was, it was gonna be clunky. Um, but because we had that mindset built in, that really came in handy, especially when we got to the reporting phase. So while we delivered in sprints, we still kind of had a final report at the end. So we had delivered interim reports every 30 days to get those, those results into Todd's hands. We compiled all of that in a final report with our overall audit opinion. Now, where we went wrong was the auditors went back to our desks and did that in a silo. And that was not the way we had conducted the rest of the audit. We had been doing everything together. And when we delivered that report to Todd and his team, it was a surprise. And it was not this kind of surprise that's like chocolate or champagne, it's audit findings and, uh, an audit report that nobody loves. A surprise audit report. So it got really clunky. It got really, um, it was a challenge. But because we had built that, uh, continuous learning and experimentation, we had built the trust throughout the audit. We came out stronger on the other side and we worked through it

00:17:37

<affirmative>.

00:17:39

We also implemented things like making work, visible self-organizing teams, and greater collaboration. I'm gonna kind of breeze through that just looking at the time. I wanna make sure we stay on time. Um, but we talked about some of the benefits that we received. We talked about this throughout, but I wanna highlight them. The first was greater collaboration and engagement within both teams. Within that collective team, we were able to focus on the areas of greatest priority and highest risk to the organization. We were able to successfully adapt to change. So in delivering in sprints, if something changed in the environment or we learned something and we were able to pivot and say, we thought this was important to focus on next, it's actually not. Let's look over here. We had greater buy-in because we were collectively as a team. Uh, there was a lot of input from both Todd's team and my team. So the whole team bought into the work we were doing and the results that we were delivering, and those results were delivered, delivered more timely, and we reduced the amount of time that we wasted.

00:18:35

So that's great in theory, but what does that actually look like? We got faster. The amount of calendar days that we spent auditing Todd's team reduced by 10.5%. We also, um, increased the speed with which we delivered those gaps to Todd and his team from when we identified them to when we got them in, in his hands so he could do something with them. In addition to, to getting faster, we got better. So because we delivered those results sooner to Todd, when we got to that final report that showed everything that we had identified together throughout the audit, we were able to show progress. So instead of saying, Todd, here are a bunch of findings, good luck, key stakeholders, look at all the stuff Todd has to do, it was look at all of the assurance that we collectively provided, and here are the things that we found that need improvement.

00:19:23

And look at all of the progress that Todd and his team has made already on those completely different story. We also got more coverage. So we were able to cover more identity, identity do domains throughout the audit than we did, um, in 2019. So 2019 we did waterfall. 2021 was when we did this better way of working. And then we submitted, um, we sent a survey at the end of the audit to Todd and his team to talk about, um, and identify what went well and what we can improve on going forward. So we collected throughout, but then also kind of wrapped it up with a bow. And the satisfaction rating that Todd and his team gave us improved two levels from 19 to 21.

00:20:02

Yeah. So how can you get there is really, you know, partnering with your auditors. Help them get through that learning curve. Uh, if you have a way of working that's very, you know, you're in the dev, DevOps, and agile methodologies, you know, partner with them to, to kind of go there with you and sort of take, take that journey. Um, demonstrate to them how to run effective standups. So that was one of the first things we started. 'cause it was the biweeklies. And then we said, you know what? Those really don't work for us. Let's, let's go to our daily standups. Uh, teach 'em how to use the tools that you're using. Uh, whether it's in Jira or whatever you use for backlog, or your Kanban or scrumban or whatever methods you're, you're, you're gonna use and bring them into your team. Uh, keep an open mind, have empathy like we talked about earlier, um, and encourage your teams to participate with the otters as they are one team. Uh, because really when you look at it, they, they are just here to help the company and help protect you, uh, from things that are going on.

00:20:51

Yeah, we all have the same objective. We're here to help the organization make sure that they can achieve their objectives. And management is trying to achieve those objectives. So instead of working against each other, let's work together. So we have some help that, that we're looking for here. Um, I would, as much as it will break my heart, I do need to hear some of these audit horror stories. So connect with me and let me know your, your worst, what went wrong. Um, I'd also love to know if you love your audit experience and just totally forgot to bring your auditors with you today. What went really well? What are some of those things that, um, you know, I, I can learn from you and take back and experiment with at Nationwide?

00:21:31

Yeah. And mine is actually away from the audit thing. So we're on a journey to just eradicate passwords from our environment, both for our associates and our, our members and our business partners. If you are on a passwordless journey, would love to hear the things that you're doing, what your plans are, blockers, uh, because we are hot and heavy into moving forward with that right now.

00:21:51

So I appreciate all of you hanging out with the auditor in the room today. Um, next year, bring your auditors, teach them about DevOps, and, uh, just really appreciate it. Looking forward to connecting with, with each of you afterwards. Great. Thank you. Thanks everyone.

00:22:09

Awesome. Thank.