Las Vegas 2018

Nationwide's DevOps Journey: The Tipping Point

Building on past presentations at DOES, Jim Grafmeyer and Jared Speno plan to update the group on the tremendous progress that was made in 2018.


Jim Grafmeyer is a forward-thinking architect at Nationwide Financial with a passion for challenging teams to deliver faster through innovation. He has a history of delivering meaningful digital experiences with a focus on the customer. His background in development is the fuel for leading DevOps adoption with an eye towards automating the world. Jim has presented at DevOps Enterprise Summit 2016 and 2017 as well as other industry conferences.


Jared is a Senior Technical Consultant for Nationwide Software Development Services. Over the course of his career, Jared has worked in development, operations, and architecture. In his current role, Jared travels around the company evangelizing DevOps, and helping Nationwide's 200+ development lines adopt DevOps practices.

JG

Jim Grafmeyer

Director, IT Architecture, Nationwide

JS

Jared Speno

Senior Technical Consultant, Nationwide

Transcript

00:00:05

So a bit about Nationwide. I'm sure everybody has seen the Peyton Manning and Dale Earnhardt commercials. My message here is we're way more than just an auto insurance company. We do a whole lot of things. Um, outside of that, uh, especially on in my side of the, the company, nationwide Financial, we're number one in a lot of things, including 4 57 retirement plans, pet insurance, corporate life insurance. Um, we've also been around a long time. We are celebrating our centennial in 2026, and we are a fantastic place to work for. Um, we've been on the Fortune 100 Best Places to Work in the last four years, so let's double click into Nationwide IT and what that looks like. My message here is that when we do things at, at Nationwide, we tend to do it at scale, and we do it across all of nationwide, which will be a theme that you'll see, um, through this deck.

00:00:48

When we talk about some of the changes we're making, some of the cultural changes, they're not done in some, uh, small part of, of the enterprise. They're, we do it across, um, all of our lines. We were very early adopters of Lean and Agile. We've probably made our agile transformation about a decade ago, which resulted in the 250 lines that you see here. We have 250 standing development teams that work on all of our software. Those lines support 65, um, primary technologies, which presents some very interesting challenges that Jared will talk about, um, in a few slides. And then those, those teams are supporting over 2500, 2500 applications that, um, that nationwide it has.

00:01:26

So this is actually as, as Jared kind of mentioned, this is our fourth year speaking at DevOps Enterprise Summit. Um, the, the last year's 20 seventeens, uh, talk kind of ended in this slide. And what we were doing is, is we were, had a lot of grassroots effort up until that point, and we were really getting lots of people knocking on our door saying, okay, I, I wanna do this. I wanna go faster. I want to take my own journey up to DevOps Mountain. And we didn't know how to convey that. We produced this one pager, so this one pager, if you walk around any of our buildings in, in Columbus, Ohio, you'll see this everywhere. And the, the purpose was that we could hand any line, any of those 250 lines a map of, Hey, if you wanna get faster, here's what we would recommend.

00:02:07

And recognizing that each person's path up there at Mountain would be different. Uh, for example, you know, some things poly skilling in, in that north camp, there might be useful to some teams, but not useful to other teams. We gave teams a set of tools that we wanted them to be on. We called them the Outfitters, uh, sticking with the mountain metaphor tools that they needed to be on before they even started their journey. Things like GitHub, things like New Relic, things like Splunk, uh, things that we recommended that you start your journey with. At the very bottom there, Jared will hit on it, uh, in a, in a slide or two, is the DevOps handbook. We saw great value by teams reading the DevOps handbook together. Finally, you know, we got a lot of questions, how do I measure this? How do I know if I'm doing better? And that's where this compass comes into play, where we are very much focused on the Ford DevOps metrics. With our North Star being lead time, we've ac we've actually operationalized, um, a standard way of those 250 teams to see what their

00:02:58

Lead is. So as Jim mentioned, we asked teams to take this, to make it their own and to find their own path up the mountain. And we didn't quite realize how far they'd take that. It started with the picture that you see on your right over here. One of our development lines created this picture where they mapped out all the different practices, figured out how they related to each other, and then used this to kind of chart their journey up the mountain. Uh, they posted this on our internal chat platform and everybody loved it. So it kind of just spread from there. And since then, teams all over the company have built out these boards that show what their journey looks like. And the really cool thing about this is we never asked anybody to do this. We never even suggested it. But what it does show is everybody's commitment to, uh, adopting DevOps into what we're trying to do at Nationwide.

00:03:38

So in 2017, we took a very prescriptive approach. There were kind of a, a core group of leaders that were helping push DevOps practices across the company. And we had a, a lot of opinions about what we wanted our teams to do. Moving into 2018, that model's completely flipped our teams because they're so engaged, they're solving their own problems, they're coming up and generating their own great ideas. And our role has now become, instead of kind of telling people what to do, just collecting stories and then spreading them across the company to help, uh, everybody mature more quickly. So, uh, I'm gonna talk about a couple of stories that kind of illustrate this. The first is this idea of hyperlocal, uh, hack days. So one of our teams in particular was really struggling with getting any of their DevOps backlog done. They were just struggling to get CI done in general.

00:04:17

So they came up with this idea to have a hack day. Uh, with this, they basically took the entire team, they kind of sheltered them from all the normal business and daily work, got off via email, got off of chat, and focused an entire day on just working on DevOps and, uh, CI backlog items. And over the course of that day, they actually accomplished what they would've normally accomplished in about six months, all in just one day. So, uh, other teams heard about this and they loved it. They thought it was great, and it's starting to spread across the entire company. And the really great thing about this is that when you focus the entire team on doing this for one day, they make a tremendous amount of progress. And it makes the conversation a lot easier when you go and talk to your business partners and your leaders and you can say, Hey, we, we increased our delivery speed by X percent, or we, uh, increased our quality by y percent.

00:04:58

When you have stories like that, it makes it really easy to justify getting more time to work on things like this. So Jim mentioned a little bit, we, we use the book club to kind of drive the culture. And the reason for this is we wanted all of our teams and all of our developers to be grounded in industry best practices. We wanted everybody to have the same understanding of what we were trying to do. And, uh, we could have nationwide eyes this and created all of our own material to talk about this. But quite honestly, the DevOps Handbook does such a good job explaining it, that there's really no point in us doing that. So, uh, we wanted everybody to start there. And this year we actually gotta see what happens if a team doesn't read the book. First. In one area of the company, the leaders read the book and they were completely on board with what the book was saying.

00:05:35

So they kind of pushed down from the top a bunch of changes that they wanted to see in their team. It was things like focusing on products instead of projects aligning to a value stream and entire value stream instead of just a piece of it, which is where most of our teams are today. And then a focus on poly skilling. And the result of this was a lot of confusion and pain within the teams. They didn't understand why these changes were happening, and to them, what it looked like was a bunch more work being pushed on, on top of them, uh, with no change to their, their capacity or anything else. So it was just more work. It was, uh, an engagement issue. It was a problem with work life balance, and it eventually boiled over and a developer venting to one of the leaders saying, Hey, we don't understand what you're doing, and this is really hurting us.

00:06:12

So the leader took that feedback really well and turned it around and said, well, have you read the DevOps handbook none of the developers had? So to help fix this, the leader actually used their training budget to allocate them time to take away from the rest of the work that they already had to do so that they could read the book. And as they read it, they started to understand why the changes were being asked for. And even more than that, they were fully supportive of them. So after reading the book, they came back and talked to the leaders and they said, Hey, we get what you're doing. We love it. It's the right direction. We have some ideas. If we could just tweak it a little bit, it'd be a little less painful for the team. And since then, they've had no problems. Engagements back up, work-life balances back where it should be, and the team's moving forward really fast.

00:06:49

So I guess the lesson here is, uh, a change like this, it really does need to be grassroots. Your, your teams need to understand why. And if you try to push it from the top, it's gonna have negative reactions and it might fail. One more story. So, uh, we had our core group of DevOps leaders in the company trying to solve problems. And one of the big problems that kept coming up was this idea of the way that we do data deployments not working. So for, I'm sure a lot of you work at big companies, I'm not sure how it works for you, but for us, if an app team has a data change, they write up a SQL script, and then they submit a ticket to a DBA team who runs it for them, it's, uh, there's a handoff. There's wait states, there's a multi-day, SLA.

00:07:24

So it's all around not a great process. So we kind of tackled this as one of our first, uh, good big problems to solve for the company. And we ended up settling on a tool called Liquid Base, which is open source, and basically set a standard and said, Hey, uh, if you guys wanna use this, it's out there. Go ahead and pick it up and start running with it. We stopped there. We'd intended to go back and kind of help push out the rollout of it, but before we could, teams started adopting it right away. Uh, and one example, we had a team that's working on a large transformational effort on a packaged application, and every time they do a release, they have somewhere in the, you know, 18 or a couple dozen SQL changes that have to be pushed with each release. That's a lot of wait states, that's a lot of pain.

00:08:02

And for them, uh, what they found is things would be run out of order or missed. So it was really slowing them down and they, they were spending a ton of time troubleshooting problems. So, uh, we announced this tool as an option, and before we could ever even roll it out, they picked it up and started running with it. And now for them, they can deploy a data change to all of their environments in the time that it used to take them to just submit a que a request to get it done in one environment. Uh, after that, other teams heard this story, and a lot of, or several teams have adopted this in less than a day. So I guess the story here is, if you provide your developers solutions to things that are painful for them, they'll run with it. They'll just adopt it. We didn't have to do anything to help roll this out. So come up with solutions that really help your teams.

00:08:40

So, uh, one of the other problems we encountered early on was this idea that if I'm not a Java developer or I'm not an Angular developer, I don't do modern web development, then DevOps doesn't apply to me. And we heard this from a lot of teams, you know, I'm a COBOL developer, I do Informatica development, so this doesn't really mean anything to me. So we created this picture to kind of help combat that. And what you see here is the top six technologies at Nationwide, and then a bunch of tools that meet some of the capabilities that we want to implement in the CICD pipeline. And I wouldn't really focus on the tools here. This is kind of just a snapshot in time. In fact, several of these have already changed, so we're not even using them anymore. But the point is that there are tools that meet the capabilities that can enable you to, to implement DevOps practices on every, uh, application stack imaginable. We've actually even had really great success in packaged apps. Uh, that story of the, the team with the data deployments problem packaged application, they're doing really well with their DevOps journey. And all we had to do is kind of help make sure that everybody understood this does apply no matter what technology you're using.

00:09:37

So on the, the journey slide that I showed in the bottom, you may not have noticed it said, one of the things that we needed to work on was support for our delivery pipeline and to start treating that like a product. And one of the problems we ran into at Nationwide was all those tools Jared just showed in that previous slide actually were spread across three different CIOs and six different application owners. And it was very, very hard to drive any sort of continuous improvement across your value stream when you had these organizational boundaries that got in the way. And each one of these silos, uh, you know, the, the app owners for that silo through no fault of their own we're optimizing for what they controlled. And it was very hard to optimize for the whole and apply systems thinking. So one of the things I'm most proud of in, in 2018 is we actually formed a platform team finally where we could, uh, have a, a centralized team that owned all the tools that that hit a developer and hit, hit the, those 250 development lines that I talked about. And you could have a single backlog with single product ownership, single funding to start driving continuous improvement across that. And this team's main purpose is to make sure that those 250 lines out aren't out solving all these problems on their own, that what they're learning is being harvested by this platform team. And then they're disseminating the information out as, uh, as we find new learnings, replace tools, upgrade things.

00:10:52

So if that last slide was Jim's favorite thing, this is probably mine from this year in March, we rolled out these challenges to our teams. And the idea here was to pick a, a concept or a principle or a tool or one of our metrics and have the teams kind of compete in the challenge and tell us how they're meeting the need for whatever that month's challenge was. The focus of this really is just to have the teams talk about how they're doing their work and really think about if they're doing it the best way possible. By the way, does anybody work at Verizon in here? Couple. Okay. So last year I had the opportunity to sit in on the Verizon Cup, and actually Carmen, who's sitting up front, got to judge it. And we loved what you guys were doing with the Verizon Cup.

00:11:25

It's basically a similar competition except a lot more grand than ours. We started a little smaller, uh, but we kind of used that as our, our launch board for what we did here. So we've been running these challenges for a while and we've had pretty good, uh, participation. I think somewhere around a third or maybe more of our lines, half participated. And the really cool thing about this is, uh, when we started these challenges, we didn't know what good looked like in a lot of cases or what maturity looked like for the tools or for the metrics across the company. I'm sure many of you can, uh, relate to this. We look in, we work in a very large company, we're distributed all around the country, and it's very difficult to know what's going on outside of your department. So we have teams participate in these challenges, and then when we pick a winner, we share the stories, we make sure that we tell the story so that everybody can understand what's happening.

00:12:08

And now everybody kind of knows where the bar is. And one of the really cool things that's been happening is whenever we announce a winner, we hear from the winner that they get, uh, teams from all over the company reaching out to them and asking for tips or pointers on how to get mature more quickly. And it's speeding up maturity across the entire company. Uh, often it's teams that would never even know each other existed because they don't work in the same location or the same department. And we're seeing that best practices are organically spreading between our teams much better.

00:12:35

So you heard almost in the, uh, in the Microsoft, um, talk this morning about, uh, PaaS offerings and, and PaaS cloud offerings. So Nationwide's very early in their, in our cloud journey, and we, we quickly realized that if we were going to move to the cloud quickly, we needed to have some sort of opinionated architecture that would allow us to get us there. So we, we brought in a, a platform as a service offering, and what that allowed us to do is gave developers a, a very clear path to how they can run their apps in the cloud. And what we were finding is if we didn't do this and we unleashed our, our workforce on say, native AWS services, they would be paralyzed by all the micro decisions they had to make because we just weren't there from a, from an associate staff. So the, the message here is, don't be afraid to buy a platform like this to get you speed, to help springboard your move to the cloud. And that's exactly what this is providing us. And we're seeing great results from both developer efficiency as well as operator, um, efficiency on the platform.

00:13:31

Okay, so a bit of a a a case study last year we talked a lot about, um, on our property and casualty side. We had a lot of great wind when Hurricane Harvey hit and we were able to quickly adapt our claims process to respond to Hurricane Harvey. This year I wanna highlight, uh, um, a major large transformational program on the nationwide financial side, uh, of our business. So very large transformational program. All the things you'd expect from, you know, audience, from people in the audience here was happening to nationwide. It, we had large, large costs. We were incurring very little software was actually being delivered and we just kept doing. The thing that we tend to do is add more people, add more people, add more people. And you can see where this is going. We got to a point where 65% of the program costs were what I'll call overhead.

00:14:13

And not in a bad way, they just weren't, uh, roles that were contributing working software on a, on a development line. And, uh, we, we had a, a very real realization that we had a number of SMEs say five or six SMEs in this space that really controlled everything. They knew the systems, they knew the business, and because of how large the program grew, they got pulled out of those development teams and put in some centralized pool where they weren't actually able to contribute where they can, can, can contribute best. So of course this killed decision velocity. We were in this mode of consensus building, uh, and, and everything just took drastically longer than it should have. So what did we do? We empowered the teams in, in, in the old world. We made the project, the center of the universe. Everything we did was to support that project.

00:14:59

Everything we did supported the work of that project. And that's how you got to all these additional roles because the project was the center of the universe. You added project managers, you added program managers, you added architects, you added, um, requirements analyst. You had all these roles because you made the work the center of what you were doing. The mental switch we made, although subtle was, was pretty transformational. We started making the teams the center of the universe. And when you do that, you start asking yourself different questions. How do I empower the teams? How do I make sure the teams can make whatever decisions they want? How do I make sure the teams have a lifeline to call if they need help? How do I make sure the teams have the technology they need in place, um, to get their job done quickly? So really switch that mindset, which resulted in, in some of the things you see here on the slide is, is we actually flipped that ratio that we talked about on the previous slide. So now 65% of this program's costs are all in the development teams contributing to working software. And of course, you know, just based on how this was done, you actually reduced the overall program cost by up to 50% because you right-size the teams to the number of SES you had. Those six or seven SMEs I mentioned, we, that basically dictated to us that we can only have six or seven teams. So we rightsized to the number of SMEs and got them back into development lines.

00:16:13

Alright, so on Jim's side of the company, we have a business crisis right now. And maybe crisis isn't quite the right word because it's actually a positive crisis. Our business has so much work that they want done that we're, we're kind of struggling to figure out how to get it all done quickly enough. Um, our CIO on Jim's side of the company recognized this problem. And what he wanted was for it not to be the long pole in the tent anymore. Uh, I'm sure many of you can relate to this as well, but you know, business has an idea and then a bunch of stuff has to happen. Then eventually you have something that gets out into the hands of your customers and it is involved in a lot of that. So traditionally what we find is that we tend to be the, the, the constraint in getting work done.

00:16:50

So Jim, CIO wanting to change this throughout this challenge, essentially a mandate for all of his teams to get 40% faster and the number 40%. There's really nothing magical about that, by the way. What really matters is that it isn't a number like 10%. If you want to get 10% faster, you can tweak things. And with focus you can probably get 10% faster just on making small fixes. But if you wanna get 40% faster, you have to fundamentally rethink the way that you deliver software. And that's exactly what what he wanted our teams to do. So he threw out this challenge and basically said, all teams gimme ideas. Tell me what you need to go faster. And there's really no rules to this. You know, whatever constraints you think you may have because of organizational issues or policy problems or anything else, just ignore 'em.

00:17:29

Tell me what you need to be able to go 40% faster. So through this process, the teams threw out somewhere around a thousand different ideas, which consolidated down to about 200. And we found that they fell into three, pretty much three distinct categories. The first were local changes that teams could solve on their own. They just hadn't had the time to do it. And now we have a list of things and our CIO is basically saying, if you have something you can do to go faster and it's just something that your team needs to do, just do it. You have a permission slip now, uh, even if it means you have to slow down delivering business value work for a little while, it matters. It's very important because it's gonna speed us up in the long run. So just do it. Category number two, where things that our CIO needed to solve himself, and these are kind of the broader organizational problems.

00:18:08

Uh, maybe two areas aren't working together as well as they could, or maybe there's a policy change, anything like that, that really requires an executive leader to help. And an example of item that falls into this space, something that he got early was, uh, related to our Jenkins CI server provisioning process. Our infrastructure organization actually had a wholly 100% automated solution to delivering CI boxes to our development teams. But for whatever reason, it was never made available to the developers directly. So they still had to submit a request and wait a little while, and then eventually they'd get it back manual process in front of automation. So, uh, despite whatever teams tried to to do, to convince the infrastructure organization to turn this over to them, they didn't really have any success. So bubbled up to our CIO, he went and talked to his peer on the other side of the company and the infrastructure organization, and then within the next day it was made available to everybody.

00:18:55

It was just a, a mismatch of getting the communication to the right people. So those are the things he's working on. The final category or things that we would call general broad capability uplift, things that are probably too big for one team to work on, but they could have value to the entire organization if it was solved. And to, to work on these types of problems. Our CIO created a team specifically designed to have that as their backlog, no business value work, and just solved those problems. And that was put together in, uh, sometime over the summer and they've been working since then on kind of these capability uplift type issues. Examples of things in this space include fully automating our RFC process, making our release management paperwork type process completely no touch, which they're actually pretty close to rolling out. I think they're actually testing it with a line right now.

00:19:38

And then other things like automating or streamlining our, uh, our firewall rule process, which tends to be very painful for developers today. So they're kind of working on these general capability uplift problems. And I guess one thing to take away from that is obviously this isn't a team. Maybe you wouldn't have that exist forever, but there's nothing wrong with having a team like that for six months. That basically just helps bring up the, you know, the general maturity of your enterprise. And one other thing about this, our goal with getting 40% faster, it's for delivering for our business, but Nationwide is a mutual company. So we don't have shareholders, we're not publicly traded. So really our goal here is to provide our members the best experience possible and to give them the best products to meet their needs.

00:20:17

So one other thing that we're working on, uh, it's been a focus recently is this, this idea of a leader experience. There's a lot of change happening. We've talked a lot about how we're moving towards empowered teams and, uh, autonomy teams having full control over everything they need to do to deliver software. And then of course, moving to the cloud and PAs platforms. And those all represent very large changes to it. So when you think about, uh, developers and how they're gonna learn this, they're gonna get to experience it hands-on, right? They're gonna live in that world. But what does that mean for our leaders? They're expected to support these teams and they're not gonna have that type of experience. So we need to find a way to, to make sure that they understand the changes that are happening and understand what their role is in it.

00:20:54

And, uh, above all else, just make sure they have empathy for what their teams are going through. So we've been working on developing a leader experience that's gonna help them answer questions like, uh, or move from mindsets like, you know, how do I deliver projects to, am I doing the best thing for my product? Or instead of how do I optimize for cost? Am I doing the right things to make sure that I can deliver software as quickly as possible? Um, additionally, you know, how do I empower my teams? What does it look like if I'm a leader of a team that's really supposed to be autonomous and be driving exactly how they do the work, what does that mean as a leader? So, uh, we're, we're working on developing this curriculum and I'd say we're close and hopefully we'll be rolling it out soon.

00:21:31

So, uh, this is our where we still need help slide, and it's fighting this mindset. Uh, we've been, we've been working on rolling out DevOps across the enterprise for a few years now, but when we go and talk to teams, we still get a lot of pushback like this. They're so focused heads down on delivering business value that still sometimes it's hard for us to find a way to get them to, to work on CI and on working on, uh, DevOps backlog type items. So if anybody solved this problem at your company, please come up and talk to us afterwards. We'd really love to hear your stories. Alright. All right. That's it. Thank you very much. Thanks.