Overcoming All or Nothing DevOps

My name is Kevin Chaloupecky. I'm currently a product manager with Principal Financial Group, which is based out of Des Moines, Iowa. I've been in the IT field for 20 years or so, held a lot of different roles. I've been an engineer, I've been a team leader, I've been a technical lead. But currently, I serve as a product manager, so I've gotten the opportunity to be a lot more exposed on the business side and get an opportunity to work directly with our IT teams now and what we need to deliver. Ross?

Yep. And my name's Ross Sickora. I'm a software engineering coach with Principal. Been doing the software development thing for about eight years now. Three of those years I've spent with Principal. I get the opportunity to work with advancing our engineering practice throughout the organization. I've had the privilege of working on some pretty large monolith Java web applications, helping to rewrite a greenfield mobile experience for our organization, and then now this opportunity to be an engineering coach. It's a really exciting time to be a software developer. Really exciting opportunity to come into a large organization like this and drive change.

So very briefly, I'll introduce Principal Financial Group, if you're not familiar with it. To tell this story, I'm going to take you back to the year 1879. Two things happened in the year 1879. First, the light bulb was invented. Very appreciative of that. The second thing is that a banker in Des Moines, Iowa, realized the hardships that families would go through when his fellow bankers passed away unexpectedly. To try to help with these hardships, he created a life insurance company called Banker's Life. That company has grown over the last 140 years into what we know today as Principal Financial Group, a Fortune 500 global organization, working to help people live their best lives through financial services.

A little bit more about just real quickly what we do. We focus on planning for the future, so retirement planning. We have a lot of 401(k) plans and ESOP plans and annuities, things like that. We work on protecting the present through insurance, life insurance, disability insurance, et cetera. And we do that through what we believe to be an outstanding workplace culture, a culture that's often been recognized as an outstanding culture to work in.

A little bit about our technology footprint. We have about 2,900 technologists in the organization, about 65% of those US-based, another 25% in our captive group in Pune, India. We have been, Computerworld, one of those that I'll talk about from a culture perspective, on the best places to work in IT. But like many of you, we are constantly faced with challenges, new challenges coming at us. One example: we've recently acquired the Wells Fargo institutional retirement and trust business. That will double our retirement administration, the business that we have to support. It'll make for some outstanding opportunities for us, and challenges for us in how we provide technical solutions to be able to handle changing business needs like that.

Okay, so we've kind of painted the backdrop of who Principal is and where we work and the framework in which we operate. So let's get into why you're really here, and that's to hear about DevOps, right? So from a DevOps adoption perspective, it varies widely across the organization. I just talked about how varied our organization is. It's a large organization. It's a diverse organization. Our adoption also varies a great deal across the organization. We have some teams that are very far in the journey. We have other teams that are not nearly as far today. But even in those areas where we have made some great progress, Ross and I both have an opportunity to work in areas where we've made some really good progress when it comes to the DevOps adoption. A lot of significant opportunities still remain, even in those areas, for improvement. If you were to look at the DevOps metrics, the deployment frequency, mean time to recovery, some of those that I'm sure you're all very familiar with, collectively, across the organization, we're probably in that low to medium performing against those metrics today.

Yep. So a little bit more about what we're hoping that you can walk away from here today with. So we want to share with you where we started on our DevOps journey. We also want to share with you what's going well right now as we see it for us, what's slowing us down, what's staying in our way, and then finally, what do we wish we knew a few years ago when we kind of got going down this path of adopting some of these DevOps practices and principles?

So without any further ado, where did we start? So I think we probably started in a place very similar to a lot of you in here, where there were a few wild hares in the organization that started to hear about this new way of delivering software. Like, they started to blur those lines between dev and operations. So once those people started to kind of spin up and get going, they started getting the books, right? The IT Revolution books. Who in here has been given one of those books?

Okay. Now, who has given those books out to someone else in their organization? Right? So what's really interesting is once we start giving out these books, it starts to look like Halloween, right? Around corporate America, we just hand these things out like they're candy. So once they started getting handed out to our engineering teams, they would look at us like we just asked them to climb a mountain, right? And they're like, "How do I get there?" And there's a lot of great steps in the books, but ultimately, our groups were paralyzed. They're like, "Well, we've never climbed a mountain before, so what are we going to do now?"

Well, then we started extolling the virtues of DevOps to these folks, right? And we're like, "Dude, Facebook is doing it. Amazon's doing it. Netflix, Google, Microsoft," any other high-performing technology company that you hear about there, and our engineers are like, "Uh, dude, we're not Google."

Yeah, man. But there's a journey to this, right? We had to start reshaping that conversation once we started to understand how big this mountain seemed to all of our engineers. So what we started doing is re-articulating it as: you're not trying to climb up Amazon's mountain. You're not trying to scale Everest. What you're trying to do is you're trying to find the mountain that works for you now, and you're going to go climb it, and you're going to go do that thing. And then you're going to go journey off to another mountain. And our mountain that we're trying to climb up today is different than all your mountains that you're all trying to climb today. And tomorrow, it's going to be a different one again. And what we're trying to help people understand is the journey of getting there is really the reward.

So what's going well for us? What have we done that's been successful so far? First, we created what we call engineering teams. So we found some of these like-minded people, some of these early adopters, some of these people that were anxious to go climb that mountain. Let me go. All right? Give me my climbing tools and just let me go. We found some of those people, and we put them together on teams. And we said, "All right, go for it." And they can deliver some amazing things. When these people that are like-minded together and you give them the tools that they need, and you just let them go, they can do amazing things. So we've created some engineering teams around the organization. Ross and I both had an opportunity to work with some of those forward-thinking engineering teams.

We created what we call a Principal Dojo, so an immersive learning opportunity where we can take a team, and we can separate them a little bit from their day-to-day work. We can give them some coaching. We give them an engineering coach, somebody like Ross. We give them a product coach, and we can really work with those teams every single day, day in and day out, in trying to get them to understand what it means to do some of these things that are DevOps. And every day, we just get a little bit better. Every day, we just get a little bit better, and that's so important.

Solving for some pains. So question for all of you. You all have jobs, right? I'm assuming you all have jobs. So you all have jobs, and you all expect to be paid for those jobs, right? Imagine what it would be like if you didn't trust your employer to get that money in your account on the day that it was supposed to arrive in your account. What wouldn't you be able to get done if that was the case, right? It's a little bit similar to where we found ourselves, okay?

The part of the organization that I work in is responsible for paying out all of the financial advisors for the work that they do. We have some amazing financial advisors out there. They work with their clients. They want to sit with their clients. They want to understand their clients' needs. They want to understand what their clients are trying to achieve with their financial planning, right? They're out there, and that's what they want to do, okay? We sit on the back end of that, and we make sure that they're fairly compensated for the work that they do.

We pay out over a billion dollars a year in compensation to all of these various financial advisors, and we get it right 99.9% of the time. Actually, more than that. But 99.9% of the time, we get it right. But if you're in that 0.1%, that doesn't matter, right? And we found ourselves making some mistakes. We found ourselves struggling with a system that was built on older technology. It was built on brittle technology. It often broke, and we didn't necessarily know that something happened midway through when something got lost, right? And so our customers, our financial advisors, would call us a couple of weeks later and say, "Hey, you didn't pay me right." Then we'd have to go do a bunch of research, and then that would take time from us, and they were frustrated, and we were frustrated. It was a bad situation, right?

Well, so as a product manager, what I would love to do is say, "You know what? Get rid of that system. It's built on brittle technology. It's not going to scale. Let's just replace it," right? And so I'd love to get one of these engineering teams to go focus on that. That would be ideal. That's going to take time. That's going to take money. It's going to take a lot of effort to get there, and we'll go down that path. But in the meantime, what we did is we said, "You know what? How can we at least make sure that we know we have a problem before our customers know that there's a problem?" Okay? How can we put some system telemetry in place to make sure that we understand our systems? Okay? And so that's what we did. We put some telemetry in place. We put some warnings in place. We put some checks in place. We did all these things to try to make sure that we understood what happened before it affected anybody outside of our walls. Now we've got a lot safer environment in which to operate, and now we can focus one of these engineering teams on truly delivering excellent software, which is what they'll be good at.

Yeah. So this next point here of show, don't tell, I'm kind of a living embodiment of this for us, right? Our organization saw value in bringing in individuals who are technical, that can help our engineering teams advance their practices. Adopting practices such as test-driven development, integration testing, and implementing your ELK Stack and all those other great tools that we have out there to get the problem solved that we need to solve. So I'm one. We've got a couple of other folks that are full-time with the organization. And then we also bring in consultants to demonstrate these practices to our engineering teams who haven't been exposed to these ways of working. What we found is this is way more effective for us than just giving them a book and saying, "Here, go do the DevOps, man." You know?

So what's slowing us down now? Good question, right? Well, we ask for permission a lot. We are a financial services organization. We have 140 years of history. We have a lot of folks who have spent a lot of their career with this organization and have learned that asking for permission is the best way for them to sort of get things done up to this point. So because we have this permission-asking culture, there's a lot of things that just have to go through a lot of red tape and a lot of bureaucracy to be approved. Well, we want people to go out, try some different things, swing for the fences, and if we have a culture that avoids risk and we have to ask permission, those experiments don't happen often enough for us.

So there's another component to our culture where there's an adverse reward for the risk that people take to try different experiments within our ecosystem. So what I've heard countless times from our engineers is like, "There's no reward for me doing it better. If I just ship stories, then that's good enough. I don't have to make the system better. It doesn't matter." Whoa, man. We need to reassess that right now, and we're working on that. We're working to understand how we can reward teams for trying different things and going further and working harder to deliver a better system instead of just output.

So another thing that we need to pay attention to is the way that we have designed systems in the past. We are an investment company, right? We want to put money in place and let it grow. So we want to put money into a software system and let it flourish. We just want it to sit there for 20, 30 years and give us a return on that investment. Well, that's a different paradigm than where we're working now. We want to have modular systems that we can change out different components of. We want to be able to make changes on the fly when we come up with a better way to deliver software. Well, that's a really different paradigm than where we've been coming from for the last 30, 40 plus years that we've been building software systems.

Oops. Can you go back one? Sorry. I hit the button a little too quick. The last one up here is about outputs versus outcomes. So many of you probably work in a large organization of some kind and may have some familiarity with Scaled Agile Framework, SAFe. Very popular framework, and SAFe is phenomenal. It's great. It's got some excellent tenets to it: make all work visible and take an economic view, and all those things are fantastic. The way we chose to implement SAFe, we missed the mark a little bit. Because what we did is we took all these teams, we said, "All right, we got a bunch of teams together," and we put all the features up on a wall, and people started taking things, and it was exciting, and it was great.

But it was how we measured it where we made some mistakes. We looked at, well, did you get the thing done? Committed versus completed was a metric that I heard about over and over and over again. Well, this is what you committed to get done when the PI started. Here we are at the end of the PI. Did you get it done? And so a lot of teams would say, "All right, we just got to get this thing done." Because there wasn't this sense of ownership, because next PI, I'm probably going to work on something different. And the phrase I heard over and over and over again from teams was, "Well, it's done, but..." I found that scary. "It's done, but we skipped a few things along the way. It's done, but we haven't really tested." Whatever that but was, that's what I heard over and over again, and that's where we really struggled with our implementation of SAFe. And that lack of ownership is really what cost us. This quote is one that we really like: "Anti-ownership is anti-DevOps," and that's where we found ourselves.

So then what we did after that is we said, "You know what? We're going to skinny this down. We're going to take a smaller number of teams and put them on a set of capabilities that they're going to own start to finish." And we still implemented some of those SAFe tenets that I was talking about earlier, but now they own it. So now they own the whole thing, and we quit talking about committed versus completed, and we quit talking about arbitrary PI boundaries, and we started talking about the outcomes that we wanted to achieve. And those teams, that small set of teams, is interested in those outcomes that they're trying to deliver. That's worked really well for us.

Okay, so I'm back with what we wish we'd known a couple of years ago. So for anyone who's starting down their DevOps journey right now, these are three things that I think you should pay close attention to as you go forward in your journey. So first, realize the place that you're doing some things right. So when I started with Principal, I came from a smaller company where I got to run some sweet Ant scripts on my computer to build our deployable, and I used to ask a guy on the phone, "Hey, where do you want me to put this deployable so you can get it on the server for me?" And I drop it over there, and it was pretty great. I thought, "This is how it's done."

So then I get this role with Principal where I'm building some of our larger Java web applications, and they've got Maven, and they've got Bamboo, and they've got all these cool tools that I didn't know existed at the time. I was like, "Wow. I can just ship this thing. That's great." Then I got this on-call call. And I knew from what I was getting called about that we were just going to have to restart a server. So I got on the horn with our ops guys. "Hey, ops guys, can you guys go ahead and just bounce this server node for me there? It'd be great." "Dude, check your email." What? Okay, so I checked my email. I got a link. I got this link. They built self-service. They had a self-service web portal. It was awesome. I'd never seen such a thing. So I could restart my own server. I could check my own logs. I didn't have to call anyone.

So there was a great mindset there that had that thing built. We want our developers to be able to self-serve, get the things they need. They shouldn't need to put a ticket in. Now today, three years later, all the developers look at that thing like, "Why do we still have this thing? Why isn't it better?" Well, because we treat it like a project. We built this capability out for interacting with our web servers so that we could self-serve, and then we didn't keep going because it was just a project. We just weren't supporting it anymore. So because of that, we have to bring those things to the front. Those things that you've built that actually allow for self-service, those things that have the right mind, that have shown that you can progress, and you have the mentality of continuous improvement, you need to bring those things up and bring them to light. Like, look, we were already doing these things. We were already taking these steps to be better. We just have to keep going. We have to take what made that possible and go forward.

So then, in addition to that, another thing that we wish we'd known a couple of years ago is that we were really not focusing on the right people. I'm sure anyone in here who's read the books that are out there, we need our early adopters to be the first ones to make change happen. Well, in our organization, what we did is we tried to craft the message for everyone. Kumbaya. DevOps is going to be great. We all need to go along on this journey together, rather than going and picking out our best engineers and doing the political work to get those folks together to start creating a critical mass of greatness for us. So we started doing that now. We've got some more progress to make there with finding those right people in those right areas to create more centers of critical mass within our organization to move us forward.

And then finally, one thing we are still struggling with, and we are trying to get better with, but we've seen a lot of growth, is actually celebrating the victories along the way. So saying thank you to the folks who implemented the telemetry or came up with the idea that telemetry could solve this problem faster and cheaper than the other option of rewriting the whole dang system. So think of those folks that are in your organization, that are driving change within your organization. When you go back at the end of this conference, tell them thanks. Tell them thanks for doing the hard work of driving change in your organization, because it's going to mean a ton to them.

So where do we go from here? As I said in the opening, our company is as old as the light bulb, so we have 140 years of history. That's a firm foundation on which to build. That's fantastic. But we also know that what got us here won't keep us here. So what are we going to do to make sure that we stay here? And that's some of the stuff that we've been talking about so far.

We're going to accomplish that with this third one, which is a focus on culture. Okay? You heard that this morning in the talks. We're saying the same thing. We have to focus on the culture that's going to get us better every single day. Culture is hard. Culture is very difficult to change. There was a quote I heard the other day, and it was, "You don't change culture with memos and emails. You change it one conversation at a time." And I thought that was so true. So to Ross's point, we tried to bring everybody along. We tried to send out this big email, "Here's what we're going to do." That's not going to be effective. How are you going to get that culture to just build on itself and just continue to grow? That's one of the things that we're focused on moving forward.

The last one: great people will do great things. We have outstanding people. We know that. It's been recognized over and over again by being a great place to work and all those other accolades that we've received. But outside perspective can help, too. Sometimes you need that consultant or that person with some outside perspective or things that you're gathering at a conference like this one. That's powerful stuff that you can take back to your team to make sure that they get the help that they need.

So here's our ask for you. Do you have stories? Do you have successes? Do you have challenges? Where can we learn from each other? We would love to have a conversation with you. The advantage of having the very first slot here on a Monday is that we now get a chance to talk to all of you for the next three days about your journey, about what you've learned, some things that you could share with us, and some things that we could share with you, potentially in more detail. So we would love to have those conversations with you. Now, after this talk, the next several days, seek us out. We'd love to have those conversations with you.

So thank you very much. The clock down here tells me that I have about six minutes left. So questions, and I'll just repeat them, and we'll go from there. So what questions do you guys have? Yeah, right back there in the back.

Audience: You mentioned the dojo concept.

Kevin Chaloupecky: Yeah.

Audience: How many weeks do you spend? You mentioned that you bring them in. Is it like a specific time limit?

Kevin Chaloupecky: Yeah. So the question, just for the recording, is all about the dojo and how long do teams come, and what does that look like? Ross can probably speak to this better than I can. I know that we support about six teams at a time right now, so it's still relatively small. Trying to kind of grow that and get that bigger, but maybe you can talk about some of the specifics.

Ross Sickora: Yeah. So currently what we do is we do a six week in, six week out, six week in rotation with teams that are selected to be in our dojo. So what we found is that when teams were in for six to eight weeks and then didn't ever come back, our back-in-the-wild surveys were showing that our teams were regressing their practices back to where they'd been previously. So this is an effort to help reinforce those concepts, where we go six in, six out, and six in. So we can do six teams at a time, and that iteration seems to be working well. Where it's not set in stone. We've done it anywhere from six to 12 weeks in a single go before as well. It's finding what works for the folks in your organization and what drives change in your organization for sure.

Kevin Chaloupecky: I think you had a question right down here.

Audience: How do you show value? At engineering level, usually it's like all excitement, DevOps. At the same time, executive support needs to be in there. What was your journey like?

Kevin Chaloupecky: True. So the question is about how do you show value, right? So how do you show value to the rest of the organization about some of the DevOps practices and things like that, that we want to implement, right? That's kind of the question. And it comes down to being able to articulate that value from a business perspective, right? So as a product manager, I really want to hear my engineers tell me why they want to do something, right? What's that benefit going to get, right? Oftentimes, they feel a little bit paralyzed at times. They go, "Oh, we got all this work to do, all this pile over here." As a product manager, I'm more than willing to give up some things over here if you can articulate what that's going to give me over here, right? Is that going to be a safer system? Is that going to make it easier to resolve when there are issues? Does that mean my meantime to resolution is going to go down? What are those things that it's going to give us? So it's been varied, but it's really being able to articulate that value that is so very important for us.

Ross Sickora: Yeah. And one of the things as well with the dojo concept, it takes a lot for an organization to say, "Yeah, take 12 weeks of our time," to essentially go slower to emphasize learning, right? Our goal in the dojo is to help teams learn. So making that business case over and over and over again for why it's better for the teams to go slower for a period to learn these new techniques and practices and implement new tools, and showing that acceleration on the back end. So that's a lot of how I articulate to teams. You have to show the acceleration on the back end.

Kevin Chaloupecky: Yep. Perfect. Actually, I think you had your hand up first. You go for it, James. Go for it. You're in the front row. Go for it.

Audience: Oh, it was on. So, your dojos, how were you structuring them? Was it like a classroom format? Was it more of an extreme programming piece, where you had people kind of sitting down, embedding with the team to kind of talk about and cement what was real for them in their day-to-day?

Ross Sickora: Yeah, absolutely. So, with our dojo, what we do is we take the teams out of their everyday environment, and we put them in a specified dojo space. We give them a product coach and engineering coach, and we generally ask out of the gate that they work together using mob programming techniques. And depending on how advanced the team is or how open to change they are, they may progress to more just standard XP practices as far as pairing, test driving. But for the most part, we like them to mob program together, so they all have the product conversations and all come up at the same level together.

Kevin Chaloupecky: And we include in those, the product management folks are right there with them, too, right? So we bring product owners and product managers and business folks in there to participate in that dojo right along with them, right?

Ross Sickora: Yeah, absolutely.

Audience: I've been through things like that before. I'm just wondering how your folks address that, because there are things everyone doesn't want to do.

Kevin Chaloupecky: Yeah, they do. Yep. They do. I'm going to get... Somebody right there had a question. Two more questions. Right there.

Audience: You mentioned about pulling your best engineers together. Can you elaborate a little bit about what that looked like? Did you pull them into a center of excellence? Did you put them into a dedicated team to work a CI/CD pipeline? Something like that.

Kevin Chaloupecky: Yeah, sure. I'll go first, then you can go. So in my particular area, we have, in my area, eight or nine teams and several delivery in India as well. And so what we did is we took the best of that and kind of created one team out of it and got them very focused on, okay, making sure they're building software the right way. We looked at strangulation patterns. We looked at CI/CD pipeline. We looked at all those things to try to get them to focus on building it the right way. The hard part of that is the holes that it left where they came from, and that's the political journey I think that Ross was speaking about, is that's not free, right? And so you have to give up some things to get some things. Right? We see great value in the getting, right? So that's why we've stuck with that. But it's not easy. Anything you'd add?

Ross Sickora: Yeah, also rotational opportunities is another way that you can make that happen. Sort of, you're not taking them away. We're just giving them an opportunity to try something new. That's worked well for us as well on new greenfield projects. So sort of a reward, if you will, for being a distinguished engineer in our organization is, we'll bring you over and pull you into some brand-new project, product, whatever, so that you can have that opportunity to see something from the ground up and build it the right way.

Kevin Chaloupecky: One last question in the last 10 or 15 seconds here.

Audience: So it's kind of a follow-on there, and it's the other side of the coin. You talk about rewarding improvement, and you talk about focusing on early adopters, but how do you combat, say, animosity in those who might be left behind, who aren't part of that early adopter crew?

Kevin Chaloupecky: That's a great question. And I'm going to be totally transparent and honest and say I'm not sure we've completely cracked that nut, if you will. So yes, there can be. I think it's showing that there's going to be that opportunity for everybody if you want it. And it's being able to articulate to them, "Hey, here's what you're going to have to do to be able to be in that environment." So there might be some animosity. "Oh, I was left behind. I don't feel like..." Okay, it's talking about, here's the opportunity that you do have to get there, and it's making sure that they see this is the future, right? Everybody's going to get there eventually. That's what we want. We want to create that. And so what we've started to see is a lot of... It started with some animosity, but it's grown into, "Okay, what do I need to do to be over there? Because that's really cool stuff." Right? Yeah. So that's what we're starting to see that transition into.

And we are out of time, so thank you very much. Again, we'll be around. Catch us, love to talk more to you about it. Thanks, everyone.

Ross Sickora: Thank you.