Las Vegas 2019
Overcoming All or Nothing DevOps
For teams in our organization, DevOps has felt like a mountain which must be scaled, and arriving at the summit was the measure of success. Many teams felt this was too daunting an effort and struggled to start. We lost sight of the journey and were too focused on the (perceived) destination.
We realized we were focused on the wrong thing.
Now we have stopped talking at teams about the values and importance of DevOps. Instead, we are meeting teams and leaders where they are today, and using DevOps practices to solve problems they care about. In this talk I'll share some of the starting points for teams we have worked with, and how we have helped them to overcome their fear of the unknown and embrace DevOps as a journey rather than a destination.
KC
Kevin Chaloupecky
Product Manager, Principal Financial Group
RS
Ross Sickora
Software Engineering Coach, Principal Financial Group
Transcript
My name is Kevin . I'm currently a product manager with principal financial group, which is based out of Des Moines, Iowa. Um, I've been in the it field for 20 years or so a hell of a lot of different roles. I've been an engineer, I've been a team leader. I've been a technical lead. Uh, but currently I serve as a product manager. So I've gotten the opportunity to be a lot more exposed on the business side and get an opportunity to work directly with our it teams now and what we need to deliver. Right?
Yep. My name is Ross Sikora. I'm a software engineering coach with principal, uh, been doing the software development thing for about eight years now. Uh, three of those years I've spent with principal, um, get the opportunity to work with advancing our engineering practice throughout the organization. Uh, I've had the privilege of working on some pretty large monoliths Java web applications, uh, helping to rewrite a Greenfield mobile experience for our organization. And then now this opportunity to be an engineering coach. Um, so really exciting time to be a software developer, um, really exciting opportunity to come into a large organization like this and drive change. So,
So mayor very briefly I'll introduce, uh, principal financial group. If you're not familiar with it to tell this story, I'm going to take you back to the year 18 79 2 things happened in the year 1879. First the light bulb was invented very appreciative of that. Uh, the second thing is that a banker in Des Moines, Iowa, uh, realized the hardships that families would go through when his fellow bankers passed away unexpectedly to try to help with this, these hardships. He created a life insurance company called banker's life. Uh, that company has grown over the last 140 years into what we know today as principal principal, financial group, a fortune 500 global organization, uh, working to help people live their best lives through financial services.
A little bit more about just real quickly what we do. Uh, we focus on planning for the future. So retirement planning, we have a lot of 401k plans and Aesop plans and, uh, annuities, things like that. We work on, uh, protecting the present through, uh, insurance, life insurance, disability, insurance, et cetera. And we do that through what we believe to be an outstanding workplace culture, a culture that's often been recognized as an outstanding culture to work in a little bit about our technology footprint. Uh, we have about 2,900 technologists in the organization about 65% of those. US-based another 25 in our, uh, captive group in Kuna India. Uh, we have men computer world. One of those that I'll talk about from a culture perspective on the best places to work in it. Um, but like many of you, we are constantly faced with challenges, new challenges coming at us. One example, uh, the, we recently acquired the Wells Fargo, institutional retirement and trust business that will dump double our retirement administration, that, uh, business that we have to support it'll make for some outstanding opportunities for us, um, and challenges for us and how we provide technical solutions to be able to handle changing business needs like that.
Okay. So let's see. So we've kind of painted the backdrop of who principal is and where we work and, and the framework in which we operate. Uh, so let's get into the why you're really here and that's to hear about dev ops, right? So from a DevOps adoption perspective, it varies widely across the organization. I just talked about how varied our organization is. Um, it's a large organization, it's a diverse organization. Um, our adoption also varies a great deal across the organization. We have some teams that are very far in the journey. We have other teams that are not nearly as far today. Um, but even in those areas where we have made some great progress, Ross and I both have an opportunity to work in areas where we've made some really good progress when it comes to, uh, the DevOps adoption, a lot of significant opportunities, uh, still remain even in those areas for improvement. If you were to look at the dev ops metrics, uh, the, the deployment frequency, meantime to recovery, some of those that I'm sure you're all very familiar with, uh, collectively across the organization, we're probably in that low to medium performing, uh, against those metrics today.
Yep. So a little bit more about what we're hoping to, that you can walk away from here today with, so we want to share with you where we started on our dev ops journey. We also want to share with you, what's going well right now, as we see it for us, uh, what's slowing us down what staying in our way. And then finally, what do we wish we knew a few years ago? And we kinda got going, uh, down this path of adopting some of these DevOps practices and principles. Um, so without any further ado, where do we start? So I think we probably started in a place very similar to a lot of you in here, where there were a few wild hairs in the, in the organization that started here about this new way of delivering software. Like this started to blur those lines between dev and operations. So once those people started to kind of spin up and get going, they started doing, they started getting the books, right? The it rev books who in here has been given one of those books.
Okay. Now who has given those books out to someone else in their organization. Right. So what's really interesting is once we started giving out these books, it starts to look like Halloween, right? Uh, we just around corporate America, we just hand these things out like they're candy. So once they started getting handed out to our engineering teams, they would look at us like we just asked them to climb a mountain. Right. And they're like, uh, how do I get there? And there's a lot of great steps in the books, but ultimately our, our groups were paralyzed. They're like, well, we've never climbed a mountain before, so where are we going to do now? Well, then we started stalling, the virtues of dev ops to these folks, right? And we're like, dude, Facebook is doing it. Amazon's doing it. Netflix, Google, Microsoft, any other like high performing technology company that you hear about there.
And our engineers were like, uh, dude, we're not Google that man, but, but there's, but there's a journey to this, right? Uh, we S we had to start reshaping that conversation. Once we started to understand like how big this mountains seemed to all of our engineers. So we started doing is re articulating it as you're not trying to climb up Amazon's mountain. You're not trying to scale Everest. What you're trying to do is you're trying to find the mountain that works for you. Now, you're going to go climate and you're going to go do that thing. And then you're going to go journey off to another mountain in our mountain that we're trying to climb up today is different than all your mountains. That you're all trying to climb today. And tomorrow it's going to be a different one again. And what we're trying to help people understand is the journey of getting there is really the reward.
So what's going well for us. What have we done? That's been successful so far. First, we created what we call engineering teams. So we found some of these like-minded people, some of these early adopters, some of these people that wanted that, that were anxious to go climb that mountain. Let me go, all right, give me, give me my, my, my, my climbing tools and just let me go. We found some of those people, and we put them together on teams and we said, all right, go for it. And they've can deliver some amazing things. When these people that are like-minded together and you, you give them the tools that they need and you just let them go. They can do amazing things. So we've created some engineering teams around the organization, Ross and I both had an opportunity to work with some of those, uh, forward thinking engineering teams.
We created what we call the principle dojo. So an immersive learning opportunity where we can take a team and we can separate them a little bit from their day to day, day work. We can give them some, uh, some coaching. We give them an engineering coach, somebody like Ross. We give them a product coach, and we can really work with those teams every single day, day in and day out and trying to get them to understand what it means to do some of these things that are dev ops. And every day we just get a little bit better every day. We just get a little bit better and that's so important, uh, solving for some pains. Um, so a question for all of you, you all have jobs, right? And swimming, you all have jobs, so you all have jobs and you all expect to be paid for those jobs, right?
Imagine what it would be like if you didn't trust your employer to get that money in your account on the day that it was supposed to arrive in your account, what wouldn't you be able to get done? If that was the case? Right. It's a little bit similar to where we found ourselves. Okay. Um, the part of the organization that I work in, uh, is responsible for, uh, paying out all of the, the, uh, financial advisors for the work that they do. We have some amazing financial advisors out there. They work with their clients. They want to sit with their clients. They want to understand their client's needs. They want to understand what their clients are trying to achieve with their financial planning, right. They're out there. And that's what they want to do. Okay. We sit on the back end of that, and we make sure that they're fairly compensated for the work that they do.
We pay out over a billion dollars a year in compensation to all of these various financial advisors, and we get it right. 99.9% of the time, actually more than that, but 99.9% of the time we get it right. But if you're in that 0.1%, that doesn't matter. Right. And we found ourselves making some mistakes. We found ourselves struggling with a system that was built on older technology. It was built on brutal technology. It often broke, and we didn't necessarily know that that something happened midway through when something got lost. Right. And so, uh, our customers, our financial advisors would call us a couple of weeks later and say, Hey, you didn't pay me. Right. And we'd have to go to a bunch of research. And then that would take time from us. And they were frustrated and we were frustrated. It was a bad situation.
Right? Well, so as a product manager, what I would love to do is say, you know what? Get rid of that system. It's built on brittle technology. It's not going to scale it. Let's just replace it. Right. And so I'd love to get one of these engineering teams to go focus on that. That would be ideal. That's going to take time and it's going to take money. It's going to take a lot of effort to get there. And we'll, we'll go down that path. But in the meantime, what we did is we said, you know, what, how can we at least make sure that we know we have a problem before our customers know that if there's a problem, okay, how can we put some system telemetry in place to make sure that we understand our systems? Okay. And so that's what we did. We put some telemetry in place. We put some mornings in place. We put some checks in place. We did all these things to try to make sure that we understood what happened before it affected anybody outside of our walls. Now we've got a lot safer environment in which to operate. And now we can focus one of these engineering teams on truly delivering excellent software, which is what there'll be good at.
Yeah. So this next point here of show, don't tell I, I'm kind of a living embodiment of this for us, right? Um, our organization saw value in bringing in individuals who are technical. They can help our engineering teams advance their practices, adopting practices, such as test driven development, integration, testing, and implementing your elk stack and all those other great tools that we have out there to get the problem solved that we need to solve. So I'm one, we've got a couple other folks that are full-time with the organization. And then we also bring in consultants to demonstrate these practices, to our engineering teams, who haven't been exposed to these ways of working. What we found is this is way more effective for us than just giving them a book and saying, here, go do the dev ops, man. You know, so what's slowing us down now.
Good question. Right. Well, we asked for permission a lot. We are a financial services organization. We have 140 years of history. We have a lot of folks who have spent a lot of their career with this organization. And I've learned that asking for permission is the best way for them to sort of get things done at up to this point. So, because we have this permission asking, there's a lot of things that just have to go through a lot of red tape and a lot of bureaucracy to be approved. Well, we want people to go out, try some different things, swing for the fences. And if we have a culture that avoids risk, and we have to ask permission, those that those experiments don't happen often enough for us. So there's another component to our culture where there's an adverse, uh, reward for the risk that people take to try different experiments within our ecosystem.
So what I've heard countless times from our engineers is like, there's no, there's no reward for me doing it better. If I just ship stories, then that's good enough. I don't have to make the system better. It doesn't matter. Whoa, man, we need to re we need to reassess that right now. And we're working on that. We're working to understand how we can reward teams who are trying different things and going further and try and working harder to deliver a better system instead of just output. So another thing that we need to pay attention to is the way that we have designed systems in the past we're corporate of wine. We are here an investment company, right? We will have put money in a place and let it grow. So we want to put money into a software system and let it flourish. Right? We just want it to sit there for 20, 30 years and give us a return on that investment. Well, that's a different paradigm than where we're working. Now. We want to have modular systems that we can change out different components of right. We want to be able to make changes on the fly when we come up with a better way to deliver software. Well, that's, that's a really different paradigm than where we've been coming from for the last 30, 40 plus years that we've been building software systems.
Can you go back one? Sorry, hit the button a little too quick. Uh, the last one up here is about outputs versus outcomes. Um, so many of you probably work in a large organization of some kind. It may have some familiarity with scaled agile framework, safe, right? Very popular framework and safe is, is phenomenal. It's great. It's got some excellent tenants to it, right? Make all work visible and take an economic view. And all those things are fantastic. Okay. The way we chose to implement safe, we missed the Mark A. Little bit. Okay. Uh, because what we did is we took all these teams. We said, all right, we got, uh, a bunch of teams together. And we put all the features up on a wall and people started taking things and it was exciting and it was great, but it was how we measured it, that where we made some mistakes we looked at, um, well, did you get the thing done?
Right? Committed versus completed was a metric that I heard about over and over and over again. Well, this is what you committed to get done. When you're the PI started here, we are at the end of the pie. Did you get it done? And so a lot of teams would, would say, all right, we're just got to get this thing done. Right. Because there wasn't the sense of ownership because next BI I'm probably going to work on something different. Right. And so they, they, they would, and the phrase I heard over and over and over again from teams was, well, it's done, but I found that scary it's done, but we skipped a few things along the way it's done, but we haven't really tested whatever that, but was right. That's what I heard over and over again. And that's where we really struggled with our implementation of safe.
And that, that lack of ownership is really what cost us. Right? Um, this quote is one that we really like anti ownership is anti dev ops. And that's where we found ourselves. So then what we did after that is we said, you know what? We're gonna, we're going to skinny this down. We're going to take a smaller number of teams and put them on a set of capabilities that they're going to own start to finish. And we still implemented some of those safe tenants that I was talking about earlier, but now they own it. Right? So now they own the whole thing. And we quit talking about committed versus completed. And we quit talking about arbitrary PI boundaries. And we started talking about the outcomes that we want to do achieve. And those teams that small set of teams is interested in those outcomes that they're trying to deliver. That's worked really well for us.
Okay. So I'm back with what we wish we'd known a couple of years ago. So for anyone who's starting down their dev ops journey right now, these are, these are three things I think you should pay close attention to as you go forward in your journey. So first realize the place that you're doing some things, right. So when I started with principal, I came from a smaller company where I got to run some sweet ant scripts on my computer to build our deployable. And I used to ask a guy on the phone, Hey, where do you want me to put this a deployable? So you can get on the server for me. And I drop it over there and it was pretty great. I thought this is how it's done. So then I CA I get this role with principal where I'm, uh, building some of our, uh, larger Java web applications.
And they've got Maven and they've got bamboo and they've got all these cool tools. I didn't know existed at the time. I was like, wow, ha I can just like ship this thing. That's great. Then I got this on-call call. And I knew, I knew from while I was getting called about that, we were just going to have to restart a server. So I got on the horn with our ops guys. Hey, ops guys, like, can you guys go ahead and just bounce this server node for me? It'd be great, dude. Check your email. Okay. So I checked my email, I got a link. I got this link. They build self-service. They had like a self-service web portal. It was awesome. I'd never seen such a thing. So I could restart my own server. I can check my own logs. I didn't have to call anyone.
So that was a great, there was a great mindset there that had that thing built. We want our developers to be able to self-serve get the things they need. They shouldn't need to put a ticket in. Now today, three years later, all the developers look at that thing. Like, why do we, why do we still have this thing? Like, why isn't it better? Well, because we treat it like a project, right? We built this, this capability out for interacting with our web servers so that we could self-serve and then we didn't keep going. Cause it was just a project we were out of. We just weren't supporting it anymore. Right. So because of that, we have to bring those things to the front. Those things that you've built, that actually allow for self-service those things that have the right mind, that have shown that you can progress and you have the mentality of continuous improvement.
You need to bring those things up and bring them to light. Like, look, we were already doing these things. We were already taking these steps to be better. We just have to keep going. We have to take what made that possible and go forward. So then in addition to that, we let, uh, the other thing that we wish we know a couple of years ago is that we were really not focusing on the right people. I'm sure anyone in here has read the books and that are out there, right? We need our early adopters to be the first ones to make change happen well in our organization, what we did is we tried to craft the message for everyone. Kumbaya dev ops is going to be great. We all need to go along on this journey together rather than going and picking out our best engineers and doing the political work to get those folks together, to start creating a critical mass of greatness for us.
So we started, we started doing that. Now we've got some more progress to make there with finding those right people in those right areas to create more centers of critical mass within our organization to move us forward. And then finally, one thing we S we are still struggling with, and we are trying to get better with, but we've seen a lot of growth is actually celebrating the victories along the way. So saying thank you to the folks who implemented the telemetry or came up with the idea that telemetry could solve this problem faster and cheaper than the other, the other option of rewriting the whole dang system. Right? So think of those folks are in your organization. They're driving change within your organization. When you go back at the end of this conference, tell them, thanks, Tom, thanks for doing the hard work of driving change in your organization, because it's going to mean a ton to them.
So where do we go from here? As I said in the opening, uh, we are, as our company is as old as the light bulb. So we have 140 years of history. That's a firm foundation on which to build that's fantastic, but we also know that what got us here, won't keep us here. So what are we going to do to make sure that we stay here? And that's some of the stuff that we've been talking about. So, so far we're going to accomplish that with this third one, which is a focus on culture. Okay, well, you heard that this morning in the talks, we're saying the same thing. We have to focus on the culture. That's going to get us better every single day. Um, culture is hard. Culture is very, very, very difficult to change. There was a quote I heard, um, the other day.
And it was, you don't change culture with memos and emails. You change it, one conversation at a time. I thought that was so true. Right? So, so to Ross's point, right? We tried to bring everybody along. We tried to send out this big email, here's what we're going to do, right? That's not going to be effective. How are you going to get that culture to just build on itself and just continue to grow? That's one of the things that we're, that we're focused on moving forward. Um, the last one, great people will do great things. We have outstanding people. We know that it's been recognized over and over again by being a great place to work and all those other accolades that we've received, but outside perspective can help too. Sometimes you need that consultant or that person was a outside side perspective or, or things that you're gathering at a conference like this one, right? That's, that's powerful, uh, stuff that you can take back, uh, to your team to make sure that they get the help that they need.
So here's our ask for you. Do you have stories? Do you have successes? Do you have challenges? Where can we learn from each other? We would love to have a conversation with you. That is the advantage of having the very first lot here on a, on a Monday is that we get a chance to talk to all of you for the next three days about your journey, about what you've learned. Um, some of the things that you could share with us, and some things that we could share with you, uh, potentially in more detail. So we would love to have those conversations with you. Um, now after this talk, the next several days, seek us out. We'd love to have those, those conversations with you. So thank you very much. Um, the clock down here tells me that I have about six minutes left, so questions, um, and I'll just repeat them and we'll go from there. So what questions you guys have? Yeah. Right back there in the back. Okay.
You mentioned devil audio concept. What was, I mean, how many weeks do you spend you make sure that you make that specific?
Yeah. So, so the question just for the, for the recording is all about, uh, the, the, um, the, the dojo and how long do teams come and what does that look like? Uh, Ross can probably speak to this better than I can. Um, I know that we support about six teams at a time right now. So it's, it's still relatively small trying to kind of grow that and get that bigger, but maybe you can talk about some of the specifics.
Yeah. So currently what we do is we do a six weekend, six week out, six weekend rotation with teams that are selected to be in our dojo. So what we found is that when teams were in for six to eight weeks, and then didn't ever come back, uh, are back in the wild surveys were showing their teams were regressing their practices back to where they'd been previously. So this is an effort to help reinforce those concepts where we go six in six, out in six in, so we can do six at a sixteens at a time. And that, um, that iteration seems to be working well where it's not set in stone. Um, we've done it anywhere from six to 12 weeks in a single goal before as well. Uh, it's finding what works for the folks in your organization and what drives change in your organization. For sure.
I think you had a question right down here.
How do you show value of engineering level? Usually it's all excitement at the same time, um, exited the support needs to be in there, like getting that.
And she, so the question is about how do you show value, right. So how do you show value to the rest of the organization about some of the dev ops, uh, practices and things like that that we want to implement? Right. That's kind of the question. Um, and it, it comes down to being able to articulate that value from a business perspective, right? So as a product manager, I really want to hear my engineers, tell me why they want to do something, right. What's the benefit going? What's that what's that benefit going to get, right. Um, oftentimes they feel a little bit paralyzed at times they can say, oh, you know, we got all this work to do all this pile over here. Um, as a product manager, I'm more than willing to give up some things over here. If you can articulate what that's going to give me over here. Right. Is that going to be a safer system? Is that going to, um, make it easier to resolve when there are issues? Does that mean? My meantime resolution is going to go down, what are those things that is going to give us? So it's been varied, but it's really being able to articulate that value. That is so very important for us.
And one of the things as well with, uh, like the dojo concept, it takes a lot for an organization to say, yeah, take 12 weeks of our time to essentially go slower to emphasize learning, right? Our goal in the dojo is to help teams learn. So making that business case over and over and over again for why it's better for the teams to go slower for a period to learn these new techniques and practices and implement new tools, uh, and showing that acceleration on the back end. So that's a lot of how I articulate to teams. You have to show the acceleration on the back end. Yep.
Perfect.
I think you had your hand at first,
Before you're in the front row. Go for it.
So, uh, how are they, your dojo is how are you structuring them? Was it like a classroom format? Was it more of an extreme programming piece where you had people kind of sitting down embedding with the team to kind of talk about and cement was real for them in their day to day?
Yeah, absolutely. So, uh, with our dojo, what we do is we take the teams out of their everyday environment and we put them in, um, our spec, a specified dojo space. We give them a product coach and engineering coach. And we generally, uh, ask out of the gate that they, uh, work together using mob programming techniques. And depending on how advanced the team is or how open to change they are, they may progress to more just standard XP practices as far as like pairing test driving. Um, but for the most part, we're S we're saying we'd like them to mob program together. So they all have the product conversations and all, all come up at the same level together.
And we answer your questions and we include in those, but the product management folks are right there with them too. Right. So we bring product owners and product managers and business folks were in there due to participate in that dojo right. Along with them. Yeah, absolutely.
We're just wondering how your folks address that cause
Yeah, they do. Yep. They do. I'm going to get somebody right there headed question two more questions right there.
You mentioned about pulling your best engineers together. I, can you elaborate a little bit about what that looked like? What is it you, uh, did you pull them into a center of excellence? Did you put them into a dedicated team to work a CIC pipeline? Something like that?
I'll, I'll go first. You can go. Um, so in, in my particular area we took, um, so we have in my area, uh, eight or nine teams and several living in India as well. And so what we did is we took, um, kind of the best of that and kind of created one team out of it and got them very focused on, okay, making sure they're building software, the right, that way. We looked at strangulation patterns, we looked at CIC pipeline. We looked at all of those things to try to get them to, to focus on building it the right way. Um, the hard part of that is the holes that it left, where they came from. And that's, that's the political journey. I think that Ross was speaking about is that's not free. Right. And, and so you have to give up some things to get some things, right. Um, we see great value in, in the getting right. Um, so that's why, that's why we've stuck with that. Um, but it's not easy. Anything you'd add.
Yeah. Also rotational opportunities is another way that you can make that happen sorta you're not taking them away. We're just giving them an opportunity to try something new. Uh, that was that's worked well for us as well on like new Greenfield projects. So sort of a reward if you will, for being a distinguished engineer in our organization as well, we'll bring you over and pull you into some brand new project pro product, whatever, so that you can have that opportunity to see something from the ground up and build it the right way.
One last question then the last 10 or 15 seconds here.
So it's kind of a follow on there. And it's the other side of the coin. You talk about rewarding improvement and you talk about focusing on early adopters, but how do you combat say animosity and those who might be left behind, um, who aren't part of that early adopter crew?
That's a great question. Um, and I'm going to be totally transparent, honest and say, I'm not sure we've completely cracked that nut if you will. Um, so yes, there can be. I think it's showing that there's going to be that opportunity for everybody, if you want it. And it's being able to articulate to them, Hey, here's what you're going to have to do to be able to be in that environment. So there might be some animosity, oh, I was left behind. I don't feel like, okay, it's talking about, here's the opportunity that you do have, um, to get there. And it's making sure that they see this is the future, right. Everybody's going to get there eventually. That's what we want. We want to create that. And so what we started to see is a lot of it started with some animosity, but it's grown into, okay, what do I need to do to be over there? Because that's really cool stuff. Right? That's what we're starting to see that transition into. Yep. And we are out of time. So thank you very much. Um, again, we'll be around catches. Love to talk more to you about it. Thanks everyone. Thank you.