00:00:02

I'm Jeremy castle. I'm a director at state farm, and I have Kevin O'Dell with me. Who's also a director. Um, today we're going to talk to you about state farm's dev ops journey. Um, and I'm going to kind of talk about all the great stuff that we've been doing to enable dev ops. And then Kevin's going to kind of come in and talk to you about, Hey, this is the reality of what it took to actually implement at state farm. So to give you a little context on the size of state farm, um, we're 36 on the fortune 50, a 100 company. Uh, we have about, uh, 19,000 agents and 5,800 or 58,000 employees. And we're based in four locations. Our headquarters is in Bloomington, Illinois, but we have hub locations in Atlanta, Dallas, and Phoenix. So we're a large company that spread across the U S and multiple locations.

00:00:48

Um, to give you a little context on the size of our enterprise technology department, uh, we have approximately 6,000 employees a little bit more than that. We're probably closer to 7,000, um, 2,700 of those would classify as software developers or infrastructure analyst. Um, we have approximately 2000 web apps, and then we still have a ton of COBOL P uh, and PD-L1, um, any type of technology. We probably use it at state farms. We're a very diverse, um, uh, set of technologies. Um, we have about 1200 products and about 15 different business areas and in our enterprise technology. So just large department with a lot of challenges in dev ops is, um, kind of an interesting journey for us. So, um, a little bit about me. I am a, uh, I've, I have responsibility over a horizontal area at state farm. So I enable what we call a lot of our developer experience. So things like Jenkins or CIC D tools get lab, um, best practices for a software development life cycle. And Kevin actually works in a business area. So yeah, I'm in the, um, automated

00:01:57

Fire or I'll say auto and homeowners area. So we actually build the products that we give to our customers using a lot of this stuff that Jeremy and his team kind of enabled for us.

00:02:06

Yeah. So here's what I'm gonna start my talk, I'm going to start with a little bit of a story. So back in 2015, I was a developer and I worked on a small persistent team and all of us, you know, we had at least 15, 10 to 15 years of experience were senior developers, and we're sitting around, we'd just finished rewriting our auto quote and purchase application. We're getting ready to put it on state farm's internal cloud. We all kind of started talking to each other like, Hey man, when when's the last time you went out to production and one of my coworkers was like, wow, you know, God made four or five years ago. And someone else is like, I've been here like 10 years. I don't think I've ever put anything out in production. And I'm like, same thing for me. I'm like, it's been made three times in my career.

00:02:47

So you could kind of see at state farm DevOps, you know, really for us was very long releases. You were on projects for potentially years. I would never get implemented. We did a lot of great stuff. We had a lot of CIC D practices, but it's really hard for us to release. So a couple of months after that, I had gotten promoted to an it architect. And, you know, I got, you know, ascended into my ivory tower with whiteboards and expos. And my first thing my boss asked me and my boss at the time is one of the greatest people I've ever met in my life. But she, she turned to me and she goes, here's the first thing I want you to do. I want you to figure out how to get code from your workstation out to production in less than 24 hours.

00:03:31

And after I kind of picked myself off the floor, you know, I was like, I don't even know how to tackle this problem. I really understood from a developer standpoint, how to get code from my workstation onto a test environment. But after that, it was very throw it over a wall to a test organization, they'll test it and then they'll throw it over operations and they'll get it out to prod. I had literally no sense of what it took to go out to production. So where do we actually start? I was like, why I really need to understand how we're delivering stuff from workstation out to production. And that's where I start with a value stream map. And this is a picture of our value stream map. And this is about 20 feet long, and this is actually on, on Lisa's office at the time.

00:04:13

Um, this was a very interesting exercise because what we found was it was approximately 1500 hours to go from workstation out to production. And that included about 150 steps, um, about 45 different handoffs. And there was about 15 or 16 different, um, unique roles just to get something out to production. Um, so that was, uh, you know, when I actually kind of looked at this and you started seeing some really, um, some bottlenecks in your value stream. So here's a really good example for us from a security standpoint, I, I used to have to sit there and talk to an analyst. They would run a security scan, they would send it over to an InfoSec analyst who would analyze it, then send it back to the person who ran the scan, which would come back to me. That was the 72 hour SLA just to get one, one single security scan.

00:05:05

So were able to actually use this value stream map to actually come into, um, you know, meetings and say, Hey guys, you might think this process is great, but when I put this all together, it's just not going to work, right. We, we don't have 72 hours of wait for potentially a scary scan. Um, so what actually ended up doing is I took the numbers from, oh, I'm going to have to go back here. But I applied the numbers, um, from that auto release. And we actually came out to almost 9,000 hours a little bit over 9,000 hours just to get that out to production that was with like, you know, 130 test deployments, several out to our pre-prod environments. And you'd walk into meetings and say, Hey guys, we're spending 9,000 hours just to get something out to production. People would, you know, their jaw dropped.

00:05:48

So that's not possible. Like now when you, when you look at it, this is really what it's taken and it's not all manual work, some it's around, but still it's still a bottleneck for us. So as we applied that, um, you know, our delivery time was two to three weeks. We were actually able to get that down to, um, two to three hours using that value stream map to actually drive out automation, uh, making investments in more modern cloud native platforms like cloud Foundry, public cloud. Um, so the way we tackled that value stream is the first half was really getting stuff out the test environments. And we, we spent a lot of time automating our pipelines, um, breaking through process, find a lot of unnecessary process. But the second half of that really had to do with our change process. That was a beast of a process to walk through.

00:06:38

So just some numbers to talk through on this. Prior to this, we had a manual change process. It took, um, we had 13 different processes. You had to go through an HP service manager. Sometimes there was over a hundred manual fields you had to enter, um, six different roles were involved to see the change process. And then you have three different approvals. And that just took a lot of time and energy and a lot of handoffs. So the other thing we focused on was how can we actually automate our change process? And that's where we ended up just burning the whole thing down because state farm we're heavily regulated. We have a lot of auditors. We have to prove we have controls. What we ended up doing was I think we can, we built API APIs around some of the things humans could do were doing previously.

00:07:22

And then we ended up extracting a lot of the data that went in the change records and actually pulling those from developer tools. And pre-populating our change records. We were actually able to get that down to one approval for manual fields, changed process, went from weeks down to hours for us. So, um, the cool thing about this is we've been able to use this process. We've picked one platform at state farm, and then we turned around we'd, we'd genericized, basically our change process with through automation. So as we build out other platforms like public cloud or somewhere else, we want to go Kubernetes environment. We were able to just bolt on this change process and get big lift, um, down to sometimes people now can go to production 15 minutes, which I was a week long thing before. And now it's, it's a much quicker process for us.

00:08:10

Well, then what happened? We got to the point where like, okay, we've done all this cool stuff, but we have, you know, seven, 8,000 people we need to get on board with this. How do we, how do we bring this out and roll it out to the masses? That's where, um, I actually stumbled across some videos from this conference three, four years ago, where target came and talked about it and I'll give target, you know, a pat on the back, they were very welcoming to us. We reached out to them like, Hey, can we visit your dojo? And I was able to fly up with our team and actually, you know, we walked away and they shared a ton of information with us. We're like, Hey, I think this is something that's really going to work well for us. And this isn't, this is in 2016, um, learned a ton about, um, the dojo for what target did decide.

00:08:51

We're going to apply it at state farm. And that's kind of interesting story for us because we had really no support. It was just kind of a grassroots thing. And my team was like, Hey, how are we going to do this? We don't have room. We didn't have space. And I'm like, Hey guys, I actually know a room that's over in a building somewhere to no one really knows about clear schedule off the rest of the week. I got a team lined up next week. We're going to figure out this playbook as far as we can go. And then we're just going to start rolling with it and see what happens, which is actually a really exciting thing to do. It's how often do you at a company almost get to do a startup inside of it? So our first team went through and were like, I think this is really going to work well for us.

00:09:29

Cause we just, you could kind of see the confidence in the team to apply agile, apply continuous delivery principles to their real work. Um, slowly that built out and over 2017, we ended up building out an all four low all four of the locations. So we ended up having capacity to have about 16, 17 teams at once. Um, we sent over 170 teams through and we, we track metrics through surveys and we've, we've seen a pretty big jump, um, in our agile adoption and continuous delivery. So, um, you know, the next thing we kind of found at state farm was, Hey, that's great. We got CICB pipelines. We turned around and we did some amazing stuff with our change process. We've got dodos in place, but we're still working in projects and that's really hard. So that's where state farm started going through a digital transformation. And we shifted last year from projects over to products. And that's, you know, I think that's where Kevin kind of comes in, is Kevin. Kevin can now talk about some of the challenges that they face for us switching over to projects or from projects to products and then how we adopted some of our stuff. So Kevin.

00:10:37

Yep. All right. Thank you, Jeremy. Um, yeah, so, um, all good stuff, uh, Jeremy and his team did, and it was pretty awesome. Kind of watching them off in the distance, you know, blaze the trail, learn from the industry and then supply that stuff for us. But let's, let's talk reality, right? So I'm in an area where we have to build product and get it out to customers. And we've got these timelines, we've got modernization efforts, we've got all these, uh, these things that we need to do. Right. And so we've got dates. And on top of that, we've got a whole bunch of people that are really experienced waterfalls, right? At state farm. You get folks that have been there for 10, 15, 20 to 25 years and it's kind of a norm, right? So it was really about, okay, we have to have a different way of working different way of looking at things.

00:11:21

And that's really where Jeremy's Jeremy stuff really kind of enabled it for us. So I'm going to step back even farther though. Um, before we actually made our journey from project to product and kind of talk about, uh, once we really started getting kind of a whiff of this whole dev ops mindset and different ways of working and kind of hearing what Jeremy's teams were doing. Um, we really started kind of challenging the ways that we were working at state farm. So, um, I've got a whole bunch of learnings, a whole bunch of learnings. We've got a lot of documentation and I'm going to walk through five things that I thought were difference makers in the way that we, um, are difference makers in the way that either we worked or things that we did that really kind of helped us get to where we need to be and really kind of set us up for that movement from project to product.

00:12:05

So first thing, um, if everybody owns something, nobody really owns it. So, um, I'm going to jump right to this doodle here. Um, uh, so this may look familiar to a lot of folks. When we start scaling agile, you start seeing your organization kind of put layers on top of the layers on top of layers that really match those waterfall layers on top of layers on top of layers. And you kind of see the Plinko ball or the Plinko chip kind of go down and then finally hit a dev team at the end when everything's kind of specked out, like just go build this, right. Well, what happened with that when we're trying to adopt all of Jeremy's stuff with the CIC D and with the pipelines we looked over and we've got this big solution and we see 40 or 50 red pipelines, and we go over to the developers and we're like, Hey, what's up with all these red pipelines?

00:12:52

And they said, well, we don't really own it because in the way we're doing scaled agile, we're just the next feature that comes up. We're just going to go work on that. And it may not be in the code base that we were just in, right. The next team would get it. And they'll, they'll pretty up the pipeline and they'll make a green pipeline. Um, so we stepped in real quick, got the developers together and said, okay, how do we, how do we get green pipelines? How do we, how do we do the things that Jeremy is teaching us to do? And it really said ownership. We want to own a product. We want to own this thing from initiation all the way until we put it into production all the way into sunset. So we said, let's do it. Um, so got the, got our folks together.

00:13:27

We sliced up our, our big solution at the time. Um, developers are like, yeah, we built, we started building this. And so we w we want to own this, our team's going to have this. So, um, we, uh, we said, all right, let's do it. Um, then I had the, uh, the luxury of really kind of changing the whole, um, I'll say effort at the time and kind of saying, Hey, we're going to have this product ownership. This component ownership is how we're going to change up the way we're working. Well, I got to go over to our product owners, uh, an area that I thought they would go, oh yeah, this is cool. We own a product. Um, it wasn't really that great though. Um, I actually had product owners say, well, now we actually have to know the product that my team's working on.

00:14:04

And I just kind of sat back and looked around the room, looked at some consultants is like, is this the behavior that we want? Do we really want product owners, not even knowing what their teams are building now we do. Right. So that was kind of a proof like, Hey, this is, this is where we need to go. Right. So, uh, we made the movement, um, lo and behold, couple of weeks later, everything's green people were refactoring, uh, updating their tests. It was just, it was, it was almost like a kumbaya moment where like, oh, this actually feels good and Jeremy's going to be really happy now. So, so we're, uh, so we're all good. Um, the cool thing here though, is we really started to see teams, um, get out of that kind of mercenary mindset where, Hey, they're just going to tell me what to build.

00:14:43

I'm going to build it to more of that missionary mindset. So if you guys are familiar with, uh, like Marty Keegan and, uh, his inspired book to kind of talk about that you want your teams to be missionaries, or I give them a product to own that they can drive. We started seeing that with our teams, they really started driving up and they wanted to be closer to the business and say, Hey, we, this is, this is a piece of the system that we own, and you want to really know how business uses it. And we want to be in those conversations. So we saw a lot of mature maturity from our teams. Um, the next thing was, it might be a little embarrassing. Um, this is actually goes back to Marty Keegan as well. And he, he has a statement that says something to the effect of, um, if you're not embarrassed by your first release, then you released too late.

00:15:23

So that's really saying, Hey, we really need to scale down this business functionality. We're going to provide to just get something out to our customer. Um, this is really hard though, when you want to put in CIC D right? So it's going to take some time upfront to, to build in the automation and building the pipelines. Um, and then you got business saying, Hey, I'm used to getting these big chunks of functionality. What's up. Like I need to get all this stuff. So we really had to, to, um, go and talk to our business folks about, Hey, this is the benefits that we're going to get, right? Well, you, you, as you can imagine, when it's like, Hey, my new functionality is very light, but this automation's huge businesses is just not real happy. And, and, and we actually had business in one of our meetings.

00:16:03

They actually said, oh my God, this is embarrassing. We are giving them nothing. So they were afraid to go out to the agents or the other folks that were using our systems, because they didn't want to say, Hey, this is all you're going to get right now. Right. So, um, we actually did to take us up to our executive staff and talk them through it and say, Hey, this is a different way of working. This is going to benefit us, benefit us in the future to be able to make some smaller, incremental, little changes as we learn from our customers. Right. Um, so executives, like, I think so go for it. So they supported us, which was fantastic. Um, that next year after we actually released that small functionality, that next year we went from doing about 12 releases a year to, to over 250 releases.

00:16:45

And almost half of those were using Jeremy's automated software change. So that was a developer checking, something in one approval, boom, it's in production within hours, rather than putting it in a test environment forever. So, um, as you can imagine, we now had a very happy business partner. Um, and the, the cool thing about this is the business was all over it. And they got to make a lot of changes. You know, we were like, Hey, we're going to make these small changes. And, you know, seven months ago, you said you wanted this functionality. Now you've seen your users use the system. What do you think? Well, now we don't want that functionality anymore. We want this, well, that's why we're doing the smaller changes. Right? So business really started buying in and going, wow, okay, this is cool. This is actually benefiting us. This isn't just some systems, shiny object thing that they're doing over there.

00:17:28

So, so that was a really good success story for us. So this one, I love this. You get dev ops and you get dev ops and you get dev ops. This is where I wish I could say that there's a box under each one of your guys has chairs as a key to dev ops in it. Um, but I can't do that. Uh, the point here though is, um, and we we've seen this, uh, both ways at state farm. So, um, you'll see, uh, some folks say, Hey, we need a dev ops team. I need a DevOps team. That's going to do this automation. I need this dev ops team. That's going to do, uh, the deployments for us. So they're, they're going to package this up for us. Or on the other side, you can say, Hey, we want our products team, our product teams to be doing the dev ops.

00:18:08

They need to be our dev ops teams, right? So, um, we've seen a lot of benefit of having dev ops enablement teams. Um, and I'll say, I'll even say adoption or adoption teams. These are the folks that are leading edge on the new technologies and leading edge on the new ways to maybe write test automation and get things into production, but they're not doing the work. They go to the product teams and enable the product teams to do the work. They may build some, um, some, some technology for them to use, but it's really the product teams that become your dev ops teams. They're the ones that are putting stuff into production. Um, not a separate team, cause all that does is creates a silo. And we've actually seen it in a couple areas that took the dev ops team route. All they did was created a new silo and now everybody's pointing fingers at dev at the dev ops team because the production deployment didn't work.

00:18:54

Right. And really that dev ops team, they don't know anything about your application. Why are they putting that in production? Give it to your product team, give it to those developers that built it before they know how it should run in production. And when they have to do those tasks, they take a different mindset, right? They're building a better logging, better automation. So they know that, Hey, what they're building now they're on the hook for, for that 2:00 AM call, right? If something goes wrong, um, number four, I call it the embrace, the dotted line. So we've heard you hear a lot at this conference. Um, we need to really kind of flatten our layers, right? We need to, um, go to more of a flattened organization. Um, when you move from project to product, you're really putting more on your, your product teams, right? They've got more accountability, more responsibility.

00:19:37

Um, the, the problem with this is folks are so used to used to horizontal teams, especially in the waterfall, um, uh, kind of ways of working, um, that they kind of default to that and they go, well, uh, we've got to have, we've got to have a team that's looking across all these product teams to make sure that they're all working, right. The problem with the horizontal team that I've found is once you have a horizontal team across, uh, multiple product teams, then your product teams don't talk. They only talk to the horizontal team and that horizontal team will go in and talk to the other product team. And then it goes kind of back and forth. And then you go talk to a product team and like, Hey, have you talked to the other team? No, I just talked to the horizontal team. And that can be, that could be a test team.

00:20:17

It could be an architecture team, could be a kind of a business type requirements team. It's really, really, really important to get, get all that knowledge on the product team, but then draw some dotted lines, draw some dotted lines across those product teams and say, Hey, you need to be looking left and you need to be looking right. You need to be working with your product team with the, your brother and sister product teams, because you're all on this end to end solution together. Right? So really trying to get them to be focused on, Hey, we're a team, even though I'm at a product team and I only own this product, maybe I'm, uh, I'm in a dotted line, that's quote unquote architecture. I need to be aware of the architecture completely around my system too. I need to be working with those guys just like I did before, if I'm on a separate architecture team, right.

00:21:01

So really trying to get the, um, get the autonomy down to the product teams, allow them to make decisions, but make sure that they're making decisions with, with the mindset of, of looking all around and, and understanding who they impact. Um, one of the key things I will say through this whole process, you have to learn to unlearn. You've got folks that have been doing stuff for a long, long time, right? And you've been, and they've been doing it really, really well. I mean, it's being a state farm where we're a leader in auto and home insurance. We've done some really good things in waterfall has actually worked out pretty well from us for us. But the realities of the industry is we've gotta be, be nimble. We've gotta be able to change in the future. So we have to be building our products the right way.

00:21:43

We have to be working in a different way. Um, so you have to invest in learning. You have to work with folks to make sure that they're educated and they understand a new way to, to work, right? So just, I mean, this is an example you guys be in here, this is awesome. The conversation we have and we can share is great. Um, but the folks that are back at the office, they may not be getting this education and they're not having these conversations. And then you come back and you say, Hey, I heard this at this, at this company, at this company. Right. Do they believe you or not? Right. So you've gotta be very intentional to really make sure that you're, you're educating your folks. And they're hearing this stuff from the industry, everything we're doing is from the industry, right. We're not making this stuff up that I've, I've heard that before at state farm, that dev ops is just a made up thing.

00:22:27

It's that state farm made up and I'm like, whoa, wow. That's, that's harsh. Like use Google, you know, you'll learn a lot. Um, so, but, but then you just have to realize, you have to have empathy for those folks and say, Hey, I need to enable, I need to enable this for them. So, um, if you guys, uh, all day dev ops is something that's coming up next week, um, state farm is actually sponsoring that, which is pretty cool. Um, that's actually where the shirt is from. Um, that's an all day, uh, online dev ops conference. Right. And it's gonna be talking about a lot of the same stuff that, um, we're talking about today. That's an opportunity for folks for free to learn what we're learning. Um, we did, we participated last year, had a lot of folks in, in my Dallas office, um, participate in this year.

00:23:09

We're stepping it up a notch and we have our whole, uh, ITI organization we've said, Hey, this is your day. Take it learn, go to, go to one session, just go to one session and learn something. But if you want to take the whole day and go to seven sessions and learn more about Devoss, do it. This is our new operating model. This is the new way that we need to be working. Um, so that was a, that's actually a picture of Derek weeks. And Mark Miller, you guys have probably seen them. They're the guys that created all day DevOps. So, um, we've been working with close with them, but they sent us a personal message last year that we were able to show to our folks. And our folks were like, wow, this is really cool. Like, we feel like we're part of the community.

00:23:43

So it was good learnings. Um, but after that we had multiple different events that were dev ops focused, right? So we have, we have hack days. We have, um, what we call an it symposium, which is a two and a half day internal conference. And that's good, but it covers everything. Um, we took it upon ourselves to say, we need to have some really focused new way of working hashtag dev ops, um, events that kind of focused on let's let's change the way that we're working in that we're behaving. So, um, the disrupt is, is something that we've started at state farm. Um, it it's we brought in speakers, J Paul Reed, KEMU grudge, um, have come in, John Wells has come in, um, just to give our folks a different perspective. I can sit there and harp on things like you need to do CGI. Like this is what really CIA is, but if they hear, um, can be grades coming in and say, well, CIA is this.

00:24:29

And he says the same thing. I said, they're going to listen to him because he comes from the industry. He just has that cred that I don't have. Um, so we've, we've seen that really work wonders. So I really encourage you to, to have some events and, and get your folks more engaged, give them the education. The other thing we did is we started a, um, a dev ops cohort for our leadership. We saw, we did have a, a gap in our leadership, the folks that you really need to be enabling this. Um, so I just went out there, all this stuff that I've kind of kept in my cubbyhole of, of stuff. I just put it all in a, I put it out and get, and just said, Hey, go through all this material and let's get a group together. So we have, we actually have four cohorts running now, um, two in our Dallas two in Dallas, two in Bloomington, kinda starting at small in our areas.

00:25:11

And we're, we're adjusting as we go a lot of the stuff on there. I mean, actually everything that's on there is from the industry, right? So it's, it's not really anything we made up, but there's training, there's videos, a lot of videos from here, articles, just the things that we need our folks to be looking at and reading to get them excited, to learn more. And we've seen a lot of success from that, but that's an intentional thing that we did to get our folks going. But now we have, we have more leaders and more people coming to us saying, when's the next cohort, you know, kicking off. Like we want to be part of this. We're we're hearing some good things. So it's, it's pretty cool. Um, with that, um, the dev ops community is great. Um, you meet people here. I'm amazed how I can send an email to somebody that I met at a conference.

00:25:53

Um, and, and the responses I get and the help that we get this community is unbelievable. People want to help. So if you reach out to us, we'll talk to you, um, reach out to your friends. Like they're there, there's always people willing to, to answer your questions for you. So, um, the community has been awesome, but with this slide, I'm just gonna give a shout out to our doodler. So you notice a lot of doodlers, um, that is Lisa Fry that she works with us. This is kind of a hidden talent of her. This presentation was ugly when we first created it, she made it a lot better. So I just wanted to give her props for that job.

00:26:29

Um, so we're gonna jump into the, the challenges that lie ahead, um, for us, uh, autonomy versus alignment is a big one for us. Um, as we say, Hey, we've got empowered product teams that need to have that autonomy. Um, we're really struggling with where is that line of autonomy? Um, where is the alignment, um, what do we feel comfortable with those product teams making a decision on, um, what are some things where the, these kind of curves where we should say, oh, well, you know, let's stay, let's stay between here and here. Right. Um, we don't wanna, we don't want to put too much rigor on things, but we also don't want things to kind of fly off the, um, the Ana bars. Um,

00:27:07

We've had to pick our battles there with like a lot of things we did with our automated software change process. We kind of funnel everyone into that, but it's a good thing because you get a consistent experience and we've actually seen really good results out of that. I think our, um, um, adverse change rate for people that for manuals about two, 3% people that use our automated, um, are like 0.3%. So we've just seen big wins too, versus kind of saying, Hey, do it this way. We're going to give you curbs other places. So, yup.

00:27:38

Um, helping our teams get better. So this one's really about metrics. Um, there's a whole bunch of metrics out there and you can read books on agile and metrics and all that. Um, there's so much out there that we hate to just say, Hey, teams need to do all this stuff. Right. So we're really trying to like focus, you know, we've got the Ford dev ops metrics, right? Like, are those rights or what else is there? And it really all comes back to how do we enable our teams to get better? Um, so we've got some things going on, um, going on in a couple of our areas that try some different things, but that's something that we've been kind of keeping an ear out for, um, at the conference to see kind of what other folks are doing.

00:28:10

Yeah. We've, we've definitely struggled with metrics because state farm traditionally really likes to have, you know, this, your defect rate, this is the number of hours it's going to take for you, you know, for this project to end, we've had a lot of challenges where people want to take velocity points, convert those to hours and project out two years of work. And that doesn't, that doesn't really work. Right. So,

00:28:30

And that, that actually gets us to the last one. So modernize, you kind of hear that you shouldn't necessarily have monetization efforts, right? I do it kind of small and, um, that's great, but that's really hard to do in practice, uh, especially when you've got a lot of folks that are kind of used to these, these big efforts, big waterfall roll-outs um, so really trying to figure out ways to enable the, the new mindset of working in smaller, smaller, incremental deployments, especially when you're talking about modernization. And we talk about modern modernizing, large, you know, maybe policy, admin systems, things like that. Those things are huge. Those are really hard. So it's really hard to say all is going to give a piece of it. Right. Cause it's all kind of intermingled today. Um, so really trying to figure out the best ways for us to kind of, kind of do the modernization, but then also modernizations that naturally kind of reach out years after years after years, while we're some numbers, I see a team velocity, boom, let's take that times that by 17 and boom, there, we're going to deliver here. Right? We're trying to get out of that kind of some of those actions and, and figure out what's the best way to have a, a roadmap ish look out to give our, at least our, our folks, an idea of when we're gonna give them some functionality, but not strangle our teams and say, Hey, you got to tell us when you're going to deliver things.

00:29:41

So just to add on that from a dev ops enablement standpoint, or kind of where my space is at, we've made some investments in things like cloud Foundry and you know, public cloud. And we've actually had, you know, just the last month we had over a thousand production deployments to our PCF environment and that's, everyone's using automated software change to go out there. That's an amazing story. We have teams that sometime go to production multiple times a day. And that was absolutely unheard of for me back in 2015. So to see where we're at and where we're, you know, where we were and where we're at today has been night and day, it's actually been extremely exciting for us. So pretty awesome.

00:30:18

All right. Cool. Well, that's, uh, that's all we had for you today and we have 2019 seconds 18, so we hit our time. But if you guys have questions, send us emails, whatever.