Las Vegas 2018

Scaling Continuous Delivery Across the Enterprise

Like DevOps, continuous delivery is talked about everywhere. Like DevOps, continuous delivery is used all over. However, like DevOps, continuous delivery practices are often not scaled out across the broad enterprise technology portfolio or distributed teams. This session will focus on the opportunities and benefits of scaling continuous delivery across the enterprise and review some recommendations to achieve this.

SW

Scott Willson

Product Marketing Director - Continuous Delivery, CA Technologies

Transcript

00:00:05

<silence> Probably one of the first things to know is who I am. I'm Scott Wilson. Two Ls in Wilson. If you don't get that second l you're not gonna find me anywhere on the internet. Much less <laugh> within my own com company's directory, right? Um, that's my twiddle ha Twitter handle, uh, email address, all that kind of stuff. I actually run product marketing for continuous delivery at ca currently. Um, a little bit of background for me. Way back when, when I started my career before I had too many grays, I got my first gray when I was eight, right? So you gotta bear that in mind, <laugh>. But when I first started my career, I actually started as a software engineer, right? Writing code code jockey. Uh, truth be told, I still do a little bit. Um, I'm actually in the middle of writing a, a little college football analytics program, and I've recently picked up the little language of Python, which was pretty cool doing some, uh, red rejects trying to parse some of the stats.

00:00:58

But anyways, enough about that. But suffice to say, yes, start as a software engineer. Now, I, I do a little bit of marketing with that, uh, that skill and that background. So let's jump into this, okay? Shall we? Uh, first things first. Today is a very momentous day, and it's very apropos that we're in Las Vegas. I dunno if you guys realize how important today is, but it's actually pretty big. Okay? Today's the mega millions jackpot. It is the biggest of ever, right? $1.6 billion <laugh>, I can't even imagine. $1.6 billion. And it got me thinking, since we're here in Vegas and we're at the DevOps Enterprise Summit, what in the world could you do with $1.6 billion? What kind of DevOps things? And I thought about DevOpsy and call me crazy, and I pardon my very quick distraction here, but it got me thinking a little bit about Dr.

00:01:48

Seuss, right? Oh, the DevOpsy things I could do with a billion or two. I'd put rockets in all the pockets and code words for nerds too. <laugh>. I don't know. I had a a DD moment. Perhaps most developers are a DD, but whatever. They just kind of cracked me at one point. $6 billion. We might change the whole DevOps world if you won, if you play right? 'cause in theory, if you don't play, you don't win. Alright? So a here typical statement here. Typical question here, right? You probably wanna stone me and being a heretic, even asking this question, okay? But is DevOps the anti-pattern to digital transformation? Just something to think about. And most of you're like, are you kidding? Of course not. DevOps is what makes digital transformation possible, right? That's what makes things go. Why would he even ask that question? I ask that question because of the many executives I talk to, the many companies I go and visit, I seem to see that this is a bit of a, an actual thing.

00:02:47

Why is this? I think it's because we in the DevOps world, are so busy with our heads down doing this DevOps thing that we fail to pull our heads up and take a look at the broader problems of the business. The broader context, particularly a lot of DevOps stuff I find tends to be done in a siloed manner. The evil silos that we've been talking about in it for forever. But right now it's the cloud silo. 'cause it's easy. You can do agile, you can do DevOps, things in the cloud, containerized, microservices, great. And we get focused on doing this rather than taking a look to seeing, well, ultimately, why are we doing what we're doing? And as I visit with executives and they, we talk about digital transformation and what digital transformation really means, and it really doesn't mean just going to the cloud, right?

00:03:35

Um, although I do find many CEOs get hired to take their company to the cloud, and that somehow is digital transformation. But really that isn't. But the pushback we get, yeah, we're doing digital transformation. We got this, right? What do I, so here's to put it all together, right? Who's read the DevOps? Um, the, the state of DevOps report, everyone, A lot of you have read it. Okay? Key thought from that along these lines. Doing continuous delivery for the sake of doing continuous delivery is not enough. If you want your organization to succeed, this is why I postulate that DevOps, at least for the sake of just doing DevOps, might actually be an anti-pattern to what you're trying to ultimately accomplish. Where the business is ultimately trying to go. You gotta do more, right? You gotta think, you gotta answer the great why, right? The why are we doing it?

00:04:22

What are we trying to accomplish? Those type things. You gotta get at the root of it. Otherwise, you're just doing it for the sake of doing it. Now, just to put this in a thought, a a another frame. I like stories. Who here has heard of Michael Jordan or perhaps LeBron James right now, right? Moved to la Um, of course, I've been kind of laughing about the disaster supposed disaster. LeBron James happening just from one game, <laugh> in Los Angeles long season. But ask yourself this. What if Michael Jordan just played basketball for the sake of playing basketball, would've ever made it to the NBA? Would he have ever been a champion? Right Name the athlete, formula one racer, whatever it is, some endeavor. If you did it just for the sake of doing it as hobbyists do their things, would they've ever achieved something great? And that's kind of the point, right?

00:05:11

I think that's the essence of what Nicole Forsgren found in her study this year. You've gotta think of the broader, the broader context. So here are six of the seven, uh, challenges that I identify and getting out and scaling out this thing we called continuous delivery. Okay? First thing is value stream clarity. There have been several sessions this week, right? This week, I guess it's day two, it seems like Wednesday for some reason. <laugh>, I dunno if anyone else seems to say same way, but, uh, value stream is very important. Cash to concept, right? But not just per product, but really from a business standpoint, what is the value that you're doing? Why are you doing the, the great why? Uh, how about cloud migration? I've, I've come across many organizations I talk to and they're going to the cloud, right? And there's kind of, um, in fact, going to the cloud is such a strong thing.

00:06:07

I'd almost say there's kind of a, we're going to a cloud explorative, right? We're just gonna do it. And so there's this great thing that's called lift and shift that is happening, right? Who's here is familiar with lift and shift? Anyone? Everyone. Okay? It's a disastrous idea. 'cause what happens is, what I found at Reinvent last year is everyone just takes their big monolithic middleware applications and they lift it up and put it in the cloud, right? And you know, the questions I heard as I walked around the booths and everything at Amazon reinvent last year. Hey, uh, do you know of anybody who can do job scheduling in the cloud? Can anyone do any kind of batch scheduling in the cloud? And why are they looking to do these things in the cloud? Because those apps have a lot of care and feeding a lot of repetitive clear, you gotta clear those darn logs, right?

00:06:57

You gotta do all kinds of things, reset the caches, all the stuff that you did. And those things didn't really matter when all you had to pay is the electrical bill for these apps, right? But now that you're on the Amazon or the Assure pricing model, these CPU heavy memory thirsty applications can actually cost you more money than actually keeping them OnPrem. And indeed, many companies are finding this out. If they just lift and shift, they find that they have to have the same processes and toolings that they had on site, right? The world of the cloud isn't all unicorns and rainbows and four leaf clovers. Turns out these, these apps still need the care and feeding that they had on-prem. All you've essentially done is just changed your, uh, data center, have you not? That's it. Lift and shift means I now have a co-located data center.

00:07:46

Hooray. That's exciting. I gotta manage two different places. My, I gotta somehow do continuous delivery across two different, uh, domain sets, right? That's not really, uh, what we wanna do. But the reality is, as far as cloud MI migration goes, most enterprises, at least the ones I talk to, what a lot of surveys and data say, is that the enterprise is going to have a co-located cloud strategy. And for nothing else for the data, the data's heavy. They're not throwing up, you know, terabytes of petabytes of data up into the cloud. A lot of times it's just too much. They're gonna keep it OnPrem. So at some point, you're gonna have to contend with trying to do continuous delivery across multiple data centers, essentially is what they work out to be. What about, uh, fragmented tools and teams? Big challenge, lots of open source, lots of cool tools to get the job done.

00:08:39

But, um, well, I mean, I guess if I just, the, uh, elephant in the room, I'm with ca, who, who here has heard ca is getting acquired? Anyone? No. Yeah, yeah. Lots of you. Okay. M and as mergers and acquisitions is a reality of the world. So you get these nice tools that you do two years later, your company makes an acquisition and they have their own sets of tools. Right? Now, how do you reconcile that? In fact, I have found that some of the companies I have gone to, they don't even need mergers and acquisitions, right? They already, they just distribute, uh, a big retailer who I will not name. They have a big office in Atlanta where I'm from, and they also have a group in the Midwest and somewhere out in California. And they found that these distributed teams have their own sets of tooling, and they're trying to create a continuous delivery best practice, right?

00:09:32

If they don't consider the larger use case and come to some type of consensus on how they're gonna work together and have a standardized place, then really all they got is the Atlanta based continuous delivery practice and the LA based continuous delivery practice, right? And they move at different cadences, different speeds, and they use different tools, standards, and so forth. And executives, when you get up to a certain level, they kinda like standards. They like repeatability. And we're gonna get into why that is in a moment. Uh, legacy versus modern. That's actually a real thing. I find a lot of times you go to conferences like this, we tend to think, oh yeah, we can do all this stuff. But the little secret is it's if you're using containers, microservices in the cloud, right? At the modern enterprise, you've got all this other stuff to deal with.

00:10:17

And we're gonna talk about that in a second. Um, visibility and measurement. I mean, that's a DevOps thing, right? You gotta, you can't improve what you can't measure. But how do you measure all this stuff and even the stuff that you're not actually writing, right? And then, I mean, security and compliance, you really don't need to say much about that. Obviously, security and compliance have to be first class citizens. It can't be something you go do after the fact. Um, auditors like to have a nice single place to go see what actually happened. They like to be able to know what happened without variableness. Uh, one of the things I have, I've always said something I've lived by is that, uh, computers are great at doing what you say, not what you mean. Human beings, we're great at doing what you mean, okay? And so security and compliance has got to be a first class citizen.

00:11:05

We're gonna jump in on that in a second. So I wanted to talk a little bit about, about legacy to microservices, because it is a big problem. This picture, as simple as it is, uh, symbolizes the enterprise, right? You've got mainframe and, uh, hey, mainframes have developers too. Sometimes I think we forget this. Um, you know, to use a, perhaps a political parlance, the inconvenient truth is that there's these other little tools, commercial off the shelf things like Siebel, people Soft, SAP, they too have developers and they too have their own thing. Maybe we don't like to admit that they're actually real developers, but, but the enterprise has these, and they have their own cadence and their own things. Um, and then of course we got the emerging stuff that we're all excited about. You know, yay, we can all do development in the cloud.

00:11:54

And it's so cool. I get Visual Studio code and with a couple of commits, I get pushed things up and it's so awesome. But, um, what happens if we don't respect the entire portfolio, if we don't pull our heads up and take a look at this horizon, what we have is the hurry up and wait. I don't know whoever's been to a doctor in the last couple years, but I swear a few years ago, doctor's office had a really great innovation moment, and they realized people don't like waiting in the lobby, right? I remember this, it must have been about 10 years ago. I don't know. I went to the doctor, I think with my son or something, and man, I couldn't believe it. They, we signed in, they pulled us into the back room within a couple minutes. And, but see, then the problem was I sat in the back and waited in the waiting room, right?

00:12:40

Where I used to wait in the lobby and then be seen, I'd be seen immediately with the doctor. Now they just make me feel better, sort of by having me go immediately to the treatment room and wait forever. And that's what happens. If we just focus on just the cloud stuff, then every 50th, every 200th deployment, we now have to wait for those darn Siebel guys to catch up with us or the mainframe guys, right? True. Digital transformation happens when you've got really continuous everything. And let's be honest, the, the, the CEO lifespan is two years, right? So why aren't people just gonna wholesale, rewrite all of this stuff? We'll rewrite some. I mean, some of you guys, some of you guys are actually rewriting some of your major apps. Cloud native show of hand to anyone, several you. Okay? So what happens when I, uh, in the mid nineties, I was part of doing some of this too.

00:13:29

I rewrote some new apps, I rewrote it several apps several times. It was this whole calm dcom thing that Microsoft was pushing. And then, then it was whole, you know, then it was Java and we're going middleware, and it was soa and we rewrote things. And what happened is we'd rewrite an app and write a couple new ones. And then over 10 years you look back and your portfolio has kind of ballooned. 'cause you don't get rid of the old stuff. Now, from a business context, why don't we get rid of the old stuff? Okay? I talked to a railroad company a few years ago. I don't know why in my head I'm forgetting who they are all the sudden. But they took the unprecedented, uh, adventure, shall we say, to go ahead and rewrite their mainframe code. They wanted to get off the mainframe, okay?

00:14:17

And I cannot remember how many hundreds of thousands of microservices that they had at the time that they explained this. But at the time of they're explaining what their journey was, this gentleman tells me they had spent eight years and $200 million, and they weren't close to being done. And I thought, eight years, how many CEOs was this <laugh>? I mean, and if you're a public company, what board's gonna go tell you what we want you to do? We want you to ignore revenue and go ahead and rewrite all this legacy stuff for about 10 years. Spend all of our money, and maybe we'll become profitable on the other side of this. It's just, it's just not gonna happen, right? So we have to live with this stuff. So rather than trying to ignore that, this is the reality. Let's deal with the reality and make everything fast.

00:15:02

Do continuous everything. Continuous delivery of the mainframe, continuous delivery of SAP, your Oracle systems, whatever it is. If it's an app, it can be delivered continuously. And we should be thinking about that. And I think the time to consider this is now because three quarters of all companies say they're doing continuous delivery, but they're doing it, like I said, for the mostly the cloud native stuff. So that's great. We've experimented and have some success doing this. So now let's take those lessons learned and let's work back, right? Let's go to this left and start figuring out how we can apply these things to the mainframe and to other legacy systems, not just the mainframe security. I could have brought lots of different statistics, but, um, it would've been too easy for me to get distracted, <laugh>, because the numbers are, uh, something else. But we all know this, right?

00:15:50

Uh, application layer attacks are the leading cause of database breaches. But the only thing I wanted to jump in here on, on security, 'cause we all, we all know, right? Security's gotta be a conti, uh, a first class citizen with continuous delivery. But the key thought I wanna share with this is that everyone is so interested in scanning their code, right? Oh, we, we got all these different tools that scan the code and it goes out and it gets updated. And we, we know that the code we're pushing is secure. But I, I met with a, uh, pretty big shoe company and I talked to their, their security auditors one point, and I asked him just a simple question about security. I said, so, you know, my experience, I used to run a small, uh, it shop somewhere along the lines of my career. I said, um, when we do deployments, I don't know every third update or something, you know, we're always changing the app, putting features and things.

00:16:42

And every so often some new feature required some new access, some credential to some other file system or some database or something, right? He goes, yeah. I said, so what do you do when that happens? I mean, you probably grant them access 'cause the deployment fails, right? He goes, oh yeah, it fails, and then you hurry and grant the permission so the deployment will work. He goes, yep. And then you take that permission back, right? And the guy looked at me, he's like, well, no. So in other words, you got a God mode user that's designed to be destructive to your environment that just out there, should it ever get compromised, can wreak havoc on your environment, right? So the idea is, what I'm trying to say here is it's one thing to make sure your apps are are secure. It's another thing to secure the automation mechanics themselves.

00:17:34

And we need to make sure that those are secured. 'cause those credentials and those mechanics are designed to make destructive changes. Now they, they do beautiful destruction and construction for us, right? But we don't want those to be compromised. Okay? So some other thoughts. What about scaling things out? So we already covered security, we already covered the, the whole portfolio. What about leadership? Okay, in the book, I assume almost everyone has read this book. Yeah, there was another key thought in here when it is near the end of the book, I thought was very interesting and very candid of Gene to admit this, but basically says that a lot of the change agents that made all this great progress in their company, they have an opportunity to go work somewhere else. And so they take it and upon his or her departure, the company reverts back to where they were and they go back to their old ways. So that means the continuous delivery practice, and indeed the DevOps practice was not sustainable, right? So to be a leader, we have gotta focus on sustainability. So things, practices, this goodness that we talk about here, survives your departure when you leave. Otherwise, we're just doing change at the end of the day for the sake of change, right?

00:18:50

Another great book about, uh, leadership, it's a great book, Robert M. Gates, but he's talking about if you wanna make change and you wanna make sustainable change, then you need to focus on pulling people outside of their silos, right? The quote here is, the best way to get access to and use internal talent and ideas is to implement reform. To implement reform is to get people from different parts of the organization working together outside their normal bureaucratic environment. We kind of heard that this morning, didn't we? I mean, we had a lawyer talk to us this morning. <laugh>, I've been to, I think to every DevOps, uh, enterprise summit. I've never heard a lawyer give a keynote before. I thought, wow, we have arrived. If the lawyers are talking about DevOps, maybe, uh, maybe we really have arrived. But that's kind of to this point, right? We're pulling people outside of their, their own individual silos.

00:19:40

It really takes the expertise of everybody if we're gonna do something sustainable way. And when it comes to expertise and externalizing it, we need to avoid group think DevOps has a, has an opportunity or has the, the myopic tendency, I should say, to become very thinking upon itself. Dev, no ops, just engineering best practices, right? Um, now according to this book, another great book, highly, highly recommended, uh, James Wicki. He says that we need to avoid group thinking. Homogenous groups he says are dangerous, right? Because they think and only see things within their own bubble. They cannot see nor think or comprehend anything other than themselves. You know, sort of like those, uh, uber driverless cars and bicyclists, those AI learn machine learning algorithms with back propagation. Only know the data that you teach 'em. And when the weird thing that happens that they were not taught like a bicyclist running in front of the car, disaster strikes, right?

00:20:48

Uh, the same thing happens with people. I'll read the quote here. It's really good. Non-polarized groups consistently make better decisions and come up with better answers than most of their members. And surprisingly often, the group outperforms even its best member. This is because you have a diverse set of opinions, a diverse set of perspectives that are being applied to a particular problem. Okay? The book goes in to understand how this works in the NFL or actually does not work in the NFL, how it works in stock markets. And, uh, even in the political races, it's a really good read. But another thought, another use case that he brings up is, uh, you know, the disastrous, uh, shuttle crash in the atmosphere, right? Uh, the management mission team said the performance of the MMT is an objective lesson on what? On how not to run a small team or group, okay?

00:21:42

A powerful demo. Uh, instead of making people wiser, being in a group can actually make them dumber. That myopic group think a homogenous group is very dangerous because there's no diversification of thought. Now, what happened in this crash, when the, um, when the shuttle went up, there was a group that looked at the foam, they had fallen down and they did the simple math. It's just foam. But at the amount of tonnage of foam that it was, they realized, Hey, wait a minute, there actually might be a problem. Well, the group that they kept trying to talk to did not give them equal footing to express their opinion. So their opinion got ignored. And of course, disaster struck. I thought it was great to have a lawyer here today. You should have a business analyst on your DevOps team. You should have other people on your DevOps teams that have those perspectives.

00:22:31

Non-technical perspectives can actually help avoid disaster. You know, I remember, um, being part of a group and everyone was saying, um, well, I won't tell you the group, but, um, actually, 'cause it's, uh, it'll hit a little too close to home. But suffice it to say it was with this group and the the, as we're thinking about papers to write and what we were gonna author, there was a concern with people, some angst that we don't understand why businesses don't think it's okay for us to fail in production. And we should be allowed to experiment. It should be okay to fail. I actually worked for a broker dealer, okay? And I'll remember I reported to the CEO of this broker dealer and he said, Scott, something you need to know about a, a brokerage. I said, what's that? Something goes wrong if we are out of compliance.

00:23:17

I, me personally, I get fined, not the company. If we're found fraudulent, whatever, I go to jail, the company gets nothing. So I don't wanna go to jail. And it's your job to make sure I don't, right? Failing in production in that environment is actually not a good thing. There's a, there was an, was an investment company called Knight Capital who had a failure in production, right? Some of you're nodding, you remember it, within minutes, they accumulated a 9 billion point position in the market. It was a false position because of some errant code that went into production. So I say it's, it's one thing to be able to, to fail, but you need to find safe environments in which to fail and experiment in. There are some environments, healthcare, airline industries, uh, wall Street and others where it's probably not okay to fail in production. Uh, production actually, things can actually go very wrong, very bad for people.

00:24:11

Okay? My last challenge here to talk about, who here has heard things like this? It's an application company every, uh, application economy, right? No, it's a data driven economy. I've heard that every application, every business is a software business, right? It's about the app. Stupid. Has everyone heard these kind of things, right? Uh, I will tell you, I've run my own little small business and some other ventures along those same lines. I'm gonna tell you a little secret. Now, you guys have heard what I'm about to say here, but it's not organized for you. So I'm gonna break it down. There are really only three things that a business cares about. Okay? Under this same guise here, I could tell you guys, um, almost sometimes what we talk about here at DevOps sounds like a manufacturing line. If I ever worked for an auto manufacturer thing, I was like, no, every company's a manufacturing company, right?

00:25:03

I can even take it for every company's a cog company and a wheel company, right? A conveyor belt. 'cause the reality is, if suddenly we couldn't make cogs, the world would stop. But how much value does the business put on those cogs? Lemme just think about that. The cogs make the conveyor belt. You couldn't even get an iPhone without cogs and gears on those manufacturing floors. But yet they pay pennies for these things. What does the business care about? They don't care that much about cogs, but I'll tell you what they do care about. They care about increasing revenue, period. That's it. How do I make more money? In fact, the whole reason you're in business is to make money. Kind of obvious. If you ever did a lemonade stand sold cookies when you were a kid, you didn't do it to lose money. Although I did have a cousin who thought, so I remember a real quick, oh man, this kid, he's had an interesting life, but that's a whole other perhaps comic session.

00:25:54

You know, you can come see me at some bar tonight and I'll give you the story, but he said, Scott, I got this idea, man. We're, we're gonna go buy some Kool-Aid and we can sell every glass for a penny. And I worked out the math for him. I was like, dude, we're gonna lose money on this. He just did not understand his efforts would actually lose money, but that's not good business. Anyways, next thing businesses care about in decreasing their costs, they wanna save money. Make as much money as you can, because the difference between these two are what your margins. And there's a reason why I'm telling you guys all this. And there's one third thing that any business owner, any business executive cares about, which is mitigating and managing risk. The reason I bring this up, because every time I come to these DevOps shows, when I speak, when I talk to you, all, all these things, we speak geek and we try to do this evangelist thing at our companies.

00:26:47

I mean, gene Kim talked about it. We'll come here and we'll hear all this good stuff. We'll get so excited and think we can last another year. That was his opening comment, right? But the problem is we go back to the, our boss and we start speaking geek, right? Oh, I got this tool in this widget. We can go so much faster, we can do this thing. And he's like, these three things. How does that help me do accomplish these three things? And you're like, well, I'll give you a quick example. The last two minutes, we can go to market faster. What does that mean?

00:27:19

What does it mean to your, your business by going to market faster? A very simple way to look at it is, right, if I were in the business of selling Snickers and I could get a Snickers bar out to market by September, how much more money could I make versus putting it out in the market in November? That three month gap means what? I can sell more Snickers than I could otherwise. You have to frame it in this kind of way. These three things. That's it. That's all any business is ever, ever doing. Just those three things. It's kind of simple. But having met with several C level folks, um, this is what they care about. Okay? That's it. So key thoughts to think about here as we wrap up. And uh, I'll, I'll add one extra thing to this. I remember a meeting with a, with a CIO of a big bank and we were talking to him about all these neat things we could help his company do, right? Make his people more efficient in this stuff. And he says, you know, Scott, this is all great, but I gotta be honest, and this is gonna sound bad, but none of this matters to me. I said, what do you mean none of it matters?

00:28:29

This, he says, my, my people and it, their salaries are fixed. So whether they work 80 hours or they work 30, I don't care. He says, now I know that sounds bad because I do care, right? And his, uh, his chief technician was right there. He goes, I do care about these people. I truly do. He says, but from the balance sheet, unless you can address these three things against a fixed salary base, there's no business case. There's no sense of doing it. Okay? So key thoughts to walk away from. Be a leader persuade guide. Respectfully argue. Don't be afraid to argue your point, but be respectful. But you gotta persuade people. You gotta kinda lead them so they understand and can get to where the business case is. The simple example I gave you about Snickers going to market, isn't the end going to market faster is not the end.

00:29:17

Answer that financial question. What do you get by going to market faster? Think more broadly about continuous delivery. The entire portfolio. Go for a sustainable practice, not just a siloed practice. Embrace the entire portfolio. I already mentioned that continuous everything. Not just continuous something, continuous everything. Make the mainframe go faster. You'll watch how quickly things change. That's, that's a real digital transformation. Secure the automation mechanics themselves think in terms of really including ops and making ops. Agile learn to speak the language of business risk, cost and revenue. That's all a business cares about ever. The only the whole reason you can invest in Wall Street, 'cause everything boils down to these three things. You can look at any company and figure out what are they doing with those three things and determine if you're gonna invest in them or not. Uh, DevOps plus multiple disciplinary teams.

00:30:08

Multidisciplinary teams are the key. I think that's important. Start including folks. The, the most efficient team I had in my career is, was made up of, uh, some developers. We had some QA people, we had some business power users and business analysts, and that was our team. And man, we pushed this out so quick before Dev develop, DevOps was ever called DevOps. And it was always high quality because we had this diverse perspective. And the business, uh, power users would always find weird bugs that weren't weird. Not technically bugs, but they would always have this perspective on the business that would help us. And then, um, just last thing here and ending, uh, it's a journey, not a destination. It's a continual, uh, cycle of, uh, continuous improvement, right? It's not just saying, we did continuous delivery, woo, we're done. You gotta continually tweet it in a sort of like working out something you need to do more of. You can't just reach at a certain point, you gotta continually work out, right? And then, uh, here's just once again contact information and uh, yeah, thanks for your time. Thank you.