Las Vegas 2020

DevOps Lessons from Executive Leadership: Why DevOps Matters to Corporate Performance

Over the last 7 years at DOES, Scott Prugh and Erica Morrison have walked us through their amazing transformation. This included a transformation in both people and processes that have returned amazing results. Scott also discussed the amazing work that CSG has done with technology and modernizing their 40-year heritage application stack.


This year, Ken Kennedy, EVP and President, Technology and Product, will discuss the amazing transformation from his perspective. Ken will cover:

1) why DevOps is important

2) the results that DevOps can bring

3) how to convince executives to get started.


We will also discuss the importance of leadership and DevOps and our journey to develop and grow leaders.

KK

Ken Kennedy

Executive Vice President and President, Technology and Product, CSG

SP

Scott Prugh

Chief Architect & SVP Software Engineering, CSG

Transcript

00:00:07

Thanks so much Maya and Ross. If you're like me, you were blown away by Doug Parker, CEO of American airlines, talking about how transformation is everyone's responsibility. And certainly not just its, if you are interested in learning more about the American airlines journey and the talent they're looking for, please join a session they're hosting during networking time. We'll post the exact details in slack right now. Okay. One of the next speakers is the only person who has spoken every year here at DevOps enterprise summit, all the way back to 2014, that person is Scott Prue, chief architect, and SVP software engineering at CSG, a leading provider of customer engagement solutions. And they are a publicly listed company over the years. He has spoken about one of the most amazing transformations I've ever seen. The focus of his stories have primarily been around the systems that generate the majority of revenue for the company, which includes changing how the teams work, modernizing mainframe code that goes back over 40 years, migrating away from hostile vendors whose business models are in conflict with their customers and much, much more.

00:01:17

I'm so excited that this year he is speaking with his boss, Ken Kennedy, executive vice-president and president of technology and product. Ken will be speaking about how he, as a profit and loss owner values, the work that Scott and the technology teams have done and why it's so important that Scott continues to build more leaders within the technology organization and his advice to this community on how to convince executive management on the importance of dev ops. Also, Scott is on the programming committee for this conference and his also talks about the leadership lessons learned, which helped enable all the amazing outcomes he's talked about in previous years. Here is Ken and Scott.

00:02:01

Jane, thank you for that introduction. Hello everyone. My name is Ken Kennedy and as Jean mentioned, I'm the executive vice president at CSG and the president of technology and product. And in that capacity, I'm responsible for our product management organization, our engineering organization, our public and private cloud operations and our corporate it functions. And my goals center on revenue, growth, profitability and customer satisfaction. And where I spend the majority of my time is, is looking to optimize the flow of work so we can deliver, deliver the greatest value to our customers and the shortest sustainable lead time. And I'm joined today by my colleague, Scott Prue.

00:02:37

Hi folks. My name is Scott Prugh. I'm chief architect and SVP of engineering at CSG. And I spent the last couple of years really focusing on high performance engineering organizations that both build and operate their software.

00:02:53

And today Scott and I would like to share our story. Uh, we will share with you some of our experiences over the past 10 years, the importance of strong leadership and how to convince executives like me, that dev ops matters when it comes to corporate performance. Before we do that, I'd like to briefly share some information on CSG. CSG is a provider of innovative customer engagement solutions that help businesses acquire, monetize, engage, and retain their customers. We've been in business for about 35 years and we have 500 clients spread out over a hundred different countries. Our annual revenue is just shy of a billion dollars and we have 4,000 employees around the globe and we spend about $140 million each year in R and D.

00:03:34

The solutions we provide, uh, fall into three main categories. The first is around digital monetization or revenue management is a more common term. And that is basically enabling service providers to be able to monetize their, the services that they deliver, deliver to end customers. The second is around customer engagement, which is really driving the best possible customer experience by delivering the right message to the right individual at the right time, through the right channel, with the right context. And the third is our digital payment solutions, which enable the transfer of funds, whether it's through electronic funds, transfer credit card payments, debit card payments. And we have a payment gateway solution that enables our customers to get compensated for the services they deliver. But one of the things I'm most proud about is CSG is repeated recognition by Gartner as a leader and their magic quadrant that comes out every year. And then by being recognized as a leader, CSG is, uh, highlighted for not only their completeness of vision, but also their ability to execute

00:04:36

Our journey over the last, uh, 10 years has been a very interesting one. And, uh, we, we look very different from when we did, when we entered the, the last decade, uh, over the past 10 years, we've created accountable cross-functional high-performing teams that have allowed us to be more efficient. We've introduced a lean portfolio management that enables our product management to focus on those features that are going to drive the greatest value and manage our work in process. We've also invested heavily in, uh, to the tune of tens of millions of dollars in automation and technology modernization, and we've achieved faster throughput and significantly improved quality by some of the changes we've made, embracing dev ops principles and optimizing flow. And finally we've fostered a culture of continuous improvement and learning by making this a safe environment where we learn from our mistakes rather than punishing people for mistakes. But to get into a little more detail, I'd like to hand that off to Scott Peru,

00:05:35

Talk a little bit now about really our journey and what we've discussed over the last few years at DevOps enterprise. You know, we started in 2014 really talking about our agile transformation and then really carry that the next couple of years through our dev ops transformation. And that focus was really around the dramatic people and process changes we made to higher performance last year at, at does 2019 Jean reach out to me. And he asked for me to talk about our modernization journey and really how we went about, uh, implementing CGI automated testing, uh, and even large-scale code conversion and porting across a lot of our infrastructure. And that even included, modernizing our mainframe and basically starting to convert a lot of the code off and actually converting from a traditional V Sam, uh, systems to, um, modern relational databases. And this year, both Ken and I will be talking to you about, um, executive leadership and how dev ops matters to corporate performance.

00:06:39

So a little look back at some of the metrics we've covered the last couple of years, uh, through our agile transformation, kind of early dev ops, we did, uh, grow our subscriber base, um, quite a bit. And then also at the same time transactions of our, our system grew dramatically. And as you can see, really kind of up to date, we've grown from 50 million subscribers to date to about 67 million over that same period of time, we've grown from 750 transactions on our platform to over 8,000. That's almost a 10 X increase over that period of time. And we have done all that with keeping our costs constant. Additionally, when we look at incidents, our impact on the system, we made a dramatic improvements there. We've gone from close to 1600 incidents per month and drop now to, uh, under 300 to 2 86, that's an 80% improvement.

00:07:30

Additionally, we measure something called release impact it, which is when we put a release into production, uh, what impact or incidents come out of that release? We've dropped that dramatically, um, over 97% from, uh, 507 to 13, uh, through, um, the improvements that we've made and the final thing, and the most compelling is really to look at two things here. One is released on demand, which basically is the deployment frequency. And we put code into production. Basically we we've decoupled, uh, many of our feature deployments from the release so that we don't release in a big batch. We release when they're ready. And we call that release on demand. And we've been measuring that percentage is 2018. Additionally, we measure something called impact minutes, which is, uh, which is basically the, the recovery time multiplied by a blast area rate is. And as you can see, we've reduced that from 22,000 to 4,300 over this period of time.

00:08:26

So at least on demand has increased dramatically. The deploy frequency has gone up and the impact on the system has, has basically gone down. It's now kind of a quick summary of all these metrics and, and really interesting to us is how they tie to, uh, accelerate and the high performance metrics documented, uh, there by Nicole Forsgren our change failure rate, uh, the change failure percentage ties to what we were tracking the incidents per month, or at least impact. And also part of the impact minutes, meantime to recovery is directly tied to impact minutes, then feature cycle time and release on demand really affect that, that lead time. And then finally deploy frequency, um, is basically what we were driving with release on demand, which is deployed faster when the code is done. And really all of this basically enabled the outcomes below us. The ability for us to grow subscribers, grow transactions on the platform without increasing costs. And what I'm really most proud about is basically our employees are five times happier having done all this

00:09:29

CSC delivers our software as a service. Some of our products include revenue management or billing for services, field service management, which is managing technicians and trucks as they do installations and service calls and output solutions, which includes the composition and delivery of billing statements. We expose these capabilities through a common service layer called SL boss, and the service layer supports 30 of the largest cable and satellite providers in north America. It has 180,000 call center representatives. It serves 67 million households across north America and over a thousand unique application integrations developed by our customers. And our first story takes bay takes place in 2010, back in 2010, our software releases were major events, uh, because we are a SAS company. When we push out a new release, 100% of our customers get the new functionality simultaneously at the time we delivered big batches of functionality simultaneously.

00:10:23

So there was often fall out and it's, it was not uncommon for us to deliver a release and spend the next two weeks triaging and correcting issues. And I'd like to share you a story of one fateful release in 2010, that I will certainly never forget the really split in very early on a Sunday morning as it usually does. And on Sunday volumes were light and we had a usual number of tickets to work through. Then on Monday morning, as volumes picked up considerably, we realized that we had a major problem on our hands. The response times for our SL bus service layer were unacceptably. So, and the switch situation was going from bad to worse. Every single application was impacted call star service desk board an all time high. And my mobile phone was blowing up with calls from CEOs at Comcast dish, time Warner cable and charter communications.

00:11:08

Just to name a few, we pulled together the development team responsible for SL boss and reviewed the changes that went into the release to our surprise. There was no smoking gun that could explain abroad performance degradation. We engaged to the separate operations team for assistance. And then we received the news operations had decided to migrate from Solaris M four thousands to IBM AIX servers to save money on hardware. The hardware changed, you know, went in as part of the release and the old hardware had been powered down and we knew we had a big problem, but we didn't know we had a massive problem. It took us 48 hours working around the clock to stabilize the situation. We had power back on the old Solera servers. Um, the proprietary boxes were underpowered to support the pent up demand. Our deployment processes were entirely configuration changes to proprietary. Middleware required us to cycle the server and the cycle time was 45 minutes long compounding the issues we rushed to restore service. We made manual mistakes, which would cause us to entirely restart the process. When the dust settled 48 hours, I knew our service layer represented a massive single single point of failure that we had to address. And at this point I turned to the development lead. I trusted most, and I said, Scott, I need you to fix this

00:12:26

Well on Ken, help me with this challenge. I was pretty apprehensive. I had heard about the problems with SL boss, the massive outages where were legendary, the daily struggles and conflicts between the development and operation teams were often discussed. So I turned to, to one of my good friends at the time and mentors. Um, Maricio some more, for some advice we call the model. Most advice with a grin is how could you make it worse? Um, he then elaborated that leaders look for opportunities to lead and solve problems, uh, for their org. And this is an opportunity to do that. He also confronted me and said, Hey, you have the skills to make this better. So I took his advice, sadly, Moe died a year after this conversation and he would never see the fantastic transformation that occurred with SL boss and at CSG. I know he would be proud of what we have accomplished.

00:13:27

So a little bit more about what we, what we did here. We basically went in and took our proprietary infrastructure and put automated builds and automated testing around that to start as a foundation. So we would understand how it worked. And so we have faster feedback to build the system. We then set off to port some over 300 transactions to commodity infrastructure. And as you can see kind of the results here, they were pretty unbelievable. We went, uh, lower the cost, some 300 X, oh, we were over $7,000 per TPS. And now we're, uh, around $20. Our, our feature development time, um, increased dramatically. The amount of time we could spend on feature development from 50 15 to 55%, build them deploys went from, uh, almost half a day to five minutes. Server recycles kind of, as Ken mentioned, we can now recycle servers. And in under two minutes where, before it took 45, and finally we ended up with automated tests.

00:14:24

The other great thing that we actually did here, um, is we able to, we started to work differently. I got introduced, um, during this project to, to Steve BARR, who's actually been featured in a lot of our other stories. He was a great transformative leader who started to think differently. He's the one who proposed that we create a shared operations team, basically that shared operations scene would, uh, run production, non production and production the same way. Um, they would do deploys every single day and get practice and they would run their production environment with the same tools and techniques. That piece was the start of really our understanding of how powerful dev ops could be and how that could drive a lot of our transformation to provide higher performance. Now, I'll turn it back to Ken

00:15:10

For our next story. I'm going to fast forward to 2016. At this point, I was now responsible for product management and development operations. How are reported to another leader within the organization? Now, one thing that has been constant with our customer demands, they wanted us to deliver new features quickly and with high quality, however, we were struggling mightily internally within our four walls. On one side of the organization, we had my development organization that that development organization was focused on delivering new features with speed. If quality suffered, that was a secondary concern. Then they complained of the need for better tooling and demanded that operations deliver environments more quickly. And they hated, hated our change management process that required an act of Congress to push through even a minor low cost change on the other side of the organization. And we had an operations team that was focused on quality after all, they were the ones who would have to get out of bed at 2:00 AM.

00:16:05

When something went wrong. I had one senior leader in the operations organization telling me that it was his job to protect customers from my organization's bad code. One thing they had in common with development was the disdain for our change management process, but it was plainly evident when you have two organizations that have different objectives. You're not going to be successful when you have two organizations that plan work separately, you're not going to be successful. And when you have two organizations that lack empathy for the other organization, you're not going to be successful. And I knew something had to be done. I knew these two teams had to come together and unified cross funk as a unified cross-functional teams with shared goals, reporting to one leader. We needed leaders skilled in both development and operations practices with the technical aptitude to get into the details, to streamline our service delivery.

00:16:53

And I felt so, so strongly. I went to my boss, our CEO, and I told them we had to change. He needed one leader sitting over development and operations. And I felt so strongly. I told him that if he had to make the decision and that didn't involve me, I would go look for another job to his credit. Our CEO made a very difficult decision because he knew what had to be done. And fortunately for me, he chose me to lead the combined organization, or I'm not, I wouldn't be here telling the story. However, our work was just beginning. This was not a popular decision among the teams. We were asking people to change the way they'd worked. And sometimes some of those people had worked the same way for decades. And we define new expectations, chief among them. We told developers that they would be on call. If there was a problem with their software, they were, they were going to be the ones to fix it. We also told the operations people that they had to learn how to write code, to automate a large number of manual processes. Once again, I turned to Scott, he was now burdened with making these combined teams successful.

00:17:56

So one of the things we kind of really thought about here was really changing the way that we were structured and going from a resource efficient organization that ran a lot of projects, uh, to, uh, to teams that were organized for flow and knowledge efficiency. We brought those, we brought the operations folks together on the same teams that were engineering, that, that software that was difficult. And, uh, it took a while for us to, to get adjusted. But our leadership team was fully in support of this and fully, um, supported our people in this transition. We worked hard to make sure that people got the training that, that, that they needed and that they were, that were ready to actually go through and create these, these cross functional teams. Even though we worked really hard to make sure that everyone succeeded, there were some folks that, that didn't made it. We had to didn't make it. And we had to make some tough choices to basically transition those folks out of the organization with this change.

00:18:54

Although it took a little while to get started, we immediately started to see some great results. And we started to understand how the structure of our organization influenced both the behavior of our teams and the quality of our software. When we started to dig in with these cross functional teams, we found that some 98% of those 1600 incidents were being handled by those level two ops teams. Those teams that previously were kind of on their own to deal with these, these operational issues. When we dug in a little bit further, the team started researching, they found some 92% of those. Six of those incidents were quick fixes. These were fixes that basically were, were the corrected once, but didn't get at the root cause of the problem. So they would occur again the next month. And come back once we got these cross-functional, uh, development operations teams to look at this, they quickly started to burn down and resolve these issues.

00:19:50

And this was one of the key drivers to that dramatic drop in incidents, uh, that, that you saw in the beginning and really kind of returning a lot of capacity to these teams because they were no longer firefighting fixing these incidents all the time. The other thing that we realized is that our problem management processes and this multi tier support architecture that we had in the organization, uh, wasn't helping us. It wasn't going to save us from this problem because the handoffs basically never got to the, uh, prevented us from getting to the root causes of the issue.

00:20:24

Over the last several decades. Scott and I have been through several battles together. CSU is the third company we've worked, worked at together from our consulting days to jointly founding a startup, to working at CSG Scott and I have shared a strong working relationship built on mutual respect and trust. And during that time I've seen Scott grow in the earlier years, Scott was an amazing, uh, the term 10 X developer applied to Scott in the early part of his career. He was 10 times as productive as his peers. And, uh, I could give him the most complex coding assignment. He'd regularly deliver the problem with the 10 X developers that he or she is limited by the number of hours in the day. And even if someone is the best program in the world, there are limits to the impact he or she can make many years ago.

00:21:11

Um, I sat down with Scott and I said, he's reaching, you're reaching the limits of what you can personally do. I told him I needed him to evolve from a 10 X developer to 10 X leader. I need him to have a broader impact. I need him to share his knowledge beyond his immediate team. I need him to cultivate and grow and learn new agile teams today. I couldn't be more proud of Scott's leadership. Not only has he been the catalyst for transformational change, but he's also built an amazing team of leaders. Some of whom have spoken at prior does summits and all of them were pivotal leaders in the transformational journey. But rather than have me tell you about the importance of leadership, let me pass it off to Scott.

00:21:55

So I want to talk now about what we learned from growing leaders over the years, I hope is to provide a framework for folks going through a transformation and a leadership journey that they can use as a guide. The first thing I'll say is decide to lead the vision. This is really about building self-awareness. You need to ask yourself the question, do you want to lead? And are you up for the challenge, then figure out what your vision needs to be to help your org reach its goals and provide clarity to your people about what that vision is going to be and where it's going to take you. The next I would say is about technical to strategic excellence. You need to figure out how you go from being a great technical leader to a strategic leader. In this case, you need to move from being great at execution or great at the domain that you were in. And instead look at how you provide strategic value and leadership for your org and your company. The next is org needs and your leadership needs. You need to determine what your org needs are and those gaps, and then you need to adjust your leadership needs to fill those gaps. This requires adjusting your leadership over time to help fill the gaps that you might have for me, this meant adjusting my leadership across many areas, setting vision on our technology roadmap, leading organizational design, doing cross org alignment, building people and teams, as well as the international talent development.

00:23:35

The next is build people and leaders. Basically, you need to make a strong investment in not just building your people, but also your next generation of leaders. You cannot execute a large transformation and a journey alone, and your people and leaders are vital to scaling your leadership. Great leaders have great people and they also have great leadership teams. Next is transparency, empowerment, and safety. You need to provide transparency for your people as well as empowering them to make the decisions aligned to the vision of where you want to go. This also includes creating an environment of psychological safety, where people are free to experiment and fail without fear and punishment, be inclusive and bring different views to the table and your decision-making process.

00:24:28

The next is build influence networks. You're not going to be able to convince all at once to follow your vision. This is especially true for peers and parallel organizations. This is where influence networks are key. You need to find like-minded individuals across your org that can help influence the change. You need to build purposeful connections and identify opposing views. You need to understand the concerns and work to influence them through their directs and other peers. This is also a great opportunity to build connections across groups, to improve the working relationships and productivity, because you will likely need to work with these folks later to deliver success for the company. For example, at CSG, we built a community of learners and leaders through our DevOps culture series. This brought together parties across CSG who are passionate about working differently and creating a high performance team. Also extend your influence network outside your company, and find practitioners in the industry who can help validate and enhance your thinking, but also influence your organization to get to where you need to go. These industry practitioners also can help form a support network to help you weather the changes as a leader, you will need to work through. For me personally, the leaders and friends in the step-ups community have been a critical part of both my influence network and support network

00:25:58

Develop a learning culture, high performance organizations, view learning as a vital investment, and also a competitive advantage note that your organization will shadow your values as a leader. And as such, you need to demonstrate that learning is vital and part of what is done every day, read and research for anxiously, build a deep and profound understanding about how complex systems and organizations work and how this applies to your organization, share your learnings with your team and create space for this learning to occur. For example, as part of our transformation at CSG, we've pulled expertise from theory of constraints, lean Toyota, agile DevOps, and even the military via works like team of, and one mission. Also this community of dev ops leader has been vital to teach us so much about the best ways to think about transforming into a high performance organization. Finally, give back to the community, share your expertise through events like this and writing, teaching. What you know is a fantastic way to give back, but it also refines your understanding and furthers your learning. The final is have courage. Leadership is very hard and success is not guaranteed. You'll need to have courage to try out many different options, fail at many of those. And try again, you will need the persistence to build a coalition to whether the transformation, the journey is very long and it only looks simple and neat. Looking back, be assured it's very hard and without courage and a support network, it's nearly impossible to do.

00:27:36

The final thing I want to talk about here is how, um, we found that that really some of these capabilities in this framework tie back, uh, to accelerate and transformational leadership and the, and the capabilities of transformational leadership, our vision, intellectual stimulation, inspirational communication, supportive leadership and personal recognition. We have found that many of, um, the, the growing leadership capabilities on our roadmap tie back to these high-performance capabilities and both Nicole Forsgren spoke accelerate, and the state of DevOps report in 2017, we highly recommend both those works to understand more about this. I'll turn it back to Ken.

00:28:20

Our final topic today is how to work with executive leadership. When Scott and I met with Jean two months ago, he described a challenge. Many of you face the challenge of convincing executive leadership to invest in dev ops. Well, for you to convince the executive leadership to invest heavily in dev ops, you must do two things. First, you have to make the correlation between dev ops and organizational performance for them. And second, you have to build credibility and trust. Let's face it. Executive leadership has focused on organizational performance. Executive leadership talks about revenue, profitability, market share, and customer satisfaction. Those are the things that are discussed in the boardroom, and those are the things that drive compensation right or wrong. Many executives do not think in terms of, uh, systems thinking, flow metrics, feedback loops, experimentation, and learning. My advice to you is to share the state of DevOps report with your executive team.

00:29:16

The report produced by Nicole Fullscreen, Jess humble, and gene Kim did a wonderful job of studying the effect of specific business practices. And the correlation with improved organizational performance is statistically proves that the correlation between software development performance and organizational performance is statistically proves how improving throughput and stability will positively affect productivity, profitability, market share, and customer satisfaction. Even more specifically, it shows how lead time deployment, frequency change, failure rate, and meantime to resolution separate top performing companies from average performing companies. Once they have this agreement, you can define your objectives in terms of throughput and stability, rather than having to go through a mental gymnastics of showing how automated deployments may drive an IM improve the market share.

00:30:06

The next step is to build credibility. And this is only achieved with a proven track record of success to PR to build credibility. You must do three things. Uh, first operate with an economic mindset, business cases matter wherever possible, articulate, the hard cost savings or the increases in revenue. These are the easy invest easiest investments for an executive like me to approve. The second thing is prioritize investments. I have no doubt that all of you have a long backlog of investments that you'd like to make. You must acknowledge that the budget is finite resource and no business has the luxury of eliminating a hundred percent of the technical debt while putting all other initiatives on hold, prioritize your investments and focus on those items that deliver the greatest value in the shortest time. And then finally deliver on the commitments. Nothing builds credibility, like delivering on commitments, show the benefits of your efforts. Don't be reluctant to broadly communicate your success and recognize the people who drove that success.

00:31:04

So gene always asks, what are the things that we're looking for? And w while we're proud of our accomplishments at CSG, we know there's more work to be done, and we strive for continuous improvement, and we're looking for help in three areas. And if you have thoughts or suggestions, please don't hesitate to reach out to Scott or me. The first is around extending dev ops to client projects while we've fully embraced dev ops within our four walls. We often struggle to convince clients to embrace dev ops principles when we're doing client specific work for them. The second is around addressing architecture and skill bottlenecks. I'm sure there are several new audience that have faced this challenge. We'd love to hear from some of you in terms of how you've successfully addressed bottlenecks. And then third is around building more fungible teams. We, we strive to prioritize features that deliver the greatest value, but often we find ourselves in developing lower priority features because the resources we need are with the domain expertise. So the technical skills are over committed for other features. So if you have input on that, please again, please reach out to Scott RI, and then finally, we'd be remiss if I didn't take a moment to thank our amazing teams. I get a lot of credit for the incredible work done by our fantastic leaders and their amazing teams. They show they work tirelessly to max maximize customer delight, and they've been relentless, relentless in their commitment to continuous improvement, to all the 4,000 employees around the world. Scott. And I say, thank you.

00:32:30

Thank you folks.